What-If Analysis with ML Studio: Testing Business Scenarios

Introduction

A deployed model answers the question "what will happen given these inputs?" But business stakeholders rarely stop there. They want to know what would happen if they changed the inputs. What if we raised the price by 15%? What if we approved this loan at a lower interest rate? What if the manufacturing temperature was 5 degrees higher?

These are What-If questions, and answering them systematically is the bridge between a model sitting in production and a model driving real business decisions. CorePlexML's ML Studio provides a dedicated environment for creating, running, and comparing What-If scenarios against any deployed model.

ML Studio Overview

ML Studio is organized around three concepts: sessions, scenarios, and baselines.

A session is tied to a specific deployed model. When you create a session, ML Studio traces the full lineage from the deployment back through the model, experiment, and dataset version to extract the feature schema. This schema defines which features you can modify and their valid types and ranges.

A baseline is the starting point: a set of feature values that represent the current state or a typical observation. ML Studio runs the baseline through the model to produce a baseline prediction.

A scenario is a variation on the baseline. You change one or more feature values and ML Studio produces a new prediction. The power comes from comparing multiple scenarios against the baseline to understand how different inputs affect the outcome.

Creating a What-If Session

Start by navigating to ML Studio in the CorePlexML interface. Click New Session and select a deployed model. ML Studio automatically loads the feature schema from the deployment's lineage chain.

Through the SDK, session creation looks like this:

from coreplexml import CorePlexMLClient

client = CorePlexMLClient(
    base_url="https://api.coreplexml.io",
    api_key="sk_your_api_key"
)

# Create a session linked to a deployment
session = client.studio.create_session(
    deployment_id="dep_xyz789",
    name="Churn Pricing Analysis"
)
print(f"Session ID: {session['session_id']}")
print(f"Features: {[f['name'] for f in session['schema']]}")

The schema response includes each feature's name, type (numeric or categorical), and for numeric features, the observed range from the training data. This information drives the form inputs in the UI and validates scenario values in the API.

Setting the Baseline

The baseline represents your reference point. For a churn prediction model, you might set the baseline to a typical customer profile:

baseline = client.studio.set_baseline(
    session_id=session["session_id"],
    features={
        "tenure": 24,
        "MonthlyCharges": 70.0,
        "Contract": "Month-to-month",
        "InternetService": "Fiber optic",
        "TechSupport": "No",
        "PaymentMethod": "Electronic check"
    }
)
print(f"Baseline prediction: {baseline['prediction']}")
# Output: Baseline prediction: 0.72 (high churn probability)

The baseline prediction becomes the reference line in all comparison charts. Every scenario is measured as a delta from this value.

Running Scenarios

With the baseline set, create scenarios by modifying one or more features. Start with single-feature changes to isolate each variable's impact, then combine changes for more complex analysis.

# Scenario 1: What if this customer signs a two-year contract?
scenario_1 = client.studio.create_scenario(
    session_id=session["session_id"],
    name="Two-Year Contract",
    features={
        "Contract": "Two year"
    }
)
print(f"Prediction: {scenario_1['prediction']}")
# Output: Prediction: 0.18

# Scenario 2: What if we also add tech support?
scenario_2 = client.studio.create_scenario(
    session_id=session["session_id"],
    name="Two-Year + Tech Support",
    features={
        "Contract": "Two year",
        "TechSupport": "Yes"
    }
)
print(f"Prediction: {scenario_2['prediction']}")
# Output: Prediction: 0.12

# Scenario 3: What if we increase monthly charges by 20%?
scenario_3 = client.studio.create_scenario(
    session_id=session["session_id"],
    name="Price Increase 20%",
    features={
        "MonthlyCharges": 84.0
    }
)
print(f"Prediction: {scenario_3['prediction']}")
# Output: Prediction: 0.78

Comparing Scenarios

The comparison endpoint returns all scenarios side by side with their predictions and deltas from the baseline.

comparison = client.studio.compare(session_id=session["session_id"])
for scenario in comparison["scenarios"]:
    delta = scenario["prediction"] - comparison["baseline"]["prediction"]
    direction = "lower" if delta < 0 else "higher"
    print(f"{scenario['name']}: {scenario['prediction']:.2f} "
          f"({abs(delta):.2f} {direction} than baseline)")

In the UI, the comparison chart uses a color-coded bar chart:

• Blue bars represent the baseline prediction
• Green bars indicate scenarios where the prediction is above the baseline
• Red bars indicate scenarios below the baseline
• Gray bars appear when the prediction is identical to the baseline

For regression models (predicting a numeric value), the chart directly compares predicted values. For classification models, the chart displays predicted class labels. Since class labels are categorical, the numeric bar comparison is replaced with a text-based summary.

Understanding SHAP Contributions

When you run a scenario, ML Studio can also return SHAP (SHapley Additive exPlanations) contribution values for each feature. SHAP values decompose a prediction into the additive contribution of each input feature, answering the question: how much did each feature push the prediction up or down from the average?

scenario = client.studio.create_scenario(
    session_id=session["session_id"],
    name="Contract Change Analysis",
    features={"Contract": "Two year"},
    include_contributions=True
)

for feature, contribution in scenario["contributions"].items():
    direction = "+" if contribution > 0 else ""
    print(f"  {feature}: {direction}{contribution:.4f}")

A typical output might show that switching to a two-year contract contributes -0.35 to the churn probability (strongly reducing it), while the fiber optic internet service contributes +0.12 (increasing it). This decomposition helps you understand not just what the model predicts, but why.

Important limitation: SHAP contributions are not available for StackedEnsemble models. If your deployment uses a stacked ensemble, the contributions field will be empty. This is a limitation of the H2O framework's predict_contributions() method, which does not support ensemble architectures. Individual models (XGBoost, GBM, Deep Learning, GLM) all support contributions.

Business Use Case Examples

Pricing Optimization

A telecom company wants to understand how price changes affect customer churn. Using ML Studio, the pricing team creates a baseline customer profile and runs scenarios at different price points: current price, 5% increase, 10% increase, 15% increase, and a bundled discount. The comparison chart immediately reveals the inflection point where churn probability jumps sharply, helping the team set prices that maximize revenue without triggering excessive churn.

Risk Assessment

An insurance company uses a claims prediction model. The underwriting team creates scenarios for a prospective policy holder, varying coverage amount, deductible, location, and vehicle age. Each scenario shows the predicted claim probability and expected loss amount. The team can identify which combinations of coverage and deductible produce acceptable risk levels, and price the policy accordingly.

Manufacturing Quality Control

A factory uses a defect prediction model trained on sensor data. The operations team sets a baseline at current operating conditions and runs scenarios that vary temperature, humidity, and line speed. The comparison reveals that a 3-degree temperature reduction decreases the predicted defect rate by 40%, while increasing line speed by 10% has minimal impact. This analysis directly informs the optimal machine settings.

Loan Approval Analysis

A lending team evaluates how changing loan terms affects default probability. The baseline is set to the applicant's current profile. Scenarios explore different loan amounts, interest rates, and term lengths. SHAP contributions reveal which factors drive the default risk for this specific applicant, enabling personalized loan structuring that balances approval rates with portfolio risk.

Best Practices

Start with single-feature changes. When you change multiple features at once, it is difficult to attribute the prediction change to any individual variable. Begin by varying one feature at a time to understand its isolated effect, then combine changes for more realistic compound scenarios.

Validate with domain knowledge. If a scenario produces a counterintuitive result (e.g., higher income increases default probability), investigate further before acting on it. The model may have learned a spurious correlation, or the feature interaction may be more complex than expected. Use SHAP contributions to understand the underlying drivers.

Document your scenarios. Give each scenario a descriptive name and record the business question it answers. ML Studio sessions persist, so you can return to a previous analysis months later. Well-named scenarios make the analysis self-documenting.

Compare across customer segments. Run the same set of scenarios with different baseline profiles. A price increase that barely affects low-tenure customers might significantly impact long-tenure ones. Segmented analysis reveals heterogeneous effects that aggregate analysis misses.

Use realistic feature values. Setting a feature to an extreme value that never occurs in your training data produces extrapolated predictions that may not be reliable. Stay within the observed ranges shown in the feature schema to ensure your scenarios fall within the model's learned domain.

Combine with A/B testing. What-If analysis generates hypotheses. A/B testing validates them. Use ML Studio to identify the most promising interventions, then design controlled experiments to confirm the predicted effects in practice. The model tells you what should happen; the experiment confirms whether it actually does.

ML Studio turns your deployed models into interactive decision-support tools. Instead of treating predictions as opaque outputs, you can explore the decision boundary, understand feature contributions, and build evidence-based business cases, all without writing a single line of model code.