Experiment analyses and how to read them

Experiment analysis takes place after the conclusion of the prescribed treatment. The goal of the post-experiment analysis is to determine if the treatment led to a change in the target metric(s). This is done through modeling what would have happened had there been no treatment, and comparing it to the observed results in the treatment geographies.

Determining Controls

For experiment analysis, we train many models to predict the counterfactual, or what would have happened in the treatment locations. The different models comprise of different control locations, different model parameters, or different models entirely. By training and comparing many models, we are able to understand the range of possible effect the treatment may have had.

Based on the models that fit well and reflect plausible results, we select the top few controls to chart and compare to the observed values.

Understanding Results

The goal of every experiment is to learn something. Unfortunately, it's not always an exciting result! Upwards of four out of five marketing experiments will disprove the hypothesis. It's fairly common to not see incremental lift, and that itself is a valuable result.

Lift

Based on the selected models, Paramark makes a determination between evidence of lift, no lift, or inconclusive results.

• Lift - The observed results in the treatment states differed from the controls (predicted results).

• No Lift - The observed results in the treatment states did not differ from the control.

• Inconclusive - The observed results did not exhibit a consistent pattern against the controls, or the controls do not fit well enough to draw conclusive results.

Certainty

Paramark will also indicate the amount of confidence in the results.

• Low - There is a lot of variance or poor fits in the control models, so the conclusion is weak.

• Medium - There may be some unexplained variance across the controls, or they do not unanimously agree on the effect.

• High - There is clear and unambiguous lift against all of the selected controls. (Or lack of lift in the observed results compared to all controls)

Recommendations

Paramark will recommend a set of next best actions. These often involve our recommendation on scaling or reducing spend on the channel, further experimentation, or new strategies.

Incorporating results into the MMM

If the results are medium or high confidence and differ greatly from the MMM, Paramark will set the priors on the next MMM to incorporate these results into the model. Setting the prior does not force the model to reflect the results exactly, rather it guides it towards the result of the experiment. The model is still using the entire timeframe to make conclusions based on correlation, and it’s possible setting the priors does not result in perfectly aligned outputs.

Want to learn more about experimentation? Check out other Paramark experimentation articles here