Randomized Experiments and Product Experimentation

This course treats randomized experiments as decision systems with design, monitoring, and careful reporting. Randomization creates credible counterfactual comparisons, and applied experiments still require careful choices about units, outcomes, exposure, power, guardrails, noncompliance, interference, and communication.

The objective is to connect experimental design to operational judgment. By the end of the course, a reader should be able to design an experiment around a real decision, explain the estimand, reason about practical significance, diagnose threats to validity, and prepare a decision summary that separates statistical evidence from launch readiness.

Figure: A clean-versus-bugged allocation diagnostic, showing why experiment validity depends on the assignment running as designed (adapted from Lecture 02: A/B Testing and Product Experimentation).

Lecture Sequence

01. Randomized Experiments

This lecture connects Randomized Experiments to assignment, power, guardrails, noncompliance, interference, and decision reporting.

02. A/B Testing and Product Experimentation

This lecture develops A/B Testing and Product Experimentation with examples that make assumptions, diagnostics, and interpretation visible.

03. Power, MDE, Sample Size, and Practical Significance

This lecture uses Power, MDE, Sample Size, and Practical Significance to clarify the analyst’s question, evidence, assumptions, and decision implications.

04. Guardrail Metrics and Multiple Testing

This lecture shows how guardrail metrics and multiple testing shape experiment interpretation and launch risk.

05. Clustered Experiments

This lecture frames Clustered Experiments as a decision problem and asks what evidence can be trusted, challenged, and communicated.

06. Noncompliance, Intent-to-Treat, and Treatment-on-Treated

This lecture builds intuition for Noncompliance, Intent-to-Treat, and Treatment-on-Treated and ties the result to model choice, uncertainty, and action.

07. Interference and Network Effects

This lecture applies Interference and Network Effects with emphasis on diagnostics, tradeoffs, and evidence limits.

08. Experiment Readouts for Business Teams

This lecture develops Experiment Readouts for Business Teams as a practical pattern for analysis, diagnostics, and decision support.