06. Causal Inference for AI Systems

AI Systems
Causal Inference
LLMOps
Lecture Notes
Causal methods for evaluating, deploying, monitoring, and governing AI products and human-AI workflows.
Published

May 3, 2026

This course flips the direction: AI systems become the intervention. It covers estimands, experiments, triggered exposure, RAG and agent evaluation, routing bias, monitoring, fairness, cost-benefit, governance, and lifecycle management.

Notebook links open rendered HTML pages generated from the source notebooks under notebooks/lectures/. Code is visible by default; rendering is configured not to execute live notebook code, so local LLM or GPU-heavy cells are not triggered during website builds.

Notebook Sequence

  1. 01. AI Systems as Interventions
  2. 02. From Model Metrics to Causal Impact
  3. 03. Estimands for AI Product Changes
  4. 04. Randomized Experiments for AI Features
  5. 05. Triggered Experiments and Exposure Logging
  6. 06. Human-AI Workflows and Post-Treatment Variables
  7. 07. Guardrail Metrics for AI Systems
  8. 08. LLM-as-Judge and Measurement Bias
  9. 09. Causal Evaluation of RAG Systems
  10. 10. Causal Evaluation of AI Agents
  11. 11. Policy Changes, Routing, and Selection Bias
  12. 12. Observational Evaluation of AI Deployments
  13. 13. Difference-in-Differences for AI Rollouts
  14. 14. Synthetic Control for Enterprise AI Rollouts
  15. 15. Off-Policy Evaluation for Logged AI Decisions
  16. 16. Interference and Feedback Loops in AI Systems
  17. 17. Fairness, Bias, and Heterogeneous AI Effects
  18. 18. Causal Monitoring, Drift, and Regression Testing
  19. 19. Cost-Benefit and Decision Analysis for AI Deployments
  20. 20. Experiment Readouts for AI Product Teams
  21. 21. Governance, Auditability, and Model Lifecycle
  22. 22. Capstone: Evaluating an AI System End to End

How To Read This Track

  • Work through the notebooks in order if you want the full course arc.
  • Treat each notebook as a lecture plus lab: read the discussion, inspect the code, and rerun locally when you want to experiment.
  • For AI-heavy notebooks, expect some brittleness when live model calls are enabled; that instability is part of the course material rather than something hidden from the reader.

The .ipynb sources remain in the matching folder under notebooks/lectures/.