Applied AIwith Sawan Ruparel

Capstone project

Build a working system, measure it, and explain its limits.

Public self-study edition

The project

Build a working AI system for a clearly scoped problem. Establish what success means before choosing a model or architecture. Keep the scope small enough to build, run, and evaluate end to end.

The guide adapts the existing Week 15 project criteria for independent study. Use the rubric for self-assessment or peer review; the public edition does not provide instructor grading.

Deliverables

  1. A working application: a Git repository with pinned dependencies and setup instructions that someone else can follow.
  2. A demonstration: up to 15 minutes, live or recorded, covering a normal case, a challenging case, and a failure case.
  3. An architecture writeup: 4–6 pages describing the problem, data flow, alternatives, selected design, and operational limits.
  4. Evaluation results: a held-out set, baseline comparison, and measured quality, cost, and latency. Include machine-readable case results.
  5. Failure analysis: concrete examples, categorized failure modes, observed frequencies, and proposed improvements.

Technical ingredients

The original project asks you to combine at least three of these ingredients:

  • Retrieval through vector RAG, GraphRAG, or hybrid search.
  • An agentic loop with tool use.
  • A custom MCP server or a substantial MCP integration.
  • Multi-agent orchestration.
  • Human approval points or persistent memory.
  • Fine-tuning, distillation, or routing across model sizes.
  • A safety layer with evaluation coverage.

Select ingredients because the problem calls for them. Compare the result with a simpler baseline and explain the cost of added complexity.

Phase assignments, decision artifacts, and capstone milestones

In the taught offerings, learners completed three graded assignments at the end of Phases 1, 2, and 3 before dedicating Phases 4 and 5 to the Capstone Project. Rather than waiting until Week 15 to assemble evidence, each phase assignment and capstone stage produces a 1-page engineering decision artifact that feeds directly into the final capstone architecture writeup and evaluation harness:

Course pointPhase / checkpointEvidence to produce
Week 4Assignment 1 (End of Phase 1)Baseline, Control-Flow & Indexing Spec: tool-using structured-output baseline score (10+ cases), fallback state diagram for invalid schema or low confidence, and keyword/vector/hybrid indexing comparison.
Week 7Assignment 2 (End of Phase 2)Retrieval & Eval Harness Report: golden evaluation dataset ($N \ge 50$ cases) comparing naive chunking against the chosen retrieval strategy, with faithfulness/relevance/precision/recall and a taxonomy of top 3 failure modes.
Week 10Capstone proposalOne-page proposal: problem scope, candidate architecture, model candidates, and evaluation plan synthesizing Assignments 1–2.
Week 11Assignment 3 (End of Phase 3)MCP Tool Contract & Recovery Trace: custom MCP server with formal tool schemas, explicit timeout/retry/durable-state bounds, human-escalation gate, and trace logs of recovery from an injected failure.
Week 12Capstone benchmark (Phase 4)Cost-quality Pareto benchmark: benchmark at least 3 model sizes or a cascade router on your capstone evaluation set with measured quality, cost, and latency.
Week 14Capstone Stage 1 (End of Phase 4) & demo freezePareto Frontier & Red-Team Audit: latency/cost/quality comparison across model tiers or adapters, layered guardrails with adversarial prompt-injection test results, and a stable demonstration build.
Week 15Final Capstone Defense (Phase 5)Final deliverables: reproducible code, 15-minute demonstration, 4–6 page writeup, and machine-readable evaluation results.

For self-study, these are sequencing suggestions. There are no public submission deadlines or staffed office hours.

Self-assessment rubric

The following weights are reproduced from the existing Week 15 rubric. They sum to 100% of the capstone assessment.

DimensionWeightLooks like
Problem clarity10%Defined scope; "could plausibly ship"; not solved by one API call
Architectural thinking15%Diagram; explicit choices; trade-offs articulated
Engineering quality15%Runs from clean clone; sensible error handling; readable code
Evaluation rigor20%Eval set built early; numbers for quality + cost + latency; honest about scope
Failure analysis15%Categorized, quantified, fixable-vs-fundamental distinction
Demo execution10%Three scenarios; one failure shown; live or seamless recording
Writeup quality10%4–6 pages; diagrams; concrete; reads like an internal design doc
Capstone ingredients used5%At least three from the required list (RAG, MCP, multi-agent, ...)

Demonstrating the system

Spend most of the demonstration showing actual behavior and evidence. Give a short problem introduction, walk through one architecture diagram, run the three scenarios, and explain evaluation results and failures. Keep credentials and personal data off screen. If the live system is unavailable, label a recording as a recording and explain what was tested.

Reproducibility and safety

Record your model versions, dataset provenance, package versions, environment requirements, and experiment dates. Test external actions against disposable resources. Include rejected requests and recovery behavior in your evaluation. State which results you actually measured and which remain unverified.

Week 15: Capstone develops the project patterns, evaluation strategy, cost model, safety review, and failure analysis in more detail. The resources page links to primary implementation documentation.