Applied AI Course
Building AI systems that can be evaluated and used
Public self-study edition · Edition 2027.1 (v3.0)
From a model that answers a prompt to a system that can retrieve, act, recover, and be evaluated. A graduate-level course in the engineering of applied AI.
Work through prompting and retrieval, build tool integrations and agent workflows, then evaluate the tradeoffs in quality, cost, latency, and safety. Bring the pieces together in a working capstone.
Taught by Sawan Ruparel at the University of Connecticut in Spring 2025 and Spring 2026. This public edition makes the course materials available for independent study.
Teaching through evidence
The course connects model behavior to engineering decisions: define success, compare a baseline, inspect failures, and explain the tradeoffs. The capstone asks learners to demonstrate those decisions in a reproducible system.
Example: How this course evaluates an architectural decision
Rather than asking “Does the agent work on a demo prompt?”, each phase requires documenting system changes against a reproducible evaluation harness (illustrative capstone decision log):
| Architecture stage | Answer faithfulness (n=250) | Tool / action success | Latency (p95) | Cost / 1k queries | Primary failure mode inspected |
|---|---|---|---|---|---|
| 0. Baseline (Zero-Shot Frontier LLM) | 61.2% | N/A | 1.4 s | $4.20 | Unverified hallucination on domain-specific policy rules |
| 1. Hybrid RAG + Reranker (Phase 2) | 86.4% | N/A | 2.1 s | $4.85 | Multi-hop retrieval miss when entities span two documents |
| 2. MCP Agentic Workflow (Phase 3) | 92.8% | 89.0% | 5.8 s | $11.40 | Infinite tool-retry loop on malformed upstream API schema |
| 3. Routed SLM + Guarded Workflow (Phase 4) | 91.6% | 94.5% | 1.9 s | $1.65 | Router misclassifies edge-case adversarial prompt to fast tier |
Every architectural promotion in the capstone must justify its delta in quality, latency, unit cost, and recovery behavior.
For faculty reviewing the course, follow this short path through its design:
- Learning outcomes and preparation establish the intended level.
- A representative evaluation exercise shows the move from concepts to evidence.
- The capstone rubric makes the assessment criteria explicit.
Institutional adaptation brief · Cite this curriculum · Discuss teaching this course
The learning path
Foundations and reasoning
Weeks 1–4Understand model capabilities and inference-time scaling, engineer reliable structured prompts, design deterministic state machines before adding autonomy, and represent domain knowledge with embeddings and hybrid indexes.
Assignment 1 (End of Phase 1, Week 4): Baseline, Control-Flow & Indexing Spec
Architect hybrid and graph-based retrieval pipelines, build automated evaluation harnesses, and measure faithfulness before scaling complexity.
Assignment 2 (End of Phase 2, Week 7): Retrieval & Eval Harness Report
MCP and agentic systems
Weeks 8–11Standardize tool integration with the Model Context Protocol (MCP), coordinate multi-step agentic workflows, engineer failure recovery, and design human-in-the-loop escalation.
Assignment 3 (End of Phase 3, Week 11): MCP Tool Contract & Recovery Trace
Efficiency and governance
Weeks 12–14Optimize unit economics and latency via small language models (SLMs) and routing, distill or fine-tune deliberately against a baseline, and red-team adversarial failure modes.
Capstone Stage 1 (End of Phase 4, Week 14): Pareto Frontier & Red-Team Audit
Capstone
Week 15Build a reproducible, end-to-end AI system and defend its architectural decisions with quantitative evaluation evidence.
Final Capstone Defense (Phase 5, Week 15): End-to-End Reproducible System & Evaluation Defense
Start with a question you can measure
You will compare models and workflows, inspect failure cases, and build a system whose behavior you can explain. Three phase assignments (Weeks 4, 7, and 11) and the capstone guide turn those experiments into a reproducible project.
Begin with Week 1: the AI model landscape, or use the schedule to find a topic.