Systematically Improving RAG Applications
About Course
Systematically Improving RAG Applications
Stop building RAG systems that impress in demos but disappoint in production
Transform your retrieval from “good enough” to “mission-critical” in weeks, not months. Most RAG systems stall in prototype purgatory: they demo well, but fail on complex queries—eroding trust and wasting engineering time. The difference isn’t just better tech, but a systematic mindset.
With the RAG Flywheel, you’ll:
✅ Pinpoint failures with synthetic evals
✅ Fine-tune embeddings for 20–40% gains
✅ Collect 5x more user feedback
✅ Segment queries to target high-impact fixes
✅ Build multimodal indices for docs, tables, images
✅ Route queries to the best retriever automatically
Week by week, you move from vague “make it better” to clear metrics, focused improvements, and compounding value. Real-world results include +20% accuracy from re-ranking, +14% with cross-encoders, and $50M revenue boosts from better search.
Join 400+ engineers applying this framework in production. Instructor Jason Liu has built multimodal retrieval and recommendation systems at Facebook, Stitch Fix, and through consulting—experience that shaped this practical, battle-tested approach.
What you’ll learn
Follow a repeatable process to continually evaluate and improve your RAG application
Analyze and Diagnose RAG System Performance
-
Evaluate retrieval quality using precision, recall, and MRR metrics to identify system weaknesses
-
Differentiate between leading metrics (experiments run) and lagging metrics (customer satisfaction) to drive actionable improvements
-
Design synthetic data generation pipelines that enable rapid experimentation without waiting for user data
Construct Data-Driven Improvement Frameworks
-
Create comprehensive evaluation datasets using LLMs to generate realistic query-answer pairs
-
Establish baselines using tools like LanceDB to benchmark different retrieval implementations
Design Specialized Search Systems
-
Develop multimodal retrieval systems that handle documents, images, tables, and structured data
-
Synthesize lexical (BM25), semantic (embeddings), and metadata-based search for optimal results
Optimize Query Understanding & Routing
-
Extract structured information from diverse data sources to enable precise filtering
-
Classify queries using domain expertise and few-shot classifiers to improve routing accuracy
Course Content
Topic 1: Cohort 1
-
Lesson 1: Cohort 1 Guest Lectures Building Dynamic Ai Mem
55:44 -
Lesson 2: Cohort 1 Guest Lectures Text Chunking In Rag Es
01:01:29 -
Lesson 3: Cohort 1 Guest Lectures Custom Rag Evaluations
53:37 -
Lesson 4: Cohort 1 Guest Lectures Multimodal Rag Hybrid
59:04 -
Lesson 5: Cohort 1 Guest Lectures Leveraging User Feedbac
55:16 -
Lesson 6: Cohort 1 Guest Lectures Cohere S Guide To Rag C
01:08:23 -
Lesson 7: Cohort 1 Guest Lectures Boosting Bm 25 With Gene
58:12 -
Lesson 8: Cohort 1 Office Hours Beyond Dense Embeddings Expl
25:31 -
Lesson 9: Cohort 1 Office Hours Building Resilient Rag Syste
01:00:08 -
Lesson 10: Cohort 1 Office Hours Building Scalable Systems Wi
59:00 -
Lesson 11: Cohort 1 Office Hours Data Flywheels And Fine Tu
29:16 -
Lesson 12: Cohort 1 Office Hours Evaluating Agent Performance
47:42 -
Lesson 13: Cohort 1 Office Hours Handling Real Time Insights
51:36 -
Lesson 14: Cohort 1 Office Hours Optimizing Planner And Feedb
44:24 -
Lesson 15: Cohort 1 Office Hours Smart Routing And Query Opti
42:22 -
Lesson 16: Cohort 1 Office Hours Using Ai To Streamline Query
44:32