Abby 2.0 Phase 3: The Knowledge Graph — She Understands Concept Relationships
Abby now understands that metformin is a drug used for Type 2 diabetes mellitus, which is a subtype of diabetes mellitus. She traverses OMOP concept hierarchies, finds siblings and related concepts, and warns researchers when they're building on sparse data. Phase 3 turns keyword matching into relational understanding.
The Problem: Flat Knowledge in a Hierarchical World
OMOP's vocabulary is inherently hierarchical. SNOMED CT organizes 350,000+ clinical concepts into parent-child trees. ICD-10 nests codes in chapters, blocks, and categories. RxNorm links ingredients to clinical drugs to branded products.
But Abby 1.0 treated concepts as flat keywords. Ask "What are the subtypes of diabetes?" and she'd search for the word "diabetes" in her RAG knowledge base — returning whatever documents mentioned diabetes, not the actual OMOP concept hierarchy. She had no structural understanding of how concepts relate to each other.
The Architecture: PostgreSQL + Redis, Not a Graph Database
The spec called for leveraging existing OMOP tables rather than adding a separate graph database. Smart call — the concept_ancestor and concept_relationship tables already contain the full hierarchy. We just needed to make them fast and queryable from the AI service.
KnowledgeGraphService
Five core operations, all backed by concept_ancestor and concept_relationship with Redis caching:
| Method | What It Does | SQL Table |
|---|---|---|
get_ancestors(concept_id, max_levels) | Walk up the hierarchy | concept_ancestor |
get_descendants(concept_id, max_levels) | Walk down the hierarchy | concept_ancestor |
get_siblings(concept_id) | Find same-parent concepts | ancestor → descendants |
find_related(concept_id, types) | Cross-domain relationships | concept_relationship |
get_concept(concept_id) | Single concept details | concept |
Every query result is cached in Redis with a 1-hour TTL. The cache key includes the operation, concept ID, and parameters — so get_ancestors(201826, max_levels=3) and get_ancestors(201826, max_levels=5) are cached independently. Hot paths (the top 1000 most-queried concepts) effectively become in-memory lookups after first access.
DataProfileService
Abby now maintains a living understanding of your CDM:
- Person count — total patients in the dataset
- Temporal coverage — observation period start/end dates
- Domain density — record counts per clinical domain (conditions, drugs, procedures, measurements, observations, visits, devices)
- Gap detection — three types of warnings:
- Critical: CDM has zero patients (all queries will fail)
- Sparse domain: Less than 1 record per patient (domain is unreliable for research)
- Temporal gap: Less than 3 years of coverage (longitudinal studies may lack follow-up)
Integration: Intent-Routed Graph Queries
The knowledge graph integrates into Abby's existing live context pipeline via intent detection:
"What are the subtypes of diabetes?"
→ graph_descendants intent detected
→ KnowledgeGraphService.get_descendants(4008576)
→ Hierarchy injected into prompt as CONCEPT HIERARCHY section
"Show me conditions related to metformin"
→ graph_related intent detected
→ KnowledgeGraphService.find_related(1140643860)
→ Related concepts injected into prompt
"How much data do we have?"
→ data_profile intent detected
→ DataProfileService.get_profile_summary()
→ CDM DATA PROFILE section with domain density and warnings
Data Quality Warnings as Safety-Critical Context
The context assembly pipeline from Phase 1 has a concept of "safety-critical" context — pieces that are always included regardless of token budget. Data quality warnings leverage this:
When a researcher asks about measurements and the Measurement domain has sparse data, Abby automatically includes the warning in her system prompt. She'll say "Note: the measurement domain in this CDM has limited data (500 records for 100K patients). Results may not be representative."
This prevents researchers from drawing conclusions on a foundation of missing records — a common and costly mistake in observational research.
What Shipped
| Component | Tests | Purpose |
|---|---|---|
KnowledgeGraphService | 10 | Hierarchy traversal with Redis caching |
DataProfileService | 7 | CDM coverage analysis and gap detection |
| Live context tools | — | 5 new intent patterns + 2 tool functions |
| Pipeline integration | 2 | Data quality warnings in chat prompts |
188 tests passing across the Python AI service.
What's Next: Phase 4 — Agency Framework
Abby can now remember, reason, and understand. Phase 4 gives her hands.
The Plan-Confirm-Execute framework will let Abby propose multi-step actions (create concept sets, build cohorts, queue analyses), present them for user approval, and execute them step-by-step with rollback capability. Supervised autonomy — she proposes, you decide.