Clinical AI workspace
Hoopcare · workspace
·
Members workspace · research use only — not for clinical decision-making · every patient on this site is synthetic
Hoopcare · Preop AI · reference architecture · draft for discussion
Preop AI Platform architecture & options
One picture of the whole system — from questionnaire and uploaded documents, through the LLM readers, the rules, the Mayo ML models and the guideline knowledge base, to the two reports the platform hands back — followed by the decisions that are still open and what each option costs.
1 · Overview
The system in one picture
Read left to right. Intake produces a structured preoperative record; three engines read that record; their outputs are assembled into modules; a clinician approves; two reports go out. Colour encodes which engine produced a thing — that encoding is used everywhere on this page.
Rules — deterministic, versioned
ML — Mayo LightGBM models, validated
LLM — reads, extracts, drafts
Knowledge base — retrieved, cited
Human — confirms, signs off
How to read it
- Documents go through the LLM; the questionnaire does not. Typed answers and the procedure choice land in the record directly. Only uploaded documents pass through a reader, and every extracted field is confirmed by a clinician before any engine sees it.
- The three engines are peers with different kinds of authority. Rules decide what a guideline says; the ML models estimate probabilities and quantities; the LLM reads, retrieves and drafts. Section 2 spells out where each is trusted.
- Sign-off precedes reports. The clinician approves the assembled results once; the provider report and its plain-language patient version derive from that approved set, and both carry the same contract version.
2 · Engines
Three engines, three kinds of truth
The architecture works because each engine is asked only the question it can answer well — and because one boundary is never crossed: the LLM does not adjust a validated number.
| Engine | Nature | Trusted for | Not trusted for | How it is validated | Where it lives |
|---|---|---|---|---|---|
| Rules | Deterministic code and lookup tables: guideline criteria coded as logic, medication hold/continue tables, CMS ASC and NSQIP list checks, risk bands. | Anything a guideline states as a criterion; anything that must be identical for two identical inputs; gating (eligibility, contraindications). | Estimating a probability; weighing many soft factors at once. | Unit tests against worked examples; each rule cites its source and version. | Your repo, versioned. Runs inside the API. |
| ML (Mayo) | Gradient-boosted trees released from the enclave as native LightGBM text: postop-complication bundle (80 models), RBC transfusion bundle (126), OR-time / LoS / ICU models, the four-axis acuity card. Calibrated, thresholded, with SHAP drivers. | Calibrated risk per outcome; expected blood units and an order-to-cover bound; time and length-of-stay estimates; ranking patients for triage. | Producing text; explaining why beyond feature contributions; anything outside its training population or the configuration's input set. | Golden test cases reproduce to documented tolerance (5e-7 / 5e-11) under pinned LightGBM 4.6; per-model leaf audit ≥ 10 patients; drift vs shipped reference distributions. | Scoring service in your container; artifacts in Blob Storage under versioned, hashed prefixes. |
| LLM + KB | Reads documents into schema-checked JSON; retrieves from versioned knowledge bases and drafts cited text; runs a consistency critic over the assembled result. | Extraction from unstructured text; turning a decision the rules or model already made into clear provider and patient wording; surfacing open questions; catching contradictions. | Changing a score. Deciding eligibility on its own. Uncited recommendations. | Extraction evals against gold labels; schema validation on every output; citation check; human confirmation of extracted fields. | Orchestrated by code in your repo (§4.2); provider behind a BAA in production. |
| Human | The clinician. | Confirming extracted fields; overriding any module with a reason; final approval. | — | Override reasons are logged and reviewed; they are the best signal for what to fix next. | Two interrupts in the flow: after extraction, before delivery. |
Why the boundary matters. The Mayo models carry a validated calibration and a set of published performance claims. A layer that nudges their output — even "within a bound" — replaces a validated number with an unvalidated one, and the badge that says which you are looking at cannot restore the property that was lost. Everything the LLM does happens before the model (extraction) or around it (text, checks), never on it. Section 4.1 draws the difference.
3 · Modules
Output modules and who produces them
These mirror the bottom row of the working sketch. Each card names the engine mix — the same colours as the picture — and the honest state of the piece today.
01
ASA class estimate
Rules over the record propose a class; the LLM explains the drivers in prose. Clinician confirms.
rulesllm
02
Acuity point card, 0–100
Four LLM-extracted acuity axes → integer points → band. Validated against ASA and the UCLA grid; behaves as a rule once fitted.
ml-derivedrules
03
Procedure risk tier
CPT → wRVU, CMS ASC list, NSQIP list, procedure family. Deterministic from reference lists shipped with the bundle.
rules
04
Complication risks (16 outcomes)
Postop-complication bundle: probability, reference rate, band, top drivers per outcome, per selected configuration.
ml
05
Blood units & order
RBC bundle: full unit distribution P(k), expected units, P90 (conformal), gate-then-size order recommendation. Calibration mandatory.
ml
06
OR time · LoS · ICU
Ops models: median and 80 % interval for durations; expected LoS and ICU time. Resource planning, not risk.
ml
07
ASC / same-day eligibility
Gate: CMS ASC list × acuity band × rules exclusions. The card supplies the patient side; the list supplies the procedure side.
rulesml-derived
08
Preop testing plan
Guideline criteria coded as rules decide which tests are indicated; the LLM retrieves the citation and writes the rationale.
ruleskbllm
09
Referrals & consults
Rules flag triggers (cardiology, sleep, anaesthesia clinic); the LLM drafts the referral question with the relevant findings.
rulesllm
10
Optimization targets
Anaemia / iron, glycaemia, smoking, nutrition, prehab. Rules detect; KB supplies the pathway; LLM words it. Never subtracted from a risk number in the headline.
ruleskbllm
11
Medication plan
Hold / continue / bridge tables from the KB applied by rules to the reconciled list; LLM produces the patient-facing schedule.
ruleskbllm
12
Anesthesia notes
Airway, access, monitoring and technique considerations retrieved from protocols and drafted for the anaesthesia team; cited.
llmkb
13
Patient instructions
Fasting, arrival, what to bring, what to stop — assembled from a versioned instruction library, EN / FR, reading level checked.
ruleskb
14
Questions for the surgeon
Open items the record cannot resolve (staging, planned approach, expected blood loss). Drafted by the LLM, listed for the clinician.
llm
15
Consistency & safety check
A critic pass over the assembled result: contradictions, missing confirmations, out-of-range inputs, uncited claims. Rules assert; the LLM explains.
llmrules
16
Follow-up & prehab tasks
Dated tasks for patient and team derived from the plan (repeat labs, stop smoking date, prehab sessions).
rulesllm
4 · Decisions
Decisions and options
Seven choices shape the build. Each is drawn or tabled as a difference, with a recommendation. Tags: decided already settled in the 2026-07-30 plan · recommended · not advised.
4.1 · Where the LLM sits relative to the model
4.2 · Orchestration: plain services, LangGraph in your repo, or vendor-hosted
| Option | Portable to Azure / AWS | PHI exposure | Human-in-the-loop & state | Effort | Lock-in |
|---|---|---|---|---|---|
| A · Plain services | Fully — a container | Stays in your cloud | Hand-built | Low for scoring, medium for the agent flow | None |
| B · LangGraph code + LangSmith | Fully — the graph is Python in your repo; LangSmith is an environment variable | Traces leave unless redacted or self-hosted; synthetic-only in staging | Built in (interrupts, checkpoints) | Medium | Low; observability is optional |
| C · Vendor-hosted / no-code builder | Only by self-hosting their runtime (enterprise terms) or rewriting | All payloads cross the boundary | Built in | Lowest to start | High |
4.3 · Hosting
| Component | Azure (chosen tenant) | AWS equivalent | Status |
|---|---|---|---|
| Front-end | Static Web Apps (Next.js static export, Entra ID gate, linked /api/*) | Amplify Hosting / S3 + CloudFront + Cognito | decided 2026-07-30 |
| API + engines | Container Apps (scale-to-zero, HTTPS) | App Runner / ECS Fargate | decided |
| Model artifacts | Blob Storage, versioned prefixes, immutable | S3 with object lock | decided |
| LLM | Azure OpenAI in-region (BAA available) — or the training-time model family for the _LLM configurations, see 4.5 | Bedrock | quota requested |
| Knowledge base store | Azure AI Search (hybrid vector + keyword) or Postgres + pgvector on Azure Database | OpenSearch / Aurora pgvector | pgvector first — one database also serves the LangGraph checkpointer |
| Workflow state | Azure Database for PostgreSQL | RDS Postgres | recommended |
| Observability | Application Insights (no payloads) + LangSmith (redacted) | CloudWatch + LangSmith | recommended |
| Later | Front Door + France Central region · AKS or Azure ML endpoints only if traffic or governance demands | Global Accelerator · EKS / SageMaker | deferred |
4.4 · Which model configuration the form can drive
| Configuration | Inputs it needs | Calculator can supply | Verdict now |
|---|---|---|---|
| light_noICD_noLLM | ~15 NSQIP-style core variables | Nearly all, directly | first target |
| standard_ICD_noLLM | ~50 + 849 ICD presence flags | Core yes; ICD flags only if checkbox→ICD mapping is documented and validated | second, after the mapping audit |
| light_ICD_LLM · standard_ICD_LLM | + LLM acuity axes | Only via the note drop-zone, and only with the training-time extractor (4.5) | after extractor parity |
| full_ICD_LLM | ~1,700 features, most EHR-only | No — imputing ~1,300 unobserved axes misrepresents the model | not a calculator model · EHR phase |
| SUPER LIGHT (21) · WEB-BASED (42) | Form-sized tiers already defined in the feature file | Yes, by construction | train them (2026-07-30 plan, Option A); SUPER LIGHT = NSQIP's 21-input budget, a like-for-like comparison |
The deployment session must produce, per configuration, the count of features that are direct, derivable, lookup (table shipped with the bundle) or EHR-only. That table — not this one — decides what the calculator serves. Silent imputation is how a demo quietly lies; every imputed field is named in the response.
4.5 · LLM provider and extractor drift
The llm_* features the models were trained on were produced by one specific instrument — the NB01b decomposed-acuity prompt on Gemini 2.5. An extractor that is a different model or a paraphrased prompt yields axes from a different distribution: no error, wrong risks. Two safe positions:
now Structured first
- Serve
noLLMconfigurations. - The LLM only pre-fills questionnaire fields from a pasted note; the clinician confirms; the badge shows what was pre-filled.
- Any provider works, behind a BAA in production.
later Parity for the _LLM configurations
- Same prompt, same model family as training, or
- a documented agreement study between the training extractor and the serving extractor on the enclave-side gold labels before trusting the axes.
- State plainly in the UI when the note path is an approximation.
4.6 · EHR integration path
4.7 · Where the "medical brain" lives: Langflow, LangGraph code, or your own console
The question behind this one: should the combined logic — sources, rules, prompts, model calls, integration options — be authored on a visual canvas such as Langflow, with a Hoopcare UI on top, and LangSmith kept only for testing models and collecting metrics? It can. Whether it should depends on which part of the brain you mean.
| Option | What it is | Good at | Weak at | Portable | Verdict |
|---|---|---|---|---|---|
| Langflow canvas as the system of record | Open-source visual flow builder (MIT, self-hostable in Docker); flows export as JSON; custom components are Python; flows run behind Langflow's own API. | Prototyping a reader prompt or a RAG chain in minutes; letting non-engineers see the flow; swapping models to compare. | Diffing and code-reviewing a JSON canvas; unit-testing branches; expressing deterministic clinical rules; versioning tied to the model contract; a Hoopcare UI on top inherits Langflow's API and component model. Its server has had a critical unauthenticated code-execution vulnerability (2025) — self-host internal-only, authenticated, patched. | Yes — Docker on Azure or AWS; JSON + Python travel | not as the source of truth |
| LangGraph code + LangGraph Studio + your own console | The graph is Python in your repo (§4.2 B); Studio renders that graph visually from the code; a thin internal console (the same Next.js stack as the calculator) shows registry, versions, KB snapshots, prompt hashes, eval results. | Testable, reviewable, versioned with the models; the picture is generated from the code so it can never drift from it; human-in-the-loop built in. | Editing requires a developer; the console is something you build. | Yes — the graph moves with the container | recommended for the brain |
| Hybrid: Langflow to prototype, code to ship | Explore reader prompts, retrieval settings and model choices on the canvas; once a chain is right, promote it into the repo as a graph node with tests and a golden set. | Speed of iteration where it helps; rigour where it matters. | Two places to look during the transition; needs a rule for what "promoted" means. | Yes | practical middle |
| LangSmith for evals and metrics | Datasets, model-versus-model comparisons, standard metrics, tracing — for any of the three above. | Exactly the "test different models and get standard metrics" job. | Traces leave your cloud unless redacted or self-hosted (§5). | Datasets and results export | yes, as a side channel |
The distinction that settles it. The brain has two kinds of content. Deterministic medical logic — ASA rules, testing criteria, medication tables, eligibility gates, thresholds — is data plus code with tests, cited to its source, versioned with the model contract; it does not belong on a canvas. LLM chains — readers, retrieval, drafting — are where a canvas helps you iterate. Prototype the second on Langflow if you like; keep the first in the repo; and let the picture people look at be generated from what actually runs, so the diagram on this page and the system in production cannot disagree.
5 · Boundary
PHI boundary and environments
| Staging (now) | Production (later) | |
|---|---|---|
| Patients | Synthetic only — the schema has no identifier fields, so there is nowhere to put one decided | Real, under BAA-covered services and a private network |
| Pasted notes | Processed in memory, never persisted; no note text in logs or traces; click-through acknowledgement before the panel opens | Same, plus retention rules and audit |
| LLM calls | Azure OpenAI in-region | Provider under BAA; the _LLM configurations require extractor parity (4.5) |
| Traces / evals | LangSmith with hide_inputs / hide_outputs or an anonymiser, or App Insights without payloads | Self-hosted observability inside the tenant, or BAA |
| Access | Entra ID + e-mail allowlist; "research use only — not for clinical decision-making" on every output | Institution SSO; role-based; audit log |
| Regulatory posture | Authenticated research tool for named collaborators | Confirm SaMD / MDR position before any patient-facing use; confirm the export approval covers external hosting |
6 · Outputs
The two reports
Provider report
- Headline risks with reference rate and ratio ("3.2 % vs 1.1 % average"), band, top drivers; expandable to all outcomes.
- Blood order recommendation with the unit distribution and P90; OR time / LoS estimates.
- ASC eligibility with the reason it passed or failed.
- Plan: tests indicated (with citation), referrals, medication schedule, optimization targets.
- Open questions for the surgeon; consistency-check findings; overrides and reasons.
- Provenance footer: model configuration and hash, prompt and rules versions, KB snapshot, contract version, "research use only".
Patient report
- Plain language, EN / FR, reading level checked; no probabilities unless the clinician opts in.
- What to stop, what to keep taking, and when — as a dated schedule.
- Fasting, arrival, what to bring; prehab and optimization actions in the patient's own words.
- When to call, whom to call.
- Derived from the approved provider plan — the same contract version, so the two never disagree.
7 · Versioning
One contract version
Every report prints one version string. It resolves to a manifest that pins everything that could change the answer:
| Component | Versioned as | Lives in |
|---|---|---|
| Model bundles | content hash per (bundle, configuration, outcome); manifest with feature order, calibrator, thresholds, reference rates, golden cases, LightGBM version | Blob, immutable prefix; registry.json in the repo |
| Field schema | field_schema.json — the form, the validation and the API contract are generated from it decided | bundle |
| Rules | semantic version; each rule cites its source and effective date | repo |
| Prompts | hash + name per reader and per drafting task; the training-time extractor prompt is pinned separately | repo (LangSmith prompt hub optional, exported) |
| Knowledge bases | snapshot id per KB with document list and dates | Blob + vector store |
| Thresholds & bands | per outcome, per configuration; derived on the test split at training, re-derived per site later | bundle |
The bundle layout from the 2026-07-30 plan — MANIFEST.json · models/{outcome}.txt · imputation.json · calibration.json · metrics.json · reference_rates.json · field_schema.json — is the unit that moves. Adding a configuration is a second bundle, not a second front-end.
8 · Path
Recommended path
| Phase | What ships | Engines live | Depends on |
|---|---|---|---|
| 0 · Artifact | Inventory the released bundle; reproduce golden cases under pinned LightGBM 4.6; per-configuration feature-availability table; locate calibrators and thresholds | — | the bundle files (deployment prompt, §2) |
| 1 · Skeleton | SWA + Container Apps + domain + Entra gate; stub predictor; CI with golden cases; API contract v1 with version selection | Rules (schema, bands) | nothing — start now |
| 2 · Scoring | Real bundle behind /predict; schema-driven form; results with reference rates, drivers, named imputations; version selector | Rules · ML | 0 + 1 |
| 3 · Reading | Note drop-zone → /extract → visible pre-fill → clinician confirm; document readers one type at a time (ECG, labs first); extraction evals | + LLM (upstream) | 2 |
| 4 · Modules & reports | KB snapshots + retrieval; testing / referral / medication / instruction modules; consistency check; provider report then patient report; sign-off flow | + KB | 3 |
| 5 · Reach | France Central + Front Door; SUPER LIGHT / WEB-BASED tiers if trained; FHIR adapter design; SaMD position confirmed before any patient-facing use | all | demand |
Open questions carried from the plan. Which outcomes form the headline set (NSQIP shows ~15 with a clear primary); French clinical wording for the 42 field labels; whether the export approval covers external hosting; who the first users are; and — new — which knowledge bases are in scope for the first release and who curates them.
Sources: working sketch "Hoopcare Preop AI-Agent, August 2026"; ai.hoopcare.com architecture plan (2026-07-30) and setup runbook (2026-08-03); NB12 / NB21 / RBC handoffs; deployment prompt for the released bundle.
9 · Code
Repositories and environments
Today: one personal GitHub account with three private repos. Target: one GitHub organisation, a small number of repos with clear ownership, and one shared brain consumed by every user-facing app as a service with a published contract — never by copying its source.
What goes where
| Repository | Contains | Publishes | Owner | Note |
|---|---|---|---|---|
| perioperative-ai | contract/ field schema, OpenAPI, FHIR mapping, thresholds, reference rates · rules/ with tests and citations · readers/ prompts + eval sets · graphs/ LangGraph flows · knowledge/ KB pipeline + snapshot manifests · models/ registry, loader, calibration, golden cases · reports/ EN/FR templates · service/ FastAPI · evals/ | container image; @hoopcare/perioperative-ai-client (TS) and a Python client, both generated from the contract; registry.json | clinical-ai team (you) · CODEOWNERS on rules/ and contract/ | The single source of truth for medical logic. Semantic versions; the report's contract version = brain release + bundle hash. |
| surgery-risk-calculator | NSQIP-like calculator, Epic-like browser simulator, demo data; two apps in one small TS workspace if they share components | SWA build | you | Synthetic only, Entra-gated. Pins a brain client version. |
| patient-app (v3) | direct-to-patient app | SWA / mobile build | you | Same brain, patient-report endpoint. Consider whether hoopcare-patients already is this repo. |
| v2 · clinical-dashboard, patient, back-office | as today | their pipeline | platform team | Move into the org unchanged; adopt the brain client when a v2 screen needs a module. |
| infra | Bicep / az scripts per environment; GitHub Environments with required reviewers; OIDC federated credentials to Azure (no long-lived secrets) | — | platform team | dev · stg · prod-us · prod-fr; the calculator demo deploys to stg-like resources. |
| rag-perioperative-guidelines | today: a separate Python repo | — | you | Fold the pipeline code into perioperative-ai/knowledge/; keep the documents and snapshots in Blob (size and licensing), with only manifests in git. |
Rules that keep it clean
- An organisation, not a personal account. Teams, branch protection, CODEOWNERS, environments with required reviewers, audit log, mandatory 2FA. Transfer the three repos (redirects are kept). GitHub Team plan is enough; environment protection on private repos needs it.
- The brain is consumed over HTTP with a pinned client version. No git submodules, no copied source. Breaking contract change = major version; consumers run contract tests against the pinned image in CI.
- Not in git: model binaries (2.2 GB exceeds release-asset and LFS limits, and access is governed by the Mayo approval) → Blob, immutable, hashed in
registry.json; guideline PDFs → Blob; secrets → Key Vault and environment secrets; anything resembling PHI → nowhere. - Region is an environment, not a branch. The same brain image deploys to prod-us and prod-fr from the same tag; only configuration differs. Data, models and the LLM endpoint stay in-region.
- Clinical logic has clinical owners. CODEOWNERS on
rules/,readers/andcontract/requires your review; every rule cites its source and effective date.
10 · Surface
Site map — one members-only subdomain first
ai.hoopcare.com is the internal workspace: one Next.js shell, one auth realm (Entra ID allowlist), route groups per section, the brain linked at /api/v1. Three ideas organise the menu: /brain is everything that defines an answer; /clinical-review is the evidence about those answers; /demo is where you try it. Nested in the menu, flat and deep-linkable in the URLs. The real patient app is the exception — different identity provider, real PHI, per-region — it gets its own host (patients.hoopcare.com) when it ships.
| Path | Section | What it shows | Reads from |
|---|---|---|---|
| / | Home | members only · environment banner · "research use only" | — |
| /docs/ | Documentation | overview · architecture (this page) · contract · model cards · changelog | repo docs |
| /demo/ | Surgical calculator + Epic-like simulator | synthetic only · version selector · fake patient flow | brain API · /clinical-review/synthetic |
| /brain/ | The brain | map engines map generated from the code · chains LLM readers & drafting flows — the visual builder · rules deterministic logic with citations and effective dates · prompts versions + linked eval sets · models registry: bundles, configurations, hashes, golden status, thresholds, reference distributions, model cards · knowledge KB ingestion, snapshots, coverage, citation checks · reports provider / patient templates EN / FR · playground run any module on a synthetic case | perioperative-ai repo · registry.json · Blob |
| /clinical-review/ | Benchmark & continuous evaluation | metrics model vs model, extraction evals, calibration, drift · queue clinicians grade outputs against a rubric; disagreements feed the next release · cases every run: inputs, version, outputs, overrides and reasons, trace link · synthetic the shared fake-patient library used by demo, evals and playground | API log · evals · LangSmith (redacted) |
| /developer/ | Developer | OpenAPI · client packages · sandbox keys · FHIR mapping · SMART-launch test in the simulator | contract |
| /status/ | Status | deployed version per environment · golden-case checks · ML drift alerts · LLM endpoint health · cost | CI · App Insights · LangSmith |
| /admin/ | Admin | allowlist · roles · environments · feature flags · region | — |
| patients.hoopcare.com | Real patient app (v3), later | separate host: Entra External ID, PHI, per-region; same repo family and pipeline | brain API · patient-report endpoint |
Observability: what LangSmith covers and what you build
| Need | Where it surfaces | Build or buy |
|---|---|---|
| Tracing — runs, latency, tokens, cost per case | /clinical-review/cases → trace link | LangSmith, redacted; App Insights for the ML/API side |
| Datasets, offline evals, experiment comparison ("test different models") | /clinical-review/metrics | LangSmith; gold sets live in the repo |
| Human grading of outputs against a clinical rubric | /clinical-review/queue | build — PHI-safe, override reasons, inter-rater agreement; push scores to LangSmith optionally |
| Online evaluators on sampled live traces (schema validity, citation present) | metrics · drift | LangSmith |
| Feedback from clinicians attached to the run | queue → metrics | LangSmith feedback API or your own table, kept in sync |
| Prompt versions and playground | /brain/prompts · /brain/playground | repo is the source of truth; hub optional |
| Trace → dataset ("add this case to the gold set") | queue → metrics | LangSmith |
| Regression tests in CI — eval score below threshold blocks the release | CI gate | LangSmith SDK + pytest |
| ML monitoring — LightGBM calibration drift, input drift vs shipped reference distributions, golden checks per configuration | /status · /clinical-review/metrics | build — LangSmith is LLM-centric |
| Rules coverage and audit — which rules fired, source, version | /clinical-review/cases | build |
| Safety evals — prompt injection on readers, out-of-scope inputs, PHI-leak check on the patient report | metrics · CI | build, run in CI |
| Region and retention — prod-fr traces stay in the EU | — | LangSmith EU region or self-hosted; short retention |