A consultation-note assistant for doctors

Context

After every consultation, a doctor writes a note. The work is indispensable — it forms the medical record, the handover to colleagues, the trace in case of dispute — and it happens when the next patient is already waiting, or in the evening, piled up. It is doctor time spent on data entry.

The consultation itself already contains everything needed: the reason for the visit, the history mentioned, the examination, the decision. The note is largely a structured restatement of what was just said — precisely the kind of task a language model does well.

I built this application end to end — Spring Boot backend, React frontend, Stripe billing — to cover the whole chain: recording, transcription, note generation, patient records, subscription. It then went through a beta, and that contact with reality rewrote a good part of the architecture.

Constraint

Three constraints shaped the project, and they pull in different directions.

The first is the nature of the data. A medical consultation is the most sensitive category of personal data there is. Every piece of data kept is data to protect, to justify, and one day to account for. The right strategy is not to encrypt better: it is to keep less.

The second is responsibility. A language model cannot make a diagnosis. The doctor is accountable for what they sign, legally and ethically. A system producing an authoritative-looking medical conclusion would be dangerous — not only because it can be wrong, but because it would quietly move the decision away from the practitioner.

The third is the most mundane, and it nearly killed the product: the doctor is in consultation. A tool that imposes a minute of staring at a frozen screen will not be used, however good what it eventually produces.

Decisions

Transcribe during the consultation, not after it. The first version did the obvious thing: the browser recorded, sent the audio file to the backend at the end, and transcription started there. On a consultation of twenty minutes or more the wait became indefensible — and it landed at the worst possible moment, when the doctor wants to wrap up and move to the next patient. The fix was not to make transcription faster but to stop doing it at that point: the browser now transcribes live, during the consultation, talking directly to the speech recognition service. When the consultation ends, the transcript is already there. So that the browser can address the service without holding a secret, the backend mints a short-lived token on demand: the master key never leaves the server. This decision, taken for latency, incidentally settled the first constraint: audio no longer passes through my infrastructure and is never stored there — the Consultation entity has no audio field, and the only file the API accepts is an optional contextual image. The trade-off is real: with no recording, there is no re-transcribing later and no way to check against the source. I accept it readily, because a server accumulating hours of recorded consultations would be by far the largest risk in the system.

Don’t transcribe silence. A consultation is full of moments without speech: the physical examination, taking blood pressure, the doctor writing, the patient getting dressed. Streaming that silence to a transcription service billed by the minute means paying for nothing. The client therefore detects voice activity and suspends the stream when nobody is speaking. The naive implementation — a single threshold on volume — produces two classic defects: it oscillates around the threshold, and it clips the beginnings and ends of sentences. The version I kept smooths the signal energy with a moving average, uses two distinct thresholds for opening and closing, retains half a second of audio before speech is detected, and holds the stream open for a second after it stops. In other words: only cut when certain, and never mid-word. The saving on the transcription bill is substantial, and the text loses nothing.

Split extraction from reasoning, and pay for the expensive model only where it counts. The pipeline makes two separate calls rather than one. The first, on a light model at zero temperature, extracts only the facts actually stated — presenting complaints, history taking, vital signs — into a strict JSON schema. The second, on a considerably stronger model, never sees the raw transcript: it receives only that JSON of facts, and from it derives hypotheses, a proposed management plan and follow-up. Doing it all in one call would have been simpler. The separation buys three things. First, safety: a hallucinated fact cannot silently become a diagnostic argument, since the reasoning step works only from facts already fixed and inspectable. Second, cost: the bulk of the token volume — a full transcript — is absorbed by the cheaper model, and the expensive model only handles a compact JSON, exactly where its advantage actually shows. Third, traceability: both intermediate JSON documents are retained, which makes every note traceable back to the facts it came from. Without that trace, “I don’t know why the system wrote that” would be the only possible answer — and in this domain, that is not an answer.

Constrain the model to propose, never to conclude. The instructions require hedging language — probable, possible, to be discussed, differential — and forbid a firm diagnosis while uncertainty remains. Hypotheses are capped at three, each carrying a probability and a justification anchored in the extracted facts: the doctor sees what the machine is relying on, and can therefore contradict it. Any medication proposal must carry the international nonproprietary name, the dosage, the duration, adjustments for renal impairment and for elderly patients, and alternatives in case of allergy — all cross-checked against the allergies and current treatments on file. A separate flag raises red-flag symptoms. Outputs are validated against their schema on parse, and any structural drift is logged rather than silently absorbed. The result is not a document: it is a draft the doctor reviews, corrects, and takes ownership of.

Make the wait legible, and the generation independent of the window. Even with transcription solved, writing the note takes tens of seconds. Rather than delivering the text in one block, the final stage streams it as it is produced: the doctor watches the note being written, which completely changes how the delay is perceived. The engineering cost goes beyond the visual effect — fragments are grouped into small batches before emission so the browser is not flooded, a regular heartbeat keeps the connection alive through proxies, and database persistence is amortised every fifty batches instead of writing on every fragment. Above all, generation lives on the server, not in the tab: if the doctor closes the window it carries on, and they find the finished note on returning. The stream is resumable — a dropped connection picks up where it left off — and startup is idempotent, so a generation already running or already finished does not restart and is not paid for twice.

Outcome

The application covers the full chain: token authentication with an httpOnly refresh cookie, strict data isolation between doctors — a practitioner reaches only their own patients, and isolation is enforced server-side from the security context, never from a client-supplied identifier — patient records, note generation, and Stripe subscriptions across three plans with idempotent webhooks and hourly reconciliation. Seventeen schema migrations and roughly two hundred classes and components, over nine months of development.

Above all, it went through a beta covering roughly a thousand patients — and that is where the three most important decisions above came from. None of them was in the initial design: transcription latency, the cost of silence, and generation being abandoned when the window closes only surfaced on contact with real use.

That beta ran on non-identifying records: no real identifying details were entered, and a generated note cannot be traced back to an identified patient. It was the condition for testing the pipeline under real conditions without building up, for the sake of a test, a body of named health data — the same reasoning as for the audio, applied to the protocol rather than to the code.

What the project taught me sits in its central constraint: in a domain where mistakes are expensive, the useful question is not “how far can the model go?” but “what do we refuse to delegate to it?”. Here, three things: custody of the audio, the diagnostic conclusion, and the signature. The rest of the architecture follows from those three refusals — and from a thousand consultations that showed where the first version was wrong.