Written by Technical Team | Last updated 21.10.2026 | 17 minute read
Many NHS digital services now sit inside clinical workflows rather than alongside them. An unavailable integration can prevent a referral from moving. A delayed message can leave a clinician working from incomplete information. A badly handled identifier can attach data to the wrong record. A technically successful deployment can quietly alter a workflow in a way that increases clinical risk.
The engineering problem is therefore different from building an ordinary business application. Availability still matters, but availability alone is a weak measure of whether the service is behaving safely. A system returning HTTP 200 responses while sending incomplete clinical information downstream is available in the infrastructure sense and failing in the operational sense.
High-reliability engineering offers a useful way to think about this class of system. The relevant lessons come from several overlapping disciplines: safety-critical engineering, high-reliability organisations, resilience engineering and modern site reliability engineering. They differ in terminology and history, but share an assumption that is particularly useful in healthcare: failure cannot be designed out completely, so systems must be designed to detect abnormal conditions early, constrain their consequences and recover without depending on somebody improvising perfectly under pressure.
The NHS already applies parts of this thinking through clinical safety management, hazard logs, safety cases, service management and operational resilience. The opportunity is to make those principles part of the engineering architecture rather than treating them mainly as assurance activities around it.
Most digital programmes become very good at describing the successful path. Requirements explain what should happen when a patient is found, a referral is submitted, a prescription is issued or a record is retrieved. Architecture diagrams show components exchanging data in the intended sequence. Acceptance tests demonstrate that the workflow produces the expected result.
High-reliability engineering spends much more time on what happens when one assumption in that sequence stops being true.
Consider a service that retrieves information from an upstream clinical system, transforms it into FHIR and sends it to another platform. Its nominal behaviour may be straightforward. The interesting engineering questions start when the source system returns an incomplete response, an API times out after processing a request, terminology cannot be mapped, a patient has been merged since the previous request, the receiving service accepts the HTTP request but rejects an individual resource, or a retry causes a transaction to be processed twice.
Each of those failures has different semantics. Treating them all as generic exceptions creates systems that are technically recoverable but operationally unsafe.
A useful starting point is to model failures explicitly. For each important transaction, identify its state transitions and ask what evidence exists that each transition completed. A referral, for example, should not simply move from submitted to complete. A more realistic state model may distinguish between locally accepted, transmitted, acknowledged by the receiving system, clinically accepted, rejected and awaiting reconciliation.
That allows the software to distinguish between “we know this failed” and “we do not know whether this succeeded”. The latter is one of the most dangerous states in distributed healthcare systems because an automatic retry may create a duplicate while doing nothing may lose the transaction.
Idempotency, correlation identifiers and durable transaction records therefore become safety mechanisms rather than implementation niceties. If a client retries a request because it did not receive a response, the receiving system should ideally be able to determine whether the logical transaction has already been processed. Where that is impossible, the system needs an explicit reconciliation route rather than silently guessing.
The same thinking applies to data quality. If an integration expects NHS number, date of birth and demographic information, the system needs a defined response when those data disagree. “Best effort” matching can be entirely appropriate in some contexts and unacceptable in others. The behaviour should follow the clinical consequences of a false positive and false negative, not simply the capabilities of the matching algorithm.
This is where a clinical safety hazard log becomes much more useful when it is connected directly to the engineering backlog. A hazard such as “clinician acts on incomplete medication information” should lead to concrete controls: provenance displayed alongside data, timestamps indicating freshness, explicit completeness indicators where available, monitoring for failed source queries, defined behaviour when one source is unavailable, and tests proving that partial responses cannot accidentally be presented as complete ones.
A hazard log that records “mitigated by system design” without tracing that mitigation to code, tests, monitoring and operating procedures provides much less protection.
High-reliability systems also distinguish independent failures from common-cause failures. Two copies of the same service in different availability zones provide redundancy against certain infrastructure failures. They do little if both run the same defective release, depend on the same identity provider or consume corrupted configuration from the same source.
This is a familiar problem in safety-critical engineering. Redundancy improves reliability only when the redundant paths do not share the failure being protected against.
For an NHS platform, useful questions include:
A resilience diagram showing failure domains is often more informative than another logical architecture diagram.
NHS digital resilience is not the same as uptime. High-reliability healthcare systems must be designed around clinical safety, failure containment and recovery as well as availability. That means understanding dependency failure domains, defining safe degraded modes, preventing duplicate or incomplete transactions, and ensuring that incidents can be detected and reconciled before a technical fault becomes a clinical risk.
Highly reliable systems are rarely designed on the assumption that every component must remain available continuously. They are designed so that individual failures do not automatically become total service failures.
This leads to the idea of graceful degradation. An application with six dependencies does not necessarily need to fail because dependency seven is unavailable. It may be able to preserve a reduced but useful service.
The difficulty in healthcare is that degradation needs clinical semantics.
Suppose a clinician-facing service normally assembles information from several sources. If one source becomes unavailable, there are several possible behaviours. The entire screen could be disabled. The remaining information could be displayed with a warning. Cached information could be shown with its timestamp. Users could be directed to the source system. Different data domains may require different responses.
There is no universally correct choice. The important part is that the choice is explicit, risk-assessed and testable.
Engineering teams should therefore define degraded states alongside normal states. This can be represented as a service degradation matrix describing which capabilities remain usable when dependencies fail and which controls become active. Operations teams can then monitor for those states rather than discovering them through user reports.
Timeouts, circuit breakers and queues are useful here, but only when configured according to workflow semantics. A five-second timeout copied from another service may be inappropriate for an API whose successful response routinely takes four seconds. An aggressive retry policy can turn a modest upstream problem into a major incident by multiplying load. Queuing can protect downstream systems but can also make latency invisible until a clinically relevant backlog has accumulated.
Queues in particular need operational metrics that describe work, not just infrastructure. Queue depth is useful, but the age of the oldest unprocessed clinical transaction is often more meaningful. Fifty messages delayed for ten seconds may be irrelevant. One urgent transaction sitting unprocessed for forty minutes may require immediate investigation.
This principle generalises to observability. CPU utilisation, memory consumption, pod restarts and database connections tell operators whether computers are healthy. They do not tell them whether the clinical service is healthy.
A referral platform might monitor:
successful referrals per minute,
rejection rates by receiving organisation,
time from submission to acknowledgement,
transactions awaiting reconciliation,
duplicate detection events,
identifier matching failures,
messages exceeding their expected processing window,
and the proportion of journeys falling back to manual handling.
These are service-level indicators with operational meaning.
They also make service-level objectives more useful. A target such as 99.9% availability is convenient but incomplete. A service can meet it while behaving badly for a small but clinically significant subset of transactions. Reliability objectives may need to cover transaction completion, latency, freshness and correctness as well as endpoint availability.
A patient-record service, for example, might have separate objectives for API availability and the age of data presented to users. A messaging platform might distinguish successful API acceptance from successful downstream delivery. A booking service might measure the proportion of transactions whose final state is known within a defined period.
This changes incident detection. Rather than waiting for infrastructure to cross a threshold, teams can detect that the service is departing from its expected operating envelope.
One of the strongest ideas from high-reliability organisations is a preoccupation with weak signals. A near miss is valuable because it exposes a failure path before the system produces serious harm.
Digital programmes generate these signals constantly, but many are discarded.
A message fails once and succeeds on retry. An engineer manually clears a queue. A deployment is rolled back because an integration test behaves strangely. A support analyst notices that one organisation is producing malformed data and creates a workaround. A clinician spots incorrect information before acting on it.
None necessarily becomes a major incident. Collectively, they reveal where the system is relying on luck or human intervention.
High-reliability operation means making these events observable and inexpensive to report. Teams should be able to distinguish routine noise from evidence that a safety margin is narrowing.
Repeated retries are a good example. If requests normally succeed first time and gradually start requiring a second or third attempt, the end-user success rate may remain unchanged. A conventional availability dashboard stays green. The system nevertheless has less resilience than it had before because a mechanism intended to absorb occasional failure has become part of normal operation.
The same applies to manual reconciliation. If a team spends twenty minutes each morning fixing transactions that became stuck overnight, the service may appear reliable because somebody is continuously repairing it. Reliability is being supplied by an informal human control that may disappear during leave, organisational change or a larger incident.
Those interventions should be instrumented wherever possible. A useful operations review looks not only at incidents but at retries, fallbacks, manual corrections, safety-related support cases and occasions when operators prevented an incident.
Post-incident analysis then needs to resist overly simple explanations. “Engineer deployed a bad configuration” may be factually accurate but tells us little. Why could one configuration change create that effect? Why did validation not detect it? Was the configuration schema machine-validated? Could it have been progressively deployed? Did monitoring identify the problem before users did? Was rollback automatic, documented or dependent on somebody remembering the procedure?
This is where modern systems thinking is more useful than searching for a single root cause. Software incidents usually emerge from interactions between individually reasonable decisions, dependencies and controls. The engineer making the final change may simply be the last participant in a failure path that was already present in the architecture.
Good corrective actions therefore alter the system. “Remind engineers to check configuration carefully” is weak. Schema validation, automated tests against representative configurations, four-eyes approval for particularly hazardous changes, progressive rollout and automatic rollback on defined indicators are stronger because they reduce dependence on perfect human attention.
Clinical safety incidents should feed the same engineering loop. If an incident exposes a new hazard or changes the estimated likelihood of an existing one, the hazard log should be updated. If the mitigation depends on monitoring, the relevant alert should be tested. If the incident demonstrates that an operating procedure is unrealistic under pressure, the procedure or architecture needs changing.
A safety case should describe the system that actually exists, including what has been learned in production, rather than the system as understood shortly before go-live.
For digital health systems used in England, clinical safety is not solely a supplier responsibility. NHS clinical risk management deliberately places responsibilities on both the organisation developing the technology and the health or care organisation deploying and using it.
The distinction is important for reliability engineering because many risks only become visible when software meets a real clinical workflow. Supplier assurance therefore needs to connect with local deployment, operational monitoring and continuing clinical risk management rather than ending at go-live.
| Standard | Who it applies to | What it means for reliability engineering |
|---|---|---|
| DCB0129 | Manufacturers and organisations responsible for developing and maintaining health IT systems. | Clinical hazards should influence system design, testing and technical controls. Safety documentation, including the hazard log and clinical safety case, should evolve as the product changes. |
| DCB0160 | Health and care organisations deploying, using, maintaining or decommissioning health IT systems. | The deploying organisation must consider risks created by its own workflows, configuration, integrations and operating environment, including risks that may not exist in the supplier’s generic product. |
High-reliability organisations tend to move decision-making towards the person with the most relevant knowledge when abnormal conditions occur. Large digital programmes often do the opposite.
During normal governance, hierarchy is manageable. During an incident involving an unusual interaction between FHIR mapping, an integration engine and a particular supplier’s implementation, the person best placed to make a technical judgement may be an engineer who has spent years working with that interface rather than the programme’s most senior technology leader.
Incident management should reflect that.
This does not mean abandoning clear command structures. Effective incident response normally benefits from separating coordination from diagnosis. An incident commander can manage priorities, communications and escalation while technical specialists investigate the failure. The commander does not need to be the person typing commands into production, and the deepest technical specialist should not simultaneously have to run stakeholder communications.
A practical incident structure might identify an incident lead, technical leads for affected domains, somebody responsible for communications and a record keeper maintaining an event timeline. For clinically significant incidents, the Clinical Safety Officer or appropriate clinical representation needs a defined route into decision-making.
The most valuable preparation happens before the incident. Service ownership should document which people or teams understand each critical dependency, including suppliers and external NHS services. Escalation routes should be usable at 02:00 rather than merely present in a contract. Runbooks should contain commands, dashboards, decision points and recovery steps rather than broad statements such as “contact infrastructure team”.
Recovery procedures also need testing.
A database backup is not evidence that a service can be restored. Restoration needs to be exercised, timed and verified against realistic data volumes. Failover capability is not proven because the architecture diagram contains a secondary region. Teams need to know whether identity, secrets, DNS, network routing, observability and external integrations actually work after failover.
The same principle applies to manual fallbacks. An operational continuity plan may state that staff will revert to a manual process, but that process may no longer be viable at current transaction volumes. High-reliability engineering tests the assumption rather than accepting the document.
Game days can be useful for complex NHS services. A team can deliberately simulate an unavailable upstream API, expired certificate, inaccessible identity service, delayed message queue or corrupted deployment. The objective is not theatrical chaos engineering in production. It is to discover whether monitoring identifies the condition, whether alerts reach the right people, whether the service degrades as designed and whether recovery instructions are accurate.
These exercises are particularly valuable across supplier boundaries. A service may be resilient internally but still depend on assumptions about how another organisation detects, communicates and recovers from failure. Exercising the boundary reveals those assumptions.
The most difficult reliability problem is not keeping a static system stable. NHS services change continuously: software releases, supplier upgrades, new organisations onboarding, terminology updates, certificate rotations, infrastructure patches, workflow changes, new integrations and changes to national APIs all alter the system’s behaviour.
Every change modifies the failure surface.
The answer cannot be to prevent change. Systems that become too difficult to change accumulate unsupported components, security vulnerabilities and brittle workarounds. High reliability depends on making change small, observable and reversible.
Progressive delivery is useful for this reason. A release can be exposed initially to internal traffic, a limited set of organisations or a small percentage of transactions. Monitoring then determines whether it should continue. Feature flags can separate deployment from activation, allowing code to reach production without immediately changing behaviour for every user.
The rollback path should be designed at the same time as the change. Database migrations are a common source of difficulty because application rollback may become impossible once the schema has changed incompatibly. Expand-and-contract migration patterns, backwards-compatible schema changes and staged removal of old fields can preserve reversibility.
Interface changes require similar care. Where multiple organisations consume an API, changing the producer and every consumer simultaneously is rarely realistic. Versioning, tolerant readers, contract testing and clear deprecation periods reduce coordination risk. Consumer-driven contract tests can identify when a provider change breaks assumptions made by specific downstream systems before the change reaches production.
Synthetic monitoring can test important journeys continuously. Rather than simply checking that /health returns a response, a synthetic transaction can exercise authentication, lookup, data retrieval and another critical path using controlled test data. This catches failures that infrastructure health checks miss.
Synthetic transactions need to be designed carefully in healthcare systems so that they cannot be mistaken for real clinical activity, contaminate production records or trigger downstream processes. Where full production journeys cannot safely be exercised, representative probes at carefully selected boundaries can still provide useful assurance.
High-reliability programmes also manage reliability as a finite engineering resource. If a service has suffered repeated instability, continuing to push functionality at the same rate is usually counterproductive. An error-budget approach can make this explicit: establish an acceptable reliability target, measure how much failure has occurred against it, and change engineering priorities when the service consumes too much of the allowed budget.
The NHS context may require more nuanced measures than a conventional web-service error budget. A severe clinical failure should not become acceptable merely because there were millions of successful transactions around it. Reliability budgets therefore need to sit beneath clinical safety constraints rather than replace them.
The useful principle is the feedback mechanism. Reliability performance should affect delivery planning. Repeated incidents, excessive manual recovery, deteriorating latency or persistent reconciliation backlogs should consume engineering capacity until the underlying resilience is restored.
This also prevents a common programme pattern in which the delivery team is heavily funded until go-live and then reduced to a thin support function just as the service begins generating the operational information needed to improve it. Production is where assumptions encounter real clinical workflows, real supplier behaviour, real data quality and real load. Maintaining engineering capability after launch is part of the reliability design.
A high-reliability NHS digital service is therefore not one that never fails. It is one whose engineers understand its failure modes, can see when it is moving towards an unsafe state, have constrained how far individual failures can propagate, and can recover without relying on extraordinary intervention.
Much of the required machinery is ordinary engineering: explicit state models, idempotent interfaces, bounded retries, observability, reconciliation, progressive deployment, tested recovery, good incident command and careful dependency management. Clinical safety adds another layer by forcing the team to consider not simply whether the software failed, but what the failure means in the care process around it.
The strongest programmes join those two disciplines together. Hazards influence architecture. Production telemetry tests assumptions made in the safety case. Near misses generate engineering work. Incident learning changes controls. Reliability measures influence delivery priorities. Operators, clinicians and engineers retain enough authority to act when the system behaves differently from the model.
That is closer to how high-reliability systems are actually built: not by making a large promise about robustness at the beginning of a programme, but by continuously reducing the number of ways in which an ordinary technical fault can become an exceptional clinical event.
Is your team looking for help with technology delivery? Click the button below.
Get in touch