Product delivery · Checklist
What production-ready means for business-critical software
“It works in staging” is not the bar for software that moves money, orders or customer data. Production readiness is a set of properties you can check before go-live. These are the eight we look for, and the questions that test each one.
Most incidents in business-critical systems are not exotic. A partner API slows down and requests pile up. A retry creates a duplicate payment. A deployment fails halfway and nobody knows how to roll back. An alert fires for a full disk while customers quietly fail to check out. Each of these has a well-understood engineering answer. The trouble is that none of them show up in a demo.
A useful definition of production-ready is therefore not “feature complete and tested”, but “behaves predictably when things go wrong, and the people who own it can tell what is happening”. The eight properties below make that definition checkable.
1. Failure handling
Every external dependency will eventually be slow, unavailable or wrong. For each one, the system needs an explicit answer.
- Timeouts on every network call, set deliberately rather than left at library defaults.
- Retries with backoff and jitter, only for operations that are safe to repeat.
- Idempotency keys on anything that creates or moves something of value, so a retry cannot charge a customer twice.
- A dead-letter queue or equivalent for messages that cannot be processed, with a way to inspect and replay them.
- Circuit breakers or load shedding where one failing dependency could exhaust shared resources.
Test question: what happens to in-flight work if your most important partner API is down for an hour?
2. Data integrity
In critical systems, a wrong answer is usually worse than no answer.
- Clear transaction boundaries, and an explicit decision about what happens when a multi-step operation fails halfway.
- Reconciliation jobs that compare your records with the systems you exchange data with, and report the differences.
- Audit trails for changes to money, orders, permissions and customer data.
- Database migrations that are reversible, or at least rehearsed, and released separately from risky code changes.
- Backups that have been restored, not just taken.
Test question: how would you find out that half a per cent of yesterday’s orders never reached the warehouse?
3. Automated tests at the right levels
The goal is not a coverage number. It is confidence that a change will not break the things that matter.
- Unit tests for business rules and their edge cases.
- Contract or integration tests for every external interface.
- A small set of end-to-end tests for the journeys that generate revenue.
- All of them running automatically on every change, fast enough that people wait for them.
Test question: could a new engineer change the pricing logic in their first week without fear?
4. Repeatable releases
Releasing should be boring.
- Builds that are reproducible from source, with dependencies pinned.
- Automated deployments through the same pipeline for every environment.
- Infrastructure defined as code and reviewed like code.
- A rollback path that has actually been exercised, and a plan for releases that include data migrations.
- Feature flags for changes whose risk lies in behaviour rather than code.
Test question: how long does it take to ship a one-line fix, and how long to undo it?
5. Observability tied to business outcomes
A dashboard full of CPU graphs does not tell you whether customers can pay.
- Structured logs with correlation identifiers, so one request can be followed across services.
- Metrics for business events: orders accepted, payments settled, messages processed, and their failure rates.
- Alerts on symptoms users would notice, routed to someone who can act, with noise low enough that alerts are trusted.
- Service level objectives for the few flows that matter most.
Test question: if checkout conversion halved right now, how long would it take someone to notice?
6. A security baseline from the first release
Security retrofitted after launch is expensive and incomplete.
- Secrets in a managed secret store, never in code or configuration files.
- Least-privilege access for services and people, reviewed when the team changes.
- Automated dependency and container scanning.
- Authentication and authorisation reviewed for the critical flows, supported by a lightweight threat model.
- Security-relevant events logged and retained.
Test question: if an engineer’s laptop were lost today, what could someone do with it?
7. Capacity you have measured
Every system has limits. The question is whether you know them before your customers find them.
- Load tests against realistic peaks, including seasonal events and batch windows.
- Known behaviour beyond those limits: queueing, back-pressure or graceful degradation rather than collapse.
- Rate limits and quotas on external dependencies understood and monitored.
- Cost visibility, so that scaling up is a decision rather than a surprise.
Test question: what happens on your busiest day of the year, multiplied by two?
8. Operability and ownership
Software is production-ready only when the people who will run it are ready too.
- Runbooks for the known failure modes, written for someone who did not build the system.
- Clear ownership, an on-call rota and an escalation path.
- Architecture decision records explaining why the system is built the way it is.
- A handover that has been practised: the receiving team has deployed, rolled back and handled a simulated incident.
Test question: could the team that will own this system recover from its most likely failure without calling the people who built it?
Using the checklist
Not every system needs every item at full depth. An internal reporting tool and a payment flow deserve different standards, and over-engineering carries its own risk. The value of a checklist like this is that it turns production readiness into an explicit decision. For each property, either it is in place, or someone has consciously accepted the risk of it being absent.
Make that decision before go-live, write it down, and revisit it when the system becomes more important to the business. Systems rarely become less critical over time.