Designing a Shipping Contingency Path for Vendor Outages
A case study in designing a degraded-mode shipping path that preserved business operations during third-party logistics outages.
The operational problem
A business-critical shipping workflow depended on a third-party logistics provider for rates and label purchases. When that provider was unavailable, shipping could stop across operational sites.
There was no contingency path.
That made a vendor outage more than an integration failure. It removed the business's ability to complete a physical operation. Warehouses and laboratories could prepare shipments, but they could not move them.
I led the design and implementation of a fallback capability for provider outages and planned maintenance windows. The objective was deliberately narrow: preserve the ability to ship when the primary platform could not provide it.
This was a business-continuity system, not a second attempt at rebuilding an entire logistics product.
Why a fallback path was necessary
Retries are useful when a dependency is slow or briefly unavailable. They do not create an independent operating path when that dependency has a sustained outage.
The workflow needed an alternative provider boundary, not increasingly determined requests to the same unavailable one.
Under normal conditions, the integration service sent rate and label-purchase traffic to the primary logistics provider. During a qualifying failure, circuit-breaker logic directed new work through a C# wrapper around a separate carrier system. That system used a TCP-based interface rather than a modern HTTP API, so the wrapper translated between the application's integration contract and the carrier protocol.
The shape of the design was simple enough to draw:
Normal path
Rate request ─────┐
Label purchase ───┴─> Integration Service ──> Primary Logistics Provider
Degraded-mode path
Rate request ──────> Integration Service ──> Rate circuit
(no valid rate options)
│
└─> C# Carrier Wrapper ──TCP──> Fallback Carrier System
Label purchase ───> Integration Service ──> Purchase circuit ──> C# Carrier Wrapper ──TCP──> Fallback Carrier System
(requires a fallback-owned rate)
Primary-owned rate + primary purchase failure ──> refresh rates ──> fallback-owned rate
Fallback scope: preserve critical shipping, not primary-system feature parity
Rate retrieval and label purchase share the same high-level route, but they do not share identical failure semantics. Treating the whole integration as one interchangeable call would hide the state created between those two steps.
Defining what “failure” actually means
The rate workflow could query multiple carriers through the primary provider. An early production issue revealed a flaw in the first definition of failure: contingency could be triggered when one carrier failed even though other valid carrier options remained available.
That was dependency-level thinking applied to a business-level decision.
The business did not require every carrier response to succeed. It required at least one usable shipping option. A single carrier error was therefore evidence of partial degradation, not proof that the shipping capability was unavailable.
The circuit-breaker criteria had to reflect the outcome the application was responsible for preserving.
Define failure based on whether the business capability is unavailable, not whether one dependency call failed.
This distinction matters anywhere an integration aggregates results. If one source fails but the system can still produce a valid outcome, failover may reduce availability rather than improve it.
Designing degraded mode instead of feature parity
The fallback system did not reproduce every carrier, service level, or workflow supported by the primary platform. That was intentional.
Feature parity would have increased delivery time, testing scope, and operational complexity. It would also have made the contingency system another large platform requiring independent upkeep, even though its purpose was temporary operation during an exceptional condition.
Instead, the fallback supported the smallest useful capability: obtain compatible rates and purchase labels so critical shipments could continue.
Degraded mode should be visibly and operationally narrower. The question is not, “Can the backup impersonate the primary system perfectly?” It is, “What must remain possible for the business to keep operating safely?”
A contingency path does not need to reproduce the primary system. It needs to preserve the critical business capability.
State ownership determines failover boundaries
Rate retrieval and label purchase form a sequence. The first step creates state that the second step consumes.
Rates returned by the primary provider contain identifiers owned and understood by that provider. The fallback carrier system cannot use those identifiers. This creates an important boundary when provider health changes between the two operations.
Consider this sequence:
- The primary provider returns a rate while its circuit is closed.
- The provider becomes unavailable before label purchase.
- Label purchase using the primary provider's rate identifier fails.
- The user refreshes and requests rates again.
- The open circuit routes that new request to the fallback system.
- The user selects the fallback-compatible rate and proceeds.
Automatically translating an in-flight primary rate into a fallback rate would require reconciling two providers' state, pricing, and identifiers during an outage. I chose not to introduce that complexity into the contingency boundary.
The failed label attempt and rate refresh are an intentional tradeoff, not an invisible claim that two different systems share state.
Failover boundaries are determined by state ownership, not just service availability.
Seamless failover is easy to promise when drawing stateless boxes. It becomes more specific once one of those boxes has already issued an identifier another box cannot interpret.
Local circuit state and acceptable inconsistency
Circuit-breaker state lives in memory on each integration-service instance. As a result, instances may briefly disagree about whether the primary provider is healthy.
One instance may have observed enough qualifying failures to open its circuit while another has not. During recovery, instances may also probe and close their circuits at slightly different times.
Centralizing circuit state could make the fleet's decision more consistent. It would also add shared infrastructure and another operational dependency to a path whose purpose is surviving dependency trouble.
For this system, local state was an acceptable compromise:
- Outages and maintenance windows were temporary.
- The fallback offered degraded operation rather than a permanent routing mode.
- The consequences of brief disagreement were judged to be bounded relative to the business risk.
- Central coordination would add complexity to design, operation, and recovery.
This is not a universal argument for local circuit breakers. If temporary disagreement could create duplicate financial actions, violate ordering, or produce unsafe behavior, the tradeoff would be different.
Here, limited inconsistency was preferable to making the resilience mechanism depend on a new coordination system.
Temporary inconsistency can be acceptable in degraded mode when its consequences are understood and the tradeoff is explicit.
Recovery probing is part of the design
Opening the circuit is only half of a circuit breaker's behavior. The system must also decide when and how to test whether the primary provider has recovered.
The first implementation attempted a recovery probe after a short cooldown. In production, that interval was too aggressive and was increased.
The important lesson was not the eventual number. A timeout copied from a library example says little about how a particular vendor fails, how long its maintenance transitions last, or whether repeated probes add pressure during recovery.
Recovery behavior is part of the integration contract. It should be tuned using observed outage and recovery patterns, then revisited as those patterns change.
A circuit breaker is not finished when it opens correctly. Recovery probing deserves the same design attention as failure detection.
What production taught me
A planned vendor maintenance window gave the contingency path something test environments could not fully provide: real operational traffic while the primary provider was intentionally unavailable.
The fallback carried shipping work, and that use exposed an issue in the wrapper's TCP response handling. The downstream carrier protocol defined an explicit response terminator, but the implementation made an incorrect assumption about when enough bytes had arrived from the stream. A targeted production hotfix corrected the framing behavior so shipping could continue.
The protocol issue is a separate story, but its appearance here matters. A contingency path can compile, pass integration tests, and still contain assumptions that emerge only under real response sizes and timing.
The lesson is not that fallback systems are unreliable. It is that rarely used systems need deliberate operational exercise precisely because production does not care how reassuring the architecture diagram looked.
A fallback path that is never exercised is an assumption, not a capability.
Planned maintenance was useful not merely because the backup switched on, but because it tested routing, protocol handling, operational response, and recovery while people were prepared to observe them.
Business impact
The contingency path was used in two separate production events.
During one provider outage, it generated approximately 750 shipping labels while the primary provider was unavailable.
During a different, planned maintenance window, it supported shipping approximately 200 cases from one operational laboratory. When the TCP response-handling issue surfaced during that window, I diagnosed it and deployed the targeted hotfix so the affected shipping work could proceed.
Those numbers should not be combined. They describe different events and different units of work. Together, they demonstrate the same outcome: the organization retained a critical shipping capability during periods when its normal provider path was unavailable.
The contingency design did not eliminate operational judgment, and it did not make every transition seamless. It created a narrower path with understood constraints—one that the business could actually use when the primary path disappeared.
Reusable engineering principles
Define failure in business terms
A dependency error matters when it prevents the system from producing the required outcome. Aggregated workflows should not fail over merely because one participant failed.
Preserve capability, not symmetry
Build the smallest contingency path that keeps the essential operation available. Full feature parity can make the backup too expensive and complicated to trust.
Follow the state
Failover is easiest before provider-specific state has been created. Once an identifier, reservation, or quote belongs to one provider, crossing the boundary may require a new business operation rather than transparent rerouting.
Make inconsistency an explicit decision
Local resilience state can be reasonable when disagreement is temporary and its consequences are bounded. Document why it is acceptable; “eventually all instances figured it out” is not a consistency model.
Design recovery, not only failure detection
Cooldowns and probes influence both upstream recovery and user experience. Tune them against the dependency's real behavior.
Exercise the path that is supposed to save you
Fallback code ages quietly. Use controlled maintenance windows and operational drills to validate the entire capability, including the work people must do when failover is not seamless.
Business continuity is not created by the presence of a second integration. It comes from explicit failure semantics, constrained scope, honest state boundaries, and evidence that the degraded path works when the normal one does not.