The Error Said “No Rates.” The Rates Were Right There.

A third-party shipping provider had scheduled maintenance for the evening.

Normally, that would have been uneventful. A contingency path existed specifically so shipping could continue while the primary provider was unavailable.

But one remote operation was still shipping late into the night, so before the maintenance window began, the team manually enabled contingency mode.

Traffic moved to the fallback path. Shipping continued.

Then the errors started.

They were intermittent at first. The application reported that no rate data had been returned. The wrapper service’s persisted error information appeared to say the same thing: no rates.

That was the story the system was telling me.

It was also wrong.

The error described a belief, not reality

The fallback shipping system was hosted in cloud infrastructure but behaved much like an on-premise integration. A C# wrapper communicated with it over TCP through a NetworkStream, translating between the application and a legacy protocol.

The application said the fallback system had returned no rates.

But the explanation did not fit what I was seeing, so I stopped treating the error message as a fact.

I took the same request shape and exercised the fallback path independently.

It returned rates successfully.

That replay did not prove that the original failed exchange had returned a complete response. It did prove that the fallback could return rates for the same kind of request, which made “the shipping system has no rates” an incomplete explanation.

There was now a useful contradiction:

The application said there were no rates.

The downstream system could return rates for the same request shape.

That changed the debugging question.

I was no longer asking, “Why isn’t the shipping system returning rates?”

I was asking, “If the rates exist, where is the application losing them between receiving the response and interpreting it?”

The error told me what the wrapper believed happened. The independent response showed that reality was more complicated.

Raw evidence wins that argument.

TCP gives you bytes, not messages

I inspected the framing-relevant response bytes alongside the wrapper’s stream-handling logic. This was also where AI became useful.

I gave an AI coding and reasoning tool the portions of the response relevant to framing and the relevant implementation context. Within roughly two minutes, it pointed out an important mismatch between the protocol and the way the NetworkStream was being read.

The independently captured response contained rates. The wrapper’s behavior showed how a production response split across reads could be classified before the complete logical message had been assembled.

The wrapper’s logic for deciding whether the response was complete was wrong.

TCP provides an ordered stream of bytes. It does not preserve the application-level messages written on top of that stream. One write on the sender does not guarantee one corresponding read on the receiver. A response may arrive across multiple reads, and the amount returned by any individual read is not proof that the logical response has ended.

The wrapper had been making assumptions about when enough bytes had arrived to constitute a complete response. Those assumptions happened to work for many smaller responses. Under larger production rate responses, the data could be split across reads or arrive with different timing.

The wrapper could then conclude that the response was incomplete even though the downstream system had returned valid rate data.

The downstream protocol already had the answer: it defined an explicit termination character marking the end of a logical response.

The wrapper needed to keep assembling bytes from the stream until the protocol-defined terminator arrived, then interpret the complete response.

The reusable principle is straightforward:

TCP gives you a byte stream. Your protocol gives you message boundaries. Don’t confuse the two.

This was not TCP being unreliable. TCP was doing exactly what TCP promises to do.

The application had quietly invented an additional promise and then become upset when the network declined to honor it.

AI was useful after the problem became specific

It would be convenient to summarize this as “AI solved the production incident in two minutes.”

That is not what happened.

Before I used AI, I had already:

Observed the production failure.
Inspected the persisted errors.
Exercised the same request independently.
Proved that the downstream system could return rates.
Established a contradiction between the application error and downstream behavior.
Narrowed the likely failure boundary to response handling inside the wrapper.

Only then did I provide the model with the framing-relevant response excerpt and the bounded implementation context.

Without that acceleration, the next steps probably would have involved reading a difficult protocol specification, researching C# stream behavior, searching for similar failures, comparing all of that with the wrapper implementation, and testing competing hypotheses.

That could realistically have taken hours.

AI compressed the knowledge-acquisition and reasoning step. It did not replace the investigation that made the question answerable.

Two principles from that experience have stayed with me:

AI-assisted debugging becomes much more useful after you reduce the search space.

And:

Give AI contradictions and evidence, not just error messages.

“Production is broken” gives a model an enormous possibility space.

“The application reports no rates, but this response excerpt contains rates; here is the framing protocol and the relevant stream-reading logic” is an engineering problem.

A workaround is temporary by definition

While I prepared the fix, the failures were still intermittent. Refreshing allowed users to retrieve rates successfully, so I recommended refresh as a temporary operational workaround.

That was a reasonable mitigation at the time. It kept work moving and bought investigation time.

Then the incident changed.

Failures became more frequent. Eventually, refreshing stopped recovering several cases.

The workaround was no longer reliably mitigating the problem.

Meanwhile, the primary shipping provider was intentionally unavailable for maintenance. The contingency system was not sitting idle, waiting for a safer deployment window. It was actively carrying production shipping traffic.

At that point, the decision was not simply:

Is deploying a hotfix risky?

Of course it was.

The real comparison was:

What is the risk of this targeted hotfix compared with the rapidly increasing risk of leaving the deteriorating contingency path unchanged?

The exact production traffic and network behavior could not be reproduced locally. Pre-deployment validation was necessarily limited.

But the proposed change was supported by several independent pieces of evidence:

The raw response contained rates.
Independent requests completed successfully.
The protocol specification defined an explicit terminator.
The NetworkStream implementation did not consistently assemble the response according to that protocol.

I deployed the targeted hotfix during the active incident.

This is another principle I took from the night:

A workaround buys investigation time only while the workaround remains reliable.

And one more:

Compare hotfix risk with the trajectory of the incident, not with an imaginary zero-risk “do nothing” option.

Doing nothing is still a decision. During a deteriorating incident, its risk is not static.

Deployment success was not recovery

The new service instances came online.

Production errors dropped quickly.

That was encouraging, but it was not enough.

I created a test case and successfully retrieved rates as a smoke test. The test did not reproduce the exact production network behavior, so I did not treat it as proof that the defect was fixed.

Then four orders that had become stuck and unable to complete their workflow proceeded successfully. Operational users confirmed that the workflow was functioning again.

Recovery came from multiple signals:

Technical telemetry showed the error rate falling.
A smoke test confirmed the normal path still worked.
Previously stranded business work completed.
The people using the workflow confirmed that it was functioning.

The deployment showed that the new code was running.

Together, those signals showed that the business had recovered.

Recovery includes the work stranded during the failure, not only whether new requests succeed.

That distinction matters. A system can accept new traffic while yesterday’s failed work remains stuck, duplicated, or silently abandoned.

The contingency architecture did its job

It would be easy to describe this as a failure of the contingency design.

I do not think that would be accurate.

The architecture decision was to preserve shipping capability when the primary provider was unavailable. During the maintenance window, the fallback enabled approximately 200 cases, plus four overnight cases, to ship.

It preserved the business capability it was designed to protect.

The defect was narrower: the wrapper incorrectly inferred message completion from stream behavior instead of following the protocol’s framing rules.

Architecture decision:

Preserve shipping during a primary-provider outage.

Implementation defect:

Incorrect TCP and application-message framing in the fallback wrapper.

Those should not be conflated. A resilient architecture can contain implementation defects. In fact, contingency paths often reveal defects precisely because they receive less routine production traffic than the primary path.

The right response was to correct the defect and strengthen the contingency path—not conclude that the contingency itself had failed.

What changed permanently

The hotfix changed the stream-reading behavior, but stopping there would have left part of the incident unresolved.

I also improved the logging. The hotfix introduced a more accurate and distinct error type, and the failure mode and fix were shared with the team.

The original “no rates” error had sent the investigation in the wrong direction. Rates were sometimes present; the wrapper had failed to assemble and interpret the complete response.

Error messages are interpretations produced by software. They are not sworn testimony.

When an error contradicts direct evidence, inspect the boundary that produced the interpretation.

The distance between evidence and understanding

AI was extraordinarily effective during this incident because the problem had already been narrowed using engineering evidence.

I still had to recognize that the original error was suspicious, test the dependency independently, establish the contradiction, decide what evidence to provide, verify the explanation against the protocol, assess production risk, decide whether to deploy, and validate business recovery afterward.

Those were the consequential parts of the incident.

AI did not replace the debugging process.

It dramatically shortened the distance between evidence and understanding.

And debugging became considerably easier once I discovered that the rates reported as missing were, inconveniently, sitting right there in the response.

System Note

Protocol principle

TCP provides a byte stream. Application protocols must define—and implementations must honor—their own message boundaries.

Debugging principle

When the application’s error conflicts with raw evidence, investigate where reality is being translated into interpretation.

AI-assisted engineering principle

Reduce the search space first. Give the model evidence, contradictions, protocol context, and relevant implementation—not only the final error message.

Incident principle

Evaluate hotfix risk against the current direction of the incident. Validate recovery using technical signals, business workflows, stranded work, and confirmation from actual users.