When you provision an analytics dashboard into a customer’s workspace, the customer’s first open is the entire review. Nobody reads the job log. They open the page and either the numbers are there or they are not. Every engineering decision behind that screen collapses into one impression, and the impression does not get a second try.
This is one spoke of a series on shipping inside a customer’s environment. Its rule: the first render is the product, and partial success is the path to watch.
Partial success is worse than failure
The setup: a partner’s dashboards are cloned into a client’s workspace, and the charts are re-pointed at the client’s own data connections. Some connections may be missing. The obvious accommodation is to allow a partial clone: bring across what resolves, skip what does not, let the operator fill the gaps later.
The trap is what “skip” keys on. In this system, the decision to skip a chart’s refinement was tied to the resolved connection map. Allow partial clones and that map silently changes meaning: a chart with no resolvable connection is no longer an error, it is a chart that gets published with nothing behind it. Every chart in the worst case. The result is a fully dead dashboard, delivered to a client, with no gate anywhere that would have stopped it.
The fix was a gate, not tolerance. Partial is allowed only when the clone can prove the result will render, and we rewrote the skip logic so a change in the map’s meaning cannot quietly change the outcome. A regression test pins the exact case: a clone with unresolved connections must not publish. I wrote about the general principle in pre-flight before you say “queued”; this is what happens when the pre-flight passes and the work itself can still degrade.
”Sync says done but the number is still wrong”
The second way a first open goes wrong is subtler. The clone succeeded, the data synced, the sync reported done, and the number on the chart is stale. I find this one harder to catch than a dead screen, because everything upstream is telling the truth and the customer is looking at a lie.
The cause was cache invalidation with a blind spot. The analytics layer’s invalidation endpoint could only evict cache entries it had a key record for. Entries created without one survived every post-sync flush until their own expiry, which meant a chart could show yesterday’s number after a sync that genuinely completed today.
The durable fix was structural rather than a wider sweep. We split the cache so that each type of cached thing lives in its own store, and a post-sync flush became a precise, atomic clear of exactly the store that matters, rather than a pattern scan that could miss entries and mask its own misses. When that flush cannot reach its store, it now fails loudly to the error tracker instead of returning quietly. The class “sync says done but the number is wrong” closed, because orphaned entries can no longer outlive a flush.
Prove it by driving it, not by reading it
The change from blocking to warning on the clone surface is the kind of thing a green test suite can approve while the real screen is broken. So we did the proof on the surface: the branch and the current production build were driven side by side, off the shared slot, through the same clone with the same seeded destinations, and the two results were compared as a customer would see them.
That exercise caught something no test would have: a shared vendor autoloader meant the “old” server was running a slice of new code, so the baseline was not the baseline. Fixing the comparison was the first step to trusting it. I have written elsewhere about pointing an agent at the surface and letting it try to break it; this was the manual version, and it earned its hour.
Failure arrives labelled as failure; partial success arrives labelled as success. Where in your own pipeline does “some of it worked” still ship as done?