“Sync Succeeded” Doesn’t Mean Your Data Is Correct: What I Learned Testing 97 Real Connector Bugs
After 23,000+ connector runs and a frozen blind test, a generic correctness checker held up — but automatic bug discovery fell short.I spent 9.5 days, 23,000+ connector runs, and a frozen blind test checking whether data-sync correctness can be judged generically. The judge held up. Automatic…
Annons
Annons
After 23,000+ connector runs and a frozen blind test, a generic correctness checker held up — but automatic bug discovery fell short.I spent 9.5 days, 23,000+ connector runs, and a frozen blind test checking whether data-sync correctness can be judged generically. The judge held up. Automatic bug-hunting didn’t.Here’s a bug I reproduced last week.A Google Analytics 4 connector reads a daily report. On one busy day the report has 160,000 rows. The connector asks for a page of up to 100,000 rows, gets it, and then checks a response field that doesn’t exist to decide whether there’s another page. There isn’t one as far as it can tell, so it moves on to the next day.Every sync exits successfully. The orchestrator shows green. The warehouse holds 100,000 rows for that day, and nobody gets paged because nothing failed.Sixty thousand rows are just gone.I’ve been a data engineer for over six years, and this pattern has bothered me the whole time. Most connector testing answers questions like: did the API respond, did the code throw, did the job finish, did records come out the other end? Those are good questions. But the one that matters is different:After everything settles, does the destination actually hold what the source holds?This article is about trying to test that question, generically, across very different connectors. I set up a real experiment for it: a benchmark of 97 historical bugs, a frozen engine, and a blind test. One half of my idea survived and the other half didn’t, and the half that failed taught me more.The bugs that look different but aren’tConnector bugs look wildly different at the code level. Some examples from the corpus:A record appears late. It gets created at 10:00, but the API only shows it at 10:05. The connector synced at 10:02, saved “I’ve read up to 10:02” as its bookmark, and will never look before 10:02 again.Pagination stops early. The GA4 case above.Two records share a timestamp. The connector saves updated_at > 12:00:00 as its next filter, but three records were updated at exactly 12:00:00 and only two had been read.A delete never arrives. The source deleted a record; the destination still shows it as live.A child is older than its parent’s bookmark. Conversation parts, order transactions and the like.A full refresh delivers a delta. The destination wipes the table and loads only what changed.I kept wondering whether all of these, different as they look, reduce to a handful of questions about the final state:Is every live source record present in the destination?Is each one at its latest version?Are deleted records gone when the connector promises to capture deletes?Are there records in the destination that never existed at the source?Did the sync actually finish?If that’s true, you don’t need a custom test for every bug. You need one small, generic judge, which researchers call an oracle, plus a way to produce the nasty executions for it to judge.The idea wasn’t new. The question was.Before writing code, I looked at what already exists, because “nobody has thought of this” is almost never true.It wasn’t. Deterministic simulation testing (popularized by FoundationDB and now Antithesis) already runs real code against simulated, adversarial environments. Model-based and property-based testing already check invariants instead of individual examples.Chaos engineering already injects faults. ConductorOne has written publicly about testing its identity connectors against a mock server with fault injection and deliberately malformed responses. Connector frameworks ship acceptance tests, and data-quality tools test the data once it lands.What I couldn’t find was an answer to a narrower question:Can replication connectors, written in different frameworks, against different APIs, with different cursor and pagination designs, share one generic correctness model, with all the provider-specific weirdness pushed into reusable simulated providers?That’s a testable claim. So I decided not to build a product until I’d tested it. No UI, no SaaS, no AI inside the engine. Prove it or kill it first.Three pieces: profile, explorer, oracleThe system I ended up testing has three parts.Provider profiles are simulated versions of real APIs: Stripe, HubSpot, Shopify, Salesforce, GA4, Okta, Jira, Google Sheets and about thirty more. Each one models pagination, result caps, auth, deletes and when changes become visible, using only documented behavior.If a behavior isn’t in the provider’s docs, it goes into an explicit ASSUMPTIONS list, and any result that depends on it is flagged as unverified. That rule turned out to matter a lot, as you'll see.Here’s the top of the Okta profile, to show the style:"""Okta Management API: System Log (GET /api/v1/logs) and list pagination (developer.okta.com).Documented behavior modeled here: - pagination by Link headers: rel="self" is the current request, rel="next" the next page - "polling" requests (no `until`): the next link is ALWAYS returned, even when the page is empty, so a client can keep polling for new events; an empty page's next link equals its self link"""ASSUMPTIONS = {} # nothing undocumented was needed for this providerThe connector is real. I exported the actual historical code, both the buggy revision and the fixed one, installed it with the dependency versions it shipped with, and ran it unmodified. Each run is a short life: three syncs an hour apart, then a couple of settle syncs.The oracle looks only at the end state. Here’s the core of it, slightly simplified:def evaluate(truth, ever, dest, mode, compare_fields, runs, identity, deletes_captured=True): v = {} dedup = mode.endswith("_dedup") if any(r.killed for r in runs): v["liveness"] = [r.killed for r in runs if r.killed] # a sync never finished live = {pk: ver for pk, ver in truth.items() if not ver.deleted} missing = sorted(pk for pk in live if pk not in dest) if missing: v["missing"] = missing # live at source, absent downstream stale = [pk for pk, ver in live.items() if dest.get(pk) and not _matches(dest[pk][-1], ver.data, compare_fields)] if stale: v["stale"] = stale # present, but an old version if (dedup or mode.startswith("overwrite")) and deletes_captured: undeleted = sorted(pk for pk, ver in truth.items() if ver.deleted and pk in dest) if undeleted: v["undeleted"] = undeleted # deleted at source, still live phantom = sorted(pk for pk in dest if pk not in ever) if phantom: v["phantom"] = phantom # never existed at the source if dedup: dupes = sorted(pk for pk, rows in dest.items() if len(rows) > 1) if dupes: v["duplicate_keys"] = dupes verdict = Verdict(passed=not v, violations=v) verdict.silent = bool(v) and all(r.exit_code == 0 and not r.errors for r in runs) return verdictThere are no provider names in it, no connector names, and no bug IDs. The whole file is about 165 lines. It knows a few destination modes (append, append-dedup, overwrite) so it doesn’t call at-least-once duplicates in an append table a bug, and it reports whether the failure was silent, meaning every sync claimed success.The explorer produces the nasty executions. It never reads connector code. It only knows generic things: which records the fake API has served so far in this sync (the “read frontier”), when syncs happen, and which operations the provider supports. From that it generates schedules like:update_already_read edit a record behind the read frontier, mid-syncupdate_behind_and_ahead_of_read_frontier edit one behind, then one aheadinsert_ahead_of_reader create a record while the connector is paginglate_visible_insert / late_visible_update a change that exists before the API shows itdelete_read_earlier delete something an earlier sync already readchild_ahead_and_behind_parent parent/child timestamps out of ordergeometry dataset sizes around the observed page sizefault x phase 429, 500, connection reset, revoked or expiring tokens, at the first request, after page 1, ...The read frontier is the idea that made search work. Mid-sync, the fake API knows exactly which records it has already handed over:already served: A B C------------------------------ read frontiernot yet served: D E Fedit B → the connector already read the old versionedit E → the connector will read the new versionedit both, B before E → does the saved bookmark skip B's change?A bug case ties it together. This one is a held-out Okta bug, and nearly everything in it is configuration, not logic:STREAM = "logs"IDENTITY = ["uuid"]CURSOR = "published"DEST_MODE = "append_dedup"DELETES = "not_captured"COMPARE_FIELDS = ["uuid", "displayMessage", "published"]def workload(world, rng, size): # the neutral workload: same shape for every case _seed(world, size) return [START, START + timedelta(hours=1), START + timedelta(hours=2)]One distinction became the central tension of the whole project:Judging a bad execution and generating a bad execution are two different problems.The oracle does the first. The explorer does the second. I scored them separately the entire time and never blended them:Oracle recall: given the bug’s own trigger, does the oracle fail the buggy revision and pass the fixed one?Discovery recall: given only a neutral workload and the explorer, does anything expose the bug, with the fixed revision passing the same schedule?A prototype can prove almost anything, so I collected real bugsDemos are easy to rig, even by accident. So the first real piece of work was a corpus of 97 historical connector bugs from public issue trackers: Airbyte, Singer taps, Meltano SDK, dlt, and a few others. Each entry recorded the symptom, the mechanism, the fix if there was one, whether the failure was silent, and how reconstructable it looked.They span cursor and bookmark logic, pagination, eventual visibility, auth and retries, deletes, schema handling, and destination behavior. Some are silent, some loud.Two rules shaped everything after that.Real code, both sides. Wherever possible, a case runs the actual historical connector before and after its fix. A fix that still fails, or a buggy version that passes, is a result, not something to tune away. That rule caught two maintainer-declared fixes that didn’t fix their own bug. In one, a Stripe tap PR moved its request window back five minutes but kept an emit filter that dropped exactly the records the lookback fetched.Strict evidence. A discovery counts only if the provider behavior it depends on is documented independently of the bug report. If the only evidence that “Klaviyo enriches events three hours later” is the bug reporter saying so, any finding that relies on it is reported as assumption-dependent, not counted.The corpus was split up front: 53 development bugs to build against, and 44 held-out bugs I wasn’t allowed to look at closely until the engine was frozen.Nine and a half days of iterationI worked through the development set one bug family at a time. After every change, a regression gate re-ran every earlier case under a recorded build hash, so a new feature couldn’t quietly break an old result.Prototype (5 cases). The oracle judged the known reproductions fine. Searching was the problem. A HubSpot bug, where an edit lands mid-sync and is lost, was found in 0 of 12 random-edit runs. Once the explorer could aim at the read frontier, one generic primitive (update_behind_and_ahead_of_read_frontier) found it reliably. That was the first lesson: random mutation is a weak search, and knowing where the reader is is a strong one.Family 1, cursors and bookmarks. Intercom, Mailchimp, Shopify. This added one generic parent/child timing primitive. The oracle didn’t change.Family 2, late visibility. Stripe, Klaviyo, Shopify GraphQL bulk exports. This added late_visible_update. It's also where the Stripe "fix that didn't fix" showed up.Family 3, pagination. Teamwork, Ding Connect, Simplecast, HubSpot’s 10,000-result search cap. No new mutation primitive was needed. If the observed page size is p and the provider documents a result cap c, geometry just tries the sizes where paging logic breaks:1, p-1, p, p+1, 2p, 2p+1, 3p+1c-1, c, c+1, c+p, c+p+1, c+2p+1That found them all. The HubSpot bug shows why the odd-looking sizes matter: at exactly 10,001 records the connector passed, and at 10,201, once the reset after the cap crossed a second page, it looped forever. This was the first real sign of convergence.Families 4 and 5, faults and auth. 401s, OAuth expiry, retries, single-use refresh tokens. This was a useful negative result. Every one of these bugs was loud: a sync failed visibly. I also threw 429s, 500s and connection resets at 26 connector revisions from the earlier families, and 339 of those faults actually landed. Silent data damage: 0 of 339. Faults surface as crashes, which ordinary chaos testing already catches. Auth chaos isn’t where the value is.Deletes. Stripe cancellations, Dataverse, Google Drive’s trashed files, HubSpot archives. The existing rules were enough, once each stream declared whether it promises to capture deletes. That declaration matters for trust: a cursor-based incremental stream simply can’t see a hard delete, so a deleted row surviving there is reported as DECLARED_LIMITATION, not as a bug. A tool that cries wolf about things the connector never promised won't get used.Destinations. Singer ACTIVATE_VERSION, dlt SCD2, overwrite swaps. All the warehouse-specific mechanics went into a projection layer that turns physical tables into a logical state: a row with _sdc_deleted_at set is not current; an SCD2 row whose _dlt_valid_to is filled in is closed. The oracle still asks the same questions. Again, no new correctness rule.Across the whole development phase, the oracle went from 162 to 165 lines, and custom verdict code stayed at zero.Development looked almost too goodAt the end of development:Buggy revisions detected 27 / 27Buggy vs fixed separated 24 / 24Automatic discoveries (strict evidence) 22 / 22Custom, per-bug verdict code 0 linesThose numbers would make a great launch tweet. They’re also close to meaningless as evidence, because the development bugs shaped the explorer. Every time a bug wasn’t found, I asked what generic capability was missing and added it. Of course the dev score converged to perfect. That’s what development does.The only number that would mean anything was the next one.Then I stopped changing the engineI froze V1 under a git tag, with a file-hash manifest for every frozen component: oracle, explorer, geometry, faults, session model, destination semantics, outcome classes, runner. A checker script verified the hashes before every held-out run, and it printed freeze intact every time.After the freeze, the rules were:Allowed: new provider profiles, new case mappings, runtime configuration through existing hooks.Not allowed: any new oracle rule, explorer primitive, geometry dimension, profile hook, mapping key, destination mode, fault type, session behavior, or scoring exception.If a held-out bug needed one of those, it was recorded as a gap, not fixed. If the oracle itself broke, it was recorded as a defect, not patched. The explorer always got the same neutral workload, fixed in writing before any held-out case ran, never the bug’s own scenario.Of the 44 held-out bugs, 10 were set aside by evidence triage before the freeze, without being run. That left 34. Not all 34 could be run, and I didn’t want one convenient “X/34” number hiding that:10 environment-blocked: Java, Kotlin, Go or CDC stacks I couldn’t build in the sandbox. These are not evaluated, neither passes nor misses.1 had no real implementation: no shipped connector uses the affected configuration.5 were frozen capability gaps: the frozen model couldn’t express them (more on these below).18 were runnable. Two of those are outside V1’s scope. One is a page re-read that only causes duplicate deliveries, which V1 deliberately tolerates. The other is a connector whose later syncs fail loudly, but whose only data is a constant identity row, so the destination stays correct.That leaves 16 in scope, and 14 with strict provider evidence.The result: the oracle held up, the explorer didn’tHere’s the held-out result under the frozen engine:Oracle recall, strict 14 / 14Oracle recall, extended 16 / 16Automatic discovery, strict 5 / 14 (≈ 36%)Automatic discovery, extended 6 / 16Fixed revisions passing 12 / 13Custom verdict code 0 linesFrozen capability gaps 5Oracle defects 1The oracle generalized. Without a single line of per-bug judgment, it correctly judged executions from GA4, Okta, Jira, NetSuite, Salesforce, TikTok, Google Sheets, Shopify, Stripe, and dlt, none of which had shaped it.The explorer did not. Development discovery was 22/22. Held-out discovery was 5/14.That gap is the most important result of the project.Auditing my own headlineAt this point I could have written “14/14” and stopped. Instead, I went back over how I’d counted, because some of the counting rules had been decided after I’d seen the results.The “strict” set was partly my judgment. Eight of the 14 strict cases recorded an unverified provider assumption somewhere in the run. For example, the Jira profile assumes ties in ORDER BY updated come back in ID order, which isn't documented.For each one, I decided the assumption wasn't part of the bug's mechanism, and wrote the reason down. But I made those calls after seeing results. Using a purely mechanical rule instead, where any recorded assumption disqualifies a case, the strict set shrinks to 6:Mechanical strict (no unverified assumption at all) oracle 6/6 discovery 2/6Adjudicated strict (my per-case judgments) oracle 14/14 discovery 5/14Extended (bug mechanism itself is assumed) oracle 16/16 discovery 6/16The samples are small. 14/14 has a 95% confidence interval of roughly 77–100%. 5/14 is roughly 13–65%. That’s strong evidence, not proof.There’s a selection effect. The five capability gaps are bugs the oracle never got to judge. Count them as cases it couldn’t judge and the oracle number becomes 14/19.Not every detection needed an oracle. This was the correction that mattered most.Of the 14 strict detections, three were syncs that never finished, which a timeout would catch. Three were loud failures, which an exit code would catch. You don’t need a semantic oracle for those.The other eight were silent: every sync reported success and the destination was wrong. Pardot records lost on tied timestamps. A TikTok report configured for a single day that never made a request. The GA4 page cap. dlt silently coercing a string ID to 0.A Jira query read in the user's time zone instead of UTC. Stripe charges created in the exact second a window ended. Shopify fulfillment orders emitted with 6 of their 8 nested fulfillments. Google Sheets rows past a pair of blank rows.The most meaningful result isn’t 14/14. It’s that the frozen oracle caught all 8 held-out silent-corruption bugs, where the connector itself said everything was fine.That’s 8/8, with a 95% interval of roughly 63–100%. Small, honest, and the part of this that existing tools don’t already do.Why automatic discovery failedThe nine misses weren’t random. They fell into four blind spots, and none of them is about time, which is the one dimension the explorer moves well.1. Ties. Bugs that need many records sharing one cursor value: Pardot, Zendesk Talk, HubSpot. My neutral workload deliberately gave every record a distinct timestamp, which makes for clean data and also guarantees these bugs never appear. In production, updated_at = 12:00:00 on 10,000 rows after a bulk update is completely ordinary.2. Scale. GA4 needs more than 100,000 rows in a single day’s report. Geometry explores a few pages around the observed page size, and it stopped at 1,351 rows. The oracle would have caught the missing 60,000 rows instantly. The explorer never produced the execution.3. Record shape. The explorer changes when records change, never what they look like. It never makes blank spreadsheet rows, mixed-type IDs, or an order with more nested children than fit on one API page.4. Configuration and compound triggers. The TikTok bug only happens when start_date == end_date, and the explorer doesn't vary connector config. The Shopify bug needs a connection reset and an edit before the resume. The explorer runs one operation per schedule.The lesson I’d generalize from this:The state space of a connector isn’t just temporal. It includes data geometry, value shape, configuration, and compound schedules. An explorer that only moves time around will miss most of it.Five bugs the frozen model couldn’t expressFive held-out bugs were capability gaps. They weren’t misses: the frozen vocabulary simply had no way to state the scenario.Credential lifecycle (3 bugs). A Singer tap persisting a rotated refresh token by rewriting its config file. A provider that rotates refresh tokens on a roughly 24-hour schedule rather than on every refresh. A token rotated during the connection check and then discarded by the platform.A duplicate rule for snapshot outputs (1). A client listing that repeats an event across page boundaries.Declared access scope (1). Whether records the credentials can’t read belong in the destination at all.Five gaps falling into three groups is about as clean a boundary as a blind test gives you.And there was one genuine defect in the frozen oracle. In the dlt case, the fixed revision produced records with identity keys of mixed types, 1 and "no_id". Look back at the oracle code:missing = sorted(pk for pk in live if pk not in dest)In Python, sorting 1 and "no_id" together raises a TypeError. The oracle crashed on a perfectly valid execution. I didn't fix it during the held-out run. It's logged as the first V1.1 fix, and it's exactly the kind of thing a freeze is supposed to surface.The false-positive number that almost fooled meThe explorer also ran on every fixed revision, which is the healthy control. Across 733 in-scope control runs, 41 were flagged:23 depended on unverified provider assumptions, so they count neither way.11 were my own mistakes. Seven were request-budget kills: I set a NetSuite case’s request budget lower than the connector’s one-request-per-record pattern needs. Four came from profiles that let report data for any past day change, when real providers only restate recent days.7 look like real bugs present in both revisions and unrelated to the target. One example: a Google Sheets loop condition, from_row < sheet_max_row, that seems to skip the last row when the sheet's grid is exactly full. None of these has been verified against a live provider.So the harness-side false-alarm rate was 11/733 = 1.5%, which sounds excellent. Then I counted per connector. 8 of the 15 fixed revisions got at least one red flag of some kind, and 3 of those 15 were flagged because of my own mistakes.A team testing their connector doesn’t experience “722 clean schedules.” They experience “I ran it on my healthy connector and it went red.” Both numbers are true. Only one predicts how the product would feel.And the oracle’s own logic? It never called a destination wrong when the destination matched the simulated source. But be careful with that number: the oracle compares against the simulation, so a false judgment can only come from a wrong simulation. Which brings us to the real problem.We removed per-bug code. We didn’t remove domain knowledge.Custom verdict code stayed at zero throughout. But the held-out set needed 13 new provider profiles, 1,348 lines, and the biggest were over 200 lines each (GA4 219, Google Sheets 211, Shopify fulfillment 187).The complexity didn’t disappear. It moved:per-bug verdict logic → removed (0 lines)per-provider behavior models → substantial (≈100 lines each, some 200+)And every practical false alarm traced back to a profile or a case configuration, not to the oracle. So the product question has changed from “can a generic judge work?” to:Can trustworthy provider profiles be built, reused across many connectors, and kept correct as APIs change, cheaply enough?A 200-line profile reused by fifteen connector implementations is a great deal. A 200-line profile needed for every individual connector is not. If I build this, the metrics I’ll track from day one are time to add a provider, lines per profile, share of documented versus assumed behavior, connectors per profile, and profile defects per hundred runs.What I haven’t done: a live providerEvery result here used real connector code against simulated providers. Not one held-out result was checked against the real GA4, Jira or Google Sheets.That’s the obvious next validation, and it’s cheap. The Google Sheets candidate is a real sheet with an exactly-full grid and a current connector. If the last row really goes missing, that’s a different kind of evidence: the system generated a scenario that exposed a defect nobody had reported. I’ll label it post-benchmark external validation, not fold it back into held-out recall.What 9.5 days of this actually costFor anyone curious what a project like this takes:Calendar Oct 21 → Oct 30, 2026 (about 9.5 days)Commits 42 in the benchmark repoConnector runs 23,050 recorded, ~143 hours of aggregate sync time (gates ran 6 in parallel, so wall-clock was much less)Repository 62 case files · 36 provider profiles ~97 MB of exported connector code · ~5.3 GB of Python environmentsOracle 162 → 165 lines over the whole projectCustom verdict code 0 lines, every buildI did this with an AI coding assistant as my research partner, plus 13 helper agents that reconstructed held-out cases in parallel: each one exported the historical code, built a provider profile, ran the reproduction, and reported back. My role was mostly research lead and reviewer: setting the rules, making the judgment calls, and catching optimistic framing, including in my own headline.The surviving logs record at least about 0.5 million output tokens and about 150 million cached-input tokens. Treat those as minimums: the long conversation was compacted several times, so a lot of the early history isn’t in the logs. The helper agents account for most of the recorded work.The expensive part wasn’t the oracle. It was reconstructing real historical environments, old dependency sets, and faithful provider behavior. That’s also the lesson about AI-assisted research: parallel agents made it practical to rebuild real historical environments many at a time, but they didn’t decide what counted as evidence. The rules did, and someone had to hold to them.What the failed parts taught meJudging and searching are different problems. I started out treating “catching bugs” as one problem. It’s two, and they generalize very differently.Freeze early enough to embarrass yourself. 22/22 in development became 5/14 blind. Without the freeze, I’d have kept adding primitives until the benchmark looked perfect, and learned nothing.Write the counting rules before you see the results. My cleanest-looking number, 14/14, shrank to 6/6 under a mechanical rule. I’d rather report both than defend one.Count the way the user experiences it. 1.5% per run and 8 of 15 per connector describe the same data. Only one tells you whether people will trust the tool.A negative result can sharpen the product. Auth chaos isn’t the moat, since those bugs are loud. Autonomous discovery isn’t ready. The judge is the core.Know where the complexity went. A 165-line oracle sounds tiny. The 1,348 lines of provider profiles are the other half of the story.The product I’d build after thisNot an autonomous AI bug hunter. The evidence doesn’t support that, at least not yet.What it supports is semantic correctness testing for stateful replication. The oracle is the product, and the explorer becomes one scenario source among several: developers’ own tests, historical regressions, provider-specific scenarios, and later, replayed production incidents.The first version is a CLI, roughly:$ connector-test init$ connector-test test Detected: Airbyte declarative connector · provider: Google Analytics 4 Running correctness scenarios... ✓ cursor boundaries ✓ deletes (declared: not captured) ✗ pagination stopped early Expected rows: 160,000 Destination rows: 100,000 Missing: 60,000 (unreachable from the saved checkpoint) Every sync exited successfully.Before that, there’s cleanup. The things the blind test found about my own system come first: the mixed-type identity sort, request budgets that scale with the connector’s real request pattern, and profiles that bound how far back reported data can change.Then extracting a clean core from the research harness, with no bug IDs in product code. Then CI, a small set of well-documented, widely reused provider profiles, one live-provider validation, and conversations with people who maintain connectors.For the claims, I’ll stick to what the evidence supports: cursor and checkpoint correctness, pagination and boundaries, destination and delete semantics, and final-state correctness. Late visibility is in the engine, but its held-out evidence leans on unverified provider assumptions, so I won’t lead with it.What a V2 explorer would needThe blind misses wrote the roadmap: tie generation, scale search up to declared provider caps, record-shape mutations, configuration search, and fault-plus-mutation schedules. The capability gaps add credential lifecycle, snapshot duplicates and access scope.One rule matters more than the list: those 34 held-out cases are now development data, permanently. Tuning V2 against them and calling it another blind result would be exactly the mistake this whole project was built to avoid. V2 needs a new blind corpus.What survived the experimentI started with a broad idea: maybe a system could automatically find correctness bugs across arbitrary data connectors. Nine and a half days and more than twenty-three thousand connector runs later, the evidence doesn’t support that claim yet.What survived is smaller and, I think, more useful. Once a connector execution exists, a compact generic model can tell whether the resulting replica is correct across very different providers, frameworks, cursor designs, pagination styles, delete semantics and destinations, without encoding a single historical bug into its verdict logic. On the held-out bugs where that mattered most, the silent ones, it caught all eight.The hard problem moved. It’s no longer “how do I know this execution is wrong?” It’s “which executions should I generate, and how faithfully can I model the provider?”That’s what I want to build next. Because in the end, “success” is a process status. It isn’t proof that your data is correct.If you maintain a data connector, or you’ve been burned by a sync that said “success” and wasn’t, I’d genuinely like to hear the story. Reply here or find me on X. And if you’d like to follow the build from research harness to a real tool, follow along: the next post will be the live Google Sheets test, whichever way it goes.This story is published under the Generative AI publication. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories. Let’s shape the future of AI together!“Sync Succeeded” Doesn’t Mean Your Data Is Correct: What I Learned Testing 97 Real Connector Bugs was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.