AI-assisted data validation

Validating sanity configs with an AI agent: methodology notes

Context. CoRE Stack's data sanity framework (computing/sanity/) checks pipeline outputs against YAML rule configs: 34 configs, one per pipeline. Each config checks the pipeline's output in two forms, the vector layer and the stats Excel sheet. I did this work with Claude Code: first by hand on two pipelines, then by having agents repeat the method on all the others.

Approach:

1. walk in working through adding validation for some pipelines
2. ask claude to learn that process as a skill
3. ask claude for more insights

1. Understanding the pipelines and output

• Started with terrain, then cropping intensity.
• What is the type / shape of the output?
• Does the configuration make sense?

Few things stood out:

• Excel names differ from vector names The two forms use different names for the same data, e.g. cropping_intensity_2017 vs cropping_intensity_unit_less_2017-2018.
• Rules aimed at both forms matched only one A rule with applies_to: [excel, vector] would often only check for one format, and would be ignored in other. This is bad practice.
• Contradictory presence rules. Some rules had allow_missing_fields: true while other rules required the same fields to exist. Rules like these exist but never actually test anything, and we should avoid them.

Possible limitations:

1. Schema: Is it readable and useful? Does the schema encourage good practices and discourage bad ones?
2. Rules: Are the rules well-written? I noticed some rules that were either:
   1. not being matched with any field, or
   2. matching field in only one of excel/vector but targeting both, or
   3. (in cropping intensity) targeting hard-coded names for dynamically generated field names
3. Functionality: Are the rule types good enough to find real bugs in production data? What are we missing that we should catch?

3. Have the agent repeat the method on every pipeline

I asked Claude to do what I had done by hand, for each pipeline:

• work out the output's types and structure from the pipeline code and the export code;
• guess where problems are likely (physical bounds, units, no-data values, categories);
• encode those guesses as rules;
• test the rules against real prod data.

I gave it the prod API key so it could fetch real data: two tehsils from different regions, Masalia (Jharkhand) and Raptadu (Andhra Pradesh).

The workflow became a reusable skill, because I wanted the lessons from the first two pipelines applied properly to the rest, not rediscovered each time. The skill lives in .claude/skills/sanity-config/:

• SKILL.md: the workflow, plus the framework quirks found in the pilots.
• fetch_tehsil.py: fetches the prod vector layers and stats sheets for a tehsil.
• validate.py: runs the configs and reports rules that matched no field (avenue 2), fields no rule checks, and ids found in only one of the vector and the sheet.
• mutation_check.py: breaks the data once per rule and field, and proves the rule catches it.
• layer_map.yaml: which prod layer each config validates.

Five parallel agents applied the skill to the other 33 configs, grouped by area. Each edited only its own configs and reported findings back.

That fixed avenue 2 (usage) and worked around most of avenue 1 (representation): vector and Excel rules are now separate, patterns replace hard-coded years, and presence checks are explicit.

4. Review each finding before sharing it

Findings went into TODO.md, which doubles as the PR description: prod data problems, bugs in the export code, framework gaps, and blind spots. I reviewed each item one at a time and rewrote it myself:

• Verify before sharing. No AI-generated claim goes to colleagues unchecked.
• Make it actionable. Each item is one sentence plus a raw prod URL that shows the problem.
• Ask for provenance. "Was this caught by a rule or by extra analysis? Was the rule already there?"

5. What found the bugs: rules or exploration?

Asking for provenance gave the most important result for avenue 3.

The duplicate villages example.

• The agent noticed numbers that couldn't be right: one village with a population of 1,056,330, and 12,780 community assets when the whole layer has about 5,900.
• Checking the ids showed 60 polygons sharing village 803009.
• No rule type can express this, so even now no rule catches it.
• Asking "how could we have caught this?" traced it past the pipeline code to the census input file, which has 120 rows for that id.

What this tells us:

• Only one prod bug was predicted by a rule written in advance. The ET rule was the only case of "guess where problems are, then encode it" working.
• Exploration found most of the bugs. Plausibility checks on the data found far more than the rules did.
• The rule types are the bottleneck. Most interesting bugs can be described but not enforced. The framework has no uniqueness rule, no cross-field rule (parts sum to a total, a ≤ b), no "≥ 0 or no-data" bound, no conditional rule and no date rule. Also, not_null treats "" as present.
• Rules written after seeing the bug are regression tests. They will catch that bug again, but they aren't evidence that the rules can find new ones.
• Some checks belong on the inputs. The duplicate villages started in the census input file, so a check on the pipeline's output can only catch them after the damage is done.

6. Takeaway

The agent was very productive:

• 33 configs fixed;
• a reusable skill;
• real prod bugs found;
• export-code bugs found along the way.

But every finding needed review. The scripts made the rules testable, and checking each finding's provenance kept us honest about what the rules can actually find.

Next steps, in order of value:

1. Add uniqueness and cross-field rule types.
2. Make not_null treat "" as missing.
3. Add a "≥ 0 or no-data" bound.
4. Move the vector-vs-sheet id check into the framework.
5. Add checks on pipeline inputs such as the census files.
6. Test more tehsils.