We have updated our Privacy Policy, click here for more information.
Thank you
Missing business context is a familiar problem for engineering teams. A column name may make sense to the people who use a system every day, while leaving a newcomer unsure what it represents. The explanation may be in a feed specification, in old code, or known only to a colleague
AI-assisted development encounters the same problem. If an agent assigns the wrong meaning to afield, it can produce code that runs successfully but gives people a misleading report. Preventing that means establishing what users need the output to tell them, then finding and checking the definitions behind it.
For example, a trading desk asks an AI to calculate the value of each day’s trades from a kdb+ table.The AI groups the records by date, assuming that is when each trade happened. But here, date is the table’s partition column, based on the day each trade was reported. A £250,000 trade reported a day late ends up in the wrong day’s total, understating one day’s activity and overstating the next.
In this table, the partition date follows reporting activity. The execution timestamp answers a different question.
The desk wants to compare trading activity by the day it happened, so this report should use executedAt to assign trades to each day. A report monitoring when trades were reported could legitimately use date. Understanding how people will use the result determines which interpretation is appropriate.
The report’s purpose determines which date applies. Amounts show this trade’s contribution, not the total activity for either day.
Start with the knowledge the organisation already maintains. A data catalogue may explain what fields represent and where they come from. Feed documentation may describe reporting conventions.Existing code may reveal transformations that change what a field contains before it reaches a report.
For an existing system, establish where the definitions relevant to the change are held and whether the agent can read them. In the trade-date example, that may mean providing access to the feed documentation and the code that assigns records to daily partitions.
Where business calculations are already agreed, making those definitions available can reduce how much an agent needs to infer. A semantic layer provides one way to do this, connecting shared metric definitions to the underlying data.
In dbt Labs’ account of M1 Finance’s experience, the initial chatbot generated SQL using a data catalogue, but sometimes selected the wrong data or invented columns. The replacement asked the model to select predefined business metrics through dbt’s Semantic Layer, which generated the SQL.
Other tasks require context assembled from several places. OpenAI’s internal data agent combines expert annotations with table relationships, code analysis and company documents. This helps distinguish datasets whose structures look similar but whose contents differ.
Search services and MCP connections can expose relevant documentation and tools. Reusable skills can guide an agent through a task, including which sources to consult and which checks to perform.For dbt’s tools, the semantic layer supplies definitions, MCP provides access, and skills guide the workflow.
Access to a definition does not establish that it applies to the change in front of us. A feed specification may describe an earlier version. A metric used elsewhere may serve a different reporting purpose. An ingestion process may rename a field or transform its values.
Ask the agent to explain how it interprets each field in the calculation and to check that interpretation against documentation, code and representative records. This can also challenge the engineer’s initial understanding. If the engineer believes date records execution, but the feed documentation and ingestion logic indicate reporting, that disagreement should become part of the investigation.
The engineer could be mistaken, the documentation could be stale, the implementation could be wrong, or the behaviour could reflect an intentional decision. Tracing how the field gets its values and checking records that distinguish the possible meanings can help establish which explanation fits.
Colleagues may supply context that was never documented. Their explanation is another source to check against the relevant system and reporting requirement. The aim is to reach an interpretation supported by evidence that someone else can follow.
Some missing context can be recovered from existing sources. Other gaps concern behaviour the business has not yet decided.
Knowing when a trade happened, for instance, does not settle whether a report should revise previously published totals when a late trade arrives. If that behaviour is within the scope of the change, the reporting requirement needs to address it.
Ask the agent to show how the report would look under each interpretation, identify who would see a changed figure, and explain which part of the implementation depends on the answer. That gives the person responsible something more useful to review than a broad request for “more context”.
Concentrate clarification on unresolved choices that change the result. Where applicable evidence supplies an answer, the agent can investigate and use it. Where a business decision remains open, the proposed implementation should make that choice visible.
Once the required behaviour is established, choose tests that would expose a different interpretation.
For the trade-date example, records reported on the day they happened would be a weak check:either field could produce the same daily totals. A late-reported trade separates the two interpretations. Under this report’s agreed execution-date rule, it should contribute to the day it happened without inflating the following day’s activity.
The check must also cover how records reach the calculation. Changing the grouping column will not correct the report if an earlier filter on date still excludes late-reported trades belonging to the period being measured. An agent can trace those dependencies and help check the final output.
Expected results need their own basis in the requirement. If the same unsupported assumption shapes both the implementation and its tests, passing tests can preserve the mistake. Guan Yu Lim’s FD whitepaper on an AI-native SDLC for kdb/q describes confirming requirements and deriving tests from the specification separately from the implementation. Business definitions and examples that distinguish the intended behaviour belong in that specification.
The findings should also survive the current task. Meta’s account of learning from expert feedback describes a system that turns experts’ corrections into proposed updates to an agent’s knowledge and procedures. The system tests those updates, then domain experts review them before they are retained. The same discipline applies when an engineering investigation uncovers a missing definition or an undocumented reporting rule.
A practical starting point is the next report or calculation being changed. Establish what its users need to understand, locate the supporting definitions, and ask the agent to investigate any gaps or contradictions. Then record the agreed behaviour and its sources in the organisation’s maintained documentation or definitions, and add a test for that behaviour to the test suite. The next engineer or agent should be able to understand the decision without having to reconstruct the investigation.
Don’t let hidden definitions and ambiguous data compromise your AI-assisted engineering workflows. Equip your team and agents with the right semantic layers, documented business logic, and rigorous validation tests.
