We have updated our Privacy Policy, click here for more information.
Thank you
A Practitioner’s Framework — From Business Requirement to Tested Module in Hours, Not Days
KDB/q is the language of choice for high-performance financial data platforms — yet it has historically resisted AI-assisted development. Its terse functional syntax, thin training corpus, and the absence of a real-time feedback loop in the editor have meant that even capable frontier models produced technically correct but idiomatically poor q code. This paper presents a practitioner’s framework for closing that gap: an AI-native Software Development Lifecycle loop built around specialised agent skills, a live KDB execution environment surfaced through the Model Context Protocol, and a human-in-the-loop philosophy that concentrates human judgement at requirement sign-off and PR review rather than distributing it across every line of generated code. Two production-equivalent KDB analytics modules — transaction cost analysis (GEP-1123) and intraday market microstructure (GED-1123) — were delivered using this framework and measured at 7–8× the speed of unassisted development. This paper describes the framework, the tooling, and the lessons from building it.
The financial services industry is under sustained pressure to deliver analytical capability faster, at lower cost, and with greater auditability. KDB/q sits at the heart of this challenge — it is the language that powers real-time risk, pre-trade analytics, transaction cost analysis, and market microstructure research at the world’s leading institutions. Yet KDB development has remained a specialist discipline that resists acceleration through conventional AI tooling.
This paper introduces a structured, skills-based AI-SDLC loop that changes that calculus. The framework rests on three foundations. First, agent skills encode domain expertise for each workflow stage — business analysis, development, and documentation — so that the AI operates within well-defined boundaries rather than hallucinating idioms it does not know. Second, the human in the loop is preserved as a deliberate design choice at the workflow transitions that matter most: requirement elicitation and acceptance. Third, a live KDB execution environment, surfaced through the Model Context Protocol, closes the feedback loop between code generation and code execution inside the same conversation.
The result is a development loop that has been demonstrated to reduce module delivery time by 6–8× compared to unassisted development, based on measured delivery sessions, while increasing test coverage. It is not a replacement for KDB developers — it is a force multiplier for them. For clients of EPAM and First Derivative, it represents the fastest credible path to AI-accelerated KDB delivery without sacrificing the standards that financial regulators and internal model governance functions demand.
KDB-X/q occupies a unique position in the financial technology landscape. It is a vector-processing, functional, array-oriented language that treats time-series data as a first-class citizen. Its performance characteristics — in-memory columnar storage, built-in temporal join semantics, nanosecond-precision timestamps — make it irreplaceable for workloads where microsecond latency and billion-row datasets are simultaneously in scope.
That power comes with a steep cost. The language is deliberately terse: a single character can represent a higher-order function, a data type, or an adverb that changes the entire execution model of the expression that follows it. The standard iterators — each, over, scan, each-right, and each-left — have no direct equivalent in any mainstream language. The table query syntax (qSQL) superficially resembles SQL but diverges in ways that consistently surprise developers trained on relational databases.
The consequence for AI-assisted development is that training corpus thinness translates directly into code quality degradation. A frontier model asked to write a rolling volatility function will reach for a while loop before it reaches for mdev. It will use show for logging before it reaches for file-descriptor handles. These are not catastrophic errors — they are the kind of errors that slow down review cycles, accumulate technical debt, and create onboarding friction. In practice, they add 30–60 minutes of rework per function to a KDB-X/q developer’s review load.
Most AI coding assistants work by generating code and displaying it in the editor. The developer then switches to a REPL, executes the code, reads the output, returns to the editor, and incorporates the feedback. In KDB-X/q development, this context switch is particularly costly: the KDB REPL is a separate terminal process, the data it operates on may require a specific initialisation sequence, and the test framework (qcumber) has its own invocation pathway distinct from the standard q binary. The result is a feedback loop that is slow by the standards of an AI-native workflow.
AI coding assistants have broad knowledge but shallow depth. For a language as opinionated as q, shallow depth produces code that is functional but alien to a q developer’s eye. Closing this gap requires injecting domain-specific knowledge — idioms, naming conventions, API signatures, known gotchas — into the model’s context before it generates code. Without that injection, the assistant is a capable typist who does not speak the language.
The AI-native KDB/q SDLC framework described in this paper addresses all three challenges. It closes the feedback gap with live MCP-based execution. It closes the skills gap with agent instruction files that encode q idioms, naming conventions, and API knowledge. And it closes the context gap with structured skills that guide the AI through each stage of the workflow with role-appropriate expertise. What it does not do — deliberately — is remove the human from the decisions that require human judgement.
The framework is structured as a loop with three primary skill layers and two human decision gates. Each skill layer operates in an isolated session, with structured handover artefacts — JIRA memory files and scratchpad documents — ensuring continuity across context boundaries. The loop is not intended to be fully autonomous; it is designed so that human involvement is concentrated at the points of maximum leverage while AI handles the high-volume, rule-governed work in between.
Two MCP tools are the mechanical heart of the framework. They are registered as tools on a FastMCP Python server that bridges GitHub Copilot’s tool-calling interface to a running KDB-X 5.0 process.
| Tool | Purpose |
| kdbx_q_eval | Evaluate any q expression against the live database and return the result as structured JSON. Used for prototyping, validation, and incremental debugging throughout the development phase. |
| kdbx_q_unit_test | Accept a qcumber .quke DSL block, execute it through the KX qcumber binary inside the container, and return a structured pass/fail report. Used to encode BDD acceptance criteria as executable assertions. |
Session continuity is maintained through two artefact types. The JIRA memory file (.memory/jira/.md) is a structured Markdown document that records the current phase, what has been completed, what remains, and any open questions. JIRA is the preferred cloud-based home for the handover record. Where JIRA is not in use, local .memory files serve the same purpose — the key property is persistence across context boundaries, not the specific tool.
A skill in this framework is a role-specific instruction file that encodes the knowledge, workflow steps, and behavioural constraints appropriate to a single stage of the SDLC. Skills are not prompt templates — they are structured playbooks that the AI reads at the start of a session to acquire role-specific behaviour it would not exhibit by default.
The Business Analyst (BA) skill is the entry point for every development task. Its defining characteristic is the refusal to infer: the skill never assumes a requirement detail that the human has not explicitly confirmed. Every ambiguous dimension of a requirement — the triggering event, the data sources, the output schema, the edge cases, the performance expectations — is surfaced as a numbered question and held open until the human answers it.
This constraint exists because the cost of a wrong assumption in requirement elicitation is not paid until testing or production. By concentrating human attention at the requirement stage, the framework ensures that the subsequent automated phases operate against a correct specification.
Concretely, the BA skill:
The KDB Developer skill implements the module from the BDD specification. It operates in two distinct phases: a development phase (interactive prototyping via kdbx_q_eval) and a test-writing phase (qcumber test suite derivation from the BDD spec). These two phases must run in separate sessions by design — the test-writing phase derives tests from the specification alone to prevent circular test logic.
The skill encodes the KDB coding standards that are otherwise absent from AI training data. These standards are not fixed — they are designed to be adapted. An organisation can tailor the developer skill by scanning its own enterprise code repositories for idiomatic patterns, incorporating guidance from a lead developer, and adjusting the constraints to match its own quality bar: stronger documentation, stricter exception handling, looser validation in latency-critical paths. The skill is a starting point, not a mandate.
Default standards encoded in the open-source skill:
The KDB Doc and Review skill closes the loop with an AI-first review pass. This is not a convenience — it is a necessity. AI-native development changes the ratio of code produced to code reviewed. A single developer session can generate a complete multi-function module in hours. Without an automated first-pass reviewer, the human reviewer must absorb that entire output cold. The Doc and Review skill changes the dynamic: it runs first, flags every deviation from the coding standards, identifies knowledge drift between the specification and implementation, and delivers a structured findings report. The human reviewer then works from that report — focused on semantic correctness rather than pattern checking. The result is faster, higher-quality review without reviewer overload.
The three functions it performs: documentation completeness audit (every public function carries the required comment tags); knowledge drift detection (the BDD specification, module implementation, qcumber test suite, and README are cross-referenced and any divergence is flagged); and code quality review (naming conventions, iterator usage, logging pattern consistency).
The framework is not designed for full autonomy. It is designed for assisted development in which humans own the decisions with the highest leverage and AI owns the execution. The Business Analyst skill embodies this philosophy most directly. By refusing to proceed past an open question, it ensures that every development session begins with a complete, human-confirmed specification. The friction this creates in the requirement phase is deliberate.
The JIRA memory file is the practical mechanism by which human decisions persist across context boundaries. AI assistant sessions are stateless — each new conversation begins with no memory of previous ones. The JIRA memory file bridges this gap by recording, in structured Markdown, the decisions made in every session and the state of the ticket at handover.
The file follows a template with these sections:
Users do not select or name skills manually. The AI assistant identifies the appropriate skill from the task at hand — a requirement conversation triggers the BA skill; a development request triggers the developer skill; a review or documentation request triggers the doc and review skill. The user simply describes what they need; the AI determines which skill context to operate in and loads the corresponding instructions automatically. This keeps the interaction natural and removes the need for the user to understand the internal skill architecture.
The BA skill is most effective when it has access to contextual domain knowledge beyond the generic q language documentation — the available tables, their schemas, the partitioning scheme, the known data quality issues, and the business definitions of key terms. Building and maintaining this domain knowledge documentation is not optional groundwork; it is the investment that makes AI-assisted development increasingly effective over time.
The core developer workflow is iterative and conversational. Rather than generating a complete module and then testing it, the developer skill generates one function at a time, validates it with kdbx_q_eval, and iterates until the function behaves correctly before moving to the next. Errors are localised. Intermediate results are visible in the conversation. The live execution environment forces the model to produce syntactically correct q rather than plausible-looking q.
The workflow for a typical function:
A note on input guards: native q code rarely includes defensive input validation — the language favours performance and terseness. The input guard pattern in this framework is a skill-level requirement, not a q idiom. Teams that prioritise raw throughput can configure the developer skill to omit guards on internal functions; teams building analytics consumed by less experienced callers will find them valuable. The framework makes this a configurable choice rather than an implicit assumption.
The following exchange illustrates the interaction pattern: a prototype expression, live validation, and error-guided correction:
// Happy-path exchange
Developer skill: kdbx_q_eval["select vwap:size wavg price by sym from trade"]
MCP response: { "status": "success", "result": { "sym": ["AAPL","GOOG"], "vwap": [105.3,201.7] } }
// Error exchange — descriptive error lets Claude reason and self-correct
Developer skill: kdbx_q_eval["select update:price*1.1 from trade"]
MCP response: { "status": "error",
"message": "evaluation failed",
"kdb_error": "assign: type error — 'update' is a reserved q keyword" }
Developer skill: // Understands the error, retries with a valid column name
kdbx_q_eval["select adjPrice:price*1.1 from trade"]
MCP response: { "status": "success", "result": { "adjPrice": [110.0, ...] } } Descriptive error messages are essential to this loop. An error that says only ‘type’ forces the model to guess. An error that says ‘update is a reserved q keyword’ allows the model to reason, rename, and retry in a single step. The MCP server is designed to pass through the full KDB error string rather than abstracting it into a generic failure message.
The qcumber test-writing phase is deliberately separated from the implementation phase. A model that has just written a function will tend to write tests that confirm what the function does rather than tests that verify what the specification requires. By starting a new session and reading only the BDD specification, the skill produces a test suite that is genuinely independent of the implementation.
The error exchange pattern matters here too. When a test fails, the kdbx_q_unit_test tool returns the full qcumber failure output — the expected value, the actual value, and the test label. Claude reads this structured output and determines whether the fix belongs in the module code or in the test itself. This is the same self-correction loop as in 6.1, applied to the test-writing phase.
A representative qcumber test block from GED-1123:
feature .analytics.calcDay
before
if[not `.analytics in key `.; system "l /app/src/analytics/analytics.q"];
tradeData :: ([] date:3#2026.01.02; time:09:00:00 09:15:00 10:30:00;
sym:3#`AAPL; price:100 101 102f; size:1000 1500 2000)
should return one row per 30-min bucket with all signal columns
expect result is a table
98h = type .analytics.calcDay[`AAPL; 2026.01.02]
should raise a signal for an invalid symbol
expect error on unknown symbol
@[.analytics.calcDay; (`INVALID_SYM; 2026.01.02); {1b}] The MCP server runs inside a Docker container that packages the KDB-X 5.0 binary, the qcumber test runner, and the analytics codebase. This design is built for enterprise adoption from day one:
In a traditional development workflow, test coverage is a quality signal. In an AI-native development workflow, it is an acceptance gate. When code is generated by a model, the primary question is not “did a skilled developer write this well?” but “does this code do what it is supposed to do?” Automated tests are the mechanism by which that question receives an objective answer.
Target: A 90% test coverage target is the minimum threshold at which AI-generated code can be accepted into a production-equivalent environment with confidence. At that level, a failing test in the CI pipeline is a reliable signal that something changed in a way that was not intended.
The framework integrates automated test execution into the pull request workflow. Every PR that touches analytics code triggers a qcumber run against the full .quke suite. A PR that reduces coverage below 90% is blocked from merging. A PR that introduces a new public function without corresponding qcumber tests is blocked from merging. Without enforcement, coverage targets are aspirational. With enforcement, they become the structural constraint that shapes how AI generates code.
The separation of the development and test-writing sessions creates a natural alignment with Test-Driven Development (TDD) principles. The key property is that the tests are derived from the specification, not from the implementation — which means they serve as an independent verification of correctness. In the GED-1123 development sessions, the independent test-writing session identified that a function had been named update — a protected q keyword that causes load failures in subprocess-isolated environments even when it works interactively. Both issues were caught before any code was merged.
The long-term value of high test coverage in an AI-native workflow is continuous regression detection. Any enhancement that unintentionally changes the behaviour of an existing function is caught immediately. This continuous regression capability is what makes incremental AI-native development sustainable over time — without it, the accumulation of unverified AI changes creates a fragile codebase where confidence decays with each iteration.
The primary challenge of AI-assisted development is not that it generates bad code — it is that it generates code faster than human reviewers can process it. The KDB Doc and Review skill addresses this by performing an AI-first review pass before the human reviewer sees the code. The AI reviewer produces a structured report of findings; the human reviewer focuses on the findings rather than the full module. The AI reviewer excels at pattern matching; the human reviewer excels at semantic reasoning. Combining both produces better outcomes than either alone.
Knowledge drift — divergence between the specification, the implementation, the tests, and the documentation — is a systemic risk in any development workflow. In the GEP-1123 (TCA) development, the drift check revealed that the implementation had added a traderHoldTimeSecs column to the output schema that was not present in the original BDD acceptance criteria. The drift report flagged this and prompted the human to confirm whether the addition was intentional.
The KDB know-how knowledge base (.github/kdb-knowhow.md) serves a purpose that is easy to underestimate: it reduces LLM loop iterations. Every time an AI session re-discovers a known q gotcha — the update keyword conflict, the Docker namespace switching behaviour, the before each reset limitation — it spends tokens, latency, and cost on a problem that has already been solved. The knowledge base eliminates that waste. At session start, the skill reads it. The known gotcha is already in context. The model skips straight to the solution.
Over time, the most useful knowledge entries can be promoted: from the knowledge base file into the agent’s core instruction set, where they become standing rules rather than context that must be re-read each session. This is the incremental path from a knowledge file toward a fully informed agent that carries institutional KDB knowledge by default.
The knowledge base in this framework is a Markdown file — simple, portable, and human-readable. Teams with higher scale or more complex memory requirements can replace or augment it with a vector database, a structured embedding store, or any other long-term memory solution. The architecture is intentionally agnostic to the storage mechanism; the pattern of read-at-start, append-at-end is what matters.
Representative entries:
GEP-1123 was the first full end-to-end test of the three-skill workflow. The task was to implement a Transaction Cost Analysis capability computing fill rate, trader hold time, and execution hold time at the equity parent order level.
| Phase | Session | Key Outcome |
| Requirements | BA skill — 9 questions raised and resolved in one session | BDD spec with 10 acceptance criteria; data model confirmed (self-join on order table) |
| Development | KDB Developer skill — prototyped each metric in isolation via kdbx_q_eval | All 10 ACs verified against mock data; fill rate, hold times implemented |
| Test writing | New session — derived from BDD spec only, not implementation | 9 qcumber scenarios; drift check caught undocumented schema addition |
| Outcome | Three sessions, no rework after review | 100% BDD coverage; JIRA memory file maintained at every transition |
GED-1123 was a more complex challenge: six complementary market microstructure statistics computed over 30-minute intraday buckets, supporting both real-time streaming and T+1 batch modes.
The development phase surfaced three significant insights now part of the developer skill’s standard guidance. First, the update function name conflict — a function was originally named update, a protected q keyword, causing silent load failures in qcumber subprocess context. Second, the namespace switching issue — \d . switching failed under Docker container load; explicit ..* assignment resolved it. Third, a configurable weight dictionary required a specific access pattern that the model needed to be shown explicitly.
The independent test-writing session produced 24 assertions across 6 features and identified one additional bug: the streaming accumulator was not resetting correctly at bucket boundaries with no trades. Caught before any code was merged.
Outcome: delivered in four sessions (BA, development, independent test writing, review fix), all 24 qcumber assertions passing.
The table below compares actual AI-assisted delivery time against the lead engineer estimate for equivalent unassisted delivery (development, testing, and documentation combined):
| Metric | GEP-1123 (TCA) | GED-1123 (Microstructure) | Average |
| AI-assisted delivery | ~2 hours | ~4 hours | ~3 hours |
| Estimated unassisted delivery | ~16 hours (2 days) | ~28 hours (3.5 days) | ~22 hours |
| Time saving | ~14 hours | ~24 hours | ~19 hours |
| Speed multiplier | 8× | 7× | 7-8× |
| qcumber test assertions delivered | 9 scenarios | 24 assertions | — |
|
Note: unassisted estimates are the lead engineer’s professional judgement for a skilled KDB developer working alone, including write, test, and document phases. AI-assisted times are measured wall-clock hours across all sessions for each ticket. |
|||
What the data shows Across GEP-1123 and GED-1123, the three-skill AI-SDLC loop delivered 7–8× faster than the lead engineer’s unassisted baseline estimate. Both modules achieved full qcumber test coverage (9 and 24 assertions respectively) and passed the documentation completeness audit without rework. The speed gain was realised without any reduction in code quality, naming compliance, or test independence.
Prior to the formal skills framework, several development sessions used the core MCP tools alone, without the structured skill workflow. The pattern that emerged — prototype with kdbx_q_eval, encode acceptance criteria in qcumber, commit — became the foundation of the developer skill. Without the BA skill’s structured elicitation, requirement assumptions were made that had to be revisited during development. Without the review skill’s drift detection, implementation and documentation gradually diverged over successive iterations.
The framework described in this paper is explicitly a starting point. It is designed to be used by organisations that are beginning their AI-native KDB-X development journey and need a structured, auditable, and incrementally improvable approach. It is not designed to eliminate human oversight — and it would be counterproductive to deploy it as if it were.
The deliberate human-in-the-loop design at workflow transitions, the mandatory test coverage gate, and the JIRA memory file discipline are the properties that make the framework safe to use with production-bound code. They are also the properties that establish the trust baseline from which greater autonomy can be progressively introduced.
The progression from the current framework to higher levels of autonomy follows a clear sequence. Each step adds capability without removing the controls that precede it.
This progression mirrors the architecture described in the TPICAP Risk Transformation AI-SDLC on AWS document, which represents a mature enterprise deployment of the same principles at the scale of a risk platform transformation.
This paper has described a practitioner’s framework for AI-native KDB/q development that is grounded in real development sessions and validated against production-equivalent analytics. The framework demonstrates that specialised agent skills can close the idiomatic quality gap in AI-generated q code; that a live MCP-based KDB execution environment eliminates the feedback latency that is the primary obstacle to AI-native KDB development; that human involvement concentrated at requirement elicitation and PR review produces better outcomes than either full human control or full AI autonomy; and that automated qcumber testing at 90% coverage provides the acceptance gate that makes AI-generated code defensible to regulators.
For organisations considering adoption of this framework:
The fastest way to experience this framework is to see it running against a real KDB schema — live, with an actual analytics requirement, in the same AI conversation where requirements are elicited, code is generated, and tests pass. First Derivative’s KDB AI practice can arrange a proof-of-concept session that demonstrates this end-to-end in your environment.
To discuss a POC, speak to your First Derivative account manager or reach out through the First Derivative website. Reference this paper and ask for the KDB AI SDLC proof-of-concept engagement.
For teams who want to explore independently first: the open-source kdb-x-mcp-server repository on GitHub (KxSystems/kdb-x-mcp-server) provides the foundation. It is approximately 40% of what a production-ready deployment requires — the MCP server framework and the KDB connection layer. The tooling described in this paper — kdbx_q_eval, kdbx_q_unit_test, the Docker environment, and the three agent skills — represents the author’s validated extensions built on top of that foundation. First Derivative can accelerate that journey considerably.
First Derivative is a global technology and consulting firm specialising in financial markets data and analytics platforms. With deep expertise in KDB/q, real-time data infrastructure, and quantitative analytics, First Derivative serves clients across trading, risk, and post-trade operations at tier-1 financial institutions worldwide.
EPAM Systems is a global provider of digital platform engineering and development services. EPAM’s engineering excellence spans from foundation-model fine-tuning to production-grade AI-native SDLC tooling. The KDB-X MCP Server project described in this paper is an open-source initiative demonstrating the practical application of AI-native development principles to the KDB/q ecosystem.
The methodology described in this paper — the three-skill workflow, BDD specification writing, iterative prototyping via kdbx_q_eval, and qcumber-based acceptance testing — rests on design principles that are largely platform-agnostic. The skill workflow and JIRA memory file discipline can be adopted independently of the underlying kdb+ version.
Individual developer environments — The containerised, zero-configuration developer environment described in this paper depends on the KDB-X 5.0 Community Edition, which is freely available for individual use. This is the primary deployment model the framework targets: a fully self-contained KDB-X instance in each developer’s hands, removing the platform engineering overhead that has historically made kdb+/q development environments slow to provision and inconsistent across teams.
Enterprise kdb+ 4.x environments — Organisations with an existing enterprise kdb+ licence can run the MCP server against their licensed kdb+ 4.x process directly. The core tools (kdbx_q_eval, kdbx_q_unit_test) function against any reachable q process; the KDB-X 5.0-specific features (module auto-loading framework, .ai.* namespace) would not be available, but the interactive prototyping and qcumber testing loops remain intact.
Shared sandboxed environments (kdb+ 4.x without individual licences) — Where individual developer instances are not feasible, the recommended pattern is a sandboxed MCP environment backed by a shared, AI-accessible kdb+ process. Cloud-hosted AI platforms such as AWS Bedrock provide a natural integration point: the MCP server connects to a centrally managed kdb+ instance, and developers interact with it through their AI assistant without requiring a local licence. This architecture trades the zero-friction local setup of KDB-X 5.0 for a shared infrastructure model that fits existing enterprise licence arrangements.
