The real cost of analysing 100 million lines of COBOL is getting it wrong

A customer came to us last year with a number they had already worked out themselves. Pointing a large language model at their core banking system and having it write the documentation nobody had written in forty years would cost roughly 600,000 euros in tokens, for one pass. They asked us what the same analysis would cost with Nomain. Our first answer was 250,000 euros, and after a few weeks of work on token efficiency we brought it down to 25,000 euros in token costs.

Those numbers get attention, but they answer the wrong question. The right question is whether the output is correct. Whether you spend 600,000 euros or 25,000 on an analysis, you need to be able to act on it: plan a migration, do impact analysis on a change or find where an incident comes from. An analysis that is mostly right does not give you that, because you do not know which parts are wrong. Hence the price only means something once you know what you get for it.

What we were looking at

A global Tier 1 bank with one hundred million lines of code across COBOL, PL/I and Assembler. The usual condition applied: no clear architecture, integrations nobody had mapped, and most of the knowledge of how it fits together gone with the people who built it. The system has run reliably for decades, yet nobody can say precisely what it does.

Scale changes the problem

Analysing 100 million lines is a different problem to analysing a single module. Systems of this age and size are deeply entangled. A single business process can run through hundreds of programs, JCL jobs, copybooks and DB2 tables that were not designed to work like this, they just evolved over time. Hence analysing one directory, or one application, does not give you a smaller version of the right answer. It gives you a different answer, one that is confidently incomplete, and that is worse than no answer because people act on it.

The obvious approach is to let a language model read the code and write down what it finds. The difficulty is that no model can read it in one go. When Anthropic extended Claude's context window to one million tokens, it described that as enough for codebases of over 75,000 lines. A 100 million line estate is in the order of a thousand times larger. In other words, the code has to be cut into pieces, each piece is read in isolation, and the relationships between the pieces are reconstructed by the model afterwards. Those relationships are exactly what you need to get right.

Reading more per request does not solve this either. Liu et al. showed that model performance degrades significantly when the relevant information sits in the middle of a long input, even for models built for long contexts. Chroma tested 18 models and found that performance grows increasingly unreliable as input length grows. A bigger window moves the problem rather than removing it.

Small errors compound

This matters because the questions people need answered are chains: which programs are affected if a field changes length, or where a figure on a customer statement originates. Each answer follows a path through calls, copybooks, files and tables, and every link in that path has to be right.

If a model infers each link with 99 per cent accuracy, a path of ten links is fully correct about 90 per cent of the time. At 95 per cent per link it drops to about 60 per cent, and a path of twenty links at 95 per cent is fully correct about 36 per cent of the time. The numbers are illustrative, but the mechanism is not: errors multiply along a dependency chain, and in a system this entangled the chains are long. A migration plan built on that has gaps in places nobody can see.

An analysis that is mostly right is not mostly useful. If you do not know which parts are wrong, you cannot safely act on any of it.

Documentation starts ageing the moment it is written

Even an accurate analysis produces a document, and a document describes the code once, on the day it was generated. Six months later somebody changes a copybook and the document is quietly wrong in a way nobody can see. Making it true again means regenerating it at full cost, which in practice nobody does. They simply stop trusting it.

So the 600,000 euros buys a snapshot of a system that does not hold still, with an accuracy nobody can verify. That combination is the real problem.

Structure is computed, not inferred

Our approach starts before any model sees the code. We parse it and build an abstract syntax tree, which is a deterministic process. We know that this program calls that program, that this job executes that program and that this copybook is used in these 270 places, because it is written in the code. Nothing is estimated. We go down to paragraph level inside each program and track every variable through its def-use chain, so we know where data actually flows. We also parse implicit dependencies through IMS or databases, where one program writes and the other reads from the same database.

This is the core of the accuracy argument. Structure is the thing that is expensive to get wrong and cheap to compute. Hence the links in a dependency chain are read from the code rather than guessed, and the compounding described above does not apply to them. Once the map exists, the language model is no longer asked to work out how the system fits together. It is asked to explain individual nodes on a map that is already correct. Basically, the model does what models are good at, which is language, and the parser does what parsers are good at, which is certainty.

Cost is the other half of the equation

Token cost still matters, and at this scale it is not small. Our first estimate of 250,000 euros was better than 600,000, but still too much for a pilot project.

The reduction came from a routing layer. Not every task in the pipeline needs a frontier model. Generating a plain description of a paragraph is a different job to reasoning across a graph about a month-end closing process. We built a layer that sends each task to the smallest model that can actually do it, including open-source models and smaller providers where they are sufficient, and we re-test the model mix every month because the answer keeps changing. The final cost for the first full analysis was around 25,000 euros, a reduction of 90 per cent against our earlier approach, with no measurable loss in output quality.

In fact, accuracy and cost are not a trade-off here. The deterministic layer that makes the answers reliable is also what lets smaller models do most of the work, because they are explaining a map rather than drawing one. It has to be said that 25,000 euros is the token cost of the first analysis of that particular estate. It is not the price of the platform, and estates differ enormously in language mix and structure.

Daily use is where both problems multiply. In a typical agent set-up, every developer prompt sends agents to read code from scratch: millions of lines per prompt on an estate this size, across hundreds of developers, and every read is a fresh inference with its own chance of getting the structure wrong. In our case the structure persists as a knowledge layer in the customer's own infrastructure. When the code changes, only the changed parts are re-analysed, so day-to-day use costs a fraction of the first pass and the knowledge stays current.

We will get the token-costs down even further with the addition of decision models, but that's a topic for another blog post.

What accuracy makes possible

The knowledge is not the end-goal that our customers want. What people do with it is, and every interaction with our product depends on the answer being right.

At the bank, our knowledge layer acts as the ground truth for their cloud migration pipeline, which means the migration starts from finding the next best migration candidate. It provides impact analysis before a change is committed, so a proposed change shows what it touches. It powers root cause analysis, so an outage points at its origin instead of starting a search, and it routes bug triage with context rather than guesswork.

At another customer, a team spent three days working out which programs would be affected if the length of a field changed. Afterwards they asked the platform the same question and got the same answer in 52 seconds. The speed is the obvious headline. The more important part is that it was the same answer, checked against three days of careful manual work by people who know the system.

The question worth asking

If you are evaluating anything in this category, the demo will look good. A demo is a well-chosen question against a small code base, and at that scale most tools can read the code well enough.

So do not start with whether the tool can read your code, or even with what it costs to read all of it. Start with how it knows its answer is correct, and how your team can check it. Then ask what it costs to get there, and what it costs to stay correct next quarter, after your teams have changed two hundred programs.

That is the difference between buying a capability and buying an expensive document.