When modernizing an old information system, the natural starting point is the source code. We want to know which programs it contains, how they connect, which files they read and where they write their results.
That is necessary. It is not the whole picture.
One important perspective has stayed with me from my accounting and AS/400–COBOL background: a program cannot be judged only by whether it starts and finishes without an error. We also need to know whether the resulting numbers mean what accounting, operations and the people responsible for controls expect them to mean.
In Before Replacing COBOL, Preserve the System Intent, I argued that modernization should preserve the intent of a system before replacing its technology. The next question is:
How do we discover that intent when documentation is incomplete and the code alone does not explain every operational dependency?
One approach is to follow the data, the batch processing and the way results are reconciled. Read the system not only as a collection of programs, but also as a process that turns incoming events into accounting entries, balances, reports and decisions.
The Program Finished Successfully. Is the Result Correct?
Imagine an overnight batch that processes incoming payments.
It reports success. No technical error occurred, all required files were accessible and the program terminated as expected.
What have we actually established?
We have evidence of successful technical execution, within the scope of what that program and its controls check. We may not yet know whether:
- all expected inputs were received;
- a file arrived twice;
- payments were allocated to the correct accounts;
- the correct posting date was used;
- someone reviewed rejected items;
- the results are complete from the perspective of the next process.
Technical batch success and business correctness are two different questions.
This is not unique to COBOL. A modern application can have the same problem. When modernizing an old system, however, we need to discover where and how both questions are answered, so that replacing the technology does not also remove part of the control mechanism.
A green status in a job scheduler is useful. On its own, it does not tell us whether we can trust the processing result.
Follow One Transaction, Not Just One Program
As an illustrative example, consider an incoming customer payment.
At first glance, it seems simple: money has arrived and must be recorded. A closer examination reveals several distinct steps:
- Receive an input file or record.
- Validate its format and required fields.
- Check whether the item has already been processed.
- Identify the customer or outstanding item to which the payment belongs.
- Determine the posting date and accounting period.
- Create the relevant entries.
- Update records and balances.
- Include the transaction in control outputs and reports.
- Pass the result to another system or process.
Not every system separates these steps in the same way. What matters for analysis is being able to trace the transaction through the entire process.
If we examine only the program that creates the accounting entry, we may miss a rule applied during earlier validation. If we examine only the final report, we may not know which items were rejected before it was produced.
At every step, it is worth asking:
- What enters this step?
- What can change?
- What leaves it?
- How are rejection and ambiguity handled?
- How do we know that the step completed correctly?
This gradually turns a map of programs into a map of data movement and business meaning.
Batch Order Is Part of the System Logic
When reading individual programs, we may focus on conditions, calculations and file operations. Some system behaviour, however, lives outside those programs: in execution order, job parameters and operating procedures.
Imagine this sequence:
- import incoming items;
- validate them;
- post accepted items;
- calculate daily balances;
- generate control reports;
- deliver outputs to a downstream system.
If the control report is generated before posting has finished, each individual program might behave according to its own specification, while the process as a whole produces an unexpected result.
It also matters when the business date changes, when an accounting period closes and when reference data is updated.
Modernization therefore requires more than transferring calculations. It also requires examining timing and operational rules:
- Which jobs must finish before another job can start?
- What exactly does “processing for today” mean?
- What happens to input received after the cutoff?
- Can reference data change while a batch is reading it?
- When is a result considered final?
Execution order, timing and input state are not merely operational details. They can be part of the business logic.
We may choose to change them in a new solution. First, we need to understand which behaviour that changes.
What Happens When Processing Is Interrupted?
A successful run reveals only part of a system. Another part becomes visible when we ask what happens after an interruption.
Imagine a batch that stops after processing some of its input. Some entries have already been created; others have not.
What should the operator do?
Run the whole batch again? Resume from the last confirmed point? Restore data from a backup? Perform a check before deciding?
We cannot safely infer the answer from the program name, or from the fact that the program technically allows another run.
We need to establish, for example:
- how processed items are marked;
- which changes are committed together;
- whether another run could create duplicate entries;
- whether output has already been sent to another system;
- how a partially completed state is detected and resolved.
If an administrator knows that a particular check must happen before a rerun, that is important process knowledge. It should not disappear from the new solution simply because it was never written into the application code.
That does not mean preserving a manual procedure forever. It may be an opportunity for improvement. First, however, we need to understand the problem it solves.
Exceptions Reveal More Than the Happy Path
The normal path is easy to describe: valid input arrives on time, passes validation and produces the expected output.
Exceptions are also especially useful when trying to understand a legacy system:
- the same input arrives again;
- an item has no unambiguous identifier;
- a payment cannot be allocated;
- a correction arrives after period close;
- an already processed operation needs to be reversed;
- a batch receives empty input;
- a downstream system is unavailable.
For each situation, we need to distinguish three things:
What the system should do, what it actually does today and which evidence we use to judge that behaviour.
The answers may differ.
Existing behaviour may represent a valid business rule. It may also be a historical compromise, a temporary workaround or a defect that operations has learned to accommodate.
I would therefore not automatically treat every existing behaviour as the correct specification. It is baseline evidence of how the system works. What should be preserved and what should be corrected requires a deliberate decision.
Preserving system meaning does not mean preserving every defect.
Control Totals Are a Starting Point, Not the Whole Proof
When comparing old and new processing, checking totals is a natural first step.
Do the total amounts agree? Does the number of processed items match? Is the closing balance the same?
These checks are useful. Each answers a particular question, however.
The same total amount does not establish that every item was assigned to the correct customer. The same balance does not establish the correct allocation across accounting periods. The same record count does not mean that the records themselves are the same.
It therefore makes sense to compare results at several levels:
| Control level | Question it answers |
|---|---|
| Input completeness | Do we have all expected files and items? |
| Processing outcome | Can we account for accepted, rejected and deferred items? |
| Individual transactions | Do the significant fields and resulting states of specific items agree? |
| Allocation | Were items assigned to the correct accounts, customers and periods? |
| Aggregate controls | Do counts, amounts and balances agree under the defined rules? |
| Downstream outputs | Did subsequent processes receive complete and correct data? |
Not every comparison requires byte-for-byte file equality. A new solution may use different technical identifiers, timestamps or row ordering.
What matters is deciding in advance which differences are acceptable and which change the business result. Otherwise, we risk adjusting the comparison rules to fit whatever results we happen to obtain.
Period-End Close as a Map of Dependencies
During analysis, period-end close can also be an opportunity to understand relationships that are less visible during an ordinary day.
In our illustrative system, it might reveal questions such as:
- Have all input batches completed?
- Are any items unresolved?
- Have detailed records been reconciled with summary postings?
- Which period should contain a late correction?
- Who can confirm completion, and on what evidence?
- If an error is discovered after close, what is the approved correction procedure?
This is not a universal accounting procedure. The specific rules depend on the organization and its processes.
For modernization, the important point is different: closing procedures can show us which results must agree with one another and who is responsible for that reconciliation.
If a new system produces a similar-looking report but cannot demonstrate the relationship between its figures and the source transactions, we have not yet established that it has replaced the original control function.
Operations Staff Are a Source of Knowledge, Not a Substitute for Documentation
I would not limit the analysis to interviews with programmers.
Accountants, batch operators, support staff and administrators of downstream systems may also help answer important questions:
- Which output do you check first?
- What tells you that something has gone wrong?
- Which warning is expected, and which requires action?
- When would you refuse to rerun a batch without further checks?
- What do you verify before closing a period?
- Which problem makes you call a particular colleague?
These conversations help identify areas worth examining. A recollection or established habit alone is not sufficient evidence, however.
If someone says, “This must always be zero,” the next questions are: Why? What rule requires that? Are there exceptions? Do we have an example?
The goal is to turn experience into a verifiable rule, a test case or a documented decision. It is not to transfer dependence on one person from the old system to the new one.
What to Capture Before the First Rewrite
Before rewriting begins, I would want at least a basic evidence set against which the outcome can later be assessed.
1. A Process Map
Not just a list of programs, but the sequence from input to final output, including dependencies and responsibilities.
2. Understandable Rules
What is accepted, rejected, deferred, posted or corrected, and why.
3. Representative Scenarios
An ordinary day, higher-volume processing, period close, corrections, duplicates, late inputs and interrupted runs, as relevant to the system.
4. Expected Results
Not only the final report, but also significant intermediate states, rejected items and downstream outputs.
5. Comparison Rules
What must match exactly, where a difference is acceptable and who decides how unexplained differences are resolved.
6. Known Defects and Open Questions
Separate confirmed business rules from observed behaviour that is not yet understood.
7. Safely Prepared Test Data
Use only data we are authorized to use, with appropriate protection for sensitive information. If data is anonymized or replaced with synthetic data, verify that the relationships and edge cases needed for testing have not been lost.
This evidence set does not guarantee a flawless migration. It provides a more concrete basis for assessment than a general statement that the new system “does the same thing.”
Differences Need to Be Explained, Not Merely Eliminated
When old and new processing produce different results, that does not automatically mean the new system is wrong.
The cause might be a defect in the new solution, but it could also be a defect in the original system, a different initial state, changed reference data or an intentionally revised rule.
This is why explaining differences matters.
It is not enough to keep adjusting the new program until its output resembles the old output. We need to understand why a difference occurred and why we consider the chosen result correct.
If the new solution is intended to fix a historical defect, that is a legitimate change. It should be identified, approved and verified, rather than hidden inside a technical rewrite.
Likewise, agreement across prepared scenarios is evidence about those scenarios. It is not automatic proof of correctness for every future situation.
Modernization Starts by Asking Why We Trust the Result
COBOL is the entry point to this discussion. The underlying problem is broader: how do we replace a system while still understanding its results after the technology changes?
A new language, database or user interface can improve maintainability and introduce new capabilities. On their own, however, they do not establish that transactions were processed completely, correctly and in the right accounting period.
Before the first rewrite, I would therefore examine not only the source code, but also the data journey, batch order, exception handling and reconciliation procedures at period end.
That is where we can discover rules that are not immediately visible, but on which trust in the whole system depends.
Before replacing an old system, we need to explain not only why the batch completed, but also why we trust its result.