AI is becoming very good at helping us work on software systems that we do not fully understand.
It can read thousands of lines of code, find where a change should be made, explain an error, propose an implementation and prepare tests.
For software developers, this is genuinely useful. It can make unfamiliar projects easier to explore and tasks that once required a great deal of time easier to approach.
But it also raises a question:
After an AI-assisted change, has the system become easier for the next person to understand? Or have we simply managed to change it without fully understanding it?
AI Is Entering Systems That Are Already Complex
The problem did not begin with AI.
For a long time, we have been building software systems made up of many layers, tools, configurations, and rules accumulated over the years.
We have built powerful abstractions, but often fragile ones.
Abstractions allow us to avoid dealing with every underlying detail each time we work on something. They are also one of the reasons we have been able to build systems that are so sophisticated.
Fragility emerges when we focus on each new layer in isolation and gradually lose sight of what arises from its interactions with all the others. A component may work well on its own and still produce unexpected behaviours once it becomes part of the wider system.
Some of this complexity is unavoidable. Some of it comes from layers of past decisions, incomplete documentation, unclear responsibilities, or parts of the system that no one has had the time to simplify.
Traditionally, this complexity has had at least one obvious consequence: sooner or later, making changes became difficult.
AI can change this situation. It can act as a guide through the maze, reconstructing each time the relationships that a person might struggle to see.
This is useful. But that very ability could also make us feel less urgency to put the maze in order.
A dynamic that is only beginning
We are not claiming that most software projects already depend on AI to function.
We are still at an early stage.
But new working habits are forming now. Increasingly, we can ask a model to:
- reconstruct how part of a system works;
- find a hidden rule;
- prepare a change;
- explain why something is failing;
- remind us how to perform a complicated procedure.
The risk is that this ability may lead us to accept systems that become increasingly difficult for people to understand directly.
When AI can find the route for us every time, we may feel less pressure to leave clear directions for the people who come next.
Will we have to pay every time we need to understand our software?
Using an AI model consumes computing resources. Its use is often measured in tokens, the units used to calculate how much text the model processes and, in many services, how much the interaction costs.
We do not know whether each token will become more or less expensive in the future. We do not need to predict that to recognize the issue.
Even when individual requests become cheaper, a system may require more and more AI use:
- more files to inspect;
- longer conversations;
- more attempts;
- more reviews;
- several agents working together;
- the same understanding reconstructed again for every future change.
The important question is not only how much one request costs.
It is this:
Do we want understanding and maintaining our own software to become a service that we must repeatedly purchase?
AI can be an extremely useful accelerator. But it should not become the only practical way to understand what we have built.
A convincing answer is not proof
There is another problem.
Language models are not truth machines. They produce plausible answers from the information available to them, but they can be wrong, overlook an important detail or reason from an incomplete assumption.
An AI system might:
- miss an important file;
- know nothing about behaviour that appears only in the live environment;
- misunderstand a rule;
- ignore an unusual case;
- propose a change that looks correct but removes something necessary.
The risk becomes greater when the same interpretation is used to generate everything else.
AI makes an assumption, writes the code around that assumption, prepares the documentation and generates the tests. The result may look consistent, while being consistently based on the wrong idea.
A temporary guess can then become accepted project knowledge without ever being properly verified.
This is why we want to adopt a simple rule:
AI may propose an answer. It cannot also be the only proof that the answer is correct.
Important changes must be checked using evidence outside the conversation: the intended behaviour, the actual code, meaningful tests, observable results and the judgement of the people responsible for the system.
Use AI to reduce complexity
There is another possible path.
We can use AI not only to cross a complicated system, but to leave that system a little simpler than we found it.
After an AI-assisted session, we might leave behind:
- clearer code;
- fewer duplicated rules;
- responsibilities that are easier to locate;
- short and current instructions;
- automatic checks;
- tests based on the intended behaviour;
- better ways to diagnose problems;
- recovery procedures that a person can follow.
In this approach, AI does not merely supply missing understanding for a few minutes.
It helps transfer that understanding into the system and into the people who will maintain it later.
AI may still be useful next time, but it should be a little less necessary.
Why begin with software development?
This dynamic may eventually matter in other areas too. But we do not want to assume that our conclusions already apply everywhere.
We are beginning with software development because this is one of the areas where the change is closest and easiest to observe.
Developers are adopting these tools quickly. More importantly, what AI generates does not disappear when the conversation ends. New code remains in the project, connects to the rest of the system and may need to be understood for many years.
Software also gives us ways to run concrete experiments.
We can compare:
- how difficult a task is without AI;
- how much AI helps complete it;
- what can be simplified using that assistance;
- whether another person can later repeat the task without the original AI conversation.
The real result is not merely producing more code or finishing the task faster.
The result is improving the situation for whoever comes next.
Human-Scale Systems
Within ABC, we want to begin exploring this direction. We are provisionally calling it Human-Scale Systems.
It is not yet a new organization, a platform or a finished solution.
It is a research question and a path made of small experiments:
Can we use AI to leave software easier for people to understand, verify, change and recover without depending on the AI session that helped create the change?
We want to begin with real and observable cases, including honest accounts of what does not work.
Only later will we be able to judge whether the same ideas are useful in other settings.
More autonomy, not more dependency
AI can help us do important things and approach problems that once seemed beyond our reach.
We do not want to give up that possibility.
But while these new ways of working are still taking shape, we want to adopt one principle:
AI should leave behind more autonomy than dependency.
After AI has helped us, there should be more shared knowledge, more verifiable tools and a greater ability for people to continue the work.
Not only a convincing answer.
Not only more code.
And not the need to ask again, every time, how the systems we created are supposed to work.