It Isn’t a Competence Problem

A project comes back late, or over-built, or quietly wrong. The review finds a person to develop. Two quarters later a different person on a different project produces the same shape of failure — and the review finds them, too.

At some point the pattern stops being about people. Ask a computer scientist and a software engineer to build the same thing and you will often get the same artifact — but they arrived from opposite ends of the problem, and the route only shows up when the problem stops being routine.

One reasons upstream, toward the definition. The other reasons downstream, toward the evidence.

Both habits are right most of the time, which is why neither one ever gets corrected. The cost lands at the handoff — and only one of the two failures announces itself. The other surfaces two years later as a layer of workarounds nobody can remove, and by then it gets filed under competence. What follows is how to tell them apart: what builds each habit, how each one fails, the single question that says which kind of problem is actually on your desk — and what it costs a team to be short of one.

Trained, not chosen

The mindset comes from what gets punished, not from the syllabus

A field shapes its people through the mistakes it penalises, and the two fields penalise different ones.

Computer science punishes not being able to say why it works. Correctness is settled against a standard defined inside the field — proofs, complexity bounds, whether an optimal solution exists at all. The reward goes to whoever reduces the problem to its essential structure. So the trained habit is: strip it down until the shape is recognisable.

Software engineering punishes a system that surprises you in month six. Correctness is settled against the world — requirements, load, handovers, budgets, inputs nobody predicted. The reward goes to whoever builds something that survives five years and three staff changes. So the trained habit is: ask what breaks, and who maintains it.

Look at where the pages go

Open a textbook from each field and watch how the chapters are allocated. The allocation gives away what that field treats as the hard part — and what it treats as a detail you can work out for yourself.

Read it as an epistemic claim, not a curriculum complaint. Computer science establishes correctness by proof — once, in advance. Software engineering establishes it by evidence — repeatedly, after the fact. That is why verification carries the back half of one book and a footnote in the other.

The first move gives it away

The marking stops. The habit doesn’t

That is the first of the two places this split lives: the exam, where one specific kind of mistake is what gets penalised. The second is everything after it — the job, where nothing is scored at all and the habit simply runs. What crosses between them is a single visible act: an unfamiliar problem lands, and the hand goes either to the problem or to the tool. Same person, same habit, no examiner. The habit was priced once, and it is spent daily.

Derive, or retrieve

Watch two people meet an unfamiliar problem. The first move is the whole difference, and it does not fade with seniority.

Derive. Clarify the definition, place the problem in a known formal class, pick the structure that class implies. Slower to start. Does not fail silently when the problem turns out to be genuinely new.

Retrieve. Match against solutions that survived production before, then adapt. Faster, and usually right — most problems really are near-duplicates. The risk being managed is not optimality, it is surprise.

The gap widens instead of closing. Every project where the near-fit pattern worked reinforces retrieval. Every project where pinning down a definition exposed a hidden constraint reinforces derivation. Both sides keep collecting confirming evidence, so the priors harden — and more experience just means a bigger retrieval library, or sharper abstractions.

Neither one outranks the other

Compilers, type systems, databases, consensus protocols, schedulers, cryptographic primitives, the machine-learning frameworks — these are upstream products, and they arrive as given to the engineer building on top of them. The engineer’s entire correctness argument rests on them already being correct, which is exactly why they had to be proved rather than tested. One field’s output is the other’s floor.

Which is why “science versus engineering” is the wrong reading, however much the degree names invite it. Both of these are engineering. The real axis is distance from the application, and every mature field has it: physics sits upstream of materials, civil, electrical, mechanical — each of those the same underlying science pointed at a domain, each unmistakably engineering. Nobody reads that arrangement as science outranking engineering. It is a stack, and the only live argument is which layer a given problem sits on. Software has the identical structure. It is just younger, so it still argues about the labels instead of the layers.

A foundation is not superior to the building: nothing stands without it, and nobody lives in it.

What the difference costs you

Two asymmetries follow, and both are practical

The architect asymmetry. Architecture is deciding what a system is before any evidence about it exists — before there is a running thing to measure, a load to observe, a user to be surprised by. That is upstream work by definition, and it gets settled the way upstream problems get settled: by argument about the definition, not by evidence about the outcome. A retrieval habit produces excellent design — a composition of parts that have each already proven themselves — and most systems need exactly that. But composing proven parts is not the same act as fixing the shape of something with no precedent, and when there is no near-fit to retrieve, retrieval has nothing to reach for. So the architect role tends to land on the upstream habit. Not because the role demands a particular schooling — because the job is upstream, and only one of the two habits is trained on problems that have no evidence yet.

And two different jobs hide under that one title. Architecture over known territory — well-precedented systems, composed from parts that have each already proven themselves — is reachable from either background, and the retrieval habit is genuinely good at it; most architecture is this. Architecture that produces a novel solution is a different act. There is no precedent to compose from, so the shape has to be derived rather than selected. That one comes from research nature — which is not a credential, and not reliably a job title either. The people who introduced the primitives everyone else now builds on include Tony Hoare, a graduate in classical languages and philosophy; Rich Hickey, who studied music composition; and several who left university without finishing. What they share is not a school. The best candidate for what they do share is a first move: on meeting something they do not understand, investigate it rather than route around it. A science education is the most common place that habit gets deliberately trained — that is the whole point of penalising you cannot say why it works — which is why the degree looks like the cause. The degree is the proxy. When there is no near-fit to retrieve, only the habit trained on undefined problems has a move at all.

The conversion asymmetry. A computer scientist becomes a software engineer by practice. The retrieval library is accumulated, and time in production accumulates it whether you intend to or not — five years of shipping, handovers and incidents will build it for you. The other direction is not an accumulation. Deriving is not a bigger library; it is refusing to start until the definition holds. You cannot log hours into that. You have to stop accepting it works, ship it as a finished answer, and changing what satisfies you is harder than adding to what you know.

Whether it moves at all depends on what surrounds it. A habit shifts only when the person’s other habits do not pull against it; where they do, the trained one wins and stays. Which is why this is not a matter of effort. You are not replacing one habit — you are asking it to disagree with everything else you do.

The two habits fail differently, and that asymmetry matters more than which one is better

Derivation fails by never converging. The definitions never get clean enough to start. Work stalls at the whiteboard, or produces something elegant and general that nobody else can maintain. Visible immediately.

Retrieval fails by forcing the problem to fit. The pattern is applied to something it does not quite describe. The mismatch surfaces late, as a layer of accumulated workarounds nobody can now remove. Visible in year two.

One failure is loud and early. The other is quiet and late. That asymmetry is the whole reason this is worth naming — the cheap failure announces itself, the expensive one does not.

So the useful question is not which habit is better. It is whether the problem is actually novel. Genuinely novel problems are rarer than the computer scientist assumes, and more common than the software engineer assumes.

The same split, one industry over

Accounting is anchored to a ground truth defined inside the field. The standards say what correct means; the books balance or they do not; a close is provable, and the failure that costs you is being unable to say why a number is what it is. Finance is anchored to outcomes in a messy world — valuation, risk, forecasts, markets under no obligation to respect your model. Nothing there is proved. It is evidenced, repeatedly, after the fact.

Every asymmetry repeats. The accountant’s output is the analyst’s input; statements are the floor the analysis stands on. An accountant moves into analysis by practice, while an analyst asked to close the books has to accept a standard where approximately right is simply wrong. Same shape, different vocabulary — which is a good sign the shape is real, and not a story about programmers.

What to do about it

Which one fits the problem in front of you?

Domain is a decent signal. Where the uncertainty sits is a better one.

Match the problem, not the person

Two capabilities, different shapes — and a cost paid every single time work is handed to the wrong one. Assigning by whose mindset fits sounds efficient, but the problem then gets quietly reframed to suit whoever received it — an upstream problem handed to a retrieval habit comes back as a near-fit pattern with the mismatch buried, and it comes back looking finished. So decide what kind of problem it is first, out loud, before you decide who owns it. Most failures blamed on competence are failures of assignment.

All of which assumes both habits are in the room. If they are not, assignment is not a decision you are making — and the missing one is almost always derivation, because a team of retrievers cannot detect its own misses: a near-fit looks like a fit from inside. That is the case for keeping one derivation habit on a team whose problems are mostly routine. Not to do most of the work — to be the thing that notices when the routine problem wasn’t. Invert it for a company whose product does not exist yet: there an all-retrieval team will build a competent near-fit of somebody else’s product.

Depending on where you sit

• Choosing a degree? You are not picking a job title — both routes end up writing software. You are picking which habit gets drilled for the length of a degree, and which standard of “this works” will feel natural to you afterwards.
• Managing both? Friction between the two is usually a disagreement about what counts as done, not about competence. Name the problem type before the work starts and half the argument disappears.
• One of them? Your habit is right often enough to keep confirming itself. Learn the other’s failure mode rather than their toolkit — that is the part your own habit cannot see coming.
• Facing a problem right now? Ask what would have to be true for “correct” to be defined. If nobody in the room can answer, it is upstream — whatever the domain looks like, however routine the ticket reads. That question costs a minute, and it is the only cheap thing on this list.

One honest limit. Curricula vary enormously, and most computer-science graduates spend their careers doing software engineering work — which is exactly what the conversion asymmetry predicts.

And none of this is a claim about credentials. Retrieval needs a precedent; novel architecture has none by definition. That is a statement about what the work requires, not about who is permitted to do it — the habit is buildable, some engineers deliberately build it, and the title never required the degree.

What persists is not a personality type. It is the first move under ambiguity — what someone reaches for before anyone has told them what the problem is.

The last project that came apart on you — was the definition wrong, or was the evidence?