Software factories should amplify human judgment, not replace it

BuildPass runs construction operations on site for more than 1,000 general contractors and over 500,000 workers and subcontractors across Australia and North America.

At the start of 2026, we began exploring two new areas: helping construction companies manage their finances, and helping them price and win work. Building Win and Pay meant learning these domains while building software customers could use.

By the end of August, both new products were in customers' hands, with 100+ major releases across Win, Pay and their shared platform.

Writing code had stopped being the main constraint. Getting the right context quickly, noticing missing details and deciding which problems mattered were harder. More output was only useful if it helped us test our understanding and improve the product.

We built Foreman around those decisions. It gathers context, brings questions to the people who can answer them and carries their judgment into the work. Our engineers build and continually improve that system, as well as the products it helps deliver.

Foreman

Foreman connects finding a problem, deciding what to do and learning from the result:

• Intake: collect customer and team feedback, support threads, adviser notes and production errors
• Triage: clarify the intent, gather the relevant context, ask for missing input, and decide whether and when the signal should become work
• Build: implement the change within the patterns of the codebase
• Verify: establish confidence in the change through tests, security and domain checks, an independent review and browser testing in the real product
• Close the loop: tell the reporter what changed and what it means for them, then use their feedback to inform the next decision

Each stage leaves state and context for the next. A human still makes every merge call. The factory brings forward the initial signal, intended outcome and current evidence, so a reviewer can understand the change without reconstructing the whole run or keeping every detail in their head.

One bottleneck at a time

We did not start with a plan for Foreman. It took shape as we worked through these bottlenecks, often in parallel. Each change taught us something about the product or how to build it. None is permanently solved; we keep adjusting the system as our understanding changes.

Bottleneck 1. Writing code

Early in 2026, an agent could take a clear card to a working branch while we were in a meeting. Producing a scoped change was no longer the rate-limiting step.

“Build construction finance. Make no mistakes.” The dream one day, but today it skips a critical step: understanding the domain deeply enough to distinguish a correct answer from a plausible one.

We were learning these domains ourselves. Agents could help us explore an unfamiliar workflow or compare possible approaches, but we still needed to know which assumptions required an answer from a customer or construction adviser. Faster implementation made those decisions more important, not less.

Bottleneck 2. Doing the right work

Cheap code creates new problems. You quickly learn that not every idea should become a change. The code can be clean, tested and wrong: the taste, judgment and context teams build up over time are not easy to replace.

Our first control was a triage skill, a checklist an agent ran before touching a card. Foreman's version of triage is quite detailed, pulling together application, company and industry context. It accepts work that clears the strict rubric. Most importantly, it holds work that does not and asks for human input before uncertainty hardens into code that further agent iterations treat as concrete truths.

Triage resolves four things before an agent builds:

• Intent: the outcome behind the request
• Vision: whether the proposed direction advances the product
• Context and input: what is already known, and which evidence, expertise or human input is missing
• Decision: whether the work should enter the queue, and when

Foreman records its reasoning and confidence for each.

When Foreman cannot decide responsibly, it holds the card and sends a specific question to the reporter, product team or construction advisers. Instead of asking someone to review an entire problem, it can identify the missing answer that would change the decision. When someone answers, the card goes back through triage.

That helps us learn too. A question often points out details we had missed or narrows several plausible approaches to one we can defend. We can concentrate on shaping the parts we understand confidently, while making the remaining uncertainty explicit rather than carrying all of it in our heads.

Foreman reads a wiki assembled from customer calls, demos, adviser sessions and domain research, and records the sources used for each verdict. Relevant context can reach the next person without them having to retrace every conversation.

We also preserve selected human corrections as guidance and evaluation cases. These are deliberate engineering changes, not automatic learning from every interaction. As the product moves from greenfield modules to refinement and edge cases, the evaluation suite moves with it. Rubric changes must pass that set before they land.

Bottleneck 3. Running enough work

Once more cards cleared the gate, our own human attention became the limit again. One person could only start, steer and review a few sessions at a time. Fast agents still spent too much time waiting for the next instruction.

We separated the work. Each card runs in its own worktree, with its context attached, inside a local Linux sandbox on our hardware. Multiple agents can work concurrently, and orchestration agents can pick up eligible cards, start workers and report what needs attention.

We use Linear as the durable queue of what needs to be done. GitHub records the state of each change and detailed review evidence. We have used tools including Conductor, Superset and Orca as control surfaces for the fleet.

The useful change for our engineers is being able to get into the right context quickly. Less time goes into reconstructing a session or remembering what happened on another card. More can go into understanding the problem, comparing approaches and improving the checks that let the next piece of work proceed safely.

Bottleneck 4. Trusting the code

Parallel output made human comprehension the next problem. With hundreds of pull requests arriving each month, we needed the details that could change a decision to remain visible.

Every push gets an independent review, adapted from Peter Steinberger's autoreview skill. A separate Codex or Claude session reads the change cold and reports material findings. Higher-risk changes also get an adversarial pass from a different model. We verify the findings rather than applying them blindly.

Tests and conventional static analysis still run. For deeper security work, we use Vercel's deepsec. Static matchers locate security-sensitive files, coding agents trace the relevant data flows, and a second pass revalidates the findings. We maintain separate scopes and threat prompts for finance, authentication, money paths, observability and the other parts of the system.

One of the more important steps was building our browser-verification harness. It starts the real application, signs in, exercises the changed workflow and checks both browser errors and data writes. Before-and-after screenshots or a short video show the change in use. That evidence makes the intended behaviour easier to understand. If it cannot be produced, the pull request stays unready.

Every pull request begins with the original signal: who encountered the problem, what they were doing and why it mattered. It states the change, then shows the evidence. Review results are tied to the current commit; a new push makes previous confidence and browser evidence stale.

The evidence may support a merge, a recut, a hold or closing a change that solves the wrong problem. Closing clean code is useful if the reason informs the next decision.

Bottleneck 5. Feeding the factory

By June, the factory could work through inbound feedback quickly, but much of that feedback arrived later than it needed to. Signals already existed in Sentry, server logs, support threads and customer conversations. A person still had to notice them and assemble the context before they became work.

We began connecting those signals to the same intake systems. In-product reports can arrive with the current screen, recent navigation, a Sentry replay and scoped server-error context. A feedback sweeper triages the resulting Linear cards and routes accepted work to isolated workers.

The factory also looks for work. A daily regression scan produces a repair queue, while browser crawls test product flows against written intent. These checks help us notice UX issues, mismatched functionality and bugs without waiting for someone to encounter and report each one.

A finding still needs judgment. It may point to a defect, a misunderstanding of the workflow or a choice we need an adviser to explain. Finding it automatically gives us a question to investigate, not an instruction to produce more code.

Closing the loop means returning to the original signal. The reporter needs to know what changed and why. Their next response can confirm our understanding or expose another gap, giving the team and the factory something more useful to work from.

What we'd tell anyone building one

The models, harnesses and orchestration tools changed throughout this work. These principles helped us keep the system useful as the tools changed:

Give each stage one outcome

When agents work across too many stages or processes at once, critical thinking and adversarial judgment can get lost.

We have found that agents work best with clear responsibilities, but they should treat instructions from other agents with a grain of salt. An agent trying to be helpful can pass on an overconfident assumption that the next agent treats as an instruction rather than checking. Review agents can then pull a change into an AI-slop spiral. Explicit instructions to push back and stay pragmatic help prevent over-engineering.

Each stage has its own instructions, context and evaluations. When something fails, we can improve that stage without rewriting the whole factory. Clear boundaries also let developers take over a step without interrupting the rest of the work.

Keep the state declarative

Linear records what should be built: the original signal, gathered context, triage verdicts and unanswered questions. Anyone on the team can add missing context directly. If that changes the decision, the card goes back through triage.

GitHub records the active change. Labels and attached evidence show what has run, what passed and what needs attention. That state stays visible outside the agents' conversations, so a person can steer, stop or take over the work without reconstructing it first.

Put engineering judgment into the system

Models, runtimes and orchestration tools are available to other teams too. Choosing them does not settle which customer problem to solve, how to model a construction workflow or what evidence is enough to trust a change.

Our engineers build the parts shaped by those decisions: the triage rubric, domain context, risk checks, verification and evaluations. When a failure reveals a missing check or a poor instruction, fixing the system lets that judgment influence later work as well as the change in front of us.

This is engineering work alongside building the product, not a replacement for it. It takes people who understand both well enough to decide what should be automated, what needs a question and what should remain a human call. The factory gives their expertise more reach.

Let it work on itself, carefully

The factory can open cards against its own failures and propose changes to its tools. It could improve its own machinery indefinitely: code is an easy output to measure, and a self-healing harness will always find problems to fix, inserting slop where pragmatic judgment is needed.

Factory code also has a wider blast radius if left to be fully autonomous. A bad change to the rubric or review system can alter how every later piece of work is selected, built or judged. We treat the product values, risk boundaries and definition of acceptable evidence as invariants. This is where human judgment and taste matter most.

Results

The pieces began operating together as a factory in May. From May through August, the release timeline records 97 major releases across Win, Pay and their shared platform, out of 118 through the end of August. They included estimates, takeoffs, procurement, progress claims, variations, retention, forecasting, month-end close and accounting connections.

More than 3,000 pull requests had merged into the shared codebase's main branch by the end of August. The share of merges linked to a card from a builder, the team or the factory itself rose from under 10% in May to nearly two thirds in July. These counts describe delivery and traceability, not a measure of product quality.

The feedback pipeline came online in June. From then through 2 September, more than 1,300 cards arrived from customers using the product, our team, industry advisers and the factory's own monitoring. More than 60% were resolved: 45% shipped, with the rest closed as duplicates or declines.

The small team could take on more work while continuing to learn the domains it was building for. The factory helped bring missing context and unanswered questions into view, rather than requiring the engineers to remember every detail across every change.

Keep the learning loop open

As more work moves through Foreman, keeping the team's understanding current becomes harder. A passing check or a merged pull request does not tell us whether we understood the original problem correctly. We still need feedback from the people using what we build.

Our next improvements need to make that feedback easier to connect to earlier decisions: what we believed, what we changed and what we should do differently. Automating more of the workflow is useful when it gives the team room to work on those questions.

We want to finish a run with working software and a better understanding of the problem. The next run should start from there.