Six months ago a vendor showed you a demo. The AI read a contract, pulled the key terms, flagged the risky clauses, and generated a summary your legal team agreed with. You ran a pilot on fifty documents and the results looked good enough to build a business case around. You signed a contract, kicked off a rollout, and told the board that AI was going to save the company $400,000 a year in review time.

That was five months ago. The rollout is now "two weeks out." It has been two weeks out since February.

If any part of that sounds familiar, you are not alone, and the model is not the problem. The gap between an AI pilot that works and an AI system that runs your business is enormous, and almost nobody prices it in before they sign.

Every AI demo you see, whether from a vendor or from your own team, is optimized for the happy path. The document is clean. The user's question is well-formed. The example data was cherry-picked during setup. Nothing has timed out, nothing has rate-limited, nothing has returned an unexpected format.

That demo is showing you the model. What it is not showing you is the system that model has to live inside once real work starts flowing through it. And that system, not the model, is where AI rollouts fail.

We call this the demo tax: the eighty percent of the work that sits between a working prototype and a production system, and that almost nobody quotes for in the pilot phase.

When a real AI system goes into production inside a real business, it has to do a lot more than call a model. Here is an incomplete list of what has to exist before you can trust the output.

Data pipelines that feed the model the right thing. 

The pilot probably ran on a folder of fifty PDFs someone uploaded. Production will need to pull documents out of your document management system, your email, your CRM, or three of those at once. Someone has to build that, keep it running, and deal with the files that do not parse.

In the demo, the model always returned something sensible. In production, it will occasionally return nothing, return the wrong thing, or time out. Your system needs to know what to do in each case, and the answer is almost never "show an error and stop."

The pilot was evaluated by someone comparing the output to their gut. Production needs a repeatable way to measure whether the system is actually doing its job across hundreds or thousands of cases, and a way to catch regressions when you swap models or change a prompt.

For anything with real consequences, a human has to be in the decision path. That means a review queue, a UI, a way to override the model, and a way to feed those overrides back in so the system gets better over time.

You need to know, right now, whether the thing is working. Not "the server is up" working, but "the outputs are still accurate" working. That requires dashboards, alerts, and a team that knows what to do when an alert fires.

The pilot ran under one person's account. Production needs to enforce who can see what, log every action, and survive an audit. If your data is sensitive, add another layer for compliance.

Integration with the systems the work actually happens in. 

Nobody wants to log into a new AI tool to do their job. Production means the AI shows up inside the CRM, the ticketing system, or the inbox people already use. That integration is usually the most expensive part of the rollout, and it is almost never in the pilot scope.

That list is not exotic. It is the baseline for any production AI feature, and none of it was in the demo you saw.

Most AI vendors are paid on the pilot, not the outcome. Their incentive is to get you excited, get you to sign, and hand you off. The hard work of making the system survive your real data, your real users, and your real compliance requirements lands on your side of the line, and you usually do not discover it until the rollout is months in.

This is not malice. It is a structural mismatch between what a vendor is built to sell and what your business actually needs to run. The vendor sells a model. You need a system.

What a Technical Partner Does Differently

The reason VantaSoft operates as a technical consultancy rather than a dev shop is exactly this kind of gap. A dev shop builds what you ask for. A technical partner looks at the demo, asks what the system around it has to look like in production, and scopes the actual work before anyone commits to a timeline.

When we take on an AI rollout, the pilot is treated as a scoping exercise, not a deliverable. The deliverable is a production system, and we own the decisions that make that system work: how it handles edge cases, how it gets evaluated, how it integrates into the business, and how your team can operate it without us in the room.

If you already have a pilot that worked and a rollout that is stalling, the fix is almost never a better prompt or a bigger model. It is a system design conversation that should have happened before the pilot, and that you can still have now.

Questions to Ask Before You Greenlight the Rollout

If you are about to commit budget to an AI rollout, these questions will surface most of the hidden work before it becomes a problem.

1. What exactly happens when the model returns the wrong answer, and who notices?

2. Where does the data come from in production, and who owns that pipeline when it breaks?

3. How will we measure whether the system is still working six months from now?

4. Which systems does this need to integrate with, and who is building those integrations?

5. What does a human review queue look like, and whose team runs it?

6. What is our plan when the model, the API, or the vendor changes underneath us?

If the answers are vague, the rollout is not ready, and no amount of pilot success will change that.

The good news is that none of this is unknowable. It is just work, and it is work that needs to happen before the contract gets signed, not after. If you would rather have that conversation with a partner whose job is to own the system rather than sell you a model, that is the conversation we have every week with founders and operators who are trying to put AI into production.

Your AI Pilot Worked. Your Rollout Is Stalling. Here's the 80% Nobody Priced In.

As of 12:00 PM PT today, using your Claude Pro or Max subscription to power OpenClaw, NanoClaw, or any similar third-party agent runtime is a violation of Anthropic's Consumer Terms of Service. The ban has been in the works for months. It's now fully enforced.

If you built on OpenClaw, you've had two options for a while, and neither has been good.

Option one: pay the Anthropic API bill directly. OpenClaw-class agent runtimes are token-hungry by design. They run continuously, carry large context windows, re-ground on memory stores, and chain tool calls across long sessions. The pay-as-you-go bill for a handful of always-on agents on the API can run into the thousands of dollars a month per agent. For a small team or a bootstrapped business, that's not a rounding error.

Option two: point OpenClaw at the OAuth token from your Claude Max subscription and hope nobody notices. That's what a lot of people did. It worked for a while. Then Anthropic started rolling out server-side enforcement, accounts started getting suspended, and anyone who had built their operations on top of this setup had to scramble.

As of today, that workaround is dead. Anthropic's updated terms explicitly state that "using OAuth tokens obtained through Claude Free, Pro, or Max accounts in any other product, tool, or service, including the Agent SDK, is not permitted."

The obvious question: if the subscription path is closed and the API path is too expensive, where do you run your agents?

Here's the part most people missed. Over the last six months, Anthropic has been quietly adding features to Claude Code, its official terminal coding agent, that close the gap between "a coding assistant" and "a full agent runtime." Three features in particular:

 Claude Code now supports custom memory directories, timestamped memory files, and an indexed MEMORY.md that loads at the start of every conversation. Memory persists across sessions. It's not a vector database, it's markdown files, but for the vast majority of memory use cases (who is the user, what are they working on, what conventions apply, what happened last time) this is the right abstraction. It's auditable, editable by hand, and doesn't require running a separate service.

 The /loop command turns Claude Code from a request-response chat into a continuously running background worker. It's a lightweight cron scheduler built into the agent itself. You can hand Claude Code a prompt and a cadence, and it will execute that prompt on schedule, with its full memory and tool context intact.

 A permission-relay system that lets channel servers forward tool approval prompts to your phone or another surface. This is the piece that makes Claude Code trustworthy as a long-running service rather than something that needs a human at the keyboard. You can approve risky actions remotely, route alerts, and wire agent actions into your existing notification systems.

Each of these features is individually modest. Together, they do something specific: they replicate the load-bearing capabilities that made dedicated agent runtimes necessary, and they do it inside a client that runs under your existing Claude subscription, within Anthropic's terms of service.

This is huge news for anyone who was running OpenClaw-class infrastructure.

What an Always-On Claude Code Agent Actually Looks Like

At VantaSoft, we run a constellation of always-on Claude Code agents. Each one has a role (chief of staff, finance, marketing, coding, personal assistant), and each runs in its own tmux session with its own working directory, its own memory folder, its own CLAUDE.md describing its scope and guardrails, and its own set of installed plugins.

The architecture is almost embarrassingly simple:

That's the whole platform. No orchestrator, no queue, no separate memory service, no framework. The "runtime" is tmux and cron. Messages arrive from Telegram, the agent reads them, reasons about them, takes actions, replies. Scheduled work runs on the heartbeat. Memory accumulates as markdown. When an agent learns something about the principal, it writes it down. The next conversation starts with that file in context.

This exists under a single Claude Max subscription, within Anthropic's terms, without a pay-per-token bill that scales with usage.

We've open-sourced the scaffolding for this setup as 

. It's a template for running a team of persistent Claude Code agents, each with its own personality, memory, and Telegram interface, coordinated by a chief-of-staff orchestrator. Clone it, point it at your own Max account, and you have the skeleton of an agent stack running on your own hardware by the end of the afternoon.

Why This Matters for Anyone Building on Agents

The OpenClaw ban isn't just a legal technicality. It's Anthropic saying, loudly, that they do not want their subscription model powering third-party agent runtimes. Fine. But they're simultaneously building first-party agent runtime features directly into Claude Code. If you were relying on the subscription-OAuth workaround, the transition path isn't "pay the API bill." It's "port your agents to Claude Code."

For a technical consultancy advising non-technical founders, this shift changes the advice we give. A year ago, "build your own AI agent" meant committing to a platform, hiring people who know it, and standing up infrastructure. Today, the marginal cost of standing up a new agent, with its own memory, its own access rights, its own integrations, is measurable in hours. The ceiling is set by how clearly you can describe the job and what systems the agent can touch, not by the plumbing underneath.

The implications compound. If every function in your business can have a dedicated agent with persistent memory, running under a subscription you already pay for, the calculus of what's worth automating changes. Tasks that were previously "not worth building for" because the integration overhead dwarfed the value now clear the bar easily. A finance agent that reconciles one spreadsheet a week used to need a platform commitment. Now it needs a directory and a cron entry.

Claude Code isn't a full replacement for dedicated agent platforms in every scenario. If you need fine-grained multi-tenant isolation, complex approval workflows, enterprise audit logs at scale, or a polished end-user UI, a purpose-built system still has a role. The gap has narrowed, it hasn't disappeared.

One specific gap worth calling out: channel coverage. As of this writing, Claude Code's first-party channel support is limited to Telegram and Discord. OpenClaw and similar runtimes ship adapters for Slack, WhatsApp, Microsoft Teams, SMS, and email out of the box. If your operators live in Slack and you need the agent reachable there today, you'll either wait for first-party support, build your own channel bridge via MCP, or stand up a thin relay. The direction of travel is obvious (more channels are coming) but the current state is Telegram and Discord only.

But for the much larger set of use cases, internal agents serving specific operators, handling specific workflows, remembering context over time, the platform tax is no longer justified. Claude Code, memory markdown, tmux, and cron will get you there. The result is simpler to reason about, easier to modify, and runs under a pricing model that doesn't punish you for using it.

The OpenClaw ban closes a door. Anthropic's Claude Code roadmap, quietly, opened a better one.

Claude Code Is Replacing OpenClaw as the Default Runtime for Always-On Agents

You're in a sales pitch for a new CRM and the rep says, "Don't worry, it integrates with everything." You hear this and think: great, our tools will talk to each other, data will flow, and life gets easier. Then your dev team spends three months and $45,000 getting the CRM to sync with your billing system. And it still drops records every other Tuesday.

The word "integrates" has become one of the most misleading terms in software. It implies simplicity. Plug A into B, data flows, everyone moves on. But if you've ever been on the business side of an API integration project that went sideways, you know the reality is very different.

This isn't a computer science lecture. It's what you actually need to know about APIs and integrations so you can make better decisions, ask better questions, and stop getting surprised by timelines and budgets.

An API is a set of rules that lets one piece of software request something from another piece of software. That's it.

When you log into a website using your Google account, that website is using Google's API. When your e-commerce store automatically updates inventory in your accounting software, that's an API too. When you check the weather on your phone, the app is calling a weather service's API to get the forecast.

Think of it like ordering at a restaurant. You don't walk into the kitchen and cook your own food. You use the menu (the API), place an order (make a request), and the kitchen (the other software) sends back what you asked for. You never see the kitchen. You don't need to know how the food is made. You just need the menu to be accurate and the kitchen to be open.

That part is genuinely simple. The problems start when someone says, "We just need to connect System A to System B."

Why "Just Connect It" Is Where Projects Go Sideways

Here's what actually happens when your team sets out to integrate two systems via API.

First, both systems need to agree on how to communicate. One system might send dates as "March 31, 2026" and the other expects "2026-03-31." One might call a customer a "contact" with a field for "company name." The other calls them an "account" with separate fields for "organization" and "DBA." These aren't edge cases. This is every integration, every time.

Second, authentication. Your system doesn't just get to call another system's API freely. It needs permission. That means API keys, OAuth tokens, refresh tokens, and security protocols that expire, rotate, and occasionally break at 2 AM on a Saturday. One company I worked with had a Stripe integration that silently stopped processing refunds for two weeks because an auth token expired and nobody noticed. The total in unprocessed refunds: $23,000.

Third, rate limits. Most APIs restrict how many requests you can make per minute or per hour. If your system tries to sync 10,000 customer records at once and the other API allows 100 requests per minute, your integration needs to handle queuing, retries, and partial failures gracefully. This is not trivial engineering.

Fourth, the other API changes. Software companies update their APIs regularly. Sometimes they deprecate fields. Sometimes they change how data is structured. Sometimes they just break things. When Shopify or Salesforce pushes an API update, your integration needs to absorb that change or it stops working. This is ongoing maintenance, not a one-time cost.

And fifth, error handling. What happens when the other system is down? When a record is malformed? When the connection times out halfway through a sync? Every one of these scenarios needs to be anticipated, coded for, and monitored. The integrations that cause the most business damage are the ones that fail silently. Orders stop flowing, inventory drifts out of sync, and you don't find out until a customer calls to ask where their shipment is.

What This Means for Your Budget and Timeline

When a vendor says "we integrate with X" or your dev team says "we can connect those," they're describing the happy path. The happy path is maybe 20% of the work. The other 80% is handling everything that can go wrong, mapping data between systems that weren't designed to talk to each other, and building monitoring so you know when something breaks.

I've seen this pattern repeatedly. A company decides to connect their project management tool to their invoicing system. The estimate comes in at two weeks and $8,000. Twelve weeks later, they've spent $35,000 because the invoicing API had undocumented quirks, the data models didn't align, and every edge case spawned two more.

Industry data backs this up. Organizations spend an average of 30% of their development time on integration challenges, and costs frequently exceed initial estimates by 150 to 200 percent. That two-week estimate your team gave you? Budget for six weeks and you'll be closer to reality.

This doesn't mean integrations aren't worth doing. They absolutely are. A well-built integration between your CRM and your billing system can save hundreds of hours of manual data entry per year. Connecting your support ticketing system to your product database means your support team sees relevant customer data without switching tabs. The value is real. But so are the costs, and the costs are almost always higher than the initial pitch suggests.

How to Make Better Decisions About Integrations

You don't need to become technical to navigate this well. You need to ask the right questions.

When someone says "we'll integrate with X," ask: what exactly will sync, in which direction, and how often? The answer "everything syncs in real time" should make you nervous. Real-time, bidirectional sync is the most complex and expensive type of integration. Sometimes a nightly batch sync of the five fields you actually need is 90% cheaper and perfectly adequate.

Ask about ongoing costs. The initial build is often the smaller expense. Maintenance, monitoring, and adapting to API changes from the other system are where the long-term money goes. If your team hasn't budgeted for this, the integration will decay.

Ask what happens when it breaks. If the answer is "we'll fix it when we notice," that's not a plan. You need monitoring and alerts. The difference between a broken integration caught in five minutes and one caught in five days can be tens of thousands of dollars in bad data, missed orders, or duplicate records.

And ask whether you actually need a custom integration at all. Tools like Zapier, Make, and Tray.io handle simple integrations between popular platforms without custom code. They have limitations, but for straightforward use cases like "new form submission creates a CRM record," they're a fraction of the cost and complexity.

"Just connect it" is never just anything. Every integration is a small ongoing relationship between two systems that weren't designed with each other in mind. It requires translation, maintenance, monitoring, and a plan for when things go wrong. The best thing you can do as a business leader is stop treating integrations as a switch you flip and start treating them as what they are: ongoing operational commitments with real costs. When you do, you'll make better decisions about which ones are worth the investment, push back on unrealistic timelines, and stop getting blindsided when a "simple" connection turns into a three-month project.

APIs Explained: Why 'Just Connect It' Is Never That Simple

The Three Most Expensive Mistakes Non-Technical Founders Make

We've built software with dozens of founders. A few patterns show up so often — and cost so much — that we felt obligated to write them down.

Most of our clients are non-technical founders building real companies. They're sharp, driven, and deeply familiar with their market. What they often lack is a frame of reference for the software side — how it works, what it should cost, when to build, and who to trust.

That gap is completely normal. And it's exactly why certain mistakes repeat themselves. Not because founders are careless — but because without context, the wrong path often looks identical to the right one.

These are the three we see most often. We're sharing them because we'd rather a founder walk in the door having already avoided them.

Mistake 01 — Building before validating. Spending to learn what a landing page could've taught you.

We've had more than a few conversations that start like this: a founder comes to us with a product that's already been built — often by another agency, sometimes by an early hire. They want us to take it over, improve it, or fix what isn't working. When we dig in, the code is usually fine. The problem is upstream: the product was built before anyone confirmed the market wanted it.

The technical execution was treated as the hard part. Turns out, the hard part was the question the founder skipped: 

Before we write a line of production code with any client, we push on what evidence exists. Not a pitch deck. Not a survey. Actual evidence — waitlist signups, pre-sales, conversations where someone said "I would pay for this today." A fake landing page will show you more about demand in two weeks than six months of building will.

 The first build should never be the experiment. Earn the right to build by finding someone willing to pay before the product exists. If you can't, that's data — and it's free.

Mistake 02 — Hiring the right engineer for the wrong stage of the company.

Typical cost: $200K – $400K + 12–18 months

This one is subtle and it cuts in both directions. We've seen founders lose a year because they brought on an impressive senior engineer who was a perfect fit for a Series B company — and a poor fit for a 10-customer startup that needed to move and break things. We've also seen founders go the opposite direction: hire cheap, move fast, and spend year two rebuilding everything from scratch.

"The question isn't whether they can code. It's whether they can build what this company needs at this specific moment."

Pre-PMF, the profile you need is almost always a strong full-stack generalist: someone comfortable making judgment calls without a full spec, who defaults to shipping over architecting, and who's done it before in an early-stage context. That person is not typically a $250K principal engineer. They're also not the cheapest option on Upwork.

Working with an agency like VantaSoft has one real advantage here: you get a team that's already matched to your stage, without the risk of a bad full-time hire. When you're ready to bring development in-house, we'll tell you — and help you find the right person to do it.

 Hire for now, not for the company you want to be in three years. A $150K builder who ships beats a $280K architect who plans — until you've earned the right to plan.

Mistake 03 — Not being able to evaluate the work you're paying for.

This is the one that enables all the others. When a non-technical founder can't assess technical work, every vendor relationship, every hire, and every decision is operating on trust alone. That's a lot of exposure.

It means a two-week estimate is hard to challenge. An architectural recommendation is hard to question. A post-launch bug report is hard to contextualize. And if someone is padding scope, cutting corners, or simply making poor decisions — you're the last to know.

The fix is not to learn to code. It's to build enough literacy to ask hard questions — and to know when an answer doesn't hold up. Understand the difference between frontend and backend. Know why integrations are expensive and where risk lives in a scope document. Learn to spot vagueness that puts the cost of ambiguity on you.

Beyond that: find one trusted technical advisor who has no stake in your decisions. Not a vendor. Not a future hire. Someone who can sit across the table from an agency proposal and tell you whether what you're being sold is fair.

 One of the first things we do with new clients is help them understand what they're getting. If your current agency won't explain the tradeoffs in plain language, that tells you something.

None of these mistakes are fatal. Every one of them is recoverable. But the founders who sidestep them tend to share one quality: 

, before the cost of being wrong got priced into a rebuild, a bad hire, or a product the market never asked for.

We built VantaSoft to be the team that gives founders that early sanity check — whether that means building alongside you, or just helping you understand what you're looking at before you commit.

A founder I worked with hired an agency to build a customer portal. The scope was clear, the contract was $85,000, and the timeline was three months. At the one-month check-in, the agency showed polished mockups and said everything was on schedule. At two months, they demoed a login screen and a dashboard with placeholder data. "We're about 70% done," they said.

At month five, she'd spent $130,000 and the portal still couldn't process a single real transaction. When she finally brought in an outside developer to review the code, he told her the backend was maybe 30% complete. The pretty screens were a facade.

She told me afterward: "I didn't know what to look for. Every update sounded like progress."

That's the problem. Software projects don't blow up overnight. They erode. The warning signs are there early, but they're easy to miss if you don't know what "off track" actually looks like from the outside.

Early in a healthy project, your dev team gives specific updates. "We finished the user authentication flow. Next week we're building the payment integration with Stripe." That's concrete. You can track it.

When a project starts sliding, the language shifts. "We're working through some technical challenges." "We're making good progress on the backend." These sound like updates, but they contain zero information.

The shift from specific to vague doesn't happen because the team forgot how to communicate. It happens because they don't have good news to report and they're hoping to catch up before you notice.

Here's a simple test. After every status update, ask yourself: could I explain what was accomplished this week in one sentence to my co-founder or board? If the answer is no, that's not a style issue. That's a red flag.

This is the single most reliable diagnostic I know, and it requires zero technical knowledge. Can your dev team show you the actual product doing real things? Not mockups. Not wireframes. The real software, processing real data, doing what it's supposed to do.

A healthy project produces working software at regular intervals. At two weeks, maybe you can see a user log in and land on a real dashboard. At four weeks, maybe basic transactions work end to end. The features don't need to be polished, but they need to be functional. If your team is three months in and the only thing they can show you is screens with hardcoded data and buttons that don't connect to anything, you have a problem.

The founder I mentioned earlier fell into this exact trap. The agency showed beautiful UI every two weeks, but the login screen didn't connect to a real database and the dashboard graphs were static images. She assumed visual progress meant engineering progress. It didn't.

Ask for a live demo every two weeks. Not a slide deck. Not a screen recording they polished before sending. A live, interactive walkthrough where you click around and see what happens. If the team resists ("it's not ready to show yet," "we're focused on the infrastructure right now"), pay attention. Infrastructure work is real, but it doesn't take three months before anything is demoable.

The timeline keeps moving but the explanation doesn't change

Delays happen in every software project. That's not the red flag. The red flag is when the timeline moves and the reason stays the same.

"We hit some unexpected complexity in the integration." That's reasonable once. If you hear the same explanation three times in a row, the issue isn't unexpected complexity. The issue is that the team underestimated the work and hasn't revised their approach.

I worked with a startup building an inventory management system. The dev team said the API integration with their supplier would take two weeks. Four weeks later, it was "almost done." Eight weeks later, same answer. Same explanation every time: the supplier's documentation was incomplete.

The actual problem, which came out only after the founder demanded a detailed breakdown, was that the developer assigned to the integration was junior and had never built a third-party API integration before. The two-week estimate was based on how long it would take a senior developer. Nobody told the founder that's what was happening.

A healthy team adjusts. "This is taking longer than we estimated. Here's why. Here's our revised timeline." That's accountability. Compare it to: "We're still working on it. Should be done soon." That's not a plan. That's hope.

This one is counterintuitive. A team that agrees to every feature request and never pushes back seems responsive and accommodating.

In practice, a team that never says no is a team that's not managing scope and not being honest about tradeoffs. Every "yes" to a new feature is a "no" to shipping on time, and a team that won't tell you that is building a project that will go over budget.

The best technical teams I've worked with push back regularly. "We can do that, but it'll add two weeks. Do you want that more than launching on time?" "That feature is a nice-to-have. Can we ship without it and add it in v2?"

If your team never has that conversation with you, either they're padding the timeline so aggressively that every addition fits (you're overpaying), or they're silently absorbing scope and the timeline will eventually collapse under the weight of everything they agreed to.

Start by asking direct questions. "Can you show me this working in a live demo?" "What specifically got done this week?" "Why has this estimate changed three times?" "What would you cut if we needed to ship in four weeks instead of eight?"

If the answers are clear, specific, and honest (even if the news is bad), you're probably working with a good team that hit a rough patch. Bad news delivered early is manageable.

If the answers are defensive, vague, or keep changing, consider getting an outside opinion. A half-day code review from an independent senior developer costs $500 to $1,500 and can tell you whether the project is salvageable or whether you're pouring money into a hole. That's the best insurance policy in software development, and almost nobody buys it until it's too late.

The hardest part isn't spotting the problems. It's acting on them before the sunk cost feels too large to walk away from. A project that's $40,000 over budget and three months behind doesn't get better by spending another $40,000. Sometimes the right move is to stop, assess, and change course. That's not failure. That's the same judgment you apply to every other part of your business.

How to Tell If Your Software Project Is Off Track

You're about to hire a development agency. You've done your homework. You've reviewed three portfolios, compared hourly rates, asked for timelines, and checked a few references. You pick the one with the best-looking case studies and a price that feels reasonable.

Six months later, you're $120,000 in and the app still isn't launched. The agency keeps saying it's "almost done." Features you thought were included turn out to be change orders. And when you ask for the codebase, you find out they've been building on a proprietary framework you can't take anywhere else.

This isn't a horror story. It's a pattern. And it doesn't happen because founders are sloppy. It happens because the standard playbook for vetting agencies is fundamentally broken.

Every "how to hire a development agency" article tells you to check the portfolio, compare pricing, and ask for references. That advice isn't wrong, exactly. It's just useless. Portfolios show an agency's best work, not their average. Pricing comparisons are meaningless without identical scope (which never exists). References are curated. Nobody hands you the number of the client whose project went sideways.

The questions that actually predict whether an engagement will succeed or fail have nothing to do with what the agency has built. They're about how the agency thinks, communicates, and behaves when things get hard.

This is the single most predictive question you can ask, and almost nobody asks it. Every software project changes mid-stream. Requirements shift. You learn something from early users. A feature turns out to be more complex than anyone expected. This is normal. What matters is how the agency handles it.

The wrong version of this question is "can you stick to the original scope?" That's a fantasy. The right version is "walk me through what happens when we need to change direction three months in."

A strong agency will describe a clear process. Something like: "We document the change, estimate the impact on timeline and budget, and present you with options before we start building. You'll never get a surprise invoice." A weak agency will either promise that scope changes won't happen (they will) or get vague about how they handle them. Both are red flags.

I worked with a founder who hired an agency to build an internal operations tool. The initial scope was $45,000. Six weeks in, the team realized they needed the tool to integrate with a third-party logistics system that wasn't in the original plan. The agency treated every integration call as a change order. No conversation about tradeoffs, no discussion about what could be deprioritized to make room. Just invoices. By the time the project shipped, the total was $78,000 and the relationship was burned.

The founder told me afterward: "I never asked how they handle changes. I just assumed professionals would be reasonable about it." That assumption cost her $33,000.

How do you handle a project that's falling behind?

Every agency will tell you they deliver on time. That's not information. What you need to know is what they do when they don't.

Ask them to describe a project that fell behind schedule. What caused it? How did they communicate it to the client? What did they do to get it back on track? If they can't give you a specific, honest answer, one of two things is true: they've never managed a complex project (unlikely), or they're not willing to be honest about the ones that went sideways (more likely, and worse).

The best agencies I've worked with treat falling behind as a communication problem, not a heroics problem. They don't quietly crunch for two weeks hoping to catch up. They tell you early, explain why, and propose a revised plan. That transparency is worth more than any portfolio.

Compare this to the agencies that go quiet when things get hard. Two-week gaps between updates. Vague status reports. "We're working through some technical challenges." If you've ever been on the receiving end of that silence, you know the sinking feeling. By the time you find out how far behind things really are, it's too late to course-correct without blowing the budget.

This question sounds boring and legal. It is neither. I've watched founders discover, after spending six or seven figures, that the agency built their product on a proprietary framework or internal toolset. The code technically belongs to the founder, but it's useless without the agency's infrastructure. You're locked in. Need a bug fix in six months? You're calling them. Want to switch to a different team? That team has to learn a custom framework nobody else uses, or you're starting from scratch.

Before you sign anything, ask three things. Will we own the complete source code and all related assets? Are you building on open, standard technologies that any competent developer could pick up? Can we deploy and run this independently on day one?

A good agency will answer yes to all three without hesitating. Some will push back on the third one, arguing that their managed hosting is part of the value. Maybe it is. But you should be choosing that arrangement, not discovering it after the build is done.

Here's what most founders miss: the way an agency responds to hard questions during the sales process is exactly how they'll behave during the project. If they dodge questions about scope changes, they'll dodge them when scope changes happen. If they can't describe a project that went wrong, they won't tell you when yours is going wrong. If they get defensive about code ownership, they'll make the handoff painful.

The portfolio tells you what they're capable of building. The pricing tells you what they charge. Neither tells you what it's actually like to work with them when the project hits the inevitable rough patch.

So skip the standard questions. Instead, ask about the last time something went wrong. Ask what happens when priorities shift. Ask who owns the code and what it takes to walk away. You'll learn more from those three conversations than from any portfolio review or reference call.

Before You Hire a Development Agency, Stop Asking About Their Portfolio

Your dev team comes to you and says they need to rebuild the backend in a new framework. It'll take eight weeks. It's important. You nod, ask a couple of questions, get answers full of acronyms, and approve it because what else are you going to do? You hired them to make these calls.

Three months and $80,000 later, you're still waiting on that rebuild. The old system is held together with duct tape. And you're wondering if the whole thing was necessary in the first place.

This happens constantly. Not because founders are careless, but because most advice for non-technical leaders boils down to "trust your team" or "learn to code." Neither is useful. You don't need to amass raw technical knowledge to evaluate technical decisions well. What you need are frameworks that expose the quality of the thinking, regardless of the technology involved.

Here are three that I use with every client.

The single biggest red flag in any technical recommendation is when your team presents one option. "We need to switch to React." "We should move to AWS." "We have to rewrite this module."

Good technical thinking always involves tradeoffs. If someone is recommending an approach without explaining what they considered and rejected, one of two things is happening: they haven't done the analysis, or they're steering you toward a preference without showing their work.

When a developer or CTO brings you a recommendation, ask three questions. What other approaches did you consider? What are the downsides of the approach you're recommending? Under what circumstances would you recommend something different?

You don't need to understand the technical details of each option. You're evaluating the rigor of the process. A strong engineer will have a clear, concise answer for all three. They'll say something like, "We looked at keeping the current system and patching it, but the auth module can't support multi-tenant access without a rewrite either way. We also considered migrating just the API layer, but that creates two systems to maintain for six months. The full rebuild costs more upfront but gets us to a single system faster."

That's a decision you can evaluate. Compare it to: "We just really need to move to microservices." That's not a recommendation, that's a preference presented as a conclusion.

Every technical decision carries risk, and your team should be able to articulate it without you dragging it out of them. If they can't clearly explain what could go wrong, they probably haven't thought about it carefully enough.

This matters more than most founders realize. I worked with a company that approved a migration from Heroku to a self-managed Kubernetes setup. The team estimated four weeks in engineering time. What they didn't surface was the ongoing operational complexity. Six months later, the company was still spending on a DevOps contractor just to keep the infrastructure stable.

The founder wasn't technical. But if she'd asked "what's the ongoing cost of maintaining this after we ship it?" she would have gotten a very different picture. Or she would have gotten a blank stare, which is also useful information.

Here's the framework. For any significant technical decision, ask your team to answer three things: what's the worst realistic outcome if this goes wrong, what does this cost us beyond the initial build (in time, money, or complexity), and what are we giving up by choosing this path?

If your team treats these questions as annoying or unnecessary, that's a problem. Engineers who've been burned by bad decisions in the past welcome these questions. They know that the projects that blow up are usually the ones where nobody asked what could go wrong.

This is the one that catches the most issues. Ask your team to explain the decision in terms of what it means for the business, not the technology.

"We need to refactor the data layer" means nothing to you. But "right now, generating a single client report takes 45 seconds because of how our data is structured. After this refactor, it'll take under two seconds. That matters because three enterprise prospects told us in demos that the report speed was a dealbreaker" is a decision you can evaluate, prioritize, and fund with confidence.

If your team can't connect a technical decision to a business outcome, one of two things is true: the work isn't actually important, or the team doesn't understand why it's important. Both are problems.

I'm not saying every piece of engineering work needs a revenue justification. Some work is genuinely about system health, security, or reducing the chance of a future outage. But even those should be explainable. "If we don't fix this, we have a 30% chance of a full outage during peak traffic in Q4" is a business case. "We should fix this because it's messy" is not.

The strongest technical leaders I've worked with do this translation naturally. They think in terms of business risk and customer impact, not just code quality. When you find someone who can do that, hold onto them.

You don't need to run these frameworks as formal interrogations. Just make them part of how you engage with technical decisions. When someone brings a recommendation, ask what else they considered. When they describe a plan, ask what could go wrong. When they explain the work, ask them to connect it to something the business cares about.

Over time, you'll develop a sense for which decisions have been thought through and which ones haven't. You'll notice when estimates are suspiciously precise (real estimates have ranges) and when timelines keep growing without clear explanations. You'll get better at distinguishing "we need to do this" from "we want to do this."

None of this requires you to learn a programming language or sit in on code reviews. It requires you to do what you're already good at: asking sharp questions, reading people, and making judgment calls with incomplete information. That's what running a business is. Technical decisions are just another category of business decisions, and you already have most of the skills you need to evaluate them well.

The VantaSoft Tech Blog

Your AI Pilot Worked. Your Rollout Is Stalling. Here's the 80% Nobody Priced In.

Claude Code Is Replacing OpenClaw as the Default Runtime for Always-On Agents

The
Non-Technical
Founder's Guide

Evaluating a Dev Partner?

The VantaSoft Tech Blog

Your AI Pilot Worked. Your Rollout Is Stalling. Here's the 80% Nobody Priced In.

Claude Code Is Replacing OpenClaw as the Default Runtime for Always-On Agents

TheNon-TechnicalFounder's Guide

Evaluating a Dev Partner?

The
Non-Technical
Founder's Guide