You're about to hire a development agency. You've done your homework. You've reviewed three portfolios, compared hourly rates, asked for timelines, and checked a few references. You pick the one with the best-looking case studies and a price that feels reasonable.

Six months later, you're $120,000 in and the app still isn't launched. The agency keeps saying it's "almost done." Features you thought were included turn out to be change orders. And when you ask for the codebase, you find out they've been building on a proprietary framework you can't take anywhere else.

This isn't a horror story. It's a pattern. And it doesn't happen because founders are sloppy. It happens because the standard playbook for vetting agencies is fundamentally broken.

Every "how to hire a development agency" article tells you to check the portfolio, compare pricing, and ask for references. That advice isn't wrong, exactly. It's just useless. Portfolios show an agency's best work, not their average. Pricing comparisons are meaningless without identical scope (which never exists). References are curated. Nobody hands you the number of the client whose project went sideways.

The questions that actually predict whether an engagement will succeed or fail have nothing to do with what the agency has built. They're about how the agency thinks, communicates, and behaves when things get hard.

What happens when the scope changes?

This is the single most predictive question you can ask, and almost nobody asks it. Every software project changes mid-stream. Requirements shift. You learn something from early users. A feature turns out to be more complex than anyone expected. This is normal. What matters is how the agency handles it.

The wrong version of this question is "can you stick to the original scope?" That's a fantasy. The right version is "walk me through what happens when we need to change direction three months in."

A strong agency will describe a clear process. Something like: "We document the change, estimate the impact on timeline and budget, and present you with options before we start building. You'll never get a surprise invoice." A weak agency will either promise that scope changes won't happen (they will) or get vague about how they handle them. Both are red flags.

I worked with a founder who hired an agency to build an internal operations tool. The initial scope was $45,000. Six weeks in, the team realized they needed the tool to integrate with a third-party logistics system that wasn't in the original plan. The agency treated every integration call as a change order. No conversation about tradeoffs, no discussion about what could be deprioritized to make room. Just invoices. By the time the project shipped, the total was $78,000 and the relationship was burned.

The founder told me afterward: "I never asked how they handle changes. I just assumed professionals would be reasonable about it." That assumption cost her $33,000.

How do you handle a project that's falling behind?

Every agency will tell you they deliver on time. That's not information. What you need to know is what they do when they don't.

Ask them to describe a project that fell behind schedule. What caused it? How did they communicate it to the client? What did they do to get it back on track? If they can't give you a specific, honest answer, one of two things is true: they've never managed a complex project (unlikely), or they're not willing to be honest about the ones that went sideways (more likely, and worse).

The best agencies I've worked with treat falling behind as a communication problem, not a heroics problem. They don't quietly crunch for two weeks hoping to catch up. They tell you early, explain why, and propose a revised plan. That transparency is worth more than any portfolio.

Compare this to the agencies that go quiet when things get hard. Two-week gaps between updates. Vague status reports. "We're working through some technical challenges." If you've ever been on the receiving end of that silence, you know the sinking feeling. By the time you find out how far behind things really are, it's too late to course-correct without blowing the budget.

Who actually owns the code?

This question sounds boring and legal. It is neither. I've watched founders discover, after spending six or seven figures, that the agency built their product on a proprietary framework or internal toolset. The code technically belongs to the founder, but it's useless without the agency's infrastructure. You're locked in. Need a bug fix in six months? You're calling them. Want to switch to a different team? That team has to learn a custom framework nobody else uses, or you're starting from scratch.

Before you sign anything, ask three things. Will we own the complete source code and all related assets? Are you building on open, standard technologies that any competent developer could pick up? Can we deploy and run this independently on day one?

A good agency will answer yes to all three without hesitating. Some will push back on the third one, arguing that their managed hosting is part of the value. Maybe it is. But you should be choosing that arrangement, not discovering it after the build is done.

The real vetting process

Here's what most founders miss: the way an agency responds to hard questions during the sales process is exactly how they'll behave during the project. If they dodge questions about scope changes, they'll dodge them when scope changes happen. If they can't describe a project that went wrong, they won't tell you when yours is going wrong. If they get defensive about code ownership, they'll make the handoff painful.

The portfolio tells you what they're capable of building. The pricing tells you what they charge. Neither tells you what it's actually like to work with them when the project hits the inevitable rough patch.

So skip the standard questions. Instead, ask about the last time something went wrong. Ask what happens when priorities shift. Ask who owns the code and what it takes to walk away. You'll learn more from those three conversations than from any portfolio review or reference call.