TLDR: Most organizations pick a security vendor by comparing brand recognition, case studies, and certifications, because those are the easiest things to put side by side. They are a weak predictor of whether the engagement actually finds what matters. The stronger signal shows up earlier, and gets skipped more often: what happens in the scoping conversation before a contract is signed. A vendor who investigates and pushes back on your stated scope tends to deliver findings that map to real risk. A vendor who quotes against your spec sheet as written tends to deliver a report that satisfies the paperwork and misses the thing that would have actually hurt you.
A natural gas utility once hired a firm to test its corporate IT network. Somewhere during the engagement, the testers strayed into a segment connected to the utility’s supervisory control systems, the equipment that actually runs the plant. Nobody had scoped that connection in or out. It simply hadn’t come up.
A different case: an oil company’s remote well sites had radio enclosures wired directly into SCADA systems, left unlocked in the field. They’d been excluded from a prior assessment for what the internal writeup called operational fragility concerns, a reasonable-sounding phrase for “we didn’t want to risk breaking something remote and hard to fix.” A municipal Wi-Fi network followed a similar pattern in a separate incident: public-facing wireless had been scoped as its own island, disconnected from anything sensitive, until testing showed it actually reached internal infrastructure, including building CCTV and facility access controls.
None of these organizations hired an incompetent vendor. In each case, the test that was run got run correctly, within the boundaries it was given. The boundaries were the problem, and the boundaries got set before anyone started testing.
Reputation Answers the Wrong Question
When organizations evaluate security vendors, the comparison usually runs through brand recognition, published case studies, industry certifications, and client logos. These are reasonable filters for one narrow question: can this vendor do competent, professional work. They tell you almost nothing about a second, more important question: will this specific engagement find what actually matters to your organization.
Compliance frameworks reinforce the gap. SOC 2 and ISO 27001 both require a penetration test, but neither one defines what the test’s boundaries should actually cover. That decision gets left almost entirely to the organization being assessed. An auditor reviewing the evidence checks whether a test happened, whether a report exists, and whether findings were tracked. Auditors are rarely positioned to judge whether the scope itself was adequate for the organization’s actual risk profile. A test can satisfy the audit requirement in full while covering a fraction of the environment that matters, and nothing in the compliance process catches that gap.
This is the same idea we’ve written about before in terms of remediation behavior: the traits that predict a good outcome aren’t the ones easiest to check off. The difference here is timing. Remediation behavior tells you something after a finding exists. Scoping quality tells you something before the engagement starts, which means it’s also the cheapest place to catch a problem.
What Actually Predicts a Good Outcome
The signal worth watching for is simple to describe and easy to miss in practice: does the vendor treat your stated scope as a starting point to interrogate, or as a finished spec to quote against.
A vendor doing real diagnostic work asks uncomfortable, specific questions. Which teams deploy their own infrastructure outside of what IT tracks centrally? Are there third-party integrations that touch sensitive data? What does the API surface look like across every environment, including staging systems that never made it onto anyone’s official inventory? What’s the one piece of data that would create the most damage if it were exposed, corrupted, or deleted? These conversations are slower than reviewing a spreadsheet of in-scope IP addresses, and they require the organization to admit what it doesn’t fully know about its own environment. That discomfort is exactly why the conversation matters. A vendor willing to have it is doing the work of building an accurate scope. A vendor willing to skip it is handing you back whatever assumptions you walked in with.
Budget pressure makes the shortcut more tempting on both sides. Cost is consistently cited as one of the top reasons organizations don’t test as often as they should, and when an initial quote lands higher than expected, the instinctive response on the buyer’s side is to cut scope rather than revisit the budget. Something gets excluded because it would take too long to get sign-off from a business unit. Something else gets excluded because it’s assumed, without much evidence, to carry lower risk. The final scope document ends up reflecting internal politics and time constraints as much as it reflects actual risk, and the testing team is then handed that document and expected to produce findings that represent the organization’s real security posture. They can only test what they were told to test.
The assets that get cut for convenience are disproportionately the ones that matter. Systems assumed to be low-risk because they’re old, remote, or operationally sensitive to touch are frequently the ones an attacker would go straight for, precisely because nobody’s paying close attention to them.
What a Good Scoping Conversation Sounds Like
A few concrete questions separate a vendor doing real diagnostic work from one filling out a template, and they’re worth asking directly before signing anything:
- Does the vendor ask what success looks like before naming a price, or does the price come first?
- Understanding what keeps your team up at night, beyond satisfying a compliance box, should shape the test design before cost enters the conversation.
- Does the vendor push you to define what’s explicitly out of scope, not just what’s in?
- A scope document that only lists inclusions leaves far too much room for later disputes about whether something should have been covered.
- Does the vendor ask about your threat model, or only your asset inventory?
- An asset list tells a tester where things are. A threat model tells them what actually needs protecting and from whom, which is what determines whether the eventual findings will be useful or just technically accurate.
- Does the vendor flag when a proposed exclusion looks risky, even if it makes the engagement smaller and less profitable for them?
- A vendor willing to argue for a bigger, more uncomfortable scope than the one you proposed is signaling something about how they’ll handle findings later too.
The Same Test, Applied Earlier
The underlying test here is the same one that predicts whether a security finding actually gets fixed once it’s found: does new information change the plan, or does the plan get defended against the information. We wrote about that dynamic from the remediation side in our July 1 post, looking at how organizations respond once a finding exists.
The scoping conversation is where that same instinct shows up first, and it’s the cheapest point in the entire process to catch it. Before any budget is spent, before any report is written, the way a vendor engages with your stated scope is already telling you how the rest of the relationship will go. An organization that skips this conversation, and a vendor that lets them skip it, are setting up an engagement that will look complete on paper regardless of what it actually finds.