The MSP’s Guide To Selling Unicorns

Background

For a number of years, I ran a MSP or “Managed Services Provider.” If you are unfamiliar with MSPs, this is the type of company that you can outsource pretty much all of your business IT needs to – it’s like having your own IT department without having to pay for the full time staff necessary to run that department if you are running a smaller or medium sized company. A number of years ago, I decided to switch my business model and offer security services exclusively. Today, MSPs are one of my largest markets, because they often find themselves needing a hand here and there when it comes to security, and I’m willing to jump in and help them with whatever they need without trying to poach their customers. So, even though I’m not offering traditional MSP services anymore, I’m still involved in a number of MSP communities which are very active.

Magic Products

Many vendors that target the MSP community have a tendency to over-sell their product in some way or another – heck… let’s face it… it seems that most vendors and sales people do that. But at what point does it stop being “fluff” and become false advertising? Don’t get me wrong… there are a lot of vendors in the MSP community that have fantastic solutions and make everyone’s life easier and helps keep costs down by allowing smaller MSPs to do more with less staffing. The problem comes when these vendors try to break into industries outside of most smaller MSP’s experience. Most MSPs do a great job at covering security basics; but when it comes to highly advanced security tasks such as penetration testing, threat hunting, etc; they often have to work with an outside vendor – which is completly understandable – no one person in IT is an expert in everything. But vendors know this and build products and services which often look fancy, but are lacking behind the hood – especially when it comes to security tooling and services.

Yesterday, I came across a new product announcement from one such vendor, and just couldn’t keep quiet. They were promising a pen-test for $99 (see Figure 1). Now, to what will likely be some people’s dismay, I’m not going to name and shame this vendor… that isn’t the point of this article. I’m not trying to tear a single vendor down, I’m trying to shine a light on a problem within the industry.

[Redacted Company's Owner] - It's true - we've just released some new tools exclusive for [redacted] Partners that includes Pen Tests for just $99 !
It's our way to help our Partners make 2022 a rainmaker!

Shared post:
[Redacted Company] - WHAT!  A Pen Test for just $99.00?!?!?
YES - it's true!
Today we've released our Pen Testing, Vulnerability Assessment and Enterprise-Grade Risk Assessment suite that includes a ground-breaking new prospecting tool called [redacted]. 
[redacted] is a breach-probability engine that predicts the likelihood of your client or prospect being breached! We believe it's a game-changer. Dark Web is so yesterday. [redacted] is the future! 
These new tools are available through our [redacted] platform. 
Visit to [redacted URL] to learn more and become a Partner today.
Figure 1 – $99 pen-test

If you aren’t familiar with a pen-test, this is where a group of highly skilled hackers will attempt to break into your infrastructure in order to help you find your vulnerabilities. This is a process that takes know-how, tenacity, lots of caffeine, and weeks of prep-work, testing, and report writing. It’s not just technical skills either. Depending on the type and scope of the assessment, it may even include people showing up to buildings they aren’t supposed to have access to and figuring out how to get in either through social engineering or breaking and entering. I highly suggest listening to some of Darknet Diaries penetration testing episodes if you aren’t familiar with pen-testing. One of my favorites is Episode 59: The Courthouse; although this focuses mostly on the physical side when pen-testing is really a huge field within security with most testers having specializations in specific techniques, technologies, etc. This is why pen-testing gets to be so expensive – it has absolutely nothing to do with the tools used, and everything to do with the people behind the tools and the time they invest into each test.

Because the field is so broad, it takes more time than $99 could pay for just to scope out the test and figure out what the client wants. Beyond that, most pen-test agreements need to be reviewed by an attorney because you are literally doing something that is a federal crime that involves jail time (at least in the US) if you don’t have proper permission to perform your testing. You absolutely need to make sure your ducks are in a row before performing a pen-test, and most of us have our lawyer’s phone number memorized before we perform any test (getting arrested during a test – especially a physical test is not unheard of or even unusual). This is the type of service that this vendor is claiming to offer for $99. Does that seem realistic?

On the other hand, vulnerability scanning is another common practice in the industry, and that is much easier. But it’s not a pen-test. A vulnerability scan is an automated tool that almost never takes the step of actually exploiting a vulnerability, instead of looks at a number of things (such as version information, headers, port scans, server responses to certain inputs, etc.) to determine if a vulnerability is likely there, and it spits out a report. A good security company will then go and figure out what false positives are there in the report and give recommendations on how to fix them; but to be clear, this still is not a pen-test even with that extra human step. Vulnerability scans are wonderful. They allow quicker responses to vulnerabilities, they help find the low hanging fruit, and they often mimic the general scanning that opportunistic attackers use to scan the internet and find unpatched vulnerabilities/other low hanging fruit. There is nothing wrong with vulnerability scans, but they are not even in the same category as locking a team of 10 highly trained individuals (who do this day in and day out) in a room for a week and giving them the sole task of finding a way into a specific company’s systems – which is a pen-test.

So, all of this is to say that a $99 pen-test is just not possible. And that’s what I said in response to this. As you can see in figure 2, not only did I clarify what a pen-test is, the owner of the vendor came back and tried to argue with me that it was a pen-test, and the proceeded to describe something that doesn’t even qualify as a vulnerability scan – simply an enumeration tool to ostensibly see what type of infrastructure is there.

[Brian Semrau] - "Absolutely no way that is an actual pen-test. A vulnerability scan, yeah... That makes sense. A pen-test, no. A pen-test takes hours of highly skilled labor - and that's just for scoping. Advertising it as a pen-test is downright dangerous."

[Company owner] - "this is an external, automated pen test using the same tools hackers use to identify and determine if there is an oppty to breach."

[Brian Semrau] - "so it's an enumeration tool. That's not a pen-test. Nothing automated is a pen-test. Automation can and is used in pen-testing, but on its own can not be a pen-test... That's a vulnerability scan. And besides, enumeration is just the first step (after scoping) in a pen-test... There's a lot more to it than enumeration."
Figure 2 – the debate begins

I’m not always right

I’ll admit it, I’m human… I can be wrong sometimes. But I’m pretty sure this isn’t one of those times. On the off-chance that I am wrong, I opened the door for it to be proven on what I think are more than fair terms; and my offer still stands. I have yet to hear back with this company accepting my challenge. Here (Figure 3) is some additional back and forth between the company’s owner and I where I ultimately lay out the challenge. Note that the owner makes a differentiation between their product and a supposed “comprehensive” pen-test. Nowhere in their ad or on their website does it make that differentiation, it only lists “Penetration Testing”.

[Company owner] - "we agree that enumeration is a critical phase of a comprehensive pentest, which includes manual iterative attacks and lateral movement. That said, determining external entry points and verifying they're potential for breach via pentest methodologies and automated tools is also a pentest, albeit one that, if access is determined to be available then next steps should be taken.
And that's the point of our pentest, to provide visibility into potential external penetration so that our Partners can offer remediation services and protections."

[Brian Semrau] - "a pen-test requires penetration into the network, application, etc. How exactly is your tool doing that? Unless you are describing it poorly, it isn't even a proper vulnerability scan much less a pen-test.
Tell you what. I have an application that I developed to use when I teach pen-testing. Any actual pen-test against that application would almost certainly pop a shell in <10 minutes - that's assuming they have to figure out *where* to pop the shell (and I don't exactly hide it)... The exploit itself only takes 30 seconds (if that). If you really think this is a real pen-test, sign me up. I'll pay whatever it is you charge for the platform plus the test. But, if your tool fails to pop a shell (doesn't even need to be a root shell - I'll be happy if you can simply pop the shell), then you agree to give me my money back and I'll post a video of the failing on every MSP forum I'm in. On the other hand, if it is actually a pen-test, then not only will you have a customer for life (because that will make my life so much easier when I do perform *real* pen-tests), but I'll publicly apologize for doubting you and post a promotion for the new tool in every MSP forum I'm in.
But, I'm guessing that you probably will not go for that, because we both know it's not a pen-test. Claiming it is one is dangerous."
Figure 3 – the challenge

Ultimately, the owner ended up admitting (yet again) that this is essentially a vulnerability scan (if that). This wasn’t in direct response to my challenge, but was still in reply to me, so I’m mostly including it to make sure that I’m being fair and including all of their responses.

[Company owner] - "Vulnerability Assessments scan systems for known vulnerabilities and report them out - we have that as as a stand-alone scanner that Partners can use as well and, to some degree it informs the Pen Test. Pen Tests attempt to exploit weaknesses in both the known vulnerabilities, should any exist, as well as the construct and implementation of the networks architecture so as to determine the degree to which an attacker can gain access. Our tool attempts just that. It doesn't go deep, nor move laterally - that behavior is the purview of agent-based tools and manual hacking techniques - but it attempts to exploit and report out.
If you're a seasoned MSSP or security professional conversant with the tools and techniques, there are far more robust tools available. But for the average MSP and their clients, who have no or little visibility, this is a great start.
You may think that anything short of a full white-hat engagement is not a pen test, but any tool that an MSP can use to identify even some of the potential security holes that an attacker might pry open is worth them considering."

[Another commenter] - "I don’t think anyone objects to the tools, but to the mis-naming of a vulnerability scan as a penetration test, which it isn’t."

[Brian Semrau] - "you've lost the point. A pen-test is *never* about the tools. It's about the people behind the tools. There is no such thing as a pen-test that uses a point and shoot tool like you are describing. Look up NIST's definition of a pen-test, then tell me if you can honestly even come close to comparing this. I guarantee you can't. Stop false advertising before you get someone in major trouble.
Again, I put my challenge out there. Exploiting the software I developed is child's play in a pen-test. If this is a pen-test, prove it. If not, stop lying to your customers and to yourself before you end up with a massive lawsuit on your hands. The sad part is that the easiest way for a plaintiff's firm to get to your company will be through the MSP that sells your solution, so they will end up going down with you."
Figure 4 – more clarification that this is not a pen-test.

Ok, you say that this isn’t a pen-test, but who are you to say if it is or isn’t?

A pen-test refers to something very specific in the industry, and it can’t be completed by automated means. If you don’t believe me, listen to the Darknet Diaries episodes on penetration testing again. There are other standards that define penetration testing. For instance, NIST SP800-115 states “[Penetration testing] is labor-intensive and requires great expertise to minimize the risk to targeted systems. Systems may be damaged or otherwise rendered inoperable during the course of penetration testing, even though the organization benefits in knowing how a system could be rendered inoperable by an intruder. Although experienced penetration testers can mitigate this risk, it can never be
fully eliminated. Penetration testing should be performed only after careful consideration, notification, and planning.

Penetration testing often includes non-technical methods of attack. For example, a penetration tester could breach physical security controls and procedures to connect to a network, steal equipment, capture sensitive information (possibly by installing keylogging devices), or disrupt communications. Caution should be exercised when performing physical security testing—security guards should be made aware of how to verify the validity of tester activity, such as via a point of contact or documentation. Another nontechnical means of attack is the use of social engineering, such as posing as a help desk agent and calling to request a user’s passwords, or calling the help desk posing as a user and asking for a password to be reset.” (Scarfone et al., 2008) The publication goes on for a number of pages giving a very brief overview of the penetration testing phases, frameworks, etc. Needless to say, as far as the National Institute for Standards and Technology is concerned, penetration testing is not a $99 endeavor.

The other aspect is there are many compliance standards out there that require pen-tests in some frequency. If you tried to tell an auditor that your pen-test cost you $99, you would be laughed out of the room, and we all know it.

Ok, but what’s the harm?

Read the company owner’s last sentence in Figure 4 again, “You may think that anything short of a full white-hat engagement is not a pen test, but any tool than an MSP can use to identify even some of the potential security holes that an attacker might pry open is worth them considering.” And to be fair, I agree with the last part of that statement. I’m not even saying that the tool has no value, I’m saying that it’s being falsely advertised.

Vulnerability scans are good and very useful within their function. Enumeration so you know what you have to protect is good. There is nothing wrong with either, but they aren’t pen-tests. Think of it this way. Dogs are great. I love dogs. Dogs can be very useful and frankly they bring a lot of enjoyment to people’s lives. But I can’t sell a dog claiming that it’s a unicorn. A dog is a dog, it’s not a fricking unicorn. A vulnerability scan is a vulnerability scan. It’s not a fricking pen-test. An enumeration scan is an enumeration scan, it’s not a fricking pen-test or a vulnerability scan. Words have meaning. Use the right ones. Anything short of that is false advertising.

As an industry we need to do better. The MSPs who know better need to call out vendors when they try to pull this sort of thing. Do you expect your client to know the difference between a $99 fake pen-test and a real pen-test to the tune of $20k+? This is just going to turn into a race to the bottom. It isn’t just this one vendor. We all have seen vendors like this. It’s time to hold vendors accountable and for vendors to start being honest with themselves and their customers. If you have a new great vulnerability scanner, just call it that. I’ll probably be right there with you telling your potential clients how useful it can be if it’s a good product. But stop trying to sell unicorns. It just devalues the entire industry.

References

National Institute of Standards and Technology, Scarfone, K. A., Souppaya, M. P., Cody, A., & Orebaugh, A. D., Technical guide to information security testing and assessment. 36–41 (2008). Gaithersburg, MD; National Institute of Standards and Technology. Retrieved January 3, 2022, from https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-115.pdf.

Update 1/6/2022

I had a chat on the phone with the company owner, it sounds like they will be clarifying/updating wording based on feedback from myself and others in the industry. To be clear, I’m not upset with the company directly, I’m upset with how many vendors in the industry are muddying up what specific terms mean, and we seemed to agree that it was a problem even if we disagreed over whether or not the current marketing at this company was doing that. Overall it was a good conversation, and while whatever the wording is changed to may not be exactly what I would call it, things do sound like it will be a lot more clear about what the tool is that they are offering. I’m excited to see what the final evolution of those changes are, and I’ll give them props if it’s a significant enough change to help clarify things.

About thegeekkid

Leave a Reply

Your email address will not be published. Required fields are marked *

For security, use of Google's reCAPTCHA service is required which is subject to the Google Privacy Policy and Terms of Use.