Background

For a number of years, I ran a MSP or “Managed Services Provider.” If you are unfamiliar with MSPs, this is the type of company that you can outsource pretty much all of your business IT needs to – it’s like having your own IT department without having to pay for the full time staff necessary to run that department if you are running a smaller or medium sized company. A number of years ago, I decided to switch my business model and offer security services exclusively. Today, MSPs are one of my largest markets, because they often find themselves needing a hand here and there when it comes to security, and I’m willing to jump in and help them with whatever they need without trying to poach their customers. So, even though I’m not offering traditional MSP services anymore, I’m still involved in a number of MSP communities which are very active.

Magic Products

Many vendors that target the MSP community have a tendency to over-sell their product in some way or another – heck… let’s face it… it seems that most vendors and sales people do that. But at what point does it stop being “fluff” and become false advertising? Don’t get me wrong… there are a lot of vendors in the MSP community that have fantastic solutions and make everyone’s life easier and helps keep costs down by allowing smaller MSPs to do more with less staffing. The problem comes when these vendors try to break into industries outside of most smaller MSP’s experience. Most MSPs do a great job at covering security basics; but when it comes to highly advanced security tasks such as penetration testing, threat hunting, etc; they often have to work with an outside vendor – which is completly understandable – no one person in IT is an expert in everything. But vendors know this and build products and services which often look fancy, but are lacking behind the hood – especially when it comes to security tooling and services.

Yesterday, I came across a new product announcement from one such vendor, and just couldn’t keep quiet. They were promising a pen-test for $99 (see Figure 1). Now, to what will likely be some people’s dismay, I’m not going to name and shame this vendor… that isn’t the point of this article. I’m not trying to tear a single vendor down, I’m trying to shine a light on a problem within the industry.

If you aren’t familiar with a pen-test, this is where a group of highly skilled hackers will attempt to break into your infrastructure in order to help you find your vulnerabilities. This is a process that takes know-how, tenacity, lots of caffeine, and weeks of prep-work, testing, and report writing. It’s not just technical skills either. Depending on the type and scope of the assessment, it may even include people showing up to buildings they aren’t supposed to have access to and figuring out how to get in either through social engineering or breaking and entering. I highly suggest listening to some of Darknet Diaries penetration testing episodes if you aren’t familiar with pen-testing. One of my favorites is Episode 59: The Courthouse; although this focuses mostly on the physical side when pen-testing is really a huge field within security with most testers having specializations in specific techniques, technologies, etc. This is why pen-testing gets to be so expensive – it has absolutely nothing to do with the tools used, and everything to do with the people behind the tools and the time they invest into each test.

Because the field is so broad, it takes more time than $99 could pay for just to scope out the test and figure out what the client wants. Beyond that, most pen-test agreements need to be reviewed by an attorney because you are literally doing something that is a federal crime that involves jail time (at least in the US) if you don’t have proper permission to perform your testing. You absolutely need to make sure your ducks are in a row before performing a pen-test, and most of us have our lawyer’s phone number memorized before we perform any test (getting arrested during a test – especially a physical test is not unheard of or even unusual). This is the type of service that this vendor is claiming to offer for $99. Does that seem realistic?

On the other hand, vulnerability scanning is another common practice in the industry, and that is much easier. But it’s not a pen-test. A vulnerability scan is an automated tool that almost never takes the step of actually exploiting a vulnerability, instead of looks at a number of things (such as version information, headers, port scans, server responses to certain inputs, etc.) to determine if a vulnerability is likely there, and it spits out a report. A good security company will then go and figure out what false positives are there in the report and give recommendations on how to fix them; but to be clear, this still is not a pen-test even with that extra human step. Vulnerability scans are wonderful. They allow quicker responses to vulnerabilities, they help find the low hanging fruit, and they often mimic the general scanning that opportunistic attackers use to scan the internet and find unpatched vulnerabilities/other low hanging fruit. There is nothing wrong with vulnerability scans, but they are not even in the same category as locking a team of 10 highly trained individuals (who do this day in and day out) in a room for a week and giving them the sole task of finding a way into a specific company’s systems – which is a pen-test.

So, all of this is to say that a $99 pen-test is just not possible. And that’s what I said in response to this. As you can see in figure 2, not only did I clarify what a pen-test is, the owner of the vendor came back and tried to argue with me that it was a pen-test, and the proceeded to describe something that doesn’t even qualify as a vulnerability scan – simply an enumeration tool to ostensibly see what type of infrastructure is there.

I’m not always right

I’ll admit it, I’m human… I can be wrong sometimes. But I’m pretty sure this isn’t one of those times. On the off-chance that I am wrong, I opened the door for it to be proven on what I think are more than fair terms; and my offer still stands. I have yet to hear back with this company accepting my challenge. Here (Figure 3) is some additional back and forth between the company’s owner and I where I ultimately lay out the challenge. Note that the owner makes a differentiation between their product and a supposed “comprehensive” pen-test. Nowhere in their ad or on their website does it make that differentiation, it only lists “Penetration Testing”.

Ultimately, the owner ended up admitting (yet again) that this is essentially a vulnerability scan (if that). This wasn’t in direct response to my challenge, but was still in reply to me, so I’m mostly including it to make sure that I’m being fair and including all of their responses.

Ok, you say that this isn’t a pen-test, but who are you to say if it is or isn’t?

A pen-test refers to something very specific in the industry, and it can’t be completed by automated means. If you don’t believe me, listen to the Darknet Diaries episodes on penetration testing again. There are other standards that define penetration testing. For instance, NIST SP800-115 states “[Penetration testing] is labor-intensive and requires great expertise to minimize the risk to targeted systems. Systems may be damaged or otherwise rendered inoperable during the course of penetration testing, even though the organization benefits in knowing how a system could be rendered inoperable by an intruder. Although experienced penetration testers can mitigate this risk, it can never be
fully eliminated. Penetration testing should be performed only after careful consideration, notification, and planning.

Penetration testing often includes non-technical methods of attack. For example, a penetration tester could breach physical security controls and procedures to connect to a network, steal equipment, capture sensitive information (possibly by installing keylogging devices), or disrupt communications. Caution should be exercised when performing physical security testing—security guards should be made aware of how to verify the validity of tester activity, such as via a point of contact or documentation. Another nontechnical means of attack is the use of social engineering, such as posing as a help desk agent and calling to request a user’s passwords, or calling the help desk posing as a user and asking for a password to be reset.” (Scarfone et al., 2008) The publication goes on for a number of pages giving a very brief overview of the penetration testing phases, frameworks, etc. Needless to say, as far as the National Institute for Standards and Technology is concerned, penetration testing is not a $99 endeavor.

The other aspect is there are many compliance standards out there that require pen-tests in some frequency. If you tried to tell an auditor that your pen-test cost you $99, you would be laughed out of the room, and we all know it.

Ok, but what’s the harm?

Read the company owner’s last sentence in Figure 4 again, “You may think that anything short of a full white-hat engagement is not a pen test, but any tool than an MSP can use to identify even some of the potential security holes that an attacker might pry open is worth them considering.” And to be fair, I agree with the last part of that statement. I’m not even saying that the tool has no value, I’m saying that it’s being falsely advertised.

Vulnerability scans are good and very useful within their function. Enumeration so you know what you have to protect is good. There is nothing wrong with either, but they aren’t pen-tests. Think of it this way. Dogs are great. I love dogs. Dogs can be very useful and frankly they bring a lot of enjoyment to people’s lives. But I can’t sell a dog claiming that it’s a unicorn. A dog is a dog, it’s not a fricking unicorn. A vulnerability scan is a vulnerability scan. It’s not a fricking pen-test. An enumeration scan is an enumeration scan, it’s not a fricking pen-test or a vulnerability scan. Words have meaning. Use the right ones. Anything short of that is false advertising.

As an industry we need to do better. The MSPs who know better need to call out vendors when they try to pull this sort of thing. Do you expect your client to know the difference between a $99 fake pen-test and a real pen-test to the tune of $20k+? This is just going to turn into a race to the bottom. It isn’t just this one vendor. We all have seen vendors like this. It’s time to hold vendors accountable and for vendors to start being honest with themselves and their customers. If you have a new great vulnerability scanner, just call it that. I’ll probably be right there with you telling your potential clients how useful it can be if it’s a good product. But stop trying to sell unicorns. It just devalues the entire industry.

References

National Institute of Standards and Technology, Scarfone, K. A., Souppaya, M. P., Cody, A., & Orebaugh, A. D., Technical guide to information security testing and assessment. 36–41 (2008). Gaithersburg, MD; National Institute of Standards and Technology. Retrieved January 3, 2022, from https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-115.pdf.

Update 1/6/2022

I had a chat on the phone with the company owner, it sounds like they will be clarifying/updating wording based on feedback from myself and others in the industry. To be clear, I’m not upset with the company directly, I’m upset with how many vendors in the industry are muddying up what specific terms mean, and we seemed to agree that it was a problem even if we disagreed over whether or not the current marketing at this company was doing that. Overall it was a good conversation, and while whatever the wording is changed to may not be exactly what I would call it, things do sound like it will be a lot more clear about what the tool is that they are offering. I’m excited to see what the final evolution of those changes are, and I’ll give them props if it’s a significant enough change to help clarify things.