Last week, I video-called Keith (yes, Zoom).
He got his first managerial job and I wanted to congratulate him as he was working from home. It was 9am in the morning and my “Good Morning, Keith” did not spark a response from him. He was not in his usual buoyant spirts and looked exhausted. Normally he is quite friendly and does not get bothered by much. I used to hear calm stories about long-shifts and IT deployment fiascos, but this call started starkly enough to make me wonder what was up.
Since his usual excitement was gone, I asked, “Why are you looking so exhausted?”
“I’ve been awake since 3:30 AM”, he replied.
I could sense something more was wrong and dug deeper to figure out what was bothering him. His reply sounded worried, and he informed me that when he started his shift, he had witnessed unusually high traffic from Russia overnight.
Since he had primary responsibility acting as project lead for the first time for an AppSec Engineering Team it was probably natural for him to sound worried.
Feeling his concern, I asked, “You’re looking particularly tense. Is that all that happened?’
He replied, “I am in the midst of a mess. At first the SOC dashboard was fully green and everything was hunky dory. But, 25 minutes ago, another alert popped up on my screen. The IT technician has taken lead on this issue, but it is been more than 20 minutes now and the issue persists, and the ticket is still open. This is the 35th alert during this shift and I am sure this one will turn out to be a false alert again”.
I repeated “35 false alerts” in my mind. Saying I was sorry, I made a brief mental note and let him go.
Appsec developers and IT frontline engineers, can easily relate to a rash of false alerts, and if we add to this, the pressure of working from home during the pandemic, it can definitely take a toll on one’s mental state.
False Alerts & the story of the “boy who cried wolf”
I realized it was a bad time to be calling him as he was knee-deep in his company’s SIEM system. It was a typical “been there…seen that” moment experienced by many on-call developers.
Having said that, his situation, teeming with false positives and alert fatigue, can be compared to the story of ‘The Boy Who Cried Wolf’, and he didn’t want to risk being the one sounding the alarm, when there really wasn’t a real threat.
What does this story indicate? Even when the boy (having told lies) told the truth, no one believed him.
There is of course, another angle to this. The villagers ended up losing all their sheep as they did not believe the boy when the threat was real. If they had trusted the boy, despite the chances of a false alert, they would have saved the flock.
Coming back to Keith – his passion is coding. His passion extends to Continuous Integration, and he is always excited to share his latest code triumph at the source repository.
But he has less time for coding lately, because he is stuck in this wild goose chase of resolving false alerts. He loves being part of the DevOps culture but is constantly tasked to resolve alerts, and is now the victim of alert fatigue due to false positives. And lately, the Mean Time To Resolution (MTTR) for each IT ticket he’s called upon to assist has been increasing.
Let us understand what kind of alerts increase MTTR, first the types of alerts (or lack thereof) he might come across:
The “wolf-likelihood” model with 4 possible outcomes:
So, how do you minimize false alerts?
1. Automate & tame the wolves behind the real-time monitoring alerts
The kind of alerts Keith gets is dependent on what threshold/rule he sets (according to his chosen metric). His love-hate relationship with false alerts would secure his place in the 15% of IT folks who waste (ok, actually spend) their time in the identification of alerts/important data points in need of attention. Interestingly, if he had this job in late 2019, he would end up battling false positive at rates of 50% or higher from his security tools.
The solution? There’s a couple of possibilities. First, if the security technologies he’s using are giving him too many false positives, maybe he should look at newer technologies that have lower false positives, like deterministic security, which unlike typical security technologies, does not rely on any prior attack knowledge to recognize an attack, and if used side-by-side with existing tech, can help determine which alerts are higher priority. If Keith can’t switch out his tech, he should look at tools that assign alerts based on severity (let’s say low, medium and high or a scale of 1 to 10), then he’d be able to only look at and respond to the high severity alerts. That would free him up to be successful in not just maintaining his development speed process but also increase his speed in his releasing of software code. And maybe he’d even get to implement a CICD (Continuous Integration Continuous Delivery) pipeline?
Moreover, if he gets to see detailed attack data in real time, he would spend less time in debugging cycles and testing duration.
Let us look at the types of detailed attack data that could be discovered with the right security solution.
The following example gives details:
There is an attacker who tries to break into a server (depicted on the left) of an application (a vulnerable banking application):
K2 for example, discovers the application servers in realtime, along with the applications running on each server (shown on the right below).
2. Making AppSec Simple To Fit into the Development Process
Appsec needs to correlate false-positive alerts with the correct intelligence to help your security and development team resolve issues quickly.
In an ideal world, application security is thoroughly tested and vulnerabilities discovered fully remediated in the development phase of the Software Development Life Cycle (SDLC). But the irony is that only 10% of organizations, successfully identity and repair code vulnerabilities in the development phase .
Because security managers and software teams are judged on the parameters of operational risk management and speed respectively, we still see a lack of thorough appsec, a function of appsec ending up on the losing end of ‘accelerating time to market’.
The end results? As workload/application testing is only done “inside out” (in a non-running state, i.e. SAST), and after code development is finished, code vulnerabilities discovered during final testing slow down the delivery cycle and it takes longer to find and remediate these vulnerabilities. In other words, the time needed for true threat analysis and prioritization up significantly when security isn’t prioritized at the beginning of development.
So it shouldn’t have been a surprise, when a survey in 2019 of CISOs shared that “over 41% see more than 10,000 and that some claim to see more than 500,000 alerts daily.”
With that many alerts, it’s important to understand what contributes to false-positives. Some of the key drivers in false positives were found to be:
- Lack of context in the alert generation process
- Inability to consolidate and classify alerts
This is where an simple sidecar application security approach can help along with robust discovery capabilities that include detailed alert telemetry and information. Both pre-production and production need to be included, so teams can have some breathing space as security vulnerability information is gathered directly from an application as it runs.
Here at K2, our agent can be used in both pre-production and production. In pre-production, the agent is deployed in tandem with penetration testing/scanning tools and provides detailed vulnerability telemetry including the exact location of vulnerable code for every attack vector and vulnerability.
The information provided by K2 includes granular attack details including attack type, code module involved, and the exact line of code with the vulnerability. K2 uses a unique deterministic technology for detecting attacks and vulnerabilities, eliminating false alerts leading to improved efficiency of the security teams.
3. Adding Runtime Application Self-Protection (RASP)
Our friend Keith is a legacy appsec guy. That means he is still stuck dealing with false positives generated from legacy security technologies. There are so many Keiths out there who have no idea where to start or how to fix the problems they are facing. It’s no longer enough to secure applications with signature/pattern matching, heuristics, fuzzy logic, or even machine learning and artificial intelligence.
These typical security methods base their detection on knowledge gained from past attacks, and will miss true zero-day attacks. Signature and pattern matching results in too many false positives, and are incapable of providing any detailed information on the actual problem, when a real attack does happen.
That is why detailed attack and vulnerability information is needed for every alert. If you know the exact location of the vulnerable code for every attack, you can quickly resolve the alert. In addition to the information around the vulnerable code, you also need detailed information about the attacker. By having exact telemetry of the attacker, you can block just the attacker in real-time to prevent any further damage, allowing the application to stay up and running. The following shows the attack being blocked in real-time in conjunction with a leading firewall, while allowing the application to stay running for valid users.
Realtime IP block
4. Combine Firewall with Signature-less Workload protection
Blocking just an attacker’s IP address and sessionID in real-time at the firewall is the ideal solution. Block the attacker at the firewall before they even get into your network, allow the application to continue to run, and prevent the attacker from doing any damage. Having the detailed attack telemetry means real-time blocking and faster remediation is not a distant dream.
The K2 application workload security platform provides:
- Detection of attacks in real-time
- No dependency on attack method or underlying vulnerability for reliable detection of zero-day attacks
- K2 requires no changes to applications to secure them, reducing the overhead in deployment and simplifies security remediation for the DevOps team
- No false positives result in fewer, but actionable alerts
- Does not require expensive analytics or machine learning
Detecting attacks on applications in real time is extremely crucial. K2’s deterministic security and the ability to be used in both pre-production and production, makes it far easier to locate and remediate vulnerabilities, instead of having to continually keeping up with an infinite number of attack patterns that attackers can conceive.
K2’s Next Generation Application Workload Protection Platform addresses today’s need for runtime security in an easy to use, easy to deploy solution. K2’s unique deterministic security detects new attacks without the need to rely on past attack knowledge, is lightweight, and adds under a millisecond of latency to the running application. To aid in quick remediation of vulnerabilities, K2 also provides detailed attack telemetry including the code module and line number being in the code being attacked, while at the same time integrating with leading firewalls to do real time attacker blocking.
Change how you protect your applications.