I spent a good amount of time this week speaking to customers, partners and prospects about deploying, engineering and using our products — one topic that always seems to be part of the discussion is system throughput and scalability.  Of course our position regarding this is clear, as NetWitness technology was designed from inception to support any combined throughput and can scale out as your network grows.  Inevitably the conversation dives deeply into why we say this…

For any network recording AND analysis technology there is an INPUT and OUTPUT to consider, I think everyone knows this.

INPUT – the concept of guaranteeing packet acquisition and writing to a storage structure with no loss as fast as possible – 1Gbps, 10Gbps, 40Gbps… and so on. The vast majority of vendors out there focus on and emphasize this extensively — this may be capture acceleration, stream-to-disk, or flow/header technology in high bandwidth environments.

OUTPUT – the concept of being able to access and analyze the captured data, deeply and across days, weeks or months of data quickly, ideally in real-time.  Most vendors minimize the importance of this, and often do a poor job of providing value with data spanning more than a few hundred mega-bytes at a time, and rarely address true security needs.

What is never discussed or exposed in the market is that these requirements are in constant contention when acting on network data within a single physical system.  Or, in other words, the more you are writing to a system the less you can read. Being sensitive to this reality since the first version of our product over 10 years ago, we designed a system that optionally separates these services, and scales out on hardware to meet any deployment condition. Ultimately providing high-speed capture, retention, and real-time access to deep analytics – true situational awareness of your network – it is what NetWitness does.

Recently, I did a webcast that goes into detail about how to architect NetWitness in these environments — I invite you to take a listen, you should find that when it comes to architecture and scalability, NetWitness is one of the few in our space that can actually deliver.

Brian Girardi – Director, Product Mangement

There was a significant amount of coverage yesterday on research performed by NetWitness into a large set of stolen information recovered from a ZeuS botnet.  Some of the information, analysis, and commentary was very beneficial to the broader discussion of threats such as these.  There is, however, some information that we feel we should address.

• Kneber is a pseudonym for ZeuS:

Kneber is not a pseudonym for ZeuS. Kneber refers to one group of organized criminals, one group of Command and Control Systems, and 74,000+ infected victim systems for this particular ZeuS (primarily) botnet.   ZeuS is a tool, used by many groups to create command and control systems, and steal information.  There are hundreds of active ZeuS botnets, many of which are larger than this one. It is but one of many tools used in this particular botnet.  We have seen INTENTIONAL cross pollination of various trojans, including waledec, grum, and even tools such as packet sniffers.  When we discuss threat, we are referring to more than the tool used, but the organization behind them.

• Kneber is “nothing new”:

We have been very clear that this is a medium sized infestation when compared with all the tracked ZeuS botnets on the Internet.   What does make this very valuable is the opportunity to analyze such a large sample of stolen information, and quantitatively add to the discussion of threats to corporate security.  The number of infected and active systems behind some of the largest, most technology savvy companies needs to be considered, and our approach to security needs to change given the broad failure to identify or remediate these infestations.   In addition, trivializing the damage done is simply disingenuous by anyone who has seen the types of data stolen from threats such as these.

• Current protections and solutions can detect this type of activity:

This quote from Symantec, via the Guardian, KrebsOnSecurity, and others:

“Kneber, in reality, is not a new threat at all, but is simply a pseudonym for the infamous and well-known Zeus Trojan,” said the company. “The name Kneber simply refers to a particular group, or herd, of zombie computers, a.k.a. bots, being controlled by one owner. The actual Trojan itself is the same Trojan.Zbot, which also goes by the name Zeus, which has been being observed, analyzed and protected against for some time now.”

This quote is particularly troubling, as it seems to minimize the threat and is almost dismissive.   Moreover, when this particular variant was analyzed in late January (various services used), Symantec did NOT detect this as malicious.   To be fair, McAfee, Trend Micro, AVG, and most other mainstream anti-virus solutions also failed to recognize this as malicious.  In the past 3 weeks, Symantec has added signatures to detect this particular variant as a generic “Trojan Horse”.  However, if you were infected by this particular strain, your system has already processed an update that prevents you from contacting Symantec and others for updates.   In most cases, this will prevent future detection.   Worse, as part of normal operation of ZeuS, it attaches to running processes on victim systems in order to monitor them.   This data is logged along with other stolen information.   This set of data shows that ZeuS has actually attached to running versions of Symantec software on over a thousand victim systems.  Many other AV vendors are also present.

This example shows ZeuS monitoring a Symantec Live Update, and includes the ftp username and password used by the Symantec software during the update process.

• Are the facts overstated?:

The facts are fairly succinct in the whitepaper that we released.   We do not believe the threat is over-stated, and we were very conservative on the analysis released.   There are likely thousands of additional corporate networks affected, and analysis of this much information takes time.   And this is simply one of many similar operations in existence.  The group behind this effort can be described as sophisticated, yet also shows signs of lax effort to hide their trails.   The botnet is very actively managed, and continues in operation today.   The fact that they have been in successful operation for over 18 months also has to be considered.   We have also received several additional data points from federal contacts with additional insight into related government focused attacks.

More to come.

Tim Belcher and Alex Cox

It’s a little known fact that NetWitness has been innovating in the security field for over 11 years, which was further validated by the announcement of our recently granted US Patent # 7,634,557. Clearly, when it comes to network analysis we do it better than anyone else, and it’s really the only way to get the results you need.

Reaching back over a decade (ca.1999) when our first patent was filed, ( US Patent # 7,016,951 ), and murmurs of network forensics were swirling from a few experts in the security community, our innovation in this field was in full swing.  The technology was chartered as an analytical application to make sense of network traffic for users with no networking experience.  This in itself was no small task, as I cannot emphasize how difficult it was explaining what an IP address was to an English major. See the snapshot of NetWitness v3.5 ca. 2002, ironically it looks like some our 2010 competition.

In retrospect, NetWitness was conceived in a reverse direction from how most security products end up being developed.  Our strategy was to understand the data FIRST, then figure out how to capture it and scale it reliably into an enterprise.  Honestly, we spent several years trying to determine the best way to present complex network data to our users, which at that time was simple HTTP and SMTP sessions.  We had no idea how the network application profile of an Enterprise would evolve to what it is today.   With that said, we made sure that the advanced methods we developed were flexible enough to evolve with the Internet and the needs of our users.  These methods found their way into these two patents.

The first and most important patent is a method for traffic capture, session reassembly, metadata extraction and recursive port-agnostic service identification. Did you get all that?  Back when Firewall and IDS were tinkering with port numbers for rule logic, NetWitness was beyond that approach over 10 years ago.  The assumption to classify network traffic by port alone is prone to mistakes for reliable security analysis. It was not until recently there was a prominent increase in products that are, or at least market port agnostic support, like application firewalls and some DLP products.

The second patent, the topic of this announcement, extends the core technology by defining a system and method for organizing and describing the traffic we collect.  Yet again an example of how we designed the technology to evolve as the Internet evolved.   The patent specifically focuses on the session data model and structures that fuel the Investigator interface and the user experience.  The result is the most visible difference between NetWitness and our competitors, as well as what provides the analytical value when responding to <INSERT NETWORK PROBLEM HERE>. Another example of the product evolution can be seen in the screenshot below of NetWitness v5 ca. 2004.

Its always been my assertion that to do true network forensics, or really any good network analysis, you need a few key ingredients:

1) Reliable, scalable, and forensically sound network capture.  Unfortunately the vast majority of “network forensic” vendors stop HERE!

2) As you would expect from any forensic science, the technical ability to piece the clues or segments of an event back together is the next logical step. For network forensics its assembling the packets back into full sessions, because without this step you have disparate puzzle pieces, without a complete picture.

3) Then finally the right tools to analyze, correlate, mine and report the findings to humans. Thankfully there is an NetWitness App for that and a free API/SDK too.

These elements combined are the foundation of what NetWitness NextGen is, and the basis of our technology that is truly becoming a game-changer in security.  NetWitness Corporation was founded in late 2006, but unknown to many, the innovation and pioneering environment that fuels the technology today started 10 years earlier.  Enjoy our innovation by using Investigator Freeware, and know that before the security challenges of today really materialized we were hard at work creating solutions for today. Network security products that simply work.

Cheers,

NetWitness v9, ca. 2010.

The news is rife with discussions about systemic failures in the intelligence community.  It is a good thing we do not judge information security on the same scale of success.  I know of not a SINGLE enterprise network that is not being repeatedly compromised with a deluge of malicious code.  Can you imagine a world where we expected our anti-virus to actually protect us?  Weren’t we all talking years ago about what would happen when people began writing custom code to attack YOU.  Our most ubiquitous security problem today is certainly malicious code, and after recommending “Malware Bytes” to my 10^th family member or friend last month to undo a successful phishing or drive-by infection, this problem certainly does single out anti-virus products as “anti-success.”  So why do I pick on Intrusion Detection as the winner for institutionalizing failure in our security organizations?

I believe IDS started several negative trends that are still affecting the psyche of security personnel today.  For the first several years, all iterations of IDS were so prolific in their alerting that they have provided a decade long after-taste.  Some would argue they still are.  The very concept was flawed from the beginning, and only considered because we had lost control and understanding of our networks.   Systems and disk were simply not fast enough, or large enough to analyze or understand our networks.  We decided that we must look to technologies and solutions to determine what is bad on our network, ignoring the rest — and we turned to our first magic pill.

Whatever the case, they were the solutions that made “false positives” a mainstream security term. Think about that term for just a minute.  To have “false positives,” intrinsically implies that there could be some perfect solution.  That it is conceivable, much less possible, to actually determine what is bad on your network.  The problem was certainly not fixed as the vendors began marketing prevention, and the vast majority of IPSs employ little to no prevention because of the likelihood of false positives.

Now consider how many magic pills followed in the IDS wake.  Remember when DDOS was the threat?  A crop of DDOS mitigation products were made available to fix that problem.  Where are those products today?  Worms, Code Red and Nimda – gave rise to a swath of behavioral analytics products.  If I remember correctly – we were suppose to see Insider Threat as the next big thing a few years ago?  Weren’t data leakage and content monitoring systems supposed to plug that gap?  And all along this timeline, I need to manage this unmanageable amount of logs – so let me invest in SIM/SIEM.  SIM/SIEM not enough – perhaps you should look at the “Big Fix.”  The list could go on and on and on.

This is just a sampling of what you need to protect your networks, given the premise that you can automagically determine bad from good, based on some mythical perfect ruleset.  And oh yes, don’t forget – if you need help you can outsource it.  In the wake of “aurora”, and systemic compromises of some of the largest, most technology savvy companies in the United States, perhaps more will realize that compromise is INEVITABLE.  If sponsored adversaries want to get into your network, they will LIKELY SUCCEED.

We are chasing our tails, still looking for that magic pill that will secure our networks, and have not once stopped to reconsider our approach.  I single out IDS because it was the first, and loudest failure in the security space.  I believe it showed the world that security products do not have to work often or at all, but can be marketed and sold successfully. I believe it began the cycle of point solutions that has created a generation that believes it is possible to secure our networks without understanding them.

It is time to realize, that you will not know what is bad today, until tomorrow.  That you will not know the damage caused for hours, days, weeks or even months.  No matter what you invest in, nothing can protect you from what you do not know.  The threats we face tomorrow will not be the same as today.  And most importantly – you are doomed to failure if you rely on some magic solution to determine what is bad.

Instead of building our defenses based on a hodgepodge of point solutions, designed to fix yesterdays problems, why don’t we invest a modicum of our resources into an architecture that can analyze, interpret and record what is happening on your network – yesterday, today and tomorrow. Can we for a moment, invest in regaining an understanding of what is traversing our network, and create a capability to adapt to tomorrows problems?  That is the benefit in deploying NextGen.

A potential customer, after receiving a demo, commented recently on a single rule based alert that was added to a session during analysis called “suspicious file type” – saying the alert was a false positive because the executable downloaded was not malicious.  I corrected him, and not just semantically, that there are no “false positives” here.  There are simply flagged sessions based on intelligence, which add additional data elements.  If you choose to write a rule that alerts when someone downloads an executable, it will do so.  It does not make the assumption that that is a bad executable.  Humans do that.   Sure a single alert can be more valuable than others. Sure we can take a signature and incorporate it. However, it is the preponderance of the complete session analysis – perhaps various alerts, threat intelligence, and a deep detailed understanding of everything that happened in that session — all over time that provides your analysts the ability to ask very detailed and probing questions into your network — and get answers back immediately. Concerned about leakage?   Ask those question of the system.  Concerned about compliance, ask.  Concerned about malware downloads – ask.  Insiders?   Targeted PDFs?  Obfuscated javascript?  Proprietary information?  Law enforcement?  By analyzing it all, we give you a platform for answers.

“Not good enough” he said…  “It needs to tap my analyst on the shoulder and say Hey – look at this!”

“Ah – like an IDS” I responded.  He wanted a magic pill.   Institutionalized Failure.

We recently sent an opt-in email to our contact database talking about the significance of Operation Aurora and the continued ascendancy and lack of advanced threat prevention/detection in many government and commercial organizations.  We also offered a NetWitness proof-of-concept (POC) to security folks concerned about this issue.  And security people should be concerned.

A noted security blogger correctly observed that we were “amplifying FUD” in our email blast to get people’s attention.   His blog post raises a classic issue facing security professionals – does FUD help bring an issue like this to top of mind.  Or:  To FUD, or not to FUD.

It’s really unfortunate that FUD became a dirty word when compliance and “risk management” took over the security budget, but that’s when many organizations, began to fail at security too.  While many people, particularly some CIOs in hindsight, would argue that compliance has helped increase the focus and spending on information security, I would argue that it has distracted many security programs into performing a large number of basically low impact or worthless activities in the name of metrics, versus FUD.  And, compliance certainly has sponsored a whole class of expensive security technologies and related total ownership costs (TCO) which drain the security budget.

There’s also an unfortunate psychology involved here.  Many security professionals feel guilty or inadequate using FUD as an argument because all the other I/T people have real metrics and we don’t.  To some, it’s like when we were kids and everyone had Converse “Chuck Taylor” All-Star high tops and you were the one with the red Pro-Keds.  Security people can’t talk about how many “9’s” of network uptime we have, or how much we have improved call center response time, or improved the total cost per terabyte of enterprise storage.   Security sucks at producing decent metrics — and the ones we do produce, generally stink even more at reducing the fear of being owned by national-sponsored or organized criminal groups or the uncertainty and the doubts regarding the security of information in a world of advanced threats.  Security people cringe when some C-level executive compares the cost of information security to the cost of insurance – “No one likes to pay for it, but just like your car insurance, you have to have it.”  Ugh!  So, we hate the FUD argument – both when we have to use it as an argument, or when someone uses it to trivialize what we all do for a living.

But I do not think security professionals should feel this way.  I think that FUD still has a lot of usefulness in the toolkit of the security professional and within the enterprise security program, if applied in the right doses to the right places.  One of my favorite Websites is fudsec.com.  There are many good, bad and ugly uses of FUD cited here, for example, one of the good ones is Anton Chuvakin’s post, “A Treatise on FUD” – required reading for any committed FUDists.

With regard to advanced threats and other types of network visibility problems, I encourage the use of a combination of FUD and proof.  The FUD comes in the form of security professionals updating their discussion track to highlight the real causes of many cyber losses in 2010, and the need for more focus on threat intelligence and operational security versus other types of spending.  Current issues such as Operation Aurora should be analyzed for relevance, and briefed to senior management, and should be coupled one of the more credible surveys that show that most data losses result from advanced threat or sophisticated exploit/malware sources.

In the end, you will have to produce real evidence, however, and that’s why we put the POC offer on the table in our e-mail blast.   FUD should only go so far — you should show your colleagues the smoking gun with your own organization’s data.   We as a vendor could put out all the FUD-sounding marketing statistics in the world about how our approach will make you more effective at changing the face of FUD to a smile than other alternatives, but you will only believe it when it produces results in your organization, you can bank those results, and it actually reduce the FUD for yourself and your CEO.   This is how it should be.

Did NetWitness actually release a new product that consists of a bucket filled with sand? The answer is yes, but the real question is why? We released B.O.S. in an attempt to sound the wake-up call…

Organizations can no longer afford to rely so heavily on perimeter based technologies, on signatures for identification of threats – and they cannot hide their heads in the sand and hope that nothing goes wrong.  Every day, things are going incredibly wrong.   Prevention alone is an epically failing strategy.

2009 can easily be called the year of advanced threats. The scary thing is that the same can be said for every year over the last five. Despite all efforts, attacks and data losses are getting progressively worse, not better.  During the past five years there have been thousands of breaches reported - impacting state and local government, small and medium sized businesses, multi-national organizations and some of the most sensitive branches of the U.S. Government.   No one is immune and the sickness is literally life threatening.

Imagine for a moment how many breaches went unreported…imagine how many have gone completely undetected.  This is a frightening reality highlighted by the 2009 Verizon Business Data Breach report which found that 49% of breaches went undiscovered for a period of months…and 70% of breaches went completely undetected by internal teams. How is this possible?

The answer is both simple and frightening – the technologies on which organizations have come to rely  aren’t able to prevent, detect, and combat the advanced threats of 2010.

Today’s security technologies are better suited for fighting the cyber-war of 1995 than they are for dealing with today’s advanced threats. The cyber-criminal underground and nation-sponsored groups are using teamwork, custom-developed malware, third-party vulnerabilities via exploit kits, and code obfuscation to bypass existing security technologies and perceptions of security derived from compliance efforts. Because of the industry’s overreliance on signature based technologies, security managers are under the false assumption that they are protected. Too much faith has been placed in firewalls, IDS/IPS, anti-virus, anti-spam and other perimeter platforms to catch the threats.  The current cyber war footing is analogous to bringing a knife to a gun battle – security leaders are reliant upon technologies designed to fight the cyber-war of 10 years ago…our adversaries are fighting with weapons of today.

So, what can be done?

In today’s threat environment it is vitally important that all organizations develop an effective, real-time capability to detect, analyze and respond rapidly to advanced threats.  During the last three years, many of the top security teams in the government and commercial sectors have turned to the advanced threat intelligence and real-time network forensics provide by NetWitness NextGen. The only way to truly know what is going on within the network is to look at everything that is going on within the network. Full packet capture and session recreation are the only ways to accomplish this end.  Where NetWitness NextGen is deployed, the result is an effective threat intelligence program and continuous augmented awareness that provides in-depth visibility into network events that escape existing network security monitoring tools.

In 2010, you should not be buying a bucket of sand.  To combat the advanced threats we now face, organizations must:

1) Reject “status quo” and compliance-focused thinking and acknowledge that prevention is a failing strategy when facing advanced threats;

2) Focus on real-time detection and rapid investigation of advanced attacks to shorten the risk exposure window of any incident;

3) Build an internal security team that is tailored for advanced threat detection and that is armed with an enterprise-wide, real-time, network forensics capability to achieve optimal network visibility…

In short…when looking to combat advanced threats, organizations should be using NetWitness NextGen.

I was at the CSI show yesterday and was within earshot of one of our “competitors” who claimed that they were winning against NetWitness because they support 10Gbps and we do not.   I have heard this story frequently from this particular firm, and it’s a bunch of bull.

It amazes me that companies in this space, such as Niksun and Solera Networks, spend so much time emphasizing how they allegedly can capture at 10Gbps line rates – as if that’s the most important requirement for public and private organizations struggling to cope with critical advanced threats, complex data leakage scenarios, network forensics, designer malware and botnet infestations, and increasing insider crime and fraud.

Solera Networks publicly asserts that their product was certified by Miercom Labs as 10Gbps capable.  If you look at the report on their Website, however (http://www.soleranetworks.com/resources/Solera%20DS5100_Miercom%20test.pdf), you will see that a top capture rate of 8.1 Gbps was achieved solely when the “packet size” was forced to 1,518 bytes.  At other packet sizes, the performance dropped off steadily.

The practical application of the Miercom report for Solera Networks is dubious in the real world.  For example, let’s assume that in Miercom’s vernacular “packet size” is equal to “message transmission unit” (MTU).  1,518 is the maximum MTU for the transmission of data over IEEE 802 networks according to RFC 1042.  But it is unrealistic to imagine that every consecutive packet on a customer network would be 1,518.  In fact, the typical default MTU setting for devices such as routers and servers processing a protocol such as HTTP over TCP/IP is 576 – most network and system administrators today work under this assumption.  In a real customer environment with a large amount of HTTP traffic, the Miercom numbers would put the theoretical Solera throughput somewhere around 6Gbps, versus the claimed 10Gbps.  Real life is different than the lab, however, and the reality is that customer application-layer traffic produces actual average MTUs of far less than 576, thereby lowering the potential performance results below the assertions made by some vendors to something more like an average MTU of 300 — or a throughput of around 3Gbps according to the Miercom results.

Such misleading lab reports also do not address other concerns, such as the technical challenges associated with capturing at a 10Gbps on single network appliance, given physical bus bandwidth constraints and disk write speed limitations — and still offering meaningful and timely analytics to security users.  Every vendor who is engineering solutions in this space has confronted this dynamic — but most vendors do not address this problem in their marketecture because they do not have a solution.

As the consumer of solutions in this space, you should be aware of this:   The reason this issue is not discussed in any meaningful way by some vendors is because they have no real-time automated and interactive analytical capability beyond basic and often erroneous network statistics.

Consider this screenshot from Solera Networks post-facto user console:

Notice specifically the port assumptions made by the product:

To assume in 2009 that TCP ports 80 and 443 are inclusive to web traffic simply is ridiculous.  Not only is this type of analysis absurd from a network forensics perspective, but also would seem at odds with the term “deep packet inspection”.  Consider the following drill into the HTTP service using NetWitness Investigator, on even a single day of capture from the NetWitness corporate HQ:

This screenshot represents only a small portion of the available information about the HTTP protocol.  This level of detail is possible because NetWitness does complete port agnostic session analysis in an automated real-time manner, upon collection.  You cannot get there from here with other vendors like Niksun and Solera Networks.  One of the vendor’s Websites says it most succinctly:

“Once you find the traffic flow you are looking for, you can download a PCAP file of just that data and analyze the traffic using any tool that analyzes PCAP files. Or you can save it for later and use it to analyze when you have time or need evidentiary proof of malicious activity.”

There are some pretty big and unfortunate “IF”s that customers should consider before engaging with any vendor operating under such assumptions:

1.  How do I actually go about “finding the traffic you are looking for” within tens or perhaps hundreds of terabytes of data in post-facto analysis?

2.  Why would I “save it for later” and “analyze it when you have time” when it could be something critical that requires immediate attention?  The assumption is that nothing here has a sense of urgency.

3.  Forget about real-time incident response, automated analytics, or integration with your SIEM or other existing security tools…not addressed.

4.  Your organization’s security staff still has to use NetWitness Investigator to satisfy the “analyze the traffic using any tool that analyzes PCAP files” requirement in order to use this vendor’s product — and that after finding both the haystack and the needle, which you would have found already had you been using NetWitness.

All this discussion highlights the value of working with NetWitness, an engineering company dedicated to solving the important problems of security professionals, law enforcement, intelligence analysts and other people focused on cyber security issues.  NetWitness offers a 10Gbps solution and it is running on some of the largest networks in the world — we’ve been doing it for a while — but we do it in a sensible way.  We do not go to market trying to sell you “disk write speeds” or “appliance capture rates” – that’s a waste of your money and should not be the most important focus for you.   Unlike anyone else in this space, we provide an infinitely extensible data framework, real-time automated analytics, live data fusion and threat intelligence, and the best network forensics interface in the market today.  Without all that, you are just filling up disk drives.