I spent a good amount of time this week speaking to customers, partners and prospects about deploying, engineering and using our products — one topic that always seems to be part of the discussion is system throughput and scalability.  Of course our position regarding this is clear, as NetWitness technology was designed from inception to support any combined throughput and can scale out as your network grows.  Inevitably the conversation dives deeply into why we say this…

For any network recording AND analysis technology there is an INPUT and OUTPUT to consider, I think everyone knows this.

INPUT – the concept of guaranteeing packet acquisition and writing to a storage structure with no loss as fast as possible – 1Gbps, 10Gbps, 40Gbps… and so on. The vast majority of vendors out there focus on and emphasize this extensively — this may be capture acceleration, stream-to-disk, or flow/header technology in high bandwidth environments.

OUTPUT – the concept of being able to access and analyze the captured data, deeply and across days, weeks or months of data quickly, ideally in real-time.  Most vendors minimize the importance of this, and often do a poor job of providing value with data spanning more than a few hundred mega-bytes at a time, and rarely address true security needs.

What is never discussed or exposed in the market is that these requirements are in constant contention when acting on network data within a single physical system.  Or, in other words, the more you are writing to a system the less you can read. Being sensitive to this reality since the first version of our product over 10 years ago, we designed a system that optionally separates these services, and scales out on hardware to meet any deployment condition. Ultimately providing high-speed capture, retention, and real-time access to deep analytics – true situational awareness of your network – it is what NetWitness does.

Recently, I did a webcast that goes into detail about how to architect NetWitness in these environments — I invite you to take a listen, you should find that when it comes to architecture and scalability, NetWitness is one of the few in our space that can actually deliver.

Brian Girardi – Director, Product Mangement

Over the past two months, there has been a tremendous amount of chatter in the security community about the term ‘cyberwar’ and whether or not the US is engaged in a cyberwar. Mike McConnell (former Director of National Intelligence) wrote a pointed op-ed for The Washington Post claiming that, “The United States is fighting a cyber-war today, and we are losing.” His opinions are consistent with the current Director of National Intelligence, Dennis Blair, whose February testimony to the US Senate stated, “Malicious cyber activity is occurring on an unprecedented scale with extraordinary sophistication. While both the threats and technologies associated with cyberspace are dynamic, the existing balance in network technology favors malicious actors, and is likely to continue to do so for the foreseeable future.”

These statements spurred an excoriating response from the pages of Wired that, “The biggest threat to the open internet is not Chinese government hackers or greedy anti-net-neutrality ISPs, it’s Michael McConnell, the former director of national intelligence.” At the annual RSA security conference Howard Schmidt, the newly appointed White House Cyber-Security Coordinator stated unequivocally that, “There is no cyberwar.” Nonetheless in a Washington Post article on March 19th 2010 Ellen Nakashima dramatically points out the need for clearer cyberwar policies by pointing to US cyber operations already executed and that cyber actions are underway.

Various cyberwar definitions are hotly contested, even more nuance-laden and have a very material impact on the dramatic claims one might make. Below are several observations about cyberspace upon which all well-informed parties agree:

Click here to read the full posting on The Firewall at Forbes.com

There was a significant amount of coverage yesterday on research performed by NetWitness into a large set of stolen information recovered from a ZeuS botnet.  Some of the information, analysis, and commentary was very beneficial to the broader discussion of threats such as these.  There is, however, some information that we feel we should address.

• Kneber is a pseudonym for ZeuS:

Kneber is not a pseudonym for ZeuS. Kneber refers to one group of organized criminals, one group of Command and Control Systems, and 74,000+ infected victim systems for this particular ZeuS (primarily) botnet.   ZeuS is a tool, used by many groups to create command and control systems, and steal information.  There are hundreds of active ZeuS botnets, many of which are larger than this one. It is but one of many tools used in this particular botnet.  We have seen INTENTIONAL cross pollination of various trojans, including waledec, grum, and even tools such as packet sniffers.  When we discuss threat, we are referring to more than the tool used, but the organization behind them.

• Kneber is “nothing new”:

We have been very clear that this is a medium sized infestation when compared with all the tracked ZeuS botnets on the Internet.   What does make this very valuable is the opportunity to analyze such a large sample of stolen information, and quantitatively add to the discussion of threats to corporate security.  The number of infected and active systems behind some of the largest, most technology savvy companies needs to be considered, and our approach to security needs to change given the broad failure to identify or remediate these infestations.   In addition, trivializing the damage done is simply disingenuous by anyone who has seen the types of data stolen from threats such as these.

• Current protections and solutions can detect this type of activity:

This quote from Symantec, via the Guardian, KrebsOnSecurity, and others:

“Kneber, in reality, is not a new threat at all, but is simply a pseudonym for the infamous and well-known Zeus Trojan,” said the company. “The name Kneber simply refers to a particular group, or herd, of zombie computers, a.k.a. bots, being controlled by one owner. The actual Trojan itself is the same Trojan.Zbot, which also goes by the name Zeus, which has been being observed, analyzed and protected against for some time now.”

This quote is particularly troubling, as it seems to minimize the threat and is almost dismissive.   Moreover, when this particular variant was analyzed in late January (various services used), Symantec did NOT detect this as malicious.   To be fair, McAfee, Trend Micro, AVG, and most other mainstream anti-virus solutions also failed to recognize this as malicious.  In the past 3 weeks, Symantec has added signatures to detect this particular variant as a generic “Trojan Horse”.  However, if you were infected by this particular strain, your system has already processed an update that prevents you from contacting Symantec and others for updates.   In most cases, this will prevent future detection.   Worse, as part of normal operation of ZeuS, it attaches to running processes on victim systems in order to monitor them.   This data is logged along with other stolen information.   This set of data shows that ZeuS has actually attached to running versions of Symantec software on over a thousand victim systems.  Many other AV vendors are also present.

This example shows ZeuS monitoring a Symantec Live Update, and includes the ftp username and password used by the Symantec software during the update process.

• Are the facts overstated?:

The facts are fairly succinct in the whitepaper that we released.   We do not believe the threat is over-stated, and we were very conservative on the analysis released.   There are likely thousands of additional corporate networks affected, and analysis of this much information takes time.   And this is simply one of many similar operations in existence.  The group behind this effort can be described as sophisticated, yet also shows signs of lax effort to hide their trails.   The botnet is very actively managed, and continues in operation today.   The fact that they have been in successful operation for over 18 months also has to be considered.   We have also received several additional data points from federal contacts with additional insight into related government focused attacks.

More to come.

Tim Belcher and Alex Cox

While the world took pause to consider the implications of Operation Aurora, and Google lent considerable voice to the concept of Advanced and Persistent Threats (APT), we can ill-afford to believe even for a moment that they are alone in their sophistication or capability.   According to the FBI more than 100 nations have offensive cyber operations as part of their intelligence or national security fabric.  And the same attributes that make the Internet ideal for covert intelligence gathering make it attractive for corporate espionage and organized criminal activity.  The IT security industry commonly refers to the online activities of Eastern European and Asian organized crime as “cyber criminals” or “gangs”, which in many ways only serves to minimize the attention they deserve.  In truth the online operations of some organized crime syndicates are every bit as sophisticated, advanced and persistent as their nation-state counterparts.  They are truly expert at gaining footholds and siphoning off critical information.  And they are FAR more pervasive than Operation Aurora.

In late January, NetWitness security research were able to gain visibility into a large scale ZeuS-based botnet, taking user credentials and confidential information from thousands of organizations around the world (See The Wall Street Journal article).  Some of the information collected has been synthesized in the Kneber Bot whitepaper that you can dowload from the NetWitness website.

The sheer volume of information gathered and has forced us to reconsider the common belief that this very successful botnet is simply “financial services” related.  In fact, this particular botnet was much more concerned with culling account and network access credentials, as well as collecting as much detail about victim identities as possible.  In effect, they were less concerned with accessing any particular account, than being able to access ALL accounts related to a victim.

As we began analyzing victim information, we rapidly formed a picture of thousands of corporate compromises around the globe.  As with Aurora, many of the largest most technology savvy companies had internal compromised hosts, which had culled various corporate user level and administrative credentials.

We may attempt to broadly classify threats in the security industry, in hopes that we can make the complex more digestible to management.  However, classifying threats like this as “banking trojans” may do a large disservice to the victim companies.  Indeed, there are many that broadly dismiss threats such as these as “unsophisticated”, or less advanced than a pinpoint attack like Operation Aurora.  Rest assured, these adversaries could not care less how we classify their work.  They are well organized, have demonstrated technical sophistication on par with many intelligence services and do not forgo the opportunity for financial gain with the the information they collect.  If they are collecting network credentials, it means they are using or selling them in an active underground economy – which may include sponsoring foreign intelligence services. What is easier? Designing a campaign like Operation Aurora, or simply purchasing access to your target companies?

We are working with federal law enforcement, and continue in our efforts to notify victim organizations.  Please feel free to download our white paper and I am confident we will discuss great detail in future blog posts.

It’s a little known fact that NetWitness has been innovating in the security field for over 11 years, which was further validated by the announcement of our recently granted US Patent # 7,634,557. Clearly, when it comes to network analysis we do it better than anyone else, and it’s really the only way to get the results you need.

Reaching back over a decade (ca.1999) when our first patent was filed, ( US Patent # 7,016,951 ), and murmurs of network forensics were swirling from a few experts in the security community, our innovation in this field was in full swing.  The technology was chartered as an analytical application to make sense of network traffic for users with no networking experience.  This in itself was no small task, as I cannot emphasize how difficult it was explaining what an IP address was to an English major. See the snapshot of NetWitness v3.5 ca. 2002, ironically it looks like some our 2010 competition.

In retrospect, NetWitness was conceived in a reverse direction from how most security products end up being developed.  Our strategy was to understand the data FIRST, then figure out how to capture it and scale it reliably into an enterprise.  Honestly, we spent several years trying to determine the best way to present complex network data to our users, which at that time was simple HTTP and SMTP sessions.  We had no idea how the network application profile of an Enterprise would evolve to what it is today.   With that said, we made sure that the advanced methods we developed were flexible enough to evolve with the Internet and the needs of our users.  These methods found their way into these two patents.

The first and most important patent is a method for traffic capture, session reassembly, metadata extraction and recursive port-agnostic service identification. Did you get all that?  Back when Firewall and IDS were tinkering with port numbers for rule logic, NetWitness was beyond that approach over 10 years ago.  The assumption to classify network traffic by port alone is prone to mistakes for reliable security analysis. It was not until recently there was a prominent increase in products that are, or at least market port agnostic support, like application firewalls and some DLP products.

The second patent, the topic of this announcement, extends the core technology by defining a system and method for organizing and describing the traffic we collect.  Yet again an example of how we designed the technology to evolve as the Internet evolved.   The patent specifically focuses on the session data model and structures that fuel the Investigator interface and the user experience.  The result is the most visible difference between NetWitness and our competitors, as well as what provides the analytical value when responding to <INSERT NETWORK PROBLEM HERE>. Another example of the product evolution can be seen in the screenshot below of NetWitness v5 ca. 2004.

Its always been my assertion that to do true network forensics, or really any good network analysis, you need a few key ingredients:

1) Reliable, scalable, and forensically sound network capture.  Unfortunately the vast majority of “network forensic” vendors stop HERE!

2) As you would expect from any forensic science, the technical ability to piece the clues or segments of an event back together is the next logical step. For network forensics its assembling the packets back into full sessions, because without this step you have disparate puzzle pieces, without a complete picture.

3) Then finally the right tools to analyze, correlate, mine and report the findings to humans. Thankfully there is an NetWitness App for that and a free API/SDK too.

These elements combined are the foundation of what NetWitness NextGen is, and the basis of our technology that is truly becoming a game-changer in security.  NetWitness Corporation was founded in late 2006, but unknown to many, the innovation and pioneering environment that fuels the technology today started 10 years earlier.  Enjoy our innovation by using Investigator Freeware, and know that before the security challenges of today really materialized we were hard at work creating solutions for today. Network security products that simply work.

Cheers,

NetWitness v9, ca. 2010.

I was helping a fortune customer yesterday determine if they were targeted by Operation Aurora. From everything we know to date, they were not. How do we know this? We looked. In 15 minutes or so, we looked back over the last 6 months of every bit and byte that has left that company, and compared every hostname, IP address, and HTTP URL that have been associated with these attacks. This is the power of full network surveillance, and this is why you MUST be performing real-time continual deep analysis of your network activities.

There is a discussion today of some of the malware, and zero day exploits out of McAfee. They are now calling this Operation “Aurora”. In the post, George Kurtz discusses how APT, or Advanced Persistent Threats is changing the security landscape once again. It is a message, and the discussion we at NetWitness have been pushing for years. While this attack has gone largely unnoticed for months, NetWitness customers have all the historic evidence necessary to assess damage. For some, this means gaining rapid confidence that they were not compromised. For some, it means rapid damage assessment, capturing evidence, and using everything they learn to increase their security today. As I watch the best security teams in the world struggle to collect evidence of this attack over the course of days or weeks, I cannot help but wonder how much easier it would have been had we been in place.

In early December, we were called into one of the affected companies, in partnership with a large service provider. Within days, we had NetWitness gear recording at every major gateway in the country, and were scheduling international deployments. While it appears that the damage was already done to this company long before we arrived, we were instrumental in shutting down many other infestations, as well as identifying hundreds of systems that were displaying abnormal or concerning communication patterns. Had this company been a NetWitness client only 30 or 60 days before, I am absolutely confident we would have been able to bring this particular activity to light weeks earlier.

We have been reaching out to our customers, providing them details of the communication that Operation Aurora utilized, along with very simple instructions that allow them to look back over time and reassess their security. One thing is certain, while we may not know the vector of a specific attack until after the fact, it is imperative that we have the ability to quickly assess the damage and retain evidence.

This is why we must begin recording our network activity NOW. Giving your network some form of memory is an absolute imperative, and the foremost defense against APT. Furthermore, simple recording is not enough. Good luck if your recording architecture is IP based. Customers of NetWitness can search the URLs, hostnames, and other application level beaconing activities in seconds. Try doing this by scanning over 50 terabytes of packet data manually. Had this attack employed more sophisticated hosting or resolution techniques like fast flux, and even the IP addresses would have been useless.

George is absolutely correct in his assessment that the landscape has changed. These types of organized, sponsored attacks are here to stay. I am sure those in the financial community, used to dealing with advanced ACH fraud and highly targeted attacks by the Russian Business Network are sitting back this morning and saying “Welcome to the party, pal!”

Welcome to the party, pal!

I was at the CSI show yesterday and was within earshot of one of our “competitors” who claimed that they were winning against NetWitness because they support 10Gbps and we do not.   I have heard this story frequently from this particular firm, and it’s a bunch of bull.

It amazes me that companies in this space, such as Niksun and Solera Networks, spend so much time emphasizing how they allegedly can capture at 10Gbps line rates – as if that’s the most important requirement for public and private organizations struggling to cope with critical advanced threats, complex data leakage scenarios, network forensics, designer malware and botnet infestations, and increasing insider crime and fraud.

Solera Networks publicly asserts that their product was certified by Miercom Labs as 10Gbps capable.  If you look at the report on their Website, however (http://www.soleranetworks.com/resources/Solera%20DS5100_Miercom%20test.pdf), you will see that a top capture rate of 8.1 Gbps was achieved solely when the “packet size” was forced to 1,518 bytes.  At other packet sizes, the performance dropped off steadily.

The practical application of the Miercom report for Solera Networks is dubious in the real world.  For example, let’s assume that in Miercom’s vernacular “packet size” is equal to “message transmission unit” (MTU).  1,518 is the maximum MTU for the transmission of data over IEEE 802 networks according to RFC 1042.  But it is unrealistic to imagine that every consecutive packet on a customer network would be 1,518.  In fact, the typical default MTU setting for devices such as routers and servers processing a protocol such as HTTP over TCP/IP is 576 – most network and system administrators today work under this assumption.  In a real customer environment with a large amount of HTTP traffic, the Miercom numbers would put the theoretical Solera throughput somewhere around 6Gbps, versus the claimed 10Gbps.  Real life is different than the lab, however, and the reality is that customer application-layer traffic produces actual average MTUs of far less than 576, thereby lowering the potential performance results below the assertions made by some vendors to something more like an average MTU of 300 — or a throughput of around 3Gbps according to the Miercom results.

Such misleading lab reports also do not address other concerns, such as the technical challenges associated with capturing at a 10Gbps on single network appliance, given physical bus bandwidth constraints and disk write speed limitations — and still offering meaningful and timely analytics to security users.  Every vendor who is engineering solutions in this space has confronted this dynamic — but most vendors do not address this problem in their marketecture because they do not have a solution.

As the consumer of solutions in this space, you should be aware of this:   The reason this issue is not discussed in any meaningful way by some vendors is because they have no real-time automated and interactive analytical capability beyond basic and often erroneous network statistics.

Consider this screenshot from Solera Networks post-facto user console:

Notice specifically the port assumptions made by the product:

To assume in 2009 that TCP ports 80 and 443 are inclusive to web traffic simply is ridiculous.  Not only is this type of analysis absurd from a network forensics perspective, but also would seem at odds with the term “deep packet inspection”.  Consider the following drill into the HTTP service using NetWitness Investigator, on even a single day of capture from the NetWitness corporate HQ:

This screenshot represents only a small portion of the available information about the HTTP protocol.  This level of detail is possible because NetWitness does complete port agnostic session analysis in an automated real-time manner, upon collection.  You cannot get there from here with other vendors like Niksun and Solera Networks.  One of the vendor’s Websites says it most succinctly:

“Once you find the traffic flow you are looking for, you can download a PCAP file of just that data and analyze the traffic using any tool that analyzes PCAP files. Or you can save it for later and use it to analyze when you have time or need evidentiary proof of malicious activity.”

There are some pretty big and unfortunate “IF”s that customers should consider before engaging with any vendor operating under such assumptions:

1.  How do I actually go about “finding the traffic you are looking for” within tens or perhaps hundreds of terabytes of data in post-facto analysis?

2.  Why would I “save it for later” and “analyze it when you have time” when it could be something critical that requires immediate attention?  The assumption is that nothing here has a sense of urgency.

3.  Forget about real-time incident response, automated analytics, or integration with your SIEM or other existing security tools…not addressed.

4.  Your organization’s security staff still has to use NetWitness Investigator to satisfy the “analyze the traffic using any tool that analyzes PCAP files” requirement in order to use this vendor’s product — and that after finding both the haystack and the needle, which you would have found already had you been using NetWitness.

All this discussion highlights the value of working with NetWitness, an engineering company dedicated to solving the important problems of security professionals, law enforcement, intelligence analysts and other people focused on cyber security issues.  NetWitness offers a 10Gbps solution and it is running on some of the largest networks in the world — we’ve been doing it for a while — but we do it in a sensible way.  We do not go to market trying to sell you “disk write speeds” or “appliance capture rates” – that’s a waste of your money and should not be the most important focus for you.   Unlike anyone else in this space, we provide an infinitely extensible data framework, real-time automated analytics, live data fusion and threat intelligence, and the best network forensics interface in the market today.  Without all that, you are just filling up disk drives.