One of the lesser-known features that was recently introduced in NetWitness Investigator are Custom Actions.   Have you ever been analyzing a pcap in Investigator and thought “I wish there was an easy way to push this information into another system…”.   Custom Actions is a flexible extension system that will allow you to do just that.    Here are just a few examples of what can be accomplished:

Scenario:   You are a senior incident responder at your organization, and one of your company’s help desk analysts reports strange behavior from an executive’s workstation.   Luckily, this analyst is on his game and has provided you with a network capture from the workstation, which you import into Investigator for analysis.  What are some tasks that I might want to do with this investigation?

Once you are here, you’ll see a GUI overview of the custom action system with a few examples. You may have a few in the list that were installed by default.

cmd.exe /K c:\PROGRA~1\nmap\nmap.exe -PN -sV –top-ports 1000 192.168.2.245

cmd.exe /K c:\PROGRA~1\nmap\nmap.exe -PN -sV –top-ports 1000 ${VALUE}

http://www.google.com/search?q=dog

http://www.google.com/search?q=${VALUE}

For Task 3, we are going to use the same basic concept to search URLvoid.com for a domain’s presence on blacklists.  Just like Google, we’ll start with a search on URLvoid:

http://www.urlvoid.com/scan/cnn.com

http://www.urlvoid.com/scan/${VALUE}

Once these have been added, we can then use them to quickly investigate our scenario:

Using the nmap scan custom action, we find that our executive’s workstation at 192.168.2.246 is listening only on ports that it should be per our company’s security policy. We see that our Google custom action reveals that the involved filenames are related to a known ZeuS trojan infection, and we know by our URLvoid results that this command and control server now appears to be down.     This PC is compromised, so we then declare an incident and proceed with our incident response plans.

As you can see, custom actions are very easy to implement.  These are simple examples, but with some effort, tools, a favorite scripting language, etc., the sky is the limit.  You could do things like:

Do you have a good idea or example for a custom action?  Post about it in the NetWitness Community, we’d love to hear about it!

Happy Hunting!

I spent a good amount of time this week speaking to customers, partners and prospects about deploying, engineering and using our products — one topic that always seems to be part of the discussion is system throughput and scalability.  Of course our position regarding this is clear, as NetWitness technology was designed from inception to support any combined throughput and can scale out as your network grows.  Inevitably the conversation dives deeply into why we say this…

For any network recording AND analysis technology there is an INPUT and OUTPUT to consider, I think everyone knows this.

INPUT – the concept of guaranteeing packet acquisition and writing to a storage structure with no loss as fast as possible – 1Gbps, 10Gbps, 40Gbps… and so on. The vast majority of vendors out there focus on and emphasize this extensively — this may be capture acceleration, stream-to-disk, or flow/header technology in high bandwidth environments.

OUTPUT – the concept of being able to access and analyze the captured data, deeply and across days, weeks or months of data quickly, ideally in real-time.  Most vendors minimize the importance of this, and often do a poor job of providing value with data spanning more than a few hundred mega-bytes at a time, and rarely address true security needs.

What is never discussed or exposed in the market is that these requirements are in constant contention when acting on network data within a single physical system.  Or, in other words, the more you are writing to a system the less you can read. Being sensitive to this reality since the first version of our product over 10 years ago, we designed a system that optionally separates these services, and scales out on hardware to meet any deployment condition. Ultimately providing high-speed capture, retention, and real-time access to deep analytics – true situational awareness of your network – it is what NetWitness does.

Recently, I did a webcast that goes into detail about how to architect NetWitness in these environments — I invite you to take a listen, you should find that when it comes to architecture and scalability, NetWitness is one of the few in our space that can actually deliver.

Brian Girardi – Director, Product Mangement

There was a significant amount of coverage yesterday on research performed by NetWitness into a large set of stolen information recovered from a ZeuS botnet.  Some of the information, analysis, and commentary was very beneficial to the broader discussion of threats such as these.  There is, however, some information that we feel we should address.

• Kneber is a pseudonym for ZeuS:

Kneber is not a pseudonym for ZeuS. Kneber refers to one group of organized criminals, one group of Command and Control Systems, and 74,000+ infected victim systems for this particular ZeuS (primarily) botnet.   ZeuS is a tool, used by many groups to create command and control systems, and steal information.  There are hundreds of active ZeuS botnets, many of which are larger than this one. It is but one of many tools used in this particular botnet.  We have seen INTENTIONAL cross pollination of various trojans, including waledec, grum, and even tools such as packet sniffers.  When we discuss threat, we are referring to more than the tool used, but the organization behind them.

• Kneber is “nothing new”:

We have been very clear that this is a medium sized infestation when compared with all the tracked ZeuS botnets on the Internet.   What does make this very valuable is the opportunity to analyze such a large sample of stolen information, and quantitatively add to the discussion of threats to corporate security.  The number of infected and active systems behind some of the largest, most technology savvy companies needs to be considered, and our approach to security needs to change given the broad failure to identify or remediate these infestations.   In addition, trivializing the damage done is simply disingenuous by anyone who has seen the types of data stolen from threats such as these.

• Current protections and solutions can detect this type of activity:

This quote from Symantec, via the Guardian, KrebsOnSecurity, and others:

“Kneber, in reality, is not a new threat at all, but is simply a pseudonym for the infamous and well-known Zeus Trojan,” said the company. “The name Kneber simply refers to a particular group, or herd, of zombie computers, a.k.a. bots, being controlled by one owner. The actual Trojan itself is the same Trojan.Zbot, which also goes by the name Zeus, which has been being observed, analyzed and protected against for some time now.”

This quote is particularly troubling, as it seems to minimize the threat and is almost dismissive.   Moreover, when this particular variant was analyzed in late January (various services used), Symantec did NOT detect this as malicious.   To be fair, McAfee, Trend Micro, AVG, and most other mainstream anti-virus solutions also failed to recognize this as malicious.  In the past 3 weeks, Symantec has added signatures to detect this particular variant as a generic “Trojan Horse”.  However, if you were infected by this particular strain, your system has already processed an update that prevents you from contacting Symantec and others for updates.   In most cases, this will prevent future detection.   Worse, as part of normal operation of ZeuS, it attaches to running processes on victim systems in order to monitor them.   This data is logged along with other stolen information.   This set of data shows that ZeuS has actually attached to running versions of Symantec software on over a thousand victim systems.  Many other AV vendors are also present.

This example shows ZeuS monitoring a Symantec Live Update, and includes the ftp username and password used by the Symantec software during the update process.

• Are the facts overstated?:

The facts are fairly succinct in the whitepaper that we released.   We do not believe the threat is over-stated, and we were very conservative on the analysis released.   There are likely thousands of additional corporate networks affected, and analysis of this much information takes time.   And this is simply one of many similar operations in existence.  The group behind this effort can be described as sophisticated, yet also shows signs of lax effort to hide their trails.   The botnet is very actively managed, and continues in operation today.   The fact that they have been in successful operation for over 18 months also has to be considered.   We have also received several additional data points from federal contacts with additional insight into related government focused attacks.

More to come.

Tim Belcher and Alex Cox

While the world took pause to consider the implications of Operation Aurora, and Google lent considerable voice to the concept of Advanced and Persistent Threats (APT), we can ill-afford to believe even for a moment that they are alone in their sophistication or capability.   According to the FBI more than 100 nations have offensive cyber operations as part of their intelligence or national security fabric.  And the same attributes that make the Internet ideal for covert intelligence gathering make it attractive for corporate espionage and organized criminal activity.  The IT security industry commonly refers to the online activities of Eastern European and Asian organized crime as “cyber criminals” or “gangs”, which in many ways only serves to minimize the attention they deserve.  In truth the online operations of some organized crime syndicates are every bit as sophisticated, advanced and persistent as their nation-state counterparts.  They are truly expert at gaining footholds and siphoning off critical information.  And they are FAR more pervasive than Operation Aurora.

In late January, NetWitness security research were able to gain visibility into a large scale ZeuS-based botnet, taking user credentials and confidential information from thousands of organizations around the world (See The Wall Street Journal article).  Some of the information collected has been synthesized in the Kneber Bot whitepaper that you can dowload from the NetWitness website.

The sheer volume of information gathered and has forced us to reconsider the common belief that this very successful botnet is simply “financial services” related.  In fact, this particular botnet was much more concerned with culling account and network access credentials, as well as collecting as much detail about victim identities as possible.  In effect, they were less concerned with accessing any particular account, than being able to access ALL accounts related to a victim.

As we began analyzing victim information, we rapidly formed a picture of thousands of corporate compromises around the globe.  As with Aurora, many of the largest most technology savvy companies had internal compromised hosts, which had culled various corporate user level and administrative credentials.

We may attempt to broadly classify threats in the security industry, in hopes that we can make the complex more digestible to management.  However, classifying threats like this as “banking trojans” may do a large disservice to the victim companies.  Indeed, there are many that broadly dismiss threats such as these as “unsophisticated”, or less advanced than a pinpoint attack like Operation Aurora.  Rest assured, these adversaries could not care less how we classify their work.  They are well organized, have demonstrated technical sophistication on par with many intelligence services and do not forgo the opportunity for financial gain with the the information they collect.  If they are collecting network credentials, it means they are using or selling them in an active underground economy – which may include sponsoring foreign intelligence services. What is easier? Designing a campaign like Operation Aurora, or simply purchasing access to your target companies?

We are working with federal law enforcement, and continue in our efforts to notify victim organizations.  Please feel free to download our white paper and I am confident we will discuss great detail in future blog posts.

It’s a little known fact that NetWitness has been innovating in the security field for over 11 years, which was further validated by the announcement of our recently granted US Patent # 7,634,557. Clearly, when it comes to network analysis we do it better than anyone else, and it’s really the only way to get the results you need.

Reaching back over a decade (ca.1999) when our first patent was filed, ( US Patent # 7,016,951 ), and murmurs of network forensics were swirling from a few experts in the security community, our innovation in this field was in full swing.  The technology was chartered as an analytical application to make sense of network traffic for users with no networking experience.  This in itself was no small task, as I cannot emphasize how difficult it was explaining what an IP address was to an English major. See the snapshot of NetWitness v3.5 ca. 2002, ironically it looks like some our 2010 competition.

In retrospect, NetWitness was conceived in a reverse direction from how most security products end up being developed.  Our strategy was to understand the data FIRST, then figure out how to capture it and scale it reliably into an enterprise.  Honestly, we spent several years trying to determine the best way to present complex network data to our users, which at that time was simple HTTP and SMTP sessions.  We had no idea how the network application profile of an Enterprise would evolve to what it is today.   With that said, we made sure that the advanced methods we developed were flexible enough to evolve with the Internet and the needs of our users.  These methods found their way into these two patents.

The first and most important patent is a method for traffic capture, session reassembly, metadata extraction and recursive port-agnostic service identification. Did you get all that?  Back when Firewall and IDS were tinkering with port numbers for rule logic, NetWitness was beyond that approach over 10 years ago.  The assumption to classify network traffic by port alone is prone to mistakes for reliable security analysis. It was not until recently there was a prominent increase in products that are, or at least market port agnostic support, like application firewalls and some DLP products.

The second patent, the topic of this announcement, extends the core technology by defining a system and method for organizing and describing the traffic we collect.  Yet again an example of how we designed the technology to evolve as the Internet evolved.   The patent specifically focuses on the session data model and structures that fuel the Investigator interface and the user experience.  The result is the most visible difference between NetWitness and our competitors, as well as what provides the analytical value when responding to <INSERT NETWORK PROBLEM HERE>. Another example of the product evolution can be seen in the screenshot below of NetWitness v5 ca. 2004.

Its always been my assertion that to do true network forensics, or really any good network analysis, you need a few key ingredients:

1) Reliable, scalable, and forensically sound network capture.  Unfortunately the vast majority of “network forensic” vendors stop HERE!

2) As you would expect from any forensic science, the technical ability to piece the clues or segments of an event back together is the next logical step. For network forensics its assembling the packets back into full sessions, because without this step you have disparate puzzle pieces, without a complete picture.

3) Then finally the right tools to analyze, correlate, mine and report the findings to humans. Thankfully there is an NetWitness App for that and a free API/SDK too.

These elements combined are the foundation of what NetWitness NextGen is, and the basis of our technology that is truly becoming a game-changer in security.  NetWitness Corporation was founded in late 2006, but unknown to many, the innovation and pioneering environment that fuels the technology today started 10 years earlier.  Enjoy our innovation by using Investigator Freeware, and know that before the security challenges of today really materialized we were hard at work creating solutions for today. Network security products that simply work.

Cheers,

NetWitness v9, ca. 2010.

We recently sent an opt-in email to our contact database talking about the significance of Operation Aurora and the continued ascendancy and lack of advanced threat prevention/detection in many government and commercial organizations.  We also offered a NetWitness proof-of-concept (POC) to security folks concerned about this issue.  And security people should be concerned.

A noted security blogger correctly observed that we were “amplifying FUD” in our email blast to get people’s attention.   His blog post raises a classic issue facing security professionals – does FUD help bring an issue like this to top of mind.  Or:  To FUD, or not to FUD.

It’s really unfortunate that FUD became a dirty word when compliance and “risk management” took over the security budget, but that’s when many organizations, began to fail at security too.  While many people, particularly some CIOs in hindsight, would argue that compliance has helped increase the focus and spending on information security, I would argue that it has distracted many security programs into performing a large number of basically low impact or worthless activities in the name of metrics, versus FUD.  And, compliance certainly has sponsored a whole class of expensive security technologies and related total ownership costs (TCO) which drain the security budget.

There’s also an unfortunate psychology involved here.  Many security professionals feel guilty or inadequate using FUD as an argument because all the other I/T people have real metrics and we don’t.  To some, it’s like when we were kids and everyone had Converse “Chuck Taylor” All-Star high tops and you were the one with the red Pro-Keds.  Security people can’t talk about how many “9’s” of network uptime we have, or how much we have improved call center response time, or improved the total cost per terabyte of enterprise storage.   Security sucks at producing decent metrics — and the ones we do produce, generally stink even more at reducing the fear of being owned by national-sponsored or organized criminal groups or the uncertainty and the doubts regarding the security of information in a world of advanced threats.  Security people cringe when some C-level executive compares the cost of information security to the cost of insurance – “No one likes to pay for it, but just like your car insurance, you have to have it.”  Ugh!  So, we hate the FUD argument – both when we have to use it as an argument, or when someone uses it to trivialize what we all do for a living.

But I do not think security professionals should feel this way.  I think that FUD still has a lot of usefulness in the toolkit of the security professional and within the enterprise security program, if applied in the right doses to the right places.  One of my favorite Websites is fudsec.com.  There are many good, bad and ugly uses of FUD cited here, for example, one of the good ones is Anton Chuvakin’s post, “A Treatise on FUD” – required reading for any committed FUDists.

With regard to advanced threats and other types of network visibility problems, I encourage the use of a combination of FUD and proof.  The FUD comes in the form of security professionals updating their discussion track to highlight the real causes of many cyber losses in 2010, and the need for more focus on threat intelligence and operational security versus other types of spending.  Current issues such as Operation Aurora should be analyzed for relevance, and briefed to senior management, and should be coupled one of the more credible surveys that show that most data losses result from advanced threat or sophisticated exploit/malware sources.

In the end, you will have to produce real evidence, however, and that’s why we put the POC offer on the table in our e-mail blast.   FUD should only go so far — you should show your colleagues the smoking gun with your own organization’s data.   We as a vendor could put out all the FUD-sounding marketing statistics in the world about how our approach will make you more effective at changing the face of FUD to a smile than other alternatives, but you will only believe it when it produces results in your organization, you can bank those results, and it actually reduce the FUD for yourself and your CEO.   This is how it should be.