Knowledge of what’s really happening on your network is critical if you are responsible for the protection of your organization’s information assets.  Depending upon where you work and what you believe about both the capabilities of your security team and those of the adversary, you live somewhere on the spectrum of “really concerned about advanced threats” to feeling that “things are just about A-OK. ‘

NetWitness recently sponsored a study by the Ponemon Institute regarding the prevalence and awareness of advanced threats by security practitioners.  There have been many studies and reports over the last two years claiming that most successful data breaches result from “advanced threats” or “sophisticated attacks,” so we wanted to understand exactly what security people believed was happening in their organizations today and how they were coping with it.

A security blogger tweeted people to stay away from the study because it did not “hit the mark.”  Reason?  In a moment or two of weakness, the study used the revered term “advanced persistent threat (APT)” versus “advanced threat.”  Unfortunately, many security practitioners today cannot precisely define the difference.  Use of terminology within the study should not reduce the utility of the paper, however, especially if the end result of either advanced threat vector contains tangible risk to the corporation or government agency that might be mitigated through a better understanding of the network traffic.  Security people and vendors will get the terminology right at some point.

The other issue raised regarding the study concerned the number of respondents who would include terms such as “SQL Injection” in their definition of an advanced threat (“What other terms are used to describe an advanced threat?”).  Actually, the blogger missed the point of this question – the point was not to claim each response actually was an advanced threat, but to illustrate the relationship between common problems that security practitioners believe to be advanced in nature, and those that are simply evading their detection or mitigation capabilities.  Other questions in the study go down this path of lack of awareness.

Compared to other risk-based industries, the security industry is bereft of adequate pan-industry historical data, meaningful metrics, and comparative information.  Although imperfect as a first survey instrument, rather than ignore professional surveys such as the Ponemon Institute study, it should be used by security practitioners in an appropriate context:

1.  Prepare a non-FUD-based discussion for senior management regarding the characteristics of the current state of the threat environment.

2.  Bring forward studies from reputable national/international-level firms that illustrate the costs of data breaches, the sources and methods of these threat vectors, and now with Ponemon data, the opinions of 600 fellow security practitioners regarding the technical and administrative readiness of peer organizations to cope with the threats they are facing.

3.  Develop real evidence of what’s really happening in your world using your own corporate data.  For example, conduct a proof of concept of NetWitness and get definitive answers to some of the nagging questions you may have about advanced threat prevalence or insider threat concerns within your own I/T environment.

4.  Present a people, process and technology plan for reducing the uncertainty around advanced threats (and even APTs depending upon where you work).

If you’ve ever seen me, or any of the NetWitness crew, speak on malware, advanced threats or the current threat environment, you’ll generally hear more than one recurring theme, one of which is:

Your anti-virus solution isn’t working like you think it is.

This is occurring for a variety of reasons and is ultimately the result of a business-based exploitation cycle in the criminal underground.   This cycle includes software support, licensing, and ongoing quality assurance.  One of the best examples I’ve ever seen to illustrate this concept is in the case of “scan4u.biz”.

Brian Krebs posted about this particular cybercrime endeavor in his blog here a few months ago:

http://krebsonsecurity.com/tag/scan4u-biz/

However, recent intelligence gathering efforts have revealed that this particular business venture has been extended and improved using the same resilience concepts used in most large legitimate corporate infrastructures.

A brief overview of “scan4u.biz”

Scan4u.biz is essentially  a “criminal virustotal plus”.  That is, it is a service where a miscreant can submit a newly created malware binary to gauge the detection rate of various antivirus vendors.  While similar to virustotal in this regard, the key is that scanned binaries aren’t submitted to the antivirus vendors in question, as is done with virustotal.

Let’s surf the service for examples:

What we see here is a general overview of the service (translated from russian) with the following key points:

• The service doesn’t submit to anti-virus vendors.
• Antivirus clients are updated hourly to maintain a current definition set
• Submitted binaries are rechecked on a schedule and customers are emailed about new detections

Digging deeper we see an example of the current signature state of included antivirus engines, which includes the vendor name, signature update version number and last update time:

And it’s even affordable and easy to pay for…$25 a month or 15 cents per scan, and a discount for referrals.  As well as flexible payment options and multiple contact points (I’ve blocked the specifics out):

How long has this service been running?

“News” updates indicate that this service has been running since at least October of 2009 and is being consistently upgrade and maintained:

How do we kill it?

So to take this down, we’d just get the domain name suspended right?   Well..it appears that that has already been done as is evident with a quick dig:

Not found: scan4u.biz

>>>> Whois database was last updated on: Sun May 30 14:07:49 GMT 2010 <<<<

So how is it still accessible?

At this moment, this service is being hosted or proxied through a criminal infrastructure, known in the industry as Gumblar.  Gumblar was recently referenced in a large scale compromise of blogs at most major hosting companies and has been an ongoing presence in the malware world for the past few  years.   At last check, the infrastructure has at least 376 verified domains, mostly in the .ru tld, across at least 43 different IPs in geographically disperse locations.

This hosting model is, in effect, a content distribution network, as used by most major online presences.  In this case, it’s being used to both hide the miscreants actual operating location, as well as provide fault tolerance from ongoing takedown efforts by the security community.

Extending beyond antivirus checks

As well as antivirus checks, the miscreants running the service appear to have extended their checks into the online blacklist area:

“Domain check on presence in black list: ZeuS domain blocklist, ZeuS IP blocklist, ZeuS Tracker, MalwareDomainList (MDL), Google Safe Browsing (FireFox), PhishTank (Opera, WOT, Yahoo! Mail), hpHosts, SPAMHAUS SBL, SPAMHAUS PBL, SPAMHAUS XBL, MalwareUrl,SmartScreen (IE7/IE8 malware & phishing Web site),Norton Safe Web, Panda Antivirus 2010, (Firefox Phishing and Malware Protection), SpamCop.net and RFC-Ignorant.Org.”

This update indicates ongoing blacklist checks across a variety of services, including:

• Security researcher and community published blacklists (zeustracker, malwaredomainlist,malwareurl,phishtank,spamhaus)
• Browser-based anti-phishing technology (google safe browsing,smartscreen)
• Vendor blacklists (Norton, Panda, etc)

So in essence, miscreants using this service have a one-stop shop for both the detection of malicious binaries as well as the existence of their delivery systems in disparate blacklists across the internet.

They also understand researcher and malware analysis activity:

“Add ability to get execution result of find/pdfid/pefile/trid utility (“Save file on server” option must be on)”

• PDFID is Didier Steven’s excellent PDF analysis tool.
• PEFILE  is a python module used to assist in reverse engineering binaries to detect packing and other indicators of maliciousness.
• TRID is a tool used to identify files from their binary signatures.

What all of this should tell you is that criminal miscreants continue to upgrade and enhance their services to assist in perpetuating their business model, penetrate your networks, and make money!

Watch your network, because they certainly are!

Alex Cox, Principal Research Analyst

Overview

This technique can detect overflow exploits against software running on the x86 platform, meaning it applies to Windows, Unix, and Mac shellcode. It not only works independently of OS, but it also works for finding both stack and heap based overflows. Most interestingly, it catches most forms of polymorphic shellcode as well. (Actually, it exceeds at finding special shellcodes like polymorphic decryption engines, egg hunters, etc.)  While this definitely doesn’t work for all shellcode, it works for a lot of it.

The reason this technique applies to any operating system on x86 is simple. Shellcode is typically written in machine code (commonly called assembly, although it’s not actually the same thing), meaning shellcode is written using processor instructions – something independent of the OS it’s running on. Of course, the entire purpose of shellcode is manipulation of the OS, so shellcode is ultimately OS specific (even patch specific), but its basic primitives are independent of the OS.

One classic problem with shellcoding is addressing. Because shellcode is [typically] nefariously injected via exploitation into a process’s memory segment, and program execution is “hijacked” (without the benefit of setting up proper address pointers), the coder doesn’t know where in memory their code will be. The problem is, very little can be accomplished without knowing the logical memory address of parameters within the shellcode.

The simplest way around this issue is use of a CALL instruction. More information is available in the “Intel 64 and IA-32 Architectures Software Developer’s Manual Volume 2A: Instruction Set Reference, A-M” (and 2B: N-Z) located here: http://www.intel.com/products/processor/manuals/.

The CALL is used as a way to branch processor execution to another location in memory. It has the minor benefit of being able to use relative addressing, but it has the major benefit of PUSH’ing procedure linking information on the stack before branching to the target location. This is commonly referred to as Call Stack Setup. When executing a near call, the processor pushes the value of the EIP register (which contains the offset of the instruction following the CALL instruction) on the stack (for use later as a return-instruction pointer). The processor then branches to the address in the current code segment specified by the target operand.

There are several versions of the CALL instruction, but the one we’re interested in for this purpose is opcode 0xE8. This is a near call (near, meaning within the current memory segment) using relative address displacement with a negative offset (eg: backwards displacement). The actual instruction is 5 bytes long, with the last four bytes used for a relative offset (a signed displacement relative to the current value of the instruction pointer in the EIP register; this value points to the instruction following the CALL instruction). The CS register is not changed on near calls, so the results of these branches can be safely predicted (from a shellcoders perspective).

A section of a disassembled binary is shown here with an actual CALL. Notice the instruction is given as an 0xe8 plus a double word (32 bit) displacement pointer.

The CALL is usually needed early in shellcode execution to PUSH the virtual address contained in the IP onto the stack. (This is done because it’s not possible to access the IP directly, so it needs to be put on the stack to utilize parameters within the shellcode). However, the problem with the use of CALLs for call stack setup in buffer overflow shellcode is the CALL is generally located at an offset needing to serve as a return address after other instructions have already been executed. In other words, the CALL is generally located later in the shellcode and the processor executes the instructions sequentially from the start of the shellcode – unless a branching instruction is encountered.

Which is precisely how to solve the problem in shellcode – early in the execution of the shellcode, you simply JMP to the CALL in question, then call back into the shellcode and continue execution.

JMPs are simple instructions and easy to visibly identify and dissect. They are simply the opcode 0xEB followed by a byte indicating the number of bytes to jump.

The example below is taken from an MDaemon Pre Authentication Heap Overflow exploit:

In the first example above (the egghunter shellcode), we see a “\xeb\x21” which means, “Jump 0×21 (or decimal 33) bytes.” When you jump those bytes, you hit the green box, a CALL. The CALL performs the call stack setup, then branches backwards back into the shellcode and picks up just after the JMP (because of the negative displacement). The actual offset is [0xFF – 0xDA = 0x25]. 0×25 is 37 in decimal, however, you subtract 5 from that since the offset starts at the end of the 5-byte CALL. That lands us just after the JMP.

Simple, yet effective. Even analysis of polymorphic shellcode generators shows this technique applies to almost all them as well.

To summarize all this rambling, the technique (show in the FelxParser below) is simply to search for a JMP straight to a NEAR CALL with a short and negative displacement.

Evasion

Call with no offset

Evasion of JMP/CALL detection can be accomplished a number of ways. The most interesting evasions are techniques used in advanced NOP sleds obfuscation leveraging CALLs that started surfacing around the mid-2000’s.

One of the simplest CALL-based NOP substitutions worked as follows:

00000000    E800000000  call 0×5

00000005    58                           pop eax

In that example we have a CALL with no offset, which basically translates to “branch to the instruction after this CALL,” in this case an opcode that simply POPs the EIP into the EAX register. (Remember, when the CALL is hit, the processor runs through the call stack setup, meaning the EIP was just PUSHED onto the stack.) From a NOP perspective, this leaves the stack unchanged, but for a method to grab the EIP, this is a simple and efficient (although the use of NULL bytes makes this more difficult to use in a wide range of shellcode).

As that byte sequence is very rare in binaries, detecting this is much simpler since we have the benefit of a continuous 6-byet token to watch for. In the case the EIP is poped to EAX, the token is simply

0xE8 0×00 0×00 0×00 0×00 0×58

The above pattern should be extended to include all the general purpose POPs, including:

0xE8 0×00 0×00 0×00 0×00 0×58

0xE8 0×00 0×00 0×00 0×00 0×8F

0xE8 0×00 0×00 0×00 0×00 0×0F 0×1A

0xE8 0×00 0×00 0×00 0×00 0×0F 0xA9

Noir’s no JMP/CALL

This next technique was first described by noir@gsu.linux.org.tr  on the vuln-dev mailing list. It works as follows:

00000000    D9EE               fldz

00000002    D97424F4    fnstenv [esp-0xc]

00000006    58                     pop eax

In this case, the technique is to use FNSTENV to get the EIP of the last FPU instruction evaluated, then POP it from the stack. In the example above, the FLDZ FPU instruction is issued, then its EIP is POP’ed. This very cool technique allows for many permutations since any number of floating point instructions can be used.  Several dozen pages in the Intel Developers Instruction Reference A-M (starting around page 430) cover instructions that can be used in place of FLDZ.

Gera’s CALL into self

The final one we’ll look at is a crafty method to avoid JMP/CALLs, and works like this:

00000000    E8FFFFFFFF  call 0×4

00000005    C3                        ret

00000006    58                        pop eax

The interesting thing is the code above does not perform the actions the disassembler has labeled them as doing. In reality, the CALL (E8FFFFFFFF) is calling backwards into itself by a single byte. Therefore, the processor will hit the byte 0xFF (the tail end of the CALL) and interpret that byte as an instruction. In this case, the instruction is an INC/DEC (increment by 1 or decrement by 1). The 0xC3 is actually an operand to the interpreted 0xFF instruction, so it’s not a RET (return, normally used for call stack unwinding) in this case – it’s actually a pointer to the value stored in the EBX register as an operand for the INC/DEC instruction! After this step has been taken (the equivalent of a NOP really), the value on the stack is POP’ed into the EAX register using the 0×58 instruction. The value POPed is the EIP since it was PUSHed onto the stack when the CALL called back into itself.

While this is a very cool technique, it also provides a number of simple tokens to match on, similar to the Call with no offset example.

False positives and benign triggers

In testing of 55 GB of data (network and host based) no false positives were encountered searching for a JMP to short and near negative CALL. However, benign triggers were encountered (meaning the condition was detected, but it was a valid use of the condition). The condition was only detected inside some valid PE files, and because of that fact, they can be filtered using a number of simple and easy techniques depending on the technology used to discover them.

Flex Parser

Currently, the parser engine does not allow for one-byte tokens, so this parser is not functional as-is. (The concept presented here can easily be extended to identifying percent-encoded shellcodes, which is supported since they are represented as multi-byte tokens.) Nonetheless, and more importantly, the technique is annotated here in Flex so the reader can see how simple it is to write FlexParsers to discover a wide array of very complex conditions – such as universal shellcode detection.

<parser name=”exploit_x86_shellcode” desc=”exploit_x86_shellcode”>

<!– declaration section holds all variables used down
in the <match> section –>
<declaration>

<!– this parser will output messages to suspicious risk catergory –>

<meta format=”Text” key=”risk.suspicious” name=”suspicious”/>

<!– parser logic will hit on every 0xeb encountered
this is exactly why single-byte tokens are not supported,
but I’ll show it here anyways! –>

<token name=”jmp” value=”&#xeb;”/>

<!– some numbers we’ll use for testing below–>

<number name=”num_jmp_offset” scope=”session”/>
<number name=”num_call_1″ scope=”session”/>
<number name=”num_call_2″ scope=”session”/>
<number name=”num_call_3″ scope=”session”/>

</declaration>

<!– enter the below node when the pattern held in “jmp” is found–>

<match name=”jmp”>

<!– read the next byte and store the value in num_jmp_offset–>

<read length=”1″ name=”num_jmp_offset”>

<!– move the value stored in num_jmp_offset –>

<move direction=”forward” value=”$num_jmp_offset”>
<move direction=”forward” value=”1″>

<!– read the next byte, if it is 0xe8 (decimal 232),
then continue –>

<read length=”1″ name=”num_call_1″>
<if name=”num_call_1″ equal=”232″>

<!– skip low-order address byte –>

<move direction=”forward” value=”1″>

<!– check others for values 0xff’s, meaning we’re not going
far in this code–>

<read length=”1″ name=”num_call_2″>
<if name=”num_call_2″ equal=”255″>
<read length=”1″ name=”num_call_3″>
<if name=”num_call_3″ equal=”255″>

<!– if we get here, add the tag “exploit_x86_shellcode”
to the suspicious catergory for this session–>

<register name=”suspicious” value=”exploit_x86_shellcode”/>

</if>
</read>
</if>
</read>
</move>
</if>
</read>
</move>
</move>
</read>
</match>
</parser>

Over the past two months, there has been a tremendous amount of chatter in the security community about the term ‘cyberwar’ and whether or not the US is engaged in a cyberwar. Mike McConnell (former Director of National Intelligence) wrote a pointed op-ed for The Washington Post claiming that, “The United States is fighting a cyber-war today, and we are losing.” His opinions are consistent with the current Director of National Intelligence, Dennis Blair, whose February testimony to the US Senate stated, “Malicious cyber activity is occurring on an unprecedented scale with extraordinary sophistication. While both the threats and technologies associated with cyberspace are dynamic, the existing balance in network technology favors malicious actors, and is likely to continue to do so for the foreseeable future.”

These statements spurred an excoriating response from the pages of Wired that, “The biggest threat to the open internet is not Chinese government hackers or greedy anti-net-neutrality ISPs, it’s Michael McConnell, the former director of national intelligence.” At the annual RSA security conference Howard Schmidt, the newly appointed White House Cyber-Security Coordinator stated unequivocally that, “There is no cyberwar.” Nonetheless in a Washington Post article on March 19th 2010 Ellen Nakashima dramatically points out the need for clearer cyberwar policies by pointing to US cyber operations already executed and that cyber actions are underway.

Various cyberwar definitions are hotly contested, even more nuance-laden and have a very material impact on the dramatic claims one might make. Below are several observations about cyberspace upon which all well-informed parties agree:

Click here to read the full posting on The Firewall at Forbes.com

There was a significant amount of coverage yesterday on research performed by NetWitness into a large set of stolen information recovered from a ZeuS botnet.  Some of the information, analysis, and commentary was very beneficial to the broader discussion of threats such as these.  There is, however, some information that we feel we should address.

• Kneber is a pseudonym for ZeuS:

Kneber is not a pseudonym for ZeuS. Kneber refers to one group of organized criminals, one group of Command and Control Systems, and 74,000+ infected victim systems for this particular ZeuS (primarily) botnet.   ZeuS is a tool, used by many groups to create command and control systems, and steal information.  There are hundreds of active ZeuS botnets, many of which are larger than this one. It is but one of many tools used in this particular botnet.  We have seen INTENTIONAL cross pollination of various trojans, including waledec, grum, and even tools such as packet sniffers.  When we discuss threat, we are referring to more than the tool used, but the organization behind them.

• Kneber is “nothing new”:

We have been very clear that this is a medium sized infestation when compared with all the tracked ZeuS botnets on the Internet.   What does make this very valuable is the opportunity to analyze such a large sample of stolen information, and quantitatively add to the discussion of threats to corporate security.  The number of infected and active systems behind some of the largest, most technology savvy companies needs to be considered, and our approach to security needs to change given the broad failure to identify or remediate these infestations.   In addition, trivializing the damage done is simply disingenuous by anyone who has seen the types of data stolen from threats such as these.

• Current protections and solutions can detect this type of activity:

This quote from Symantec, via the Guardian, KrebsOnSecurity, and others:

“Kneber, in reality, is not a new threat at all, but is simply a pseudonym for the infamous and well-known Zeus Trojan,” said the company. “The name Kneber simply refers to a particular group, or herd, of zombie computers, a.k.a. bots, being controlled by one owner. The actual Trojan itself is the same Trojan.Zbot, which also goes by the name Zeus, which has been being observed, analyzed and protected against for some time now.”

This quote is particularly troubling, as it seems to minimize the threat and is almost dismissive.   Moreover, when this particular variant was analyzed in late January (various services used), Symantec did NOT detect this as malicious.   To be fair, McAfee, Trend Micro, AVG, and most other mainstream anti-virus solutions also failed to recognize this as malicious.  In the past 3 weeks, Symantec has added signatures to detect this particular variant as a generic “Trojan Horse”.  However, if you were infected by this particular strain, your system has already processed an update that prevents you from contacting Symantec and others for updates.   In most cases, this will prevent future detection.   Worse, as part of normal operation of ZeuS, it attaches to running processes on victim systems in order to monitor them.   This data is logged along with other stolen information.   This set of data shows that ZeuS has actually attached to running versions of Symantec software on over a thousand victim systems.  Many other AV vendors are also present.

This example shows ZeuS monitoring a Symantec Live Update, and includes the ftp username and password used by the Symantec software during the update process.

• Are the facts overstated?:

The facts are fairly succinct in the whitepaper that we released.   We do not believe the threat is over-stated, and we were very conservative on the analysis released.   There are likely thousands of additional corporate networks affected, and analysis of this much information takes time.   And this is simply one of many similar operations in existence.  The group behind this effort can be described as sophisticated, yet also shows signs of lax effort to hide their trails.   The botnet is very actively managed, and continues in operation today.   The fact that they have been in successful operation for over 18 months also has to be considered.   We have also received several additional data points from federal contacts with additional insight into related government focused attacks.

More to come.

Tim Belcher and Alex Cox

While the world took pause to consider the implications of Operation Aurora, and Google lent considerable voice to the concept of Advanced and Persistent Threats (APT), we can ill-afford to believe even for a moment that they are alone in their sophistication or capability.   According to the FBI more than 100 nations have offensive cyber operations as part of their intelligence or national security fabric.  And the same attributes that make the Internet ideal for covert intelligence gathering make it attractive for corporate espionage and organized criminal activity.  The IT security industry commonly refers to the online activities of Eastern European and Asian organized crime as “cyber criminals” or “gangs”, which in many ways only serves to minimize the attention they deserve.  In truth the online operations of some organized crime syndicates are every bit as sophisticated, advanced and persistent as their nation-state counterparts.  They are truly expert at gaining footholds and siphoning off critical information.  And they are FAR more pervasive than Operation Aurora.

In late January, NetWitness security research were able to gain visibility into a large scale ZeuS-based botnet, taking user credentials and confidential information from thousands of organizations around the world (See The Wall Street Journal article).  Some of the information collected has been synthesized in the Kneber Bot whitepaper that you can dowload from the NetWitness website.

The sheer volume of information gathered and has forced us to reconsider the common belief that this very successful botnet is simply “financial services” related.  In fact, this particular botnet was much more concerned with culling account and network access credentials, as well as collecting as much detail about victim identities as possible.  In effect, they were less concerned with accessing any particular account, than being able to access ALL accounts related to a victim.

As we began analyzing victim information, we rapidly formed a picture of thousands of corporate compromises around the globe.  As with Aurora, many of the largest most technology savvy companies had internal compromised hosts, which had culled various corporate user level and administrative credentials.

We may attempt to broadly classify threats in the security industry, in hopes that we can make the complex more digestible to management.  However, classifying threats like this as “banking trojans” may do a large disservice to the victim companies.  Indeed, there are many that broadly dismiss threats such as these as “unsophisticated”, or less advanced than a pinpoint attack like Operation Aurora.  Rest assured, these adversaries could not care less how we classify their work.  They are well organized, have demonstrated technical sophistication on par with many intelligence services and do not forgo the opportunity for financial gain with the the information they collect.  If they are collecting network credentials, it means they are using or selling them in an active underground economy – which may include sponsoring foreign intelligence services. What is easier? Designing a campaign like Operation Aurora, or simply purchasing access to your target companies?

We are working with federal law enforcement, and continue in our efforts to notify victim organizations.  Please feel free to download our white paper and I am confident we will discuss great detail in future blog posts.

We recently sent an opt-in email to our contact database talking about the significance of Operation Aurora and the continued ascendancy and lack of advanced threat prevention/detection in many government and commercial organizations.  We also offered a NetWitness proof-of-concept (POC) to security folks concerned about this issue.  And security people should be concerned.

A noted security blogger correctly observed that we were “amplifying FUD” in our email blast to get people’s attention.   His blog post raises a classic issue facing security professionals – does FUD help bring an issue like this to top of mind.  Or:  To FUD, or not to FUD.

It’s really unfortunate that FUD became a dirty word when compliance and “risk management” took over the security budget, but that’s when many organizations, began to fail at security too.  While many people, particularly some CIOs in hindsight, would argue that compliance has helped increase the focus and spending on information security, I would argue that it has distracted many security programs into performing a large number of basically low impact or worthless activities in the name of metrics, versus FUD.  And, compliance certainly has sponsored a whole class of expensive security technologies and related total ownership costs (TCO) which drain the security budget.

There’s also an unfortunate psychology involved here.  Many security professionals feel guilty or inadequate using FUD as an argument because all the other I/T people have real metrics and we don’t.  To some, it’s like when we were kids and everyone had Converse “Chuck Taylor” All-Star high tops and you were the one with the red Pro-Keds.  Security people can’t talk about how many “9’s” of network uptime we have, or how much we have improved call center response time, or improved the total cost per terabyte of enterprise storage.   Security sucks at producing decent metrics — and the ones we do produce, generally stink even more at reducing the fear of being owned by national-sponsored or organized criminal groups or the uncertainty and the doubts regarding the security of information in a world of advanced threats.  Security people cringe when some C-level executive compares the cost of information security to the cost of insurance – “No one likes to pay for it, but just like your car insurance, you have to have it.”  Ugh!  So, we hate the FUD argument – both when we have to use it as an argument, or when someone uses it to trivialize what we all do for a living.

But I do not think security professionals should feel this way.  I think that FUD still has a lot of usefulness in the toolkit of the security professional and within the enterprise security program, if applied in the right doses to the right places.  One of my favorite Websites is fudsec.com.  There are many good, bad and ugly uses of FUD cited here, for example, one of the good ones is Anton Chuvakin’s post, “A Treatise on FUD” – required reading for any committed FUDists.

With regard to advanced threats and other types of network visibility problems, I encourage the use of a combination of FUD and proof.  The FUD comes in the form of security professionals updating their discussion track to highlight the real causes of many cyber losses in 2010, and the need for more focus on threat intelligence and operational security versus other types of spending.  Current issues such as Operation Aurora should be analyzed for relevance, and briefed to senior management, and should be coupled one of the more credible surveys that show that most data losses result from advanced threat or sophisticated exploit/malware sources.

In the end, you will have to produce real evidence, however, and that’s why we put the POC offer on the table in our e-mail blast.   FUD should only go so far — you should show your colleagues the smoking gun with your own organization’s data.   We as a vendor could put out all the FUD-sounding marketing statistics in the world about how our approach will make you more effective at changing the face of FUD to a smile than other alternatives, but you will only believe it when it produces results in your organization, you can bank those results, and it actually reduce the FUD for yourself and your CEO.   This is how it should be.

I was helping a fortune customer yesterday determine if they were targeted by Operation Aurora. From everything we know to date, they were not. How do we know this? We looked. In 15 minutes or so, we looked back over the last 6 months of every bit and byte that has left that company, and compared every hostname, IP address, and HTTP URL that have been associated with these attacks. This is the power of full network surveillance, and this is why you MUST be performing real-time continual deep analysis of your network activities.

There is a discussion today of some of the malware, and zero day exploits out of McAfee. They are now calling this Operation “Aurora”. In the post, George Kurtz discusses how APT, or Advanced Persistent Threats is changing the security landscape once again. It is a message, and the discussion we at NetWitness have been pushing for years. While this attack has gone largely unnoticed for months, NetWitness customers have all the historic evidence necessary to assess damage. For some, this means gaining rapid confidence that they were not compromised. For some, it means rapid damage assessment, capturing evidence, and using everything they learn to increase their security today. As I watch the best security teams in the world struggle to collect evidence of this attack over the course of days or weeks, I cannot help but wonder how much easier it would have been had we been in place.

In early December, we were called into one of the affected companies, in partnership with a large service provider. Within days, we had NetWitness gear recording at every major gateway in the country, and were scheduling international deployments. While it appears that the damage was already done to this company long before we arrived, we were instrumental in shutting down many other infestations, as well as identifying hundreds of systems that were displaying abnormal or concerning communication patterns. Had this company been a NetWitness client only 30 or 60 days before, I am absolutely confident we would have been able to bring this particular activity to light weeks earlier.

We have been reaching out to our customers, providing them details of the communication that Operation Aurora utilized, along with very simple instructions that allow them to look back over time and reassess their security. One thing is certain, while we may not know the vector of a specific attack until after the fact, it is imperative that we have the ability to quickly assess the damage and retain evidence.

This is why we must begin recording our network activity NOW. Giving your network some form of memory is an absolute imperative, and the foremost defense against APT. Furthermore, simple recording is not enough. Good luck if your recording architecture is IP based. Customers of NetWitness can search the URLs, hostnames, and other application level beaconing activities in seconds. Try doing this by scanning over 50 terabytes of packet data manually. Had this attack employed more sophisticated hosting or resolution techniques like fast flux, and even the IP addresses would have been useless.

George is absolutely correct in his assessment that the landscape has changed. These types of organized, sponsored attacks are here to stay. I am sure those in the financial community, used to dealing with advanced ACH fraud and highly targeted attacks by the Russian Business Network are sitting back this morning and saying “Welcome to the party, pal!”

Welcome to the party, pal!

I was at the CSI show yesterday and was within earshot of one of our “competitors” who claimed that they were winning against NetWitness because they support 10Gbps and we do not.   I have heard this story frequently from this particular firm, and it’s a bunch of bull.

It amazes me that companies in this space, such as Niksun and Solera Networks, spend so much time emphasizing how they allegedly can capture at 10Gbps line rates – as if that’s the most important requirement for public and private organizations struggling to cope with critical advanced threats, complex data leakage scenarios, network forensics, designer malware and botnet infestations, and increasing insider crime and fraud.

Solera Networks publicly asserts that their product was certified by Miercom Labs as 10Gbps capable.  If you look at the report on their Website, however (http://www.soleranetworks.com/resources/Solera%20DS5100_Miercom%20test.pdf), you will see that a top capture rate of 8.1 Gbps was achieved solely when the “packet size” was forced to 1,518 bytes.  At other packet sizes, the performance dropped off steadily.

The practical application of the Miercom report for Solera Networks is dubious in the real world.  For example, let’s assume that in Miercom’s vernacular “packet size” is equal to “message transmission unit” (MTU).  1,518 is the maximum MTU for the transmission of data over IEEE 802 networks according to RFC 1042.  But it is unrealistic to imagine that every consecutive packet on a customer network would be 1,518.  In fact, the typical default MTU setting for devices such as routers and servers processing a protocol such as HTTP over TCP/IP is 576 – most network and system administrators today work under this assumption.  In a real customer environment with a large amount of HTTP traffic, the Miercom numbers would put the theoretical Solera throughput somewhere around 6Gbps, versus the claimed 10Gbps.  Real life is different than the lab, however, and the reality is that customer application-layer traffic produces actual average MTUs of far less than 576, thereby lowering the potential performance results below the assertions made by some vendors to something more like an average MTU of 300 — or a throughput of around 3Gbps according to the Miercom results.

Such misleading lab reports also do not address other concerns, such as the technical challenges associated with capturing at a 10Gbps on single network appliance, given physical bus bandwidth constraints and disk write speed limitations — and still offering meaningful and timely analytics to security users.  Every vendor who is engineering solutions in this space has confronted this dynamic — but most vendors do not address this problem in their marketecture because they do not have a solution.

As the consumer of solutions in this space, you should be aware of this:   The reason this issue is not discussed in any meaningful way by some vendors is because they have no real-time automated and interactive analytical capability beyond basic and often erroneous network statistics.

Consider this screenshot from Solera Networks post-facto user console:

Notice specifically the port assumptions made by the product:

To assume in 2009 that TCP ports 80 and 443 are inclusive to web traffic simply is ridiculous.  Not only is this type of analysis absurd from a network forensics perspective, but also would seem at odds with the term “deep packet inspection”.  Consider the following drill into the HTTP service using NetWitness Investigator, on even a single day of capture from the NetWitness corporate HQ:

This screenshot represents only a small portion of the available information about the HTTP protocol.  This level of detail is possible because NetWitness does complete port agnostic session analysis in an automated real-time manner, upon collection.  You cannot get there from here with other vendors like Niksun and Solera Networks.  One of the vendor’s Websites says it most succinctly:

“Once you find the traffic flow you are looking for, you can download a PCAP file of just that data and analyze the traffic using any tool that analyzes PCAP files. Or you can save it for later and use it to analyze when you have time or need evidentiary proof of malicious activity.”

There are some pretty big and unfortunate “IF”s that customers should consider before engaging with any vendor operating under such assumptions:

1.  How do I actually go about “finding the traffic you are looking for” within tens or perhaps hundreds of terabytes of data in post-facto analysis?

2.  Why would I “save it for later” and “analyze it when you have time” when it could be something critical that requires immediate attention?  The assumption is that nothing here has a sense of urgency.

3.  Forget about real-time incident response, automated analytics, or integration with your SIEM or other existing security tools…not addressed.

4.  Your organization’s security staff still has to use NetWitness Investigator to satisfy the “analyze the traffic using any tool that analyzes PCAP files” requirement in order to use this vendor’s product — and that after finding both the haystack and the needle, which you would have found already had you been using NetWitness.

All this discussion highlights the value of working with NetWitness, an engineering company dedicated to solving the important problems of security professionals, law enforcement, intelligence analysts and other people focused on cyber security issues.  NetWitness offers a 10Gbps solution and it is running on some of the largest networks in the world — we’ve been doing it for a while — but we do it in a sensible way.  We do not go to market trying to sell you “disk write speeds” or “appliance capture rates” – that’s a waste of your money and should not be the most important focus for you.   Unlike anyone else in this space, we provide an infinitely extensible data framework, real-time automated analytics, live data fusion and threat intelligence, and the best network forensics interface in the market today.  Without all that, you are just filling up disk drives.