El Jefe 2.2 - The curious case of a 3G Modem ( Tracking USB devices and malware)

We are glad to announce a new and exciting release of El Jefe!

If you are in the business of protecting networks, you certainly spend enormous amounts of time grepping through pages of network log files trying to track down the origins of any threat that hits your network. Maybe you have a "SIEM" that helps you by storing this information for a window of time, and letting you search it.

If you are lucky, you can find your Russian malware inside the inbox of your director of finance's secretary in the form of a beautiful Word 0day. But there are days that the odds are not with you, and there is no explanation - no way to find out how your network got infected. Let's add a little bit more complexity to our mental game and let's make the threat a sophisticated implant, something like INNUENDO.

It's time to think USB! And that is exactly what this new release of El Jefe want to give you. A clear and visual way to track down every USB device connected on every computer on your network.

A good way to understand your USB devices and their relationship is to open the USB Relationship Map on your El Jefe server:

USB Relationship Map

The first thing you notice is that WIN-4C072EVNM9N and Anibal-PC workstation shares three devices (two mass storage and one Huawei Mobile device).

Double clicking on the devices will give us interesting information about the type of device and who has plugged it in.

This Huawei 3G Modem has been used on two workstations. Why are we even using Huawei modems?!?

You can also obtain a list of every USB device plugged and unplugged from any given endpoint or all endpoints. This will give us a good idea of the HUAWEI mystery we want to highlight in this post.

This list of USB devices on the endpoints has beautiful Christmas colors.

It seems that the HUAWEI Mobile and the HUAWEI SD Storage are connected at the same time, so this probably means they are the exact same device.

The Events view always provides us with a good picture of how this will correlate with process generation, and as expected it doesn't let us down. Seconds after connecting the 3G modem Autorun.exe was executed on the machine. This is not pretty at all.

Event view

Binary inspection give us a thumb down, this is getting worse.

The Thai Antivirus BKav knows something that neither the Russian or the Estonian AV's know, WHAT IS THIS .exe HIDING?

BKav identifies the binary as a KeyLogger

With one click in El Jefe we can now use Cuckoo to start to analyze the binary more, and even search our entire enterprise to see where else it has been installed. But we leave that as an exercise for the reader.

We hope you enjoy the El Jefe 2.2 release!

El Jefe and Splunk Part 1

Immunity focuses on the offensive side of security, even with a defensive product like El Jefe.

Traditional endpoint client protection focused on blacklisting. This was pretty effective way back in the day but in todays ever-mutating world it is not very manageable or useful. The replacement for black list is white list. Well-managed white listing can be very effective, but managing it well is …well…difficult. Immunity’s approach at endpoint detection is to approach it differently. what if instead of focusing on bad or good, we look at attack behaviours, attack patterns and attack chains? That’s what we are going to do over the next couple of posts.

Initially, this post started as a simple question: what would it take to get El Jefe data into Splunk? The purpose is NOT to replicate what is already being done in El Jefe. El Jefe does what it does and it does it great. Instead we will use the data from E lJefe to provide dashboards, reports and alerts generated by Splunk.

First off, a warning: I am by no means a developer. Way back in the 90’s I could spin some pretty mean Pascal and dBase IV programs, but after working in desktop and server operations for many years I have developed into what I call CopyUnderstandProductionizegrammer.

A CUPgrammer’s typically has a problem that needs to get fixed in a hurry, searches for an existing solution for the problem or a similar problem, Copies the code and then if unfamiliar with the code tests it until it is Understood and known to fix the problem without introducing new ones. This is often done with copious amounts of print statements. Finally, the CUPgrammer implements the new adapted code into Production.

With access to some amazing minds at Immunity, one day I might not be a CUPgrammer and actually learn proper development practices, but for now, I followed my modus operandi and went to work.

There are many ways for Splunk to consume data. There isn’t a API for El Jefe that we can point Splunk to (yet) and while I could grab the data out of the SQL database that would mean changes if the schema changes, version upgrades, etc etc, In my mind, the best and most CUPgrammer option was to push the data to Splunk through the Splunk API.

I asked our Digital Executive Protection program team what the best place to get all of the El Jefe data from a system. Turns out all of the data is in a collected in a Python dictionary just before posting it to the XML service.

Since El Jefe is open source, on my El Jefe server, I opened the file
<location_of_El Jefe>/webapp/xmlserver/ElJefeXMLServer.py, after making a backup, of course.

In the import section add:

Then scroll down and locate… the class SecureXMLRPCRequestHandler section. Just above that line, enter in the following:

Then in the class SecureXMLRPCRequestHandler section right after the line rpc = ElJefeRPC() add the following.

Now before you start up El Jefe, make sure the username and password entered in Splunk have been created and add the following to splunk/etc/system/local/props.conf.

 Restart Splunk and the ElJefe XML service, login to Splunk and do a quick search for sourcetype=eljefe.

You should get some results. Click on the all fields button in Splunk and notice that Splunk auto extracted all the fields.

By default, Splunk will prefix each filed with a {}. . If you want to remove them, add the following between the sourcetype and KV_mode lines in props.conf.
SEDCMD-StripHeaders=s/^[^{]+//
With the data in, let’s build a quick situation dashboard.

In the not distant future, Immunity will likely release an app on Splunk apps that includes a few interesting dashboards and reports, but for now, either create an app, or add the dashboard to an existing app. This dashboard will highlight four things initially.

1. Number of events per system 
2. Binaries over time 
3. Unique binaries 
4. Rare processes 

The dashboard will include a selectable time and the option to input a system name.

Go to Splunk console, dashboards, create a new dashboard called ElJefe_OverSight, click edit source, delete the existing lines and paste in the XML from the link below.
NOTE: In our environment the {}. that precedes the json fields is removed on import to Splunk. If you did not do that in your props.conf make sure you add a {}. before each of the field names.

Save the changes, go back to the El Jefe app or wherever you created the dashboard and click on it.

You should now see something similar to this.

Now that we have the data coming in, in our next post, we will go back to El Jefe and get a little offensive with it.

El Jefe - Alerts

In the never ending cat and mouse game between offense and defense, the defenders generally represent the mice. The objective of vulnerability management is to flip the script on the proverbial felines and set functional mouse traps for them. We can do this using machine learning to predict a potential attack vector or, if you feel like being an oxymoron, by deploying a fast and effective clean up.

Polemic methodologies such as "0day feeds" are ineffective as they protect against samples, not classes. While useful to demonstrate actionable risk to management, catching a single instance of a vulnerability implementation (AKA exploit) , 0day or otherwise, does very little to effectively protect you at an enterprise scale against similar vulnerabilities. Shooting fish in a barrel generally does not equate to a sane defensive posture.

El Jefe Alerts is our approach to a more comprehensive methodology of vulnerability management.

With El Jefe Alerts you are able to use, share and extend scenario driven vulnerability heuristics for entire vulnerability classes that trigger well defined response events.

A simple example:  Suppose every time iexplorer.exe parents an executable anywhere in your enterprise I want to retrieve the executable, run it through a sandbox and then receive an event summary via email. We can do that with El Jefe Alerts.

Or suppose that every time iexplorer.exe's virtual size gets bigger than 2 GB we would like to receive an urgent SMS, as this is a likely indicator of a heap spray in progress. We can do that with El Jefe Alerts.

What if I want to be notified whenever someone in the Enterprise launches a certain executable outside of work hours? Not a problem for El Jefe Alerts.

Let's walk through a simple example of how to set up an El Jefe Alert.

If you've attended DEFCON in the last three years, you probably noticed a distinct pattern in any presentation that involves Windows exploitation. Namely that a lot of attacker demos end up executing calc.exe! It seems they consider math a very important step of their post-exploitation strategy. Lets ruin their 4th grade algebra aspirations with the following:
      
1) The first step is to create a Filter, which is the module responsible for the heuristic that will trigger the El Jefe response action. Filters are django models which you can find in the alerts/models.py file.

ExecutionFilter will be triggered when a binary filepath is on our blacklist

You create the model and you implement the heuristic in the "filter" function. In this case, we want to trigger an action if and when an executed binary is listed in our blacklist.

2) The second step is to create an El Jefe Action script. The Action script defines the El Jefe response action to a filter trigger event. You can define multiple actions per filter. In our example script, we create a simple test.py file that just prints some basic information about the event, and we place it in the xmlserver/actions/ folder.

A simple action that will print the username, ip and binary executed

3) The final step is to put everything together. Once we've implemented our Filters and Alerts, they will automatically show up in the El Jefe interface.

You can now go to the "Alerts" section in the navigation bar, and select "Add". You select your Filter and associate one or more Actions and then select the executables you want to keep track of.

Adding a working filter

4) Voila! You are now Busticati proof!

Gotcha! You should never do your math homework on a compromised machine!

In real life, this simple Alert is useful when  you have recognized a particular pattern with an attacker - for example, many of them use WMIC.exe because that is how metasploit's meterpreter and some other trojans do their WMI activity (for enumerating all sorts of important system information). Normal users almost never use WMIC.exe so this is an interesting heuristic to apply across your network. For the record, INNUENDO does not shell out to WMIC - it has WMI support built into it.

El Jefe v2.1 Release

It has been a couple of weeks since our last communication but the team is working hard to make El Jefe your first and best choice for threat analysis.

Today's release, while technically a minor release, has many new features.

Cuckoo configuration is not only fully integrated into our WebUI, but we also modified the implementation to allow you to use remote Virtual Machines! El Jefe now lets you configure where and how you want to run the sample inside the sandbox, allowing you to properly use it to look at more complex malware problems.

Configure a remote Sandbox Virtual Machine

For example, assume that you are ahead of the curve and you are using El Jefe to monitor your C-level executives. Each of these will probably have their own software set-up, operating system, etc. Once you find a potential threat (manually, or using the built in El Jefe heuristics), you will want to understand the penetration behavior of the sample in the exact same environment it was run in on the executive's computer. That's why you set up a different Virtual Machine to mimic each executive's personal environment and you can have the whole process run through El Jefe.

Select the right Sandbox Virtual Machine to run your sample

We want to be as open as possible and give you the freedom to interface El Jefe in new and powerful ways that we have not even thought of. That's why we built a plugin to allow you to use COSEINC's  CAMAL instead of Cuckoo to get even more accurate results from your sandbox. (Kudos to Thomas and the COSEINC team for their fast response and an excellent product!)

Integration with CAMAL

El Jefe clients are now fully configured via the El Jefe UI, which will certainly make your life easier when deploying across your user-base.

Set-up your client completely from the WebUI

El Jefe clients now support proxies - which is obviously important for those of you in large enterprises.

People seem to love the Event filter, so we've started adding more features to it. You can now trigger an email warning and send suspicious binaries directly to sandbox analysis. This is just an example, of course. Our new API is easy enough to learn that you can create your own event filter in less than 20 minutes - which is important if you are in the middle of an Incident. Fast reaction times when facing custom malware and an advanced penetration team are often the only hope of your IR team for containing an ongoing compromise.

In case you didn't notice, El Jefe is released by Immunity under the GPL v3 license. But, just in case, we now let you know in every file. Source code to El Jefe (for both the client and the server) is provided on the website and will soon be moved to a GitHub repository.

We also put together an installer for RHEL (widely used in enterprises) along with the Ubuntu one to make everything install automatically and we have a beautifully crafted PDF with a step by step guide on how to install El Jefe.

Whether you're currently responding to an incident or you think you may someday want to respond to an incident, we think installing and getting to know El Jefe is a great first step for you. It's Free, and it works, and it's only getting better.

Of course, we welcome any of your feedback - send it to support@immunityinc.com and we'll be happy to respond!

Download it here

sha1sum: b8ee361ecf67e76ec0888e570153f76b15dfcea5  eljefe2.1.release.tar.gz

Connecting El Jefe 2.0 with the Cuckoo malware sandbox

One of the great new features in ElJefe May release is the integration of the Cuckoo malware analysis system as part of our interface. Cuckoo runs the malware executable in a sandboxed environment inside a virtual machine and produces amazing data, which we display in the El Jefe interface for you. However, configuring it is not a trivial task. That's why we put together a little blogpost to make our users happier. Always thinking about our El Jefe users, is our motto!

We are going to setup the host machine and the guest machine. The last one is where we are going to run the malicious files.

The following commands will install the necessary files:

  $ sudo apt-get install python python-sqlalchemy python-bson python-pip libcap2-bin
  $ sudo pip install sqlalchemy bson Django

You also need to install mongodb. You can download the necessary files from http://www.mongodb.org/downloads.

When Cuckoo is analyzing a submitted files there are some modules and libraries we can install to get better and more complete analysis and reports. These modules are optional but their installation is highly recommended.

A list of the these packages can be found at: http://docs.cuckoosandbox.org/en/latest/installation/host/requirements/#installing-python-libraries

We are using setcap to give privileges to tcpdump to run as root without having to run Cuckoo as root.

  $ sudo setcap cap_net_raw,cap_net_admin=eip /usr/sbin/tcpdump

We can check the results with:

  $ getcap /usr/sbin/tcpdump
  /usr/sbin/tcpdump = cap_net_admin,cap_net_raw+eip

You will need to create a new user, you can do that with the following command:

  $ sudo adduser cuckoo

You can download Cuckoo from http://www.cuckoosandbox.org/download.html. You will need to copy the Cuckoo folder into the El Jefe root folder. You should see a directory listing like this:

client cuckoo dependencies installer webapp

You will also need to uncomment some lines of code to make cuckoo work:

webapp/settings.py lines 105 to 124

webapp/xmlserver/ElJefeXmlServer.py lines from 55 to 58.

webapp/home/views.py lines 44, 45, 1055, 1316 to 1334.

webapp/analysis/views.py lines 20 to 25.

webapp/templates/base_.html remove the {% comment %} and {% endcomment %} tags on lines 121 and 142.

Now it's time to setup the virtual machine where we are going to run and analyze our binaries. Cuckoo supports VirtualBox, KVM and VMWare for virtualization. We choose VMWare as our virtualization software, so the following steps will show you how to configure Cuckoo to work with VMWare. If you wish to use other virtualization software, follow the guidelines in the following URL: http://docs.cuckoosandbox.org/en/latest/installation/guest/.

First, we'll need to create a new virtual machine. The preferred OS is Windows XP, but you can use the OS of your preference. Obviously for future versions of El Jefe we will support automatically choosing the right target VM based on the El Jefe client's OS.

We need to install Python on the virtual machine and PIL (http://www.pythonware.com/products/pil/) if we want Cuckoo to be able to take screenshots of the binary as it runs.

Now it's time to configure the virtual machine's networking. We are going to disable the Windows Firewall first, and then create a virtual host only network for the guest and the host.

We are going to use the 192.168.100.0 network, the guest configuration will be:

IP Address: 192.168.100.100

Netmask: 255.255.255.0

Gateway: 192.168.100.1

Cuckoo supports DNS resolution, so you can use 192.168.100.1 as your DNS server. In our experience, we get better analysis results by using a public DNS server.

And the host configuration (for the vmnet adapter) will be:

Ip Address: 192.168.100.1

Netmask: 255.255.255.0

You can choose whatever network and addreses you like.

Now we need to configure packet forwarding on the host machine to give the guest machine Internet access. We can do this with the following commands ( replacing eth0 with your outgoing interface and vboxnet0 with your virtual interface).

iptables -A FORWARD -o eth0 -i vboxnet0 -s 192.168.56.0/24 -m conntrack --ctstate NEW -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A POSTROUTING -t nat -j MASQUERADE
sysctl -w net.ipv4.ip_forward=1

This concludes the networking setup. Now we need to install the cuckoo agent on the guest, by copying agent.py, located in cuckoo/agent in the El Jefe root folder, to the Windows Startup folder.

Now it's time to make a snapshot. Before executing a malware binary the snapshot is reverted and then the binary is executed and analyzed.

Next, we will see how to setup the configuration files on Cuckoo.

[cuckoo/conf/auxiliary.conf]

Change the interface to the one you are using.

[cuckoo/conf/cuckoo.conf]

Set your db connection on the [database] section and the result server IP address on the

[resultserver] section (we are using 192.168.100.1).

[cuckoo/conf/reporting.conf]

Set the mongodb settings on the [mongodb] section.

[cuckoo/conf/vmware.conf]

If you are going to use only one VM, you will need to modify the stuff inside the [cuckoo1] section only, otherwise you will have to create one additional section for every extra VM you wish to use.

Inside these sections you will have to set the VM path, snapshot name and IP address of the machine (we are using 192.168.100.100).

We are done setting Cuckoo for El Jefe. Before starting El Jefe remember to start the mongodb and postgresql services. You will also need to start the cuckoo.py service from cuckoo/cuckoo.py in the El Jefe root folder.

So in conclusion, you should now have Cuckoo set up for yourself, integrated into the world's most powerful open source host monitoring system, and ready for your incident response team to use. We welcome any comments or questions, and of course are ready to help you if you have problems.

David Arch

Revamping El Jefe

El Jefe 2.0 - Process Chain Visualization and Heuristics

One improvement in modern Anti-Virus is the move away from signatures to heuristics and program behavior analysis on any one process. This turns out to be a decent protection at first, but as attackers evolve it will rapidly become less effective. El Jefe offers a radically different leap-frog in the level of analysis done with its protective heuristics by looking at the entire chain of process creation, rather than each process creation event alone.

Walking through the events on multiple stations

Intuitively, a human being will ask questions based on process chain anomalies:

• Why is IExplorer.exe popping up commands like crazy? 
• Why does Adobe PDF process spawn another process as LOCAL/SYSTEM?
• Why did this user suddenly start using commands like wmic, which only a system administrator normally would use?

A big part of doing analysis is looking at pages and pages of text looking for patterns and ways to correlate stuff to find anomalies. This takes a tremendous amount of work, and has the risk of becoming a stultifying routine for the analyst. In other words, many companies are drowning in their own Big Security Data. 

As part of our Digital Executive Protection program, we spend hours looking at our client's high process creation data looking for potential signs of attack. One way we found to make this more effective is to add alternative ways to inspect and visualize this data.

The D3 JavaScript library provides us with  great resource for doing that. The first visualization tool we built show us easily the amount of usage of any given process. Instead of going the traditional way with big colorful circles for the most used processes, we invert it so with one fast view you can look at the processes that are hardly ever used - the exceptions which are most likely to be worth looking at further.

Process Usage: An easy way to identify process used only a small amount of time

The second graph we are experimenting with examines the relationship between all the processes executed on the system. This provides us with an easy way to move around different processes that have been run on any system and understand how they were triggered, by who and which activities were done.

Event Relationsip: Move around events and find the relationship with each other.

You might notice events are a key feature in El Jefe. One thing we want to explore is the correlation of an event's properties within multiple instances of that event. For example, when people use IE to host their trojans that IE has a very different memory and thread-count from a normal IE. This can be visualized in El Jefe and will stand out even though IE itself is not malicious.

Analyzing triggered events over time 

The second big feature we will introduce in this new version is the integration with the famous Cuckoo sandbox. We love the work that team has done over the years and it matches up perfectly with the procedure our analysts were using for El Jefe. We connect to Cuckoo within El Jefe, so now every time you find a suspicious binary, you can seamlessly ask El Jefe to grab the binary from the target machine and run it on the isolated Cuckoo sandbox. The result, for those unfamiliar with Cuckoo, is a beautiful report on what the binary has done, including files dropped, registries touched and a PCAP with everything the binary has sent over the network.

Event Inspection

Based on these new cool features we think El Jefe is a good addition to a company's security stance - given that it is both free, extensible, and more effective than traditional AV tools.

Keep tuned, because this new release is around the corner!