El Jefe and Splunk Part 1

Immunity focuses on the offensive side of security, even with a defensive product like El Jefe.

Traditional endpoint client protection focused on blacklisting. This was pretty effective way back in the day but in todays ever-mutating world it is not very manageable or useful. The replacement for black list is white list. Well-managed white listing can be very effective, but managing it well is …well…difficult. Immunity’s approach at endpoint detection is to approach it differently. what if instead of focusing on bad or good, we look at attack behaviours, attack patterns and attack chains? That’s what we are going to do over the next couple of posts.

Initially, this post started as a simple question: what would it take to get El Jefe data into Splunk? The purpose is NOT to replicate what is already being done in El Jefe. El Jefe does what it does and it does it great. Instead we will use the data from E lJefe to provide dashboards, reports and alerts generated by Splunk.

First off, a warning: I am by no means a developer. Way back in the 90’s I could spin some pretty mean Pascal and dBase IV programs, but after working in desktop and server operations for many years I have developed into what I call CopyUnderstandProductionizegrammer.

A CUPgrammer’s typically has a problem that needs to get fixed in a hurry, searches for an existing solution for the problem or a similar problem, Copies the code and then if unfamiliar with the code tests it until it is Understood and known to fix the problem without introducing new ones. This is often done with copious amounts of print statements. Finally, the CUPgrammer implements the new adapted code into Production.

With access to some amazing minds at Immunity, one day I might not be a CUPgrammer and actually learn proper development practices, but for now, I followed my modus operandi and went to work.

There are many ways for Splunk to consume data. There isn’t a API for El Jefe that we can point Splunk to (yet) and while I could grab the data out of the SQL database that would mean changes if the schema changes, version upgrades, etc etc, In my mind, the best and most CUPgrammer option was to push the data to Splunk through the Splunk API.

I asked our Digital Executive Protection program team what the best place to get all of the El Jefe data from a system. Turns out all of the data is in a collected in a Python dictionary just before posting it to the XML service.

Since El Jefe is open source, on my El Jefe server, I opened the file
<location_of_El Jefe>/webapp/xmlserver/ElJefeXMLServer.py, after making a backup, of course.

In the import section add:

Then scroll down and locate… the class SecureXMLRPCRequestHandler section. Just above that line, enter in the following:

Then in the class SecureXMLRPCRequestHandler section right after the line rpc = ElJefeRPC() add the following.

Now before you start up El Jefe, make sure the username and password entered in Splunk have been created and add the following to splunk/etc/system/local/props.conf.

 Restart Splunk and the ElJefe XML service, login to Splunk and do a quick search for sourcetype=eljefe.

You should get some results. Click on the all fields button in Splunk and notice that Splunk auto extracted all the fields.

By default, Splunk will prefix each filed with a {}. . If you want to remove them, add the following between the sourcetype and KV_mode lines in props.conf.
SEDCMD-StripHeaders=s/^[^{]+//
With the data in, let’s build a quick situation dashboard.

In the not distant future, Immunity will likely release an app on Splunk apps that includes a few interesting dashboards and reports, but for now, either create an app, or add the dashboard to an existing app. This dashboard will highlight four things initially.

1. Number of events per system 
2. Binaries over time 
3. Unique binaries 
4. Rare processes 

The dashboard will include a selectable time and the option to input a system name.

Go to Splunk console, dashboards, create a new dashboard called ElJefe_OverSight, click edit source, delete the existing lines and paste in the XML from the link below.
NOTE: In our environment the {}. that precedes the json fields is removed on import to Splunk. If you did not do that in your props.conf make sure you add a {}. before each of the field names.

Save the changes, go back to the El Jefe app or wherever you created the dashboard and click on it.

You should now see something similar to this.

Now that we have the data coming in, in our next post, we will go back to El Jefe and get a little offensive with it.

El Jefe - Alerts

In the never ending cat and mouse game between offense and defense, the defenders generally represent the mice. The objective of vulnerability management is to flip the script on the proverbial felines and set functional mouse traps for them. We can do this using machine learning to predict a potential attack vector or, if you feel like being an oxymoron, by deploying a fast and effective clean up.

Polemic methodologies such as "0day feeds" are ineffective as they protect against samples, not classes. While useful to demonstrate actionable risk to management, catching a single instance of a vulnerability implementation (AKA exploit) , 0day or otherwise, does very little to effectively protect you at an enterprise scale against similar vulnerabilities. Shooting fish in a barrel generally does not equate to a sane defensive posture.

El Jefe Alerts is our approach to a more comprehensive methodology of vulnerability management.

With El Jefe Alerts you are able to use, share and extend scenario driven vulnerability heuristics for entire vulnerability classes that trigger well defined response events.

A simple example:  Suppose every time iexplorer.exe parents an executable anywhere in your enterprise I want to retrieve the executable, run it through a sandbox and then receive an event summary via email. We can do that with El Jefe Alerts.

Or suppose that every time iexplorer.exe's virtual size gets bigger than 2 GB we would like to receive an urgent SMS, as this is a likely indicator of a heap spray in progress. We can do that with El Jefe Alerts.

What if I want to be notified whenever someone in the Enterprise launches a certain executable outside of work hours? Not a problem for El Jefe Alerts.

Let's walk through a simple example of how to set up an El Jefe Alert.

If you've attended DEFCON in the last three years, you probably noticed a distinct pattern in any presentation that involves Windows exploitation. Namely that a lot of attacker demos end up executing calc.exe! It seems they consider math a very important step of their post-exploitation strategy. Lets ruin their 4th grade algebra aspirations with the following:
      
1) The first step is to create a Filter, which is the module responsible for the heuristic that will trigger the El Jefe response action. Filters are django models which you can find in the alerts/models.py file.

ExecutionFilter will be triggered when a binary filepath is on our blacklist

You create the model and you implement the heuristic in the "filter" function. In this case, we want to trigger an action if and when an executed binary is listed in our blacklist.

2) The second step is to create an El Jefe Action script. The Action script defines the El Jefe response action to a filter trigger event. You can define multiple actions per filter. In our example script, we create a simple test.py file that just prints some basic information about the event, and we place it in the xmlserver/actions/ folder.

A simple action that will print the username, ip and binary executed

3) The final step is to put everything together. Once we've implemented our Filters and Alerts, they will automatically show up in the El Jefe interface.

You can now go to the "Alerts" section in the navigation bar, and select "Add". You select your Filter and associate one or more Actions and then select the executables you want to keep track of.

Adding a working filter

4) Voila! You are now Busticati proof!

Gotcha! You should never do your math homework on a compromised machine!

In real life, this simple Alert is useful when  you have recognized a particular pattern with an attacker - for example, many of them use WMIC.exe because that is how metasploit's meterpreter and some other trojans do their WMI activity (for enumerating all sorts of important system information). Normal users almost never use WMIC.exe so this is an interesting heuristic to apply across your network. For the record, INNUENDO does not shell out to WMIC - it has WMI support built into it.