Hashing: The Maslow's Hammer of Forensics

We get asked this question a lot:

Can GRR search for a hash on all machines?

The shortest answer is yes.  If you’re filling out a checklist you can stop reading here, complete the column, crack open a beer, and bask in the warm glow of satisfaction that comes with knowing you did a great job.

The real-world answer is much more complicated, but as a starting point lets go with “that’s almost never useful”.

Can GRR search for a hash on all machines?

Technically yes.  You can use the FileFinder flow to hash every file on the filesystem for every machine, and then check if your target hash is in the list.  You don’t want to do this, and in fact GRR has default safeguards that won’t let you without an explicit override.

The reason is that it’s incredibly expensive in terms of machine resources for the client.  It’s lots of disk IO, since you’re reading every bit of every file on every machine, and it’s lots of CPU cycles since you’re putting all of those bits through complex hashing operations.  You’re also spending network bytes to send all the hash results.

If there’s anything users hate more than having security agents on their machines, it’s having security agents that cripple their ability to do their jobs.

But say you do this anyway, and at incredible cost, you now have a snapshot of the hash of every file in your fleet.  The problem is the word ‘snapshot’.  GRR is a mechanism for capturing the state of a machine at a certain point in time.  Even before you finished hashing the disk of a single machine the hash set was out of date. So how often do you re-hash everything to pick up any new files or changes? Clearly this is impractical, and GRR wasn’t designed to be an IDS.

But collection/measurement of hashes is only part of the picture.  Why are we using hashes in the first place?  Often it’s because that’s what’s shared or bought as “threat intelligence”.

Most of this hash-checking behaviour is aimed at malware detection, but it’s a terrible way to detect malware because a one bit change in the binary means no detection. For anything more interesting than mass malware, you will basically never get a hash hit.  Any targeted payload will at a minimum contain a campaign code, be recompiled, or be completely customised for the target. Context aware "fuzzy hashing" can reduce this brittleness to some extent, but it doesn’t help with the resource consumption problem above.

So what should we do?

A better way to hunt for files

Here’s the questions I ask when someone comes to me asking to hunt a hash.  You can use it as a checklist for constructing a GRR hunt for a file that will make it as targeted and efficient as the circumstances allow.  The more files you exclude with each condition, the faster things will be:

• Where is the file most likely to be on the filesystem?  If it always lands in System32, or it’s most likely in a browser tempdir or user homedir, just target those paths in FileFinder.
• How big is the file likely to be?  e.g. if you know it will never be larger than 50MB, set a file size range in FileFinder. Or if you know the exact filesize, this tends to be very unique and is fast to check.
• Is it likely to have had a timestamp modified recently or within in a certain time window? Even if the time window is large (months/years) it can help when hunting in places like system directories.  Set a modification time range in FileFinder (be careful here if you suspect timestomping).
• Can we identify some part of the file that we could use as a binary signature?  For malware the best case scenario would be some reversing work that gives us a binary sequence that lives in a fairly predictable location of the file and gets reused across whole families or classes of malware.  But the technique applies generally: checking a byte range of a file is much faster than reading the whole content.  Set a contents match condition in FileFinder.

You could set the FileFinder action to HASH, but if something matches all of these conditions it is probably interesting, so in most cases you should just make it DOWNLOAD. GRR only transfers globally unique files. Also you’ll kick yourself if you see a really promising match but only get the hash and the machine goes offline.

GRR is smart and knows about the relative resource cost of each of these conditions.  It will apply cheaper stat attributes constraints like size and modification time before resorting to more expensive file content matches.

Before running this on all machines you can test it on your own machine to see if it’s too expensive or returns too many results.  If you’re downloading files this will also pre-populate the server-side collection of common binaries.

Better ways to use hashes

Searching for hashes isn’t a good fit for live forensics systems like GRR.  But that’s not to say hashes are useless.  Here’s a few high value ways to use hashes you should consider:

• Searching stream capture systems on the server-side, like:
• A central database of executable hashes that’s fed by on-host systems hashing every executable run and every library loaded (e.g. CarbonBlack or Bit9).
• Network security monitors that hash files downloaded, email attachments etc. that cross a network boundary (e.g. Suricata).
• Dead-disk forensics where you have fast I/O and the ability to spend machine resources with zero user impact.

Why No Stream Capture?

So if hashes work better for stream capture, why doesn’t GRR do that too?

We want to be really good at state capture. Finding the bits, transporting the bits, and making the bits available for analysis, all at scale, is a hard problem and we don’t see ourselves running out of work any time soon.

Stream capture is a distinct but similarly hard problem involving instrumentation/hooking at the kernel level.  Adding it would significantly complicate maintenance, client stability, and the guarantees we make about client performance.  Conversely, keeping a separation between response and detection means you always have a response capability.

Other products have already made good progress on detection using stream capture.  Our approach is to use those capabilities to tell GRR about interesting places to grab state.