PyFlag Forensic and Log Analysis GUI

For:	Linux Conference Australia
Date:	2008-01-31
Authors:	Michael Cohen (AFP) David Collett (DSD)

Tutorial Overview

This tutorial will be "Hands on"
- If you do not complete an exercise during this time, you can do so later. All content will be available on the pyflag wiki.

Overview

Introduction and installation
- We will cover the installation procedure of PyFlag.
Log Analysis:
- We will analyse a web server log.
- We introduce the GUI and the searching power available from it.

Overview

Disk Forensics:
- We will work through an example forensic disk image.
- We introduce the Virtual File System, and Scanners.
- We look at some standard forensic techniques (hash comparisons, deleted file recovery, time-lining).
Network Forensics:
- We examine how network forensics fits in with the PyFlag model.
- We work through an example pcap capture file.

Overview

Automating PyFlag:
- We introduce the PyFlag shell (PyFlash). We repeat some of the previous exercises by scripting them.
Case Study
- Independently analyse evidence from a compromised server.
- You can use whatever tool you like - hopefully pyflag can help.

Introduction

Why would I want to do forensics?
- Forensics is not only for law enforcement.
- Administrators may investigate an intrusion on their network.
- Security teams might conduct an internal investigation for computer misuse.
- You may need to recover some files which have been deleted.

PyFlag Design Goals

Manipulate large quantities of information efficiently.
- We use a database to store and manage information.
Perform common analysis in advance - perusal of information should be very quick.
- Script support allows automated analysis. Caching system allows for very responsive display.
Every inference must be directly referenced by the evidence.
- Every detail shown must be reproducible by other tools.

Installation and configuration

Today we will be installing PyFlag from source.
- You will need a Linux system.
- check the wiki for specific requirements.
Download Source TAR ball from: http://www.pyflag.net/
Untar, configure and install

tar -xvzf pyflag-0.86RC1.tar.gz
cd pyflag-0.86RC1
./configure
sudo make install

Configuration

PyFlag uses ~/.pyflagrc to store configuration parameters.
- The first time PyFlag is run, we are asked to fill in some important parameters:
  - DBUSER, DBPASSWD - The database password for the user.
  - RESULTDIR, UPLOADDIR - PyFlag will attempt to initialise the pyflag database using the provided credentials.
- For our purposes we will use the free GeoIP service from MaxMind - Make sure to set geoipdir correctly.

Demo

Log Analysis

Why would we need to analyse logs?
- We ultimately need to answer some questions:
  - Who was using our site, from where, what where they more interested in.
  - How did an intrusion occur? who was it?
Log File difficulties
- Log files are typically huge.
- Relevant data is spread across many entries.
- Sometimes we can grep relevant data out - PyFlag allows us to load the log into the DB. Let the DB do the hard work!!!

Log Analysis

Log formats
- Different applications produce logs in different formats.
- The same application can produce different log formats depending on configuration.

Log Analysis

PyFlags approach
- Makes it easier for users to import any log file format by allowing them to define a log preset
- The log preset can be used for any log file from the same source. It is essentially a template.
- Provide the users with the ability to issue natural language queries - but the DB does all the work
- PyFlag does not attempt to lead the analysis.
A log Driver is a piece of software which can handle a whole class of log formats depending on configuration. A Log Preset is an instance of a configured driver which is good for a single type of log.

Exercise - Log Analysis

Statistics required for IIS server
- We want to know:
- Who is using our site?
- What are they getting?
- What makes up the most downloads?

Create Log Preset

The Web Server was an IIS box
- We use the IIS log file driver to load the file
- Create a log preset by selecting the IIS log file driver from the wizard.
- If you have time later on, practice using the Simple log file driver to load the same file.
Main points:
- Select "Create Log Preset" from the "Log Analysis" menu.
- Select the pyflag_iis_standard_log file to test the preset against.

Load Log File

Using our Preset template
- Select Load Preset Log File from the Load Data menu.
- Name the table, and select the preset you have created previously.
- PyFlag will try to load the first few lines of the file using this preset

Analyse Log File

Analyse Log File

Log files are simply displayed through the powerful PyFlag table widget
- This allows searching, grouping (counting) and exporting.
- PyFlag simply provides powerful analytical capabilities - its up to the user to put them to good use.
Table GUI is used everywhere:
- A standard, consistent GUI interface

Filter Tables

Filter Tables

Tables can be filtered by use of search terms.
- Basic structure is: column operator action
- Operators depend on what the column is
  - For example netmask operator can only be applied on IP columns, before and after can only be applied on Time columns.

Filter Tables

Expressions can be combined using and, or, parenthesis to form complex filter criteria.
- e.g::
  
  Timestamp after "2006-10-01 10:10:00" and (Timestamp before '2006-11-01 10:10:00' or "IP Address" netmask "10.10.10.0/24") or "IP Address" = 192.168.1.1
Expressions are stored in a filter history.

Playtime

Grouping (Counting)

Grouping (Counting)

Grouping counts the total number of each entry in a column with the same value.
- e.g. count all the hits from the same IP address
Grouping is applied after any filtering.
- e.g. count all the hits from the same IP address for those hits which generated a 404 status

Playtime: Apply Table Controls

Use Table controls to limit rows
- Show all the largest downloads, and their count by grouping on bytes transferred.
- We can see that the largest downloads contain the word Schnuffs in them.
- Limit by that word to see all the suspicious transfers.
Which users access those files? Group by users.
- How many sessions are there?
- Who created the file? (user/IP)

Log Analysis

The Advanced Log Driver allows you to build more flexible log parsers.
- We will build a Syslog Log Parser.
- The Syslog time format is "%b %d %H:%M:%S".
- You will need the syslog parser for later.

Disk Forensics

PyFlag is primarily an analysis tool for Forensics.
Disk Forensics is a task which aims to locate and identify information in hard disks.
- Hard disk images come in lots of formats
- Lots of different file-systems.

IO Sources

Often Images are supplied in a variety of formats:
- DD Images
- Encase Images
- Split DD images (e.g. LogiCube)
- RAID images
- SGZip
Using an abstracted IO Source Driver we can support all those formats with the same tool.

File Systems

File-systems are used to present and organise lots of information:
- Users are very familiar with directory/files hierarchy
- Many forensic tasks are related to file-system.
- Hierarchical in nature
File-systems use an internal representation called Inodes, and present files/directories for users.
- PyFlag uses the Virtual File System
- Just like a real file-system, VFS maps inodes to filenames/directory names.

VFS Internals

Inode format:
- Inodes are sequences of strings seperated by | (pipe)
- Each of these strings begins with a single character referring to a registered VFS File driver. e.g.:
  - S Stream reassembler
  - P PST Driver
  - Z Zip Driver
  - G Gzip Driver

VFS Internals

When we wish to open an inode we successively pass data from driver to driver until we get the final file:
- Inet|S4/5|o456:30255|m1|T2
  - Means take the combined TCP stream 4+5, at offset 456 there is a mime message, the first attachment has a tar file we want the second file in it.
  - This allows PyFlag to achieve great reach with recursive unpacking of contained files.

The FileSystem Driver

When a FileSystem is loaded, we use a FileSystem Driver to populate the initial VFS:
- Support many filesystems through Sleuthkit
- Even have filesystems which do not really exist e.g. PCAP Filesystem (more on that later), or Mounted.

Scanning the VFS

Scanners are small pieces of code which analyse files from the VFS:
- Scanners discover new files to be inserted into the VFS.
  - e.g. ZipScanner, PSTScanner
- Scanners can collect metadata about VFS files in external tables (not in the VFS).
  - e.g. IndexScanner, IECache Scanner
Scanners can be recursive (i.e. Scanners will generally scan the files it discovers using all the other scanners).
Scanners are the main way to populate the VFS.

Architecture Overview

Exercise - Disk Forensics

Case background
- In this fictitious example, we suspect Tony Pistone of killing Don Vitto the famous godfather.
- Don Vitto was killed outside the Caesars palace in Las Vegas, on July 30, 2003.
- Tony claims he was never in Las Vegas, let alone near the palace in his entire life.
- An important family meeting, in the palace was taking place at the time, we don't know how the suspect found out about it.

Load IO Source

First we specify the IO Source
- This lets PyFlag know which IO source driver to use
- After this operation the image will be referred to by name, even though it might contain several files.
- PyFlag can handle directly supported formats without needing to convert:
  - We can just use encase evidence files directly

Loading the Filesystem

Loading the filesystem populates the VFS for the first time.
- All files from the hdd filesystem are represented within the VFS.
- Our Filesystem is a standard EXT2 so Auto is fine.
Now we can Scan the VFS
- Scanning the VFS discovers new files, which get inserted into the VFS themselves.
- Scanners also collect information about the filesystem and perform initial analysis.
- Scanners fall into general categories which may be enabled/disabled in groups.

Scanning the Filesystem

Enable extra Scanners

Some scanners default to off due to being slow
- e.g Compressed file support
- Typically these will be used more selectively
If the category is disabled all scanners within it are disabled.
If the category is enabled, only those scanners who are enabled are used.

Browsing VFS

DiskForensics/BrowseFS:
- Can see virtual folders __deleted__ and __unallocated__:
  - Deleted files are inodes which are allocated but are not referred to from any directory inodes.
  - Unallocated VFS inodes are psuedo files which contain unallocated chunks of data from the disk.
- Virtual folder rk_044.zip:
  - Represents the contents of the zip file by the same name.
  - Can you use the table view to see all the files ending with .jpg?

Browsing VFS

Viewing Inode - Statistics

Viewing Inode - HexDump

Viewing files by file type

Keyword indexing

Indexing makes searching through the image much faster
- Some forensic packages do not index - these can be very slow.
- Some packages index the entire image - this can make the index bigger than the image (unless we make assumptions like alphanumeric keywords etc).
PyFlag uses a dictionary of keywords to index only those words.
- We use a log time trie hash algorithm so indexing 100,000 words is roughly 3 times slower than doing 10 words.

Keyword indexing

Note:
- Indexing is done during the scanning phase. The dictionary must be populated before scanning.
- The dictionary may contain strings or regex.
- Words can be populated from the GUI or from a script.
Keyword Indexing has a lot of reach
- Can get to compressed/ encoded files through the VFS scanner mechanism.
- Can find many more keyword occurances than simply grepping through the image.

Network Forensics

When would I ever use this?
- Sometimes during an investigation it is possible to obtain network captures.
- System administrators might want to investigate suspicious activity by an employee for example.
- The network captures may be of an ongoing attack.
Legal aspects
- I am not a lawyer!!!!!
- There are complex legal issues regarding interception of traffic (Telcom Intercept Act, Privacy Act etc).
- Please seek advice before you obtain the network capture.

Network Forensics

Forensics on PCAP files is unique:
- Most network analysis tools concentrate on the network. Provide access to packets and protocols. (e.g. Wireshark)
Investigators typically are interested in high level details:
- Files transferred
- Social networks
- Emails
- URLs visited
- Web Pages seen
At the same time investigators need to pin point the packets linked with these high level events - we must always tie everything to the evidence.

Network Forensics

PyFlag merges the Network with the standard forensic model:
- A PCAP Filesystem driver populates the VFS with reassembled streams.
- A set of scanners are designed to operate on PCAP Filesystem nodes.
Network Scanners produce VFS nodes for further scanning:
- This merges the Disk Forensic capability with the network.
- For example, if someone has a document inside a zip file sent in an email which they downloaded over POP3 we can find it.

Network Forensics

Useful protocols currently implemented:
- HTTP (including chunked)
- SMTP, POP with RFC2822 Messages
- IRC, MSN, Yahoo Messenger.
- Stream Reassembler

Exercise: Network Forensics

Create a new case
Load the PyFlag standard capture test file.
- Its stored as SGZip data (compressed)
Proceed to loading the file just like it was a disk image.
- Select the PCAP Filesystem
Loading the filesystem automatically reassembles all the streams.
- Scan the streams to dissect higher level protocols.

Network Forensics

Stream Reassembly

VFS inodes beginning with S represent streams:
- Can view the content of reassembled stream.
- Can combine two or more streams by using / in the Inode name:
  - E.g. S1/2 is the recombined stream obtained when piecing stream 1 and stream 2 based on packet arrival times.
- Can view packets in each stream and forward/reverse combined stream.

Reassembled Streams

Packet Trace

Packet Dissection

Network Forensics - Example

In this example we load stdcapture_0.3.pcap - Scan the VFS using all the scanners.

The Flag Shell : PyFlash

Forensics is time consuming
- Need to be able to script image loading, analysis and reporting easily
- Can always write scripts in python and interact directly with the PyFlag API (see for example test/init.py).
PyFlash is most useful to:
- Script very simple automated tasks (more complex ones should be done in python).
- Copy lots of files from the VFS in batch.

Conclusions

PyFlag is emerging as a complete forensic solution
- Covering Disk forensics, Network forensics and Log Analisys
- An open and extensible framework
- A platform for implementing cutting edge forensic techniques.
The wiki can be found at http://www.pyflag.net/