Harvester Processor Data Flow Example

Overview
Data Flow Example
Message Transmission Phase
Parsing Phase
Processing Phase
Cleanup Phase

Overview

External events are captured and processed through the Harvester Processor Architecture (HPA). The HPA is a multi-threaded architecture - each plug-in can be implemented as a light-weight thread allowing for highly parallel execution. The Message Dispatcher coordinates system activity by regulating message flow through the system. The multi-threaded architecture allows for high speed categorization, parsing and critical data extraction by performing most operations in an on-demand parallel fashion.

This section summarizes the dataflow through the HPA. In general, messages are captured by the Receiver Group Plug-ins, parsed by the Parser Group Plug-ins, and processed for output and storage by the Processor Group Plug-ins.

Data Flow Example

Message Transmission Phase

Step 1 - Host Message Origination

A system generates a syslog (rfc 3164) message; this could be a firewall or other monitored system. Its Loghost configuration variable is set to the system where the Syslog Receiver plug-in resides.

Step 2 - Message Receipt

The message crosses over the network (lan) and is picked up by the Receiver plug-in and has headers placed upon it identifying it as a syslog message. The plug-in then sends the message through the Plug-in Message Dispatcher (PMD) to the Parser Group.

Parsing Phase

Step 3 - Forensic Data Archiving

The highest priority Plug-in in the Parser Group receives the message and begins processing it. In this case, it is the Archival Storage Plug-in. This plug-in takes the message and sends a copy out to Archival Storage. This allows for forensic analysis to use the original message. It then notifies the PMD that it is done with the message.

Step 4 - Message Parsing

The RFC Syslog Parser Plug-in is next in line to receive the message. It receives it from the bus and begins processing. It will pull apart the message and seperate out component fields. It will then use these fields to construct an Unified Log Format (ULF) message. It will Insert the ULF message into the PMD and dispose of the Syslog message as no further processing on that message needs to be done.

Processing Phase

Step 5 - Ranking

The PMD then delivers the ULF message to the highest priority plug-in in the Processor Group. The Rule Based Ranking Plug-in takes the message and begins comparing it to site definable rules in its data store. Based on these rules the score on the message is updated. The message is then replaced in the PMD to allow further processing to occur with the updated Score.

Step 6 - Storage

The Log Storage Plug-in receives the ULF message with the updated Score. The Log storage plug-in converts the message into the appropriate set of commands to insert it into the acompanying data store. In this case it would generate the appropriate SQL statements to insert the message into a MySQL database.

Cleanup Phase

After inserting the message, the DB plug-in issues a Dispose to the PMD as no futher processing of that message is required.