Harvester Architecture Overview

Harvester is an open source framework that allows security professionals to deploy an extensible multi-vendor, multi-platform facility for log consolidation, analysis and escalation. Harvester makes infrastructure monitoring possible through the integration of various point tools such as NIDS, HIDS, firewalls, web servers, routers, et al.

Harvester’s modular architecture allows the consolidation and analysis of security information from multiple sources such as NT Event Logs, UNIX syslogs, Snort IDS, Checkpoint OPSEC and ISS Real Secure SNMP. New data sources can be easily added to the system by adding new receiving & parsing modules. New data extraction methods can be created by adding new processing modules. The architecture accomodates serial and parallel message flow so any conceivable processing method can be accomodated in a scalable fashion.

The core of the Harvester system is the Harvester Processor. This is a multi-threaded, plug-in based engine for high speed consolidation, ranking and correlation of incoming log data. Messages are received, parsed, processed and stored in a standard format for use by backend external processes.

Harvester Processor Architecture

There are three main groups of plug-ins: The Receiver Group, the Parser Group, and the Processor Group. Each Plug-in group is responsible for a specific set of system functions:

• Receiver Plug-ins are responsible for translating raw incoming logs into a form suitable for the Parser Plug-ins to digest.
• A Forensic Archival Plug-in is provided allowing for the raw messages to be archived in a "pure" state for later forensic analysis.
• Parser Plug-ins translate the various incoming log formats into an internal Unified Log Format. This format is used by the Processor Plug-in Group to extract interesting data, perform rules-based analysis and provide for a standardized storage format.
• The Processor Plug-ins are responsible for data storage, first level analysis and statistical data extraction.

The Harvester Processor provides all of the necessary raw data for external review and reporting modules. A variety of reporting tools can be utilized to access relational databases. As everyone's needs are different, sample mechanisms are provided to allow the end site to customize the reporting to their needs.