SPLK-2002 Forwarder and Deployment Best Practices

Forwarder and Deployment Best Practices Detailed Explanation

Forwarders are one of the most essential components in any Splunk deployment. They are responsible for collecting data from source systems and sending it to the Splunk indexers.

This section will help you understand the types of forwarders, and how to deploy and manage them effectively, especially in larger environments.

1. Types of Forwarders

Splunk provides two types of forwarders. Both are used to transport data, but they have different capabilities and use cases.

a. Universal Forwarder (UF)

• A lightweight agent used only to forward data.

• It does not include a GUI (Graphical User Interface).

• It has a minimal resource footprint, which makes it ideal for installation on production systems.

• It cannot parse or index data on its own — that work is done by indexers.

Use Case:
Use Universal Forwarders to collect logs from servers, network devices, or applications, and send them to indexers. They are best for performance, scalability, and low maintenance.

Example:
Installing a UF on a Linux server to monitor /var/log/messages and forward the logs to Splunk.

b. Heavy Forwarder (HF)

• A full Splunk Enterprise instance that includes indexing and parsing capabilities.

• Can filter, route, and transform data before forwarding it.

• Uses more CPU and memory than the Universal Forwarder.

Use Case:
Use Heavy Forwarders when you need to manipulate or filter data before sending it to indexers. Also needed for some Splunk apps or add-ons that require data parsing at the source.

Example:
Using a HF to route logs from multiple locations to different indexers based on log type.

2. Best Practices for Forwarder Deployment

Proper deployment of forwarders is essential for system performance, manageability, and scalability.

Use Universal Forwarders by Default

• In most environments, Universal Forwarders are preferred because they are easy to install and manage.

• They consume fewer system resources, which is important on production servers.

Avoid Indexing on Heavy Forwarders Unless Necessary

• Indexing data on HFs consumes license volume and may result in duplicate indexing if not managed carefully.

• Use HFs only when parsing or routing is required and you understand the impact.

Secure Data Transmission with SSL/TLS

• Enable SSL/TLS encryption between forwarders and indexers to protect sensitive log data in transit.

• This is especially important in public networks or security-sensitive environments.

Configuration Tip:
Use outputs.conf to enable secure communication on forwarders.

Use Deployment Server to Manage Universal Forwarders

• A Deployment Server (DS) allows you to push configurations to many forwarders from a central point.

• You can manage:

  • inputs.conf for data inputs.

  • outputs.conf for indexer destinations.

  • Apps and add-ons required on the forwarders.

This is essential when you have hundreds or thousands of forwarders.

Group Forwarders into Server Classes

• A server class is a group of forwarders that share the same configuration or receive the same set of apps from the Deployment Server.

• Helps you organize your forwarders based on:

  • Operating system

  • Department

  • Log type

  • Location

Example:
Group all Windows UFs in one server class, and all Linux UFs in another.

Implement Load Balancing in Output Configuration

• In outputs.conf, you can list multiple indexers as data destinations.

• The forwarder will balance the data load across them.

• This avoids overloading a single indexer and improves overall indexing performance.

Example configuration:

[tcpout]
defaultGroup = indexer_group

[tcpout:indexer_group]
server = indexer1.example.com:9997, indexer2.example.com:9997

Forwarder and Deployment Best Practices (Additional Content)

1. Heavy Forwarders (HFs) Can Parse and Route Data

While Heavy Forwarders (HFs) are generally not recommended for indexing due to licensing and architectural concerns, they can perform data parsing before forwarding.

Key Behaviors:

• HFs can apply parsing logic defined in props.conf and transforms.conf.

• Typical parsing actions include:

  • Line breaking

  • Timestamp extraction

  • Field extractions

  • Event routing or filtering

• After processing, the data can be forwarded to downstream indexers.

Example Use Case:

You may deploy an HF in front of a firewall that emits noisy logs. The HF can filter out unnecessary messages and forward only relevant data, reducing index volume and license usage.

Caution:

HFs can consume license if indexing is not explicitly disabled. Always ensure index = false in inputs or configure the HF strictly as a parsing-only forwarder.

2. Deployment Server (DS) Is Not Recommended for HF or SH Management

The Deployment Server is designed for managing Universal Forwarders (UFs) at scale via server classes and deployment apps. It is not suitable for managing:

• Heavy Forwarders

• Search Heads

• Indexer Clusters

Reasons:

• HF and SH configurations are more complex, often involving clustering, role-specific apps, or dynamic replication policies.

• Deployment Server lacks awareness of clustering state, which can lead to inconsistent or conflicting configurations.

Important Warning:

“The Deployment Server should not be used to manage clustered search heads or indexers.”
This is both a best practice and a possible exam-tested point.

3. Forwarder Management App for GUI Control

Splunk provides a Forwarder Management App, which is a GUI-based interface to manage:

• Deployment clients (e.g., Universal Forwarders)

• Server classes and app assignment

• Status of check-ins and app deployments

Access:

• Found within the Settings → Forwarder Management page in Splunk Web (on the DS).

Functional Highlights:

• View all UFs connected to the Deployment Server.

• Group UFs into server classes based on IP, hostname, or metadata.

• Assign deployment apps to each class and track status centrally.

Note for Exam:

While GUI tools are rarely covered in depth on the SPLK-2002 exam, understanding their purpose and capabilities is valuable for real-world operations and may help in scenario-based questions.