SPLK-1003 Configuring Forwarders

Configuring Forwarders Detailed Explanation

Forwarders are essential in a distributed Splunk deployment, responsible for collecting and sending data to indexers. This guide explains forwarder types, configuration tasks, and best practices for setting them up effectively.

1. Types of Forwarders

Splunk provides two main types of forwarders, each designed for specific use cases.

1.1 Universal Forwarder (UF)

• Overview:

  • The Universal Forwarder is a lightweight Splunk instance used primarily for data collection and forwarding.
  • It does not parse or index data, making it highly efficient in terms of resource usage.

• Features:

  • Collects logs and forwards them to Indexers.
  • Optimized for large-scale deployments due to minimal overhead.
  • Supports most input types, such as file monitoring and network streams.

• Use Cases:

  • Collecting logs from servers, applications, or endpoints.
  • Forwarding data in environments where resource consumption must be minimal.

1.2 Heavy Forwarder (HF)

• Overview:

  • The Heavy Forwarder is a full Splunk instance capable of parsing, filtering, and indexing data before forwarding it.
  • It consumes more resources compared to the Universal Forwarder.

• Features:

  • Parses data and applies transformations.
  • Can route data to multiple destinations or index data locally.
  • Acts as a gateway for data aggregation or filtering.

• Use Cases:

  • Preprocessing or filtering sensitive data before forwarding.
  • Acting as an intermediary between data sources and indexers.

2. Configuration Tasks

Properly configuring forwarders is critical for efficient data collection and forwarding.

2.1 Configuring outputs.conf

The outputs.conf file specifies the destination indexers where the forwarder sends data.

Steps:

1. Edit outputs.conf:

   • Specify the indexer’s IP address and port:

     [tcpout]
     defaultGroup = default-autolb-group
     
     [tcpout:default-autolb-group]
     server = 192.168.1.10:9997, 192.168.1.11:9997
     autoLB = true
     

2. Key Settings:

   • server: Defines the indexer IPs and ports.
   • autoLB: Enables load balancing across multiple indexers.

3. Validate Configuration:

   • Restart the forwarder to apply changes:

     ./splunk restart
     

   • Check the connection status:

     ./splunk list forward-server
     

2.2 Configuring inputs.conf

The inputs.conf file defines the data sources that the forwarder will monitor.

Steps:

1. Monitor a File or Directory:

   • Example configuration:

     [monitor:///var/log/app_logs/]
     disabled = false
     sourcetype = app_log
     index = main
     recursive = true
     

2. Enable Network Inputs:

   • Configure TCP or UDP inputs:

     [tcp://514]
     disabled = false
     sourcetype = syslog
     index = syslog_index
     

3. Test Inputs:

   • Restart the forwarder to start data collection:

     ./splunk restart
     

2.3 Centralized Deployment Using Deployment Server

The Deployment Server is used to manage configurations for multiple forwarders.

Steps:

1. Enable Deployment Client on Forwarders:

   • Edit deploymentclient.conf on the forwarder:

     [deployment-client]
     disabled = false
     
     [target-broker:deploymentServer]
     targetUri = deployment_server_ip:8089
     

2. Create Server Classes on Deployment Server:

   • Group forwarders by use case or department:
     • Example server class: Linux_Servers.

3. Deploy Configurations:

   • Assign configurations (e.g., inputs.conf) to the server class.
   • Push configurations to clients.

4. Verify Deployment:

   • Check deployment status:

     ./splunk show deploy-clients
     

3. Best Practices

3.1 Use Universal Forwarders for Most Deployments

• Lightweight and efficient for log collection.
• Ideal for scenarios where no data preprocessing is needed.

3.2 Reserve Heavy Forwarders for Advanced Use Cases

• Use Heavy Forwarders only when data needs to be filtered, parsed, or routed to multiple destinations.
• Avoid using them unnecessarily to conserve resources.

3.3 Secure Data Transmission

• Enable SSL/TLS to encrypt data between forwarders and indexers:

  • Example configuration in outputs.conf:

    [tcpout:default-autolb-group]
    server = 192.168.1.10:9997
    sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
    sslPassword = <password>
    sslRootCAPath = $SPLUNK_HOME/etc/auth/ca.pem
    useClientSSLCompression = true
    

3.4 Monitor Forwarder Performance

• Use the Monitoring Console to track forwarder status and throughput:

  • SPL Query:

    index=_internal source=*metrics.log group=per_host_thruput
    | stats sum(kbps) as throughput by host
    

Real-World Scenarios

Scenario 1: Collecting Logs from Multiple Servers

Goal: Set up Universal Forwarders on multiple servers to collect system logs and forward them to a central Splunk indexer cluster.

Steps:

1. Install Universal Forwarders:

   • Download and install the Universal Forwarder on each server:

     wget -O splunkforwarder.tgz https://www.splunk.com/universalforwarder/download
     tar -xvzf splunkforwarder.tgz
     cd splunkforwarder/bin
     ./splunk start --accept-license
     

2. Configure outputs.conf:

   • Point the forwarders to the indexer cluster:

     [tcpout]
     defaultGroup = indexer_group
     
     [tcpout:indexer_group]
     server = 192.168.1.10:9997, 192.168.1.11:9997
     autoLB = true
     

3. Add Input Sources:

   • Monitor system logs using inputs.conf:

     [monitor:///var/log/]
     disabled = false
     sourcetype = syslog
     index = system_logs
     

4. Restart the Forwarder:

   ./splunk restart
   

5. Verify Connectivity:

   • On the forwarder:

     ./splunk list forward-server
     

   • On the indexer, search for incoming data:

     index=system_logs | stats count by host
     

Scenario 2: Preprocessing Data with a Heavy Forwarder

Goal: Use a Heavy Forwarder to filter and mask sensitive data before forwarding it to indexers.

Steps:

1. Set Up the Heavy Forwarder:

   • Install the full Splunk Enterprise instance on the designated server.

   • Configure it as a Heavy Forwarder by limiting its capabilities:

     ./splunk disable webserver
     ./splunk restart
     

2. Filter Data Using transforms.conf:

   • Add a rule to mask sensitive data:

     [mask_sensitive_data]
     REGEX = (\d{3}-\d{2}-\d{4})
     FORMAT = XXX-XX-XXXX
     DEST_KEY = _raw
     

3. Apply the Transformation in props.conf:

   • Link the transformation to the data source:

     [sensitive_logs]
     TRANSFORMS-mask = mask_sensitive_data
     

4. Forward Data to Indexers:

   • Configure outputs.conf:

     [tcpout]
     defaultGroup = indexer_group
     
     [tcpout:indexer_group]
     server = 192.168.1.10:9997, 192.168.1.11:9997
     

5. Test the Configuration:

   • Ingest a sample log containing sensitive data and confirm that it is masked in the indexed results.

Hands-On Exercises

Exercise 1: Monitor Application Logs

Goal: Configure a Universal Forwarder to monitor application logs and send them to a specific index.

Steps:

1. Edit inputs.conf:

   [monitor:///opt/app/logs/]
   disabled = false
   sourcetype = app_log
   index = app_logs
   

2. Restart the Forwarder:

   ./splunk restart
   

3. Verify Data in Splunk:

   • Run a search to ensure data is indexed:

     index=app_logs sourcetype=app_log | stats count by host
     

Exercise 2: Configure Deployment Server

Goal: Centrally manage configurations for multiple Universal Forwarders using a Deployment Server.

Steps:

1. Enable Deployment Server:

   • On the Deployment Server:

     ./splunk enable deploy-server
     

2. Define a Server Class:

   • Create a class for Linux servers:

     • File path: $SPLUNK_HOME/etc/system/local/serverclass.conf.

     [serverClass:Linux_Servers]
     whitelist.0 = *.example.com
     apps = linux_inputs
     

3. Create an App for Configuration:

   • Create an app named linux_inputs and include inputs.conf:

     [monitor:///var/log/]
     disabled = false
     sourcetype = syslog
     index = linux_logs
     

4. Deploy the Configuration:

   • Reload the Deployment Server:

     ./splunk reload deploy-server
     

5. Verify Deployment on Forwarders:

   • Check deployment status on the forwarders:

     ./splunk show deploy-clients
     

Troubleshooting Forwarder Issues

Common Issues and Fixes

Issue 1: Forwarder Not Sending Data

• Cause:

  • Misconfigured outputs.conf or network connectivity issues.

• Solution:

  1. Verify outputs.conf using btool:

     splunk cmd btool outputs list --debug
     

  2. Test connectivity to the indexer:

     telnet <indexer_ip> 9997
     

Issue 2: Data Not Appearing in Splunk

• Cause:

  • Incorrect inputs.conf configuration or data not matching the sourcetype.

• Solution:

  1. Verify inputs.conf using btool:

     splunk cmd btool inputs list --debug
     

  2. Check if the data is being indexed:

     index=_internal source=*metrics.log group=per_host_thruput
     

Issue 3: Forwarder Not Connecting to Deployment Server

• Cause:

  • Incorrect deploymentclient.conf configuration.

• Solution:

  1. Verify deploymentclient.conf settings:

     splunk cmd btool deploymentclient list --debug
     

  2. Check the connection to the Deployment Server:

     ./splunk list deploy-clients
     

Best Practices Recap

1. Use Universal Forwarders for Lightweight Data Collection:
   • Efficient for most scenarios.
2. Secure Data Transmission:
   • Always enable SSL/TLS to protect data in transit.
3. Centralize Configuration Management:
   • Use the Deployment Server to simplify forwarder management.
4. Monitor Forwarder Performance:
   • Regularly review internal metrics and ensure data flows smoothly.

Configuring Forwarders (Additional Content)

Forwarders are critical for collecting and transmitting data to Splunk indexers. This section expands on essential configurations and advanced management capabilities, particularly for large-scale, production-ready environments.

1. Forwarder Management via Splunk Web (UI-Based)

While configuration via CLI and configuration files is common, Splunk also offers a graphical interface to manage forwarders through the Forwarder Management feature.

Description:

• Forwarder Management is part of the Deployment Server and provides a UI for managing deployment apps and monitoring forwarder connectivity.

Access Path:

• Splunk Web → Settings > Forwarder Management

Key Capabilities:

• View connected forwarders with:

  • Hostname

  • IP Address

  • Last "phone home" timestamp

• Group forwarders into server classes

• Push apps (e.g., inputs.conf, outputs.conf) to multiple forwarders simultaneously

• Monitor deployment health (missing clients, failed updates)

Exam Tip:

• Expect scenario-based questions on "how to deploy an app to 500 forwarders" — UI-based management is frequently mentioned in the Splunk Enterprise Admin Certification.

2. Splunk App Structure for Deployment

Deploying configurations via apps requires a clear directory structure, especially when using Deployment Server.

Standard App Directory Structure:

$SPLUNK_HOME/etc/deployment-apps/linux_inputs/
├── default/
│   └── inputs.conf
├── metadata/
│   └── default.meta

Key Files:

• inputs.conf: Located in the default/ directory, contains input definitions

• default.meta: Located in metadata/, defines file-level permissions and visibility

Best Practices:

• Do not place config files in the local/ directory when building deployment apps.

• Include default.meta to ensure forwarders load the app securely and correctly.

Exam Note:

• You may encounter drag-and-drop questions or MCQs asking which files belong where in a Splunk app.

3. Mutual SSL Authentication (Two-Way SSL)

Beyond basic encryption, Splunk supports mutual SSL to enhance security between forwarders and indexers.

Concept:

• One-way SSL: The client (forwarder) verifies the server’s certificate.

• Mutual SSL: The server also verifies the client's certificate — a more secure handshake.

Configuration Parameters in outputs.conf:

[tcpout:secure_indexer_group]
server = indexer01.example.com:9997
sslCertPath = $SPLUNK_HOME/etc/auth/mycert.pem
sslRootCAPath = $SPLUNK_HOME/etc/auth/ca.pem
sslVerifyServerCert = true
sslCommonNameToCheck = indexer01.example.com

Implications:

• Forwarder must have a valid client certificate.

• Indexer’s inputs.conf must be configured to accept and validate the client’s cert.

Use Case:

• Highly regulated environments (e.g., finance, government).

4. Monitoring Forwarder Health and Throughput

Monitoring forwarder performance is essential for proactive operations.

Key SPL Queries:

1. Forwarder Throughput:

index=_internal source=*metrics.log group=thruput 
| stats sum(kbps) as bandwidth by series

• Shows how much data each forwarder is sending.

2. TCP Output Status:

index=_internal source=*splunkd.log component=TcpOutputProc 
| stats count by destIp status

• Identifies forwarders with failed connections to indexers.

3. Missing Phone Homes:

| deploymentserver 
| search latestTime < relative_time(now(), "-1h@h")

• Helps detect forwarders that haven’t checked in recently.

Exam Tip:

• Expect scenario-based questions like:

  “Which SPL identifies forwarders with connection failures?”

5. Common Configuration File Locations

Memorizing key file paths is helpful for both troubleshooting and certification exams.

File Location Table:

File	Common Location	Purpose
inputs.conf	$SPLUNK_HOME/etc/system/local/ or $SPLUNK_HOME/etc/apps/<your_app>/default/	Configure data inputs
outputs.conf	Same as above	Configure forwarding destinations
deploymentclient.conf	$SPLUNK_HOME/etc/system/local/ or $SPLUNK_HOME/etc/apps/	Configures forwarder to connect to a Deployment Server

Best Practices Recap

1. Use Forwarder Management UI for large-scale forwarder deployment and monitoring.

2. Package deployment apps properly, placing inputs.conf in default/ and adding default.meta.

3. Enable mutual SSL for secure, two-way communication between forwarders and indexers.

4. Regularly monitor throughput, connection health, and deployment status using internal logs.

5. Familiarize yourself with config file locations — these often appear in multiple-choice and troubleshooting questions.