SPLK-1003 Splunk Configuration Files

Splunk Configuration Files Detailed Explanation

Splunk configuration files are critical for controlling the behavior of your Splunk environment. These files determine how data is collected, forwarded, parsed, and stored. This guide explains key configuration files, how they work, and how to validate and troubleshoot them.

1. Critical Configuration Files

Splunk uses multiple configuration files, each serving a specific purpose. Let’s explore the most important ones in detail.

1.1 inputs.conf

• Purpose:

  • Defines how and where Splunk collects data.
  • Specifies data input sources such as files, directories, ports, or APIs.

• Key Fields:

  • monitor: Monitors a file or directory for changes.
  • tcp/udp: Configures Splunk to listen for data on specific network ports.
  • script: Executes scripts that generate data.

• Example Configurations:

  • File Monitoring:

    [monitor:///var/log/syslog]
    disabled = false
    index = main
    sourcetype = syslog
    

  • Network Input (Syslog):

    [udp://514]
    disabled = false
    index = syslog_index
    sourcetype = syslog
    

  • Script Input:

    [script://./bin/myscript.sh]
    disabled = false
    index = custom_index
    sourcetype = custom_script
    

1.2 outputs.conf

• Purpose:

  • Specifies where data collected by Splunk is forwarded.
  • Commonly used by Forwarders to send data to Indexers.

• Key Fields:

  • tcpout: Defines the destination Indexers.
  • group: Organizes multiple Indexers for load balancing.

• Example Configurations:

  • Single Indexer:

    [tcpout]
    defaultGroup = default-autolb-group
    
    [tcpout:default-autolb-group]
    server = 192.168.1.10:9997
    

  • Multiple Indexers with Load Balancing:

    [tcpout]
    defaultGroup = indexer_group
    
    [tcpout:indexer_group]
    server = 192.168.1.10:9997,192.168.1.11:9997
    autoLB = true
    

1.3 props.conf

• Purpose:

  • Controls how data is parsed during ingestion.
  • Defines rules for field extraction, timestamp handling, and event breaking.

• Key Fields:

  • TIME_FORMAT: Specifies the timestamp format in the data.
  • BREAK_ONLY_BEFORE: Splits events based on specific patterns.
  • REPORT: Refers to field extraction rules in transforms.conf.

• Example Configurations:

  • Timestamp Recognition:

    [apache_access]
    TIME_FORMAT = %d/%b/%Y:%H:%M:%S %z
    MAX_TIMESTAMP_LOOKAHEAD = 32
    

  • Event Breaking:

    [multiline_events]
    SHOULD_LINEMERGE = true
    BREAK_ONLY_BEFORE = ^\d{4}-\d{2}-\d{2}
    

  • Field Extraction:

    [custom_logs]
    REPORT-fields = extract_custom_fields
    

1.4 transforms.conf

• Purpose:

  • Defines advanced rules for manipulating raw data.
  • Used for tasks like masking sensitive information, filtering data, or routing events.

• Key Fields:

  • REGEX: Specifies the pattern to match.
  • FORMAT: Defines the transformation or destination.
  • DEST_KEY: Specifies where the transformation applies.

• Example Configurations:

  • Masking Sensitive Data:

    [mask_ssn]
    REGEX = \b\d{3}-\d{2}-\d{4}\b
    FORMAT = XXX-XX-XXXX
    DEST_KEY = _raw
    

  • Event Routing:

    [route_to_null]
    REGEX = .*debug.*
    DEST_KEY = queue
    FORMAT = nullQueue
    

2. File Precedence and Layering

Understanding how Splunk applies configuration settings is essential for troubleshooting and customizing behavior.

2.1 Configuration Layers

1. User Layer:
   • Located in $SPLUNK_HOME/etc/users/.
   • Contains configurations specific to individual users.
2. App Layer:
   • Located in $SPLUNK_HOME/etc/apps/.
   • Contains settings for Splunk apps, which can include data sources and dashboards.
3. System Layer:
   • Located in $SPLUNK_HOME/etc/system/.
   • Contains system-wide default configurations.

2.2 Priority Rules

1. Local Configurations:

   • Settings in the local directory of an app override those in the default directory.
   • Example:
     • $SPLUNK_HOME/etc/apps/<app_name>/local/props.conf takes precedence over $SPLUNK_HOME/etc/apps/<app_name>/default/props.conf.

2. Global vs. App-Specific Settings:

   • App-specific configurations can override global configurations depending on the layer.

3. Tools for Validation

Validating and debugging configuration files ensures your changes are applied correctly and avoid errors.

3.1 btool Command

• Purpose:

  • Helps debug and validate Splunk configuration files.
  • Lists active settings based on precedence.

• Syntax:

  splunk cmd btool <config_file> list --debug
  

• Examples:

  • List all active inputs:

    splunk cmd btool inputs list --debug
    

  • Debug props.conf settings for a specific sourcetype:

    splunk cmd btool props list apache_access --debug
    

3.2 Common Debugging Scenarios

• Missing Data:
  • Check inputs.conf to ensure the data source is enabled.
• Incorrect Parsing:
  • Verify props.conf and transforms.conf for parsing rules.
• Forwarding Issues:
  • Check outputs.conf for the correct destination Indexers.

Hands-On Examples

Configuring inputs.conf

Let’s start by configuring inputs.conf for various data inputs.

Example 1: Monitoring a Directory

Scenario: You want to monitor all log files in /var/log/app/.

Configuration:

[monitor:///var/log/app/]
disabled = false
index = app_logs
sourcetype = app_log
recursive = true

Explanation:

• monitor: Specifies the directory path.
• disabled = false: Ensures the input is active.
• index = app_logs: Data is sent to the app_logs index.
• recursive = true: Includes files in subdirectories.

Example 2: Listening on a TCP Port

Scenario: Your system needs to collect logs from a remote server via TCP on port 10514.

Configuration:

[tcp://10514]
disabled = false
index = network_logs
sourcetype = tcp_data

Explanation:

• tcp://10514: Configures Splunk to listen on port 10514.
• sourcetype = tcp_data: Ensures proper parsing of incoming logs.

Configuring outputs.conf

Now, let’s configure data forwarding to Indexers.

Example 1: Single Indexer

Scenario: Forward data to an Indexer at 192.168.1.10.

Configuration:

[tcpout]
defaultGroup = single_indexer_group

[tcpout:single_indexer_group]
server = 192.168.1.10:9997

Example 2: Multiple Indexers with Load Balancing

Scenario: Use multiple Indexers for load balancing.

Configuration:

[tcpout]
defaultGroup = indexer_group

[tcpout:indexer_group]
server = 192.168.1.10:9997,192.168.1.11:9997
autoLB = true

Explanation:

• autoLB = true: Enables load balancing across Indexers.

Configuring props.conf and transforms.conf

Parsing and transforming data is an essential part of Splunk’s ingestion process.

Example 1: Parsing Apache Access Logs

Scenario: Parse Apache access logs and extract fields like IP address, HTTP method, and status code.

props.conf:

[apache_access]
TIME_FORMAT = %d/%b/%Y:%H:%M:%S %z
MAX_TIMESTAMP_LOOKAHEAD = 32
REPORT-apache_fields = extract_apache_fields

transforms.conf:

[extract_apache_fields]
REGEX = ^(?P<client_ip>\d+\.\d+\.\d+\.\d+) - - \[(?P<timestamp>[^\]]+)\] "(?P<method>\w+) (?P<uri>[^\s]+) HTTP/\d\.\d" (?P<status>\d+)
FORMAT = client_ip::$1 timestamp::$2 method::$3 uri::$4 status::$5

Explanation:

• props.conf:
  • Specifies the timestamp format and links to the extraction rule in transforms.conf.
• transforms.conf:
  • Uses regular expressions to extract fields and assign them to specific names.

Example 2: Masking Sensitive Data

Scenario: Mask Social Security Numbers (SSNs) in logs.

props.conf:

[source::/var/log/app_logs]
TRANSFORMS-mask = mask_ssn

transforms.conf:

[mask_ssn]
REGEX = (\d{3}-\d{2}-\d{4})
FORMAT = XXX-XX-XXXX
DEST_KEY = _raw

Explanation:

• DEST_KEY = _raw: Rewrites the raw event data, replacing SSNs with masked values.

Debugging with btool

Use btool to validate and troubleshoot configurations.

Example: Checking Active Settings

Command:

splunk cmd btool props list apache_access --debug

Output:

• Displays the active props.conf settings for the apache_access sourcetype.
• Highlights file paths and precedence for each setting.

Troubleshooting Configurations

Common Issues and Solutions

Issue 1: Missing Data

Cause:

• Misconfigured inputs.conf.

Solution:

1. Verify the data source path:

   splunk cmd btool inputs list --debug
   

2. Ensure the input is not disabled (disabled = false).

Issue 2: Incorrect Field Extraction

Cause:

• Errors in props.conf or transforms.conf.

Solution:

1. Test the extraction rule using the Field Extractor tool in Splunk Web.

2. Validate props.conf settings:

   splunk cmd btool props list sourcetype_name --debug
   

Issue 3: Forwarder Not Sending Data

Cause:

• Misconfigured outputs.conf or network connectivity issues.

Solution:

1. Check Forwarder’s connection to the Indexer:

   splunk list forward-server
   

2. Verify outputs.conf for correct Indexer details.

Advanced Configurations

Dynamic Event Routing

Route events to specific indexes based on their source.

props.conf:

[source::/var/log/app_logs]
TRANSFORMS-routing = route_to_index

transforms.conf:

[route_to_index]
REGEX = .*error.*
DEST_KEY = _MetaData:Index
FORMAT = error_logs

Explanation:

• Events containing the word error are routed to the error_logs index.

Field Aliases

Create alternative names for fields to simplify queries.

props.conf:

[apache_access]
FIELDALIAS-uri_path = uri AS path

Explanation:

• The field uri is accessible as path in searches.

Best Practices for Configuration Management

Use Apps to Organize Configurations

• Store configuration files in $SPLUNK_HOME/etc/apps/<app_name>.
• Use the local directory for custom changes.

Use Comments

• Add comments to explain configurations:

  # This input monitors Apache logs
  [monitor:///var/log/apache]
  

Test in a Staging Environment

• Apply changes in a test environment before deploying to production.

Splunk Configuration Files (Additional Content)

Splunk relies heavily on configuration files (conf files) to define system behavior, from data ingestion to search-time processing. This section focuses on fields.conf, limits.conf, configuration reload behaviors, common syntax mistakes, and centralized management via the Deployment Server.

1. fields.conf – Field Behavior Configuration

Purpose

The fields.conf file defines how fields behave after extraction, especially during search-time and display.

Key Use Cases

• Control whether a field:

  • Is searchable

  • Is indexed

  • Supports multi-value behavior

  • Is case-sensitive or case-insensitive

Example Configuration

[custom_field]
INDEXED = false
SEARCHABLE = true

• INDEXED = false: Field is not indexed, reducing disk usage.

• SEARCHABLE = true: Field is available for search-time filtering and display.

Typical Location

• System-wide:

  $SPLUNK_HOME/etc/system/local/fields.conf
  

• App-specific:

  $SPLUNK_HOME/etc/apps/<app_name>/local/fields.conf
  

2. limits.conf – System Limits and Resource Tuning

Purpose

The limits.conf file is used to define performance thresholds and ingestion-related system limits.

Common Parameters

Parameter	Description
maxKBps	Limits indexer throughput (in kilobytes per second)
event_breaker	Defines event boundary behavior for certain inputs
maxDistEvents	Controls line distance for multiline event correlation
max_event_size	Specifies the maximum size for a single event (in bytes)

Example Configuration

[thruput]
maxKBps = 5120

This caps indexer throughput at 5MB/sec.

Location

• Default or custom tuning:

  $SPLUNK_HOME/etc/system/local/limits.conf
  

3. Configuration Reload vs Restart

Understanding which configuration files require a full Splunk restart vs those that support hot reload is crucial for minimizing downtime.

Behavior Summary

Config File Type	Hot Reload Supported	Notes
inputs.conf	No	Requires splunk restart to monitor new inputs
outputs.conf	Partial	May reload via CLI but restart is safer
props.conf, transforms.conf	Yes	Supports reload for field extraction and masking rules
indexes.conf	No	Must restart to apply changes

Reloading via CLI (when supported)

splunk _internal call /services/data/inputs/monitor/_reload

Or restart:

splunk restart

In Clustered Environments

Use:

splunk apply cluster-bundle

• Applies updated configs to all peer nodes in an indexer cluster

• Does not restart Splunk entirely but may restart indexing pipelines

4. Configuration Syntax Errors and Debugging

Misconfigured .conf files are a frequent cause of Splunk issues. Knowing common mistakes and how to debug them is essential.

Common Syntax Mistakes

• Misspelled keys (e.g., disable = true instead of disabled = true)

• Missing equals signs (=) between key and value

• Placing the stanza header in the wrong section

• Improper indentation or inconsistent formatting (INI format is forgiving, but consistency is advised)

Debugging Tools: btool

Splunk provides the btool command to inspect final configuration values and detect overrides or misplacement.

Examples:

splunk cmd btool props list --debug
splunk cmd btool transforms list --debug
splunk cmd btool indexes list --debug

• Use --debug to view the full path and file source of each parameter.

• Helpful to detect if settings are read from default/ or overridden in local/.

Other Troubleshooting Tips

• Always place changes in local/, not default/.

• Remember to restart Splunk if you're changing files like inputs.conf or indexes.conf.

• Use the _internal index to trace config application errors.

5. Deployment Server for Configuration Distribution

A Deployment Server (DS) is used to manage and push configuration files to a large fleet of Forwarders, such as Universal Forwarders on endpoints or Heavy Forwarders in DMZs.

Key Components

File	Purpose
deploymentclient.conf	Configures the client (Forwarder) to connect to the DS
serverclass.conf	Defines which apps/configs are delivered to which clients

Common Use Cases

• Deploy consistent inputs.conf across all forwarders

• Update outputs.conf for indexer destinations

• Distribute props.conf and transforms.conf rules for parsing at the edge

Example Scenario

You have 300 forwarders and want them all to monitor /var/log/app/.

1. Create an app (e.g., TA-app_logs) with inputs.conf:

[monitor:///var/log/app/]
index = app_index
sourcetype = app_logs

2. Define a server class in serverclass.conf:

[serverClass:AppLogs]
whitelist.0 = *
app.0 = TA-app_logs

3. All matching clients receive and apply the configuration automatically.

Best Practices

• Organize forwarders into logical groups (server classes)

• Use the Deployment Monitor app to track deployment status

• Avoid pushing large apps too frequently; consider versioning