SPLK-3003 Configuration Management

Configuration Management Detailed Explanation

Splunk is a highly configurable platform. Almost every feature in Splunk—inputs, indexing, search behavior, limits, and user roles—is controlled through configuration files. These settings define how Splunk behaves across different components and use cases.

1. Splunk Configuration Files

All configuration files in Splunk are located under the $SPLUNK_HOME/etc/ directory. These files are written in key-value pair format, similar to:

[stanza]
setting1 = value1
setting2 = value2

Each .conf file controls a specific area of Splunk functionality. These are not JSON or XML—they follow Splunk’s own format.

Example:

[monitor:///var/log/apache/access.log]
sourcetype = access_combined
index = web

2. Directory Hierarchy

Splunk organizes configuration files into a layered directory structure, which helps manage global, app-specific, and user-specific settings separately.

a. System-Level

Location: $SPLUNK_HOME/etc/system/

• default/: Contains factory settings provided by Splunk. Do not edit this folder.

• local/: Contains your overrides to system-level configurations. This folder has the highest priority among system-level files.

b. App-Level

Location: $SPLUNK_HOME/etc/apps/<app_name>/

Each app installed in Splunk has:

• default/: App-supplied settings

• local/: Custom settings or overrides created by users or administrators

Apps may define inputs, field extractions, dashboards, and more, and are self-contained.

c. User-Level

Location: $SPLUNK_HOME/etc/users/<username>/<app_name>/

• These are user-specific overrides.

• Useful for custom dashboards, saved searches, or UI preferences.

3. File Precedence Rules

Splunk must decide which configuration value to use when the same setting is defined in multiple locations. This is controlled by a well-defined precedence hierarchy.

Rule: local/ always overrides default/.

Complete Precedence Order (from highest to lowest):

1. User’s app local/

2. User’s app default/

3. App local/

4. App default/

5. System local/

6. System default/

This hierarchy ensures that user-level settings override everything else, and that defaults are only used when no override is defined.

Troubleshooting with btool

The btool command is used to inspect the merged configuration and identify the source of a setting.

Example:

splunk btool inputs list --debug

This shows which files contribute to the final input configuration and their file paths.

4. Common Configuration Files

Below are some of the most frequently used .conf files in Splunk:

inputs.conf

• Defines how and where Splunk ingests data.

• Includes file monitoring, network ports, scripted inputs, etc.

Example:

[monitor:///var/log/messages]
disabled = false
index = os

props.conf

• Controls data parsing, including:

  • Field extractions

  • Timestamp rules

  • Line breaking behavior

Example:

[source::.../apache/access.log]
sourcetype = access_combined

transforms.conf

• Works with props.conf for event transformation.

• Used for:

  • Routing events

  • Masking sensitive fields

  • Extracting custom fields

outputs.conf

• Defines how data is forwarded to other Splunk instances (usually Indexers).

• Key in Universal Forwarder or Heavy Forwarder configuration.

indexes.conf

• Defines index settings, such as:

  • Index names

  • Retention policies

  • Storage paths

limits.conf

• Sets system-wide limits:

  • Maximum search time

  • Number of concurrent searches

  • Memory thresholds

5. Deployment Server (for Forwarders)

In large environments, managing hundreds or thousands of Universal Forwarders manually is inefficient. The Deployment Server allows centralized configuration management for forwarders.

Key Concepts

• Server Classes: Logical groups of forwarders based on criteria like hostnames or IP ranges.

• Deployment Apps: Configuration packages (e.g., inputs.conf) that get pushed to clients.

• Clients: Forwarders that are registered to receive updates.

Key Configuration Files

• deploymentclient.conf: Placed on the forwarder to define its connection to the deployment server.

• serverclass.conf: Placed on the Deployment Server to define client groups and app assignments.

6. Best Practices

Following best practices for configuration management ensures stability, flexibility, and ease of scaling.

Never Edit Files in default/

• These files may be overwritten during upgrades.

• Always place custom settings in the local/ directory.

Use Version Control

• Store configuration files in Git or another version control system.

• Track changes and enable rollbacks when needed.

Structure Apps Modularly

• Build custom apps for each type of function:

  • One for inputs

  • One for parsing rules

  • One for dashboards

• Modular apps make migration and upgrades easier.

Test Before Deploying to Production

• Create a test environment that mirrors production.

• Validate all configurations before pushing them live.

Summary: Configuration Management

Topic	Key Takeaway
File Structure	Uses .conf files in $SPLUNK_HOME/etc/
Directory Hierarchy	Organized into system, app, and user levels
Precedence Rules	local/ overrides default/; user settings take highest priority
Key Config Files	Include inputs.conf, props.conf, transforms.conf, and more
Deployment Server	Central management for Universal Forwarders via server classes
Best Practices	Never edit defaults; use Git; test before production

Configuration Management (Additional Content)

1. Common Fields in outputs.conf

The outputs.conf file is essential for configuring how data is forwarded from a Universal or Heavy Forwarder to Indexers. It’s a commonly tested area, especially in questions about data routing and load balancing.

Key Fields to Know:

• indexer:
  Specifies the address (hostname or IP) of a target indexer.
  Example:
  indexer = 10.1.1.5:9997

• server:
  Specifies one or more target indexers (deprecated in favor of group and indexer in newer versions).

• group:
  Used to define logical sets of indexers to load balance across.
  Example:
  group = defaultGroup

• autoLBFrequency:
  Controls how often (in seconds) the forwarder switches between indexers in a group.
  Default is 30.
  Example:
  autoLBFrequency = 60

• useACK (for reliable delivery with Indexer Acknowledgment):
  Ensures that events are acknowledged by the indexer before being removed from the forwarder's queue.
  useACK = true

Exam Relevance:

You might be asked:

"Which setting controls how often a forwarder rotates between indexers?"

Correct Answer: autoLBFrequency

2. Modular App Naming Convention

In larger environments, Splunk encourages a modular and layered design for apps. This structure supports reusability, upgradeability, and configuration clarity.

Naming Standards:

• TA_app_input:
  Responsible for input definitions only, such as inputs.conf.
  Example: Collect logs from a specific source.

• TA_app_parse:
  Contains parsing logic, such as props.conf and transforms.conf.
  Used for field extractions, sourcetypes, masking.

• SA_common (Shared App):
  Common definitions like macros, lookups, and calculated fields that can be used by other apps.

These names follow Splunk’s Technology Add-on (TA) and Supporting Add-on (SA) conventions.

Exam Relevance:

Questions may ask:

"Which layer of an app should include props.conf transformations?"

Answer: TA_app_parse

3. conf.spec Files and Configuration Precedence

Every .conf file in Splunk is governed by an associated .spec file located in $SPLUNK_HOME/etc/system/README/.

Purpose:

• Defines all legal stanzas and fields for that .conf file.

• Serves as the blueprint for what the config engine expects.

• Helps validate syntax and detect unsupported configurations.

Example: inputs.conf.spec, props.conf.spec

Advanced Detail:

Although .spec files are not loaded at runtime, they define the expected field types, default values, and precedence rules.

This is important for administrators who build custom apps or need deep visibility into config resolution.

Exam Relevance:

This may appear as a "select the most accurate statement" type question, such as:

"What is the purpose of a .spec file in Splunk?"

Answer: To define the legal structure and precedence of configuration fields.

4. Key Fields in deploymentclient.conf

The deploymentclient.conf file is placed on Universal Forwarders (UFs) to connect them to a Deployment Server (DS) for centralized configuration management.

Common Fields:

• targetUri:
  The address (host:port) of the Deployment Server.
  Example: targetUri = ds01.company.com:8089

• clientName:
  A human-readable name for the forwarder (optional).
  Useful for filtering or targeting in serverclass.conf.

• phoneHomeIntervalInSecs:
  Controls how frequently the forwarder contacts the Deployment Server.
  Example: phoneHomeIntervalInSecs = 300

Example stanza:

[deployment-client]
targetUri = ds01.company.com:8089
clientName = web01_forwarder
phoneHomeIntervalInSecs = 600

Exam Relevance:

Questions may test your understanding with:

"Which field in deploymentclient.conf specifies how often the forwarder contacts the Deployment Server?"

Answer: phoneHomeIntervalInSecs

Summary

• outputs.conf: Know how group, indexer, and autoLBFrequency define data routing and load balancing behavior.

• App modularization: Recognize and apply naming conventions like TA_app_input, TA_app_parse, SA_common.

• .spec files: Understand they define config structure and legal fields — helpful in identifying misconfigurations.

• deploymentclient.conf: Key to linking forwarders with the DS, especially through fields like targetUri and clientName.