SPLK-1003 Monitor Inputs

Monitor Inputs Detailed Explanation

Monitoring inputs is one of the most essential tasks in Splunk, as it allows you to collect and index data from a variety of sources. This guide covers monitorable sources, configuration tips, and performance tuning strategies to optimize data ingestion.

1. Monitorable Sources

Splunk can monitor different types of data sources, from file directories to network ports and applications.

1.1 Files and Directories

Splunk can monitor individual files, entire directories, and subdirectories for logs and structured data.

Key Features:

• File Types:
  • Logs (.log), CSVs, JSON files, and configuration files.
• Recursive Monitoring:
  • Monitor all files in a directory and its subdirectories.
• Dynamic Updates:
  • Splunk tracks changes and ingests new data as it’s written.

Example Configuration in inputs.conf:

1. Monitor a Single File:

   [monitor:///var/log/syslog]
   disabled = false
   sourcetype = syslog
   index = main
   

2. Monitor a Directory:

   [monitor:///var/log/app/]
   disabled = false
   sourcetype = app_logs
   index = app_index
   recursive = true
   

3. Filter Files in a Directory:

   • Use whitelist and blacklist to include or exclude specific files:

     [monitor:///var/log/app/]
     disabled = false
     sourcetype = app_logs
     index = app_index
     whitelist = \.log$
     blacklist = error\.log$
     

1.2 Ports

Splunk can listen on TCP and UDP ports for real-time data, such as Syslog messages or HTTP requests.

Key Features:

• Ideal for collecting logs from network devices like firewalls, routers, and servers.
• Supports both structured and unstructured data.

Example Configuration in inputs.conf:

1. Syslog Over UDP:

   [udp://514]
   disabled = false
   sourcetype = syslog
   index = syslog_index
   

2. Custom Data Over TCP:

   [tcp://10514]
   disabled = false
   sourcetype = custom_data
   index = custom_index
   

1.3 Applications

Splunk can collect logs from third-party applications, either directly or via a forwarder.

Examples:

1. Apache Access Logs:

   [monitor:///var/log/apache2/access.log]
   disabled = false
   sourcetype = access_combined
   index = web_logs
   

2. MySQL Logs:

   [monitor:///var/log/mysql/error.log]
   disabled = false
   sourcetype = mysql_error_log
   index = db_logs
   

2. Configuration Tips

Proper configuration ensures that data is classified correctly and only relevant information is ingested.

2.1 Use Blacklist and Whitelist Filters

• Purpose:

  • Prevent unnecessary data from being ingested.

• Example:

  • Monitor only .log files and exclude files containing debug:

    [monitor:///var/log/app/]
    disabled = false
    sourcetype = app_logs
    index = app_index
    whitelist = \.log$
    blacklist = debug\.log$
    

2.2 Configure Metadata for Classification

• Metadata Fields:

  • Host: Identifies the source system.
  • Source: Specifies the data origin (e.g., file path or port).
  • Sourcetype: Determines parsing rules.

• Example:

  [monitor:///var/log/app/server.log]
  disabled = false
  sourcetype = app_server_log
  index = app_index
  host = app_server_01
  

2.3 Validate Configurations with btool

• Purpose:

  • Debug and validate inputs.conf settings.

• Command:

  splunk cmd btool inputs list --debug
  

3. Performance Tuning

Efficient monitoring ensures that Splunk can handle large volumes of data without being overwhelmed.

3.1 Adjust Polling Intervals

• Purpose:

  • Reduce the frequency of checks for high-frequency data sources to avoid performance bottlenecks.

• Example:

  • Monitor a file every 5 minutes:

    [monitor:///var/log/large_file.log]
    interval = 300
    

3.2 Limit Monitored Inputs

• Purpose:
  • Avoid overloading Splunk by monitoring only essential sources.
• Best Practices:
  • Use filters like whitelist and blacklist.
  • Archive or rotate logs that are no longer actively monitored.

3.3 Optimize Resource Usage

• Tips:

  1. Enable Compression:

     • Reduce bandwidth for remote monitoring:

       [tcpout]
       compressed = true
       

  2. Monitor Internal Logs:

     • Identify resource-intensive inputs:

       index=_internal source=*metrics.log group=per_host_thruput
       

4. Best Practices

1. Test Configurations in Staging:

   • Validate inputs.conf settings in a staging environment before applying them to production.

2. Use Modular Inputs:

   • Organize inputs.conf into separate apps for better scalability and management.

3. Regularly Audit Inputs:

   • Periodically review monitored sources to ensure they are still relevant.

4. Leverage Forwarders:

   • Use Universal Forwarders for efficient data collection on remote systems.

Real-World Scenarios

Scenario 1: Monitoring a Directory with Multiple Log Formats

Goal: Monitor a directory containing logs in different formats and assign appropriate sourcetypes to each file.

Steps:

1. Set Up inputs.conf:

   • Use whitelist to match log file patterns:

     [monitor:///var/log/app/]
     disabled = false
     index = app_logs
     
     [monitor:///var/log/app/]
     sourcetype = app_json_logs
     whitelist = \.json$
     
     [monitor:///var/log/app/]
     sourcetype = app_error_logs
     whitelist = error\.log$
     

2. Validate Metadata Assignment:

   • Run a search to verify that logs are categorized correctly:

     index=app_logs | stats count by sourcetype
     

Scenario 2: Listening for Real-Time Syslog Messages

Goal: Configure Splunk to collect real-time Syslog messages from network devices.

Steps:

1. Configure inputs.conf for Syslog:

   [udp://514]
   disabled = false
   sourcetype = syslog
   index = network_logs
   

2. Configure Network Devices:

   • Point the Syslog output of devices to the Splunk server's IP address on UDP port 514.

3. Test the Setup:

   • Use a Syslog generator (e.g., logger) to send test messages:

     logger -n <splunk_server_ip> -P 514 "Test syslog message"
     

4. Verify Data:

   • Search for the test message in Splunk:

     index=network_logs sourcetype=syslog
     

Scenario 3: Excluding Debug Logs Using Filters

Goal: Monitor application logs while excluding debug logs to reduce noise.

Steps:

1. Use blacklist in inputs.conf:

   [monitor:///var/log/app/]
   disabled = false
   sourcetype = app_logs
   index = app_index
   blacklist = debug\.log$
   

2. Verify Exclusion:

   • Run a search to confirm no debug logs are ingested:

     index=app_index NOT source=*debug.log
     

Hands-On Exercises

Exercise 1: Monitor a Rotating Log File

Goal: Set up monitoring for a log file that rotates periodically (e.g., access.log, access.log.1).

Steps:

1. Configure inputs.conf:

   [monitor:///var/log/httpd/access.log]
   disabled = false
   sourcetype = apache_access
   index = web_logs
   followTail = true
   

2. Restart Splunk:

   • Apply the configuration:

     ./splunk restart
     

3. Test Rotation:

   • Simulate log rotation:

     mv /var/log/httpd/access.log /var/log/httpd/access.log.1
     echo "New log entry" >> /var/log/httpd/access.log
     

4. Verify Data:

   • Search for the new entry in Splunk:

     index=web_logs sourcetype=apache_access
     

Exercise 2: Monitor CSV Files

Goal: Monitor a directory containing CSV files and ensure Splunk parses the fields correctly.

Steps:

1. Configure inputs.conf:

   [monitor:///data/csv/]
   disabled = false
   sourcetype = csv
   index = data_index
   

2. Define Field Parsing in props.conf:

   [csv]
   INDEXED_EXTRACTIONS = csv
   HEADER_FIELD_DELIMITER = ,
   

3. Ingest Sample Data:

   • Place a sample CSV file in /data/csv/.

4. Verify Field Extraction:

   • Run a search and display extracted fields:

     index=data_index | table field1, field2, field3
     

Advanced Troubleshooting

Issue 1: Data Not Appearing in Splunk

• Cause:

  • Misconfigured inputs.conf or file permissions.

• Solution:

  1. Check if the monitored file is accessible:

     ls -l /path/to/file.log
     

  2. Validate inputs.conf using btool:

     splunk cmd btool inputs list --debug
     

Issue 2: Duplicate Data in Splunk

• Cause:

  • Log rotation or incorrect CRC settings.

• Solution:

  1. Add a crcSalt value in inputs.conf:

     [monitor:///var/log/app/]
     crcSalt = <SOURCE>
     

  2. Use ignoreOlderThan to avoid re-ingesting old logs:

     [monitor:///var/log/app/]
     ignoreOlderThan = 7d
     

Issue 3: High Latency in Monitoring Inputs

• Cause:

  • Splunk is overwhelmed by high-frequency data sources.

• Solution:

  1. Adjust the polling interval for large files:

     [monitor:///path/to/large/file.log]
     interval = 300
     

  2. Monitor system performance:

     index=_internal source=*metrics.log group=queue
     

Issue 4: Incorrect Timestamp Parsing

• Cause:

  • Misconfigured TIME_FORMAT in props.conf.

• Solution:

  1. Update props.conf with the correct format:

     [custom_sourcetype]
     TIME_FORMAT = %d/%b/%Y:%H:%M:%S
     TIME_PREFIX = \[
     

  2. Test the parsing by ingesting sample data and checking _time values.

Best Practices

1. Regularly Audit Inputs:

   • Use the Monitoring Console to track input performance and identify bottlenecks.

2. Validate Configurations:

   • Test new inputs.conf settings in a staging environment before deploying them to production.

3. Use Filters Effectively:

   • Minimize noise by applying whitelist and blacklist filters.

4. Monitor Resource Usage:

   • Check internal logs to ensure Splunk instances are not overloaded by excessive inputs.

Monitor Inputs (Additional Content)

Monitor inputs in Splunk are used to continuously ingest data from files or directories. They are one of the most commonly used data input methods and provide powerful control over how files are read, indexed, and re-read.

This section elaborates on advanced behaviors of monitor inputs, particularly the CRC (Cyclic Redundancy Check) mechanism, initCrcLength, and file permission requirements—which are often misunderstood but important for exam and production success.

1. CRC Mechanism – How Splunk Identifies Duplicate Files

What is CRC in Splunk?

• CRC (Cyclic Redundancy Check) is a checksum-based mechanism used by Splunk to determine whether a file has already been ingested.

• Splunk computes a hash from the first 256 bytes of a file by default.

• This hash is used to track ingestion history and avoid re-indexing the same file unintentionally.

Implication:

• If a file is renamed but its content remains the same, Splunk will not ingest it again, since the CRC is unchanged.

• This behavior avoids duplicate indexing but can be problematic when content changes slightly.

2. crcSalt – Forcing Re-ingestion of Renamed Files

Purpose:

• The crcSalt setting in inputs.conf modifies how the CRC is calculated by adding entropy (a salt) to the calculation.

Usage:

[monitor:///var/log/myapp/]
crcSalt = <SOURCE>

• <SOURCE> uses the file path as part of the hash calculation, so:

  • Even if file content is the same, but path changes, Splunk will treat it as a new file.

When to Use:

• When rotated logs are renamed (e.g., app.log → app.log.1), but content remains the same and needs re-ingestion.

• When you copy the same file to a new path and want Splunk to treat it as new data.

3. initCrcLength – Handling Large Files with Varying Headers

What It Does:

• This setting limits how much of a file is considered when calculating the CRC.

• By default, Splunk uses the first 256 bytes.

• If the header changes frequently (e.g., timestamps or app info), but the rest is unchanged, it may cause Splunk to skip re-ingestion.

Usage Example:

[monitor:///data/archive/]
initCrcLength = 4096

When to Use:

• If you're ingesting large log files where headers (first few lines) change, but the rest does not.

• This helps Splunk more accurately detect file uniqueness and avoid false negatives in duplication checks.

4. File System Permissions – Critical for Monitor Inputs

Why It Matters:

• Splunk runs under a specific OS user (splunk by default).

• That user must have read access to all directories and files it needs to monitor.

Common Issues in Linux Environments:

• Splunk cannot read rotated logs due to:

  • File ownership by root or app users

  • Lack of read permissions

• Splunk Web shows the monitor input as “enabled,” but no data is ingested.

Troubleshooting Tip:

Use this command to verify permissions:

sudo -u splunk ls -l /var/log/myapp/

Ensure the Splunk process user can:

• Traverse directories (x permission)

• Read files (r permission)

5. Best Practices Recap

Feature	Summary
CRC (Default)	Uses the first 256 bytes of the file to avoid duplicate ingestion.
crcSalt = <SOURCE>	Forces CRC calculation to include file path; useful for rotated or copied logs.
initCrcLength	Expands the CRC hash size to better differentiate files with similar headers.
Permissions	Always ensure Splunk user has proper access to input directories and files.

Real-World Deployment Example

Scenario: Ingesting Rotated Application Logs

• Application writes logs to /opt/logs/app.log and rotates to /opt/logs/app.log.1, /opt/logs/app.log.2.

• Contents are mostly the same.

• Splunk is skipping app.log.1.

Fix Configuration:

[monitor:///opt/logs/]
crcSalt = <SOURCE>
initCrcLength = 2048
sourcetype = app_logs
index = prod_index

This forces Splunk to re-ingest rotated logs based on file path and longer CRC range.

Commands to Verify Ingestion Behavior

splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus

This shows files being monitored and their CRC status.