SPLK-1003 Splunk Admin Basics

Splunk Admin Basics Detailed Explanation

Core Components of Splunk

Splunk’s architecture relies on several key components, each with a specific role in the data lifecycle.

1. Search Head

The Search Head is the user interface component of Splunk. It’s where users interact with Splunk to:

• Perform searches: Write queries in the Splunk Processing Language (SPL) to extract insights.
• Create dashboards: Build visual representations of your data using charts, graphs, and other widgets.
• Set up alerts: Configure notifications for when specific conditions are met in the data.
• Generate reports: Summarize and share key findings from your data.

Key Features of Search Heads:

• In a distributed environment, the Search Head communicates with Indexers to retrieve and process search results.
• It does not store data itself; it only manages and executes queries.

2. Indexer

The Indexer is the backbone of Splunk’s data storage and processing capabilities. Its main responsibilities include:

• Storing data: The Indexer takes raw data and writes it into buckets (logical storage units).
• Making data searchable: It processes raw data during ingestion to create searchable events.

How it Works:

• The Indexer processes data by breaking it into events and assigning metadata such as timestamps, source, and sourcetype.
• It organizes the data into hot, warm, cold, and frozen buckets based on the age and usage of the data.

3. Forwarders

Forwarders are Splunk’s data collection agents that send data to Indexers. There are two main types:

1. Universal Forwarder (UF):

   • A lightweight, resource-efficient forwarder.
   • Used for forwarding raw data without processing.
   • Ideal for collecting logs from servers, applications, and endpoints.

2. Heavy Forwarder (HF):

   • Capable of parsing, filtering, and processing data before forwarding.
   • Useful when you need to reduce data volume or apply advanced transformations.

When to Use Each:

• Use a Universal Forwarder for basic data forwarding needs.
• Use a Heavy Forwarder when you need preprocessing capabilities (e.g., masking sensitive data).

4. Cluster Manager

The Cluster Manager is critical for managing Splunk’s clustering features. It ensures high availability and data redundancy in distributed environments.

Responsibilities:

• Manages Indexer Clusters: Coordinates data replication across multiple indexers to prevent data loss.
• Manages Search Head Clusters: Synchronizes configurations and schedules between clustered search heads.

5. Deployment Server

The Deployment Server simplifies managing Splunk instances, especially forwarders, by acting as a centralized configuration management tool.

Key Functions:

• Push configurations: Distribute configuration updates (e.g., inputs, outputs) to forwarders.
• Monitor forwarders: Track the status of connected forwarders and ensure they’re functioning correctly.

Splunk Data Pipeline

Splunk processes data through a pipeline consisting of several distinct stages. Each stage transforms and prepares the data for analysis.

1. Input Stage

• Purpose: This is where Splunk collects data from various sources.
• Input Methods:
  • Monitor files and directories (e.g., log files).
  • Collect network data via TCP/UDP or Syslog.
  • Use APIs to ingest structured data.
  • Run scripts to fetch dynamic data.

2. Parsing Stage

• Purpose: Splunk processes the raw data and prepares it for indexing.
• Key Actions:
  • Splits raw data into individual events.
  • Assigns metadata (e.g., host, source, sourcetype) based on predefined rules.
  • Resolves timestamps for proper event ordering.

3. Indexing Stage

• Purpose: This stage converts parsed data into a searchable format.
• Key Actions:
  • Stores the data in index buckets for quick retrieval.
  • Applies data retention policies to manage storage.

4. Search Stage

• Purpose: Allows users to analyze the data using the Splunk Processing Language (SPL).
• Key Features:
  • Users write queries to extract, filter, and visualize data.
  • Dashboards, alerts, and reports are generated during this stage.

Basic Administration Tasks

As a Splunk administrator, your daily responsibilities will include installation, management, and command-line operations.

Installation

1. Download Splunk:
   • Visit Splunk’s official website to download the appropriate version for your operating system.
2. Install Splunk:
   • Windows: Follow the graphical installer wizard.
   • Linux: Use a .tgz package or install via a package manager.
   • Mac: Use the .dmg package for installation.
3. Initial Configuration:
   • Set the admin username and password.
   • Verify Splunk is running by accessing the Web UI (e.g., http://<hostname>:8000).

Management

1. Restarting Splunk Services:

   • Use Splunk Web or the CLI to start, stop, or restart services.
   • Example CLI commands:
     • splunk start: Starts the Splunk service.
     • splunk stop: Stops the Splunk service.
     • splunk restart: Restarts Splunk, applying any new configurations.

2. Monitoring System Health:

   • Check Splunk’s resource usage and performance metrics via the Monitoring Console.
   • Review system logs for errors or warnings.

3. Troubleshooting Errors:

   • Examine Splunk’s internal logs (e.g., splunkd.log).
   • Identify and resolve indexing or forwarding issues.

Splunk CLI Commands

The Splunk Command-Line Interface (CLI) is a powerful tool for managing and troubleshooting Splunk instances.

1. Service Management:
   • splunk start / splunk stop / splunk restart: Control the service lifecycle.
2. System Monitoring:
   • splunk show config: Displays the current configuration.
   • splunk show license-status: Shows license usage and status.
3. Configuration Validation:
   • splunk btool: Debug configuration files and identify issues.

Core Components of Splunk (Extended)

1. Search Head

The Search Head is where all user interactions with Splunk occur, making it a critical component. Let’s break down its key functions further:

• Search Query Execution:

  • Users write SPL (Search Processing Language) queries on the Search Head.

  • The Search Head distributes these queries to Indexers for execution and collects the results.

  • Example SPL query:

    index=web_logs sourcetype=apache | stats count by status
    

• Dashboards and Reports:

  • Dashboards: Combine multiple visualizations like bar charts, line graphs, and pie charts to represent search results dynamically.
  • Reports: Static representations of search results that can be scheduled for regular delivery.

• Search Head Clustering:

  • In large environments, multiple Search Heads can form a cluster to handle high query loads.
  • Clustering requires synchronization of knowledge objects like saved searches, dashboards, and alerts.

Common Issues and Solutions:

• High CPU Usage: Caused by complex searches. Optimize queries and avoid overly large time ranges.
• Search Delays: Often due to communication issues with Indexers. Verify Indexer availability and network latency.

2. Indexer

The Indexer plays a pivotal role in data ingestion and search performance. Here’s a deeper look at its functionality:

• Indexing Process:

  1. Splits raw data into events.
  2. Assigns metadata such as timestamps, sources, and sourcetypes.
  3. Stores indexed data into buckets:
     • Hot: Actively written buckets.
     • Warm: Closed, searchable buckets.
     • Cold: Older, archived buckets.
     • Frozen: Beyond retention policy; data is deleted or archived externally.

• Indexer Clustering:

  • Ensures data availability by replicating data across multiple Indexers.
  • Replication Factor: The number of copies of data maintained in the cluster.
  • Search Factor: The number of searchable copies available.

Monitoring Indexer Health:

• Use the Monitoring Console to check Indexer performance metrics like disk I/O, memory usage, and indexing latency.

3. Forwarders

Forwarders act as data collectors and are the primary method for sending data to Indexers.

• Universal Forwarder (UF):

  • Lightweight and efficient.
  • Ideal for scenarios where minimal system impact is crucial, such as on production servers.

• Heavy Forwarder (HF):

  • Includes parsing and filtering capabilities.
  • Use cases:
    • Pre-processing data to reduce ingestion volume.
    • Masking sensitive data before forwarding.

Configuration Examples:

• Universal Forwarder:

  # inputs.conf on UF
  [monitor:///var/log/syslog]
  disabled = false
  index = main
  sourcetype = syslog
  

• Heavy Forwarder:

  # props.conf and transforms.conf on HF
  [source::/var/log/syslog]
  TRANSFORMS-anonymize = mask_ssn
  
  # transforms.conf
  [mask_ssn]
  REGEX = (\d{3}-\d{2}-\d{4})
  FORMAT = XXX-XX-XXXX
  DEST_KEY = _raw
  

4. Cluster Manager

The Cluster Manager is the control node in clustered Splunk deployments. Its main role is to manage Indexer and Search Head clusters.

• Key Features:

  • Maintains cluster state (e.g., which Indexers are active).
  • Ensures replication and search factors are met.
  • Coordinates failover in case of Indexer failures.

• How It Works:

  • Monitors and assigns data replication tasks.
  • Balances data distribution across Indexers for even load distribution.

Best Practices:

• Always monitor the Cluster Manager’s health using logs and metrics.
• Use replication and search factors aligned with your data redundancy needs (e.g., RF=2, SF=2 for high availability).

5. Deployment Server

The Deployment Server simplifies managing configurations for Splunk instances, especially forwarders.

• How It Works:

  • Splunk apps containing configurations (e.g., inputs, outputs) are created and deployed to forwarders.
  • Forwarders poll the Deployment Server for updates.

• Setting Up Deployment:

  • Define server classes to group forwarders with similar configurations.

  • Example:

    # serverclass.conf
    [serverClass:LinuxServers]
    whitelist.0 = linux_server_*
    app.0 = linux_inputs
    

• Monitoring Forwarders:

  • Use the Deployment Server’s status dashboard to track deployment success.

Splunk Data Pipeline (Extended)

Let’s revisit the data pipeline stages with more details and examples.

1. Input Stage

• Collects raw data from various sources:

  • Files and Directories: Log files, configuration files, etc.
  • Network Streams: Syslog, TCP/UDP connections.
  • APIs and Custom Scripts: Collect dynamic or external data.

• Example: Monitoring a log file.

  # inputs.conf
  [monitor:///var/log/apache/access.log]
  index = web_logs
  sourcetype = apache_access
  

2. Parsing Stage

• Tokenizes raw data into events and assigns metadata.

• Key Parsing Rules:

  • Sourcetypes: Define how data should be parsed.
  • Field Extractions: Use regex to identify key-value pairs.

• Example of Field Extraction:

  • Data: 192.168.1.1 - - [01/Jan/2025:12:00:00 +0000] "GET /index.html HTTP/1.1" 200
  • Fields extracted:
    • IP Address: 192.168.1.1
    • HTTP Method: GET
    • Response Code: 200

3. Indexing Stage

• Writes events into index buckets for efficient storage and retrieval.

• Example of Index Configuration:

  # indexes.conf
  [web_logs]
  homePath = $SPLUNK_DB/web_logs/db
  coldPath = $SPLUNK_DB/web_logs/colddb
  frozenTimePeriodInSecs = 2592000  # 30 days
  

4. Search Stage

• Allows users to query and visualize data.

• Example SPL Query:

  index=web_logs sourcetype=apache_access | stats count by status
  

Basic Administrative Tasks

1. Installation

Installing Splunk involves downloading and configuring it on your preferred operating system. Let’s dive into the process.

Step-by-Step Installation

1. Download Splunk:

   • Go to the Splunk Downloads Page and select the correct version for your OS.

2. Install Splunk:

   • Windows:

     • Run the .msi installer.
     • Follow the GUI wizard to set the installation path and admin credentials.

   • Linux:

     • For .tgz:

       tar xvzf splunk-<version>-Linux-x86_64.tgz -C /opt
       cd /opt/splunk/bin
       ./splunk start --accept-license
       

     • For .deb or .rpm:

       sudo dpkg -i splunk-<version>-Linux-x86_64.deb
       sudo service splunk start
       

   • Mac:

     • Install the .dmg file and drag Splunk to the Applications folder.

3. Initial Setup:

   • Access Splunk Web at http://<hostname>:8000.
   • Log in with the default credentials:
     • Username: admin
     • Password: changeme (prompted to change on first login).

Best Practices for Installation

• Install Splunk in a directory with sufficient storage to handle logs and data growth.
• Use a dedicated user account for running Splunk on production systems.
• Secure the Splunk instance by configuring SSL for Splunk Web and data transfers.

2. Managing Splunk Services

Managing Splunk services is crucial for ensuring uptime and applying updates or configuration changes. This can be done via Splunk Web or the CLI.

Common Service Commands (CLI)

• Starting Splunk:

  ./splunk start
  

  Use this command to start the Splunk services after installation or a shutdown.

• Stopping Splunk:

  ./splunk stop
  

  Stops Splunk safely. Use before applying significant configuration changes.

• Restarting Splunk:

  ./splunk restart
  

  Applies new configurations by restarting the service.

• Checking Status:

  ./splunk status
  

  Shows whether Splunk is currently running.

Using Splunk Web for Service Management

1. Navigate to the Settings menu.
2. Access Server Controls to restart, shut down, or view system information.

3. Monitoring System Health

Monitoring system health ensures that Splunk components are running optimally. Use built-in tools and dashboards to track performance and resolve issues.

Monitoring Console

1. Access:
   • Navigate to Settings > Monitoring Console in Splunk Web.
2. Key Metrics:
   • Indexing Performance: Monitor disk I/O and latency for indexers.
   • Search Performance: Track the speed of searches and system resource usage.
   • Forwarder Management: Verify the status of connected forwarders.

Log Files for Monitoring

• splunkd.log:
  • Contains information about Splunk’s internal operations.
  • Location: $SPLUNK_HOME/var/log/splunk/splunkd.log
• metrics.log:
  • Tracks performance metrics for indexing and searches.
  • Location: $SPLUNK_HOME/var/log/splunk/metrics.log

Common CLI Commands for Monitoring

• Show license status:

  ./splunk show license-status
  

• List configured indexes:

  ./splunk list index
  

• View forwarder status:

  ./splunk list forward-server
  

4. Troubleshooting

Troubleshooting is a vital skill for a Splunk administrator. Here are common issues and solutions:

Common Issues and Fixes

1. Splunk Service Fails to Start

   • Cause: Low memory, corrupted configurations, or port conflicts.

   • Fix:

     • Check splunkd.log for error messages.

     • Verify the port (default: 8000) isn’t in use by another process:

       netstat -tuln | grep 8000
       

2. High CPU or Memory Usage

   • Cause: Inefficient SPL queries or high data ingestion rates.
   • Fix:
     • Optimize SPL queries using tstats or summary indexing.
     • Reduce data volume by configuring filters in props.conf and transforms.conf.

3. Forwarder Not Sending Data

   • Cause: Incorrect outputs.conf or network issues.

   • Fix:

     • Verify forwarder connectivity:

       ./splunk list forward-server
       

     • Check splunkd.log on the forwarder for errors.

4. License Warnings

   • Cause: Exceeding daily indexing limits.

   • Fix:

     • Monitor license usage:

       ./splunk show license-status
       

     • Reduce data ingestion by filtering unnecessary logs.

Troubleshooting Tools

• btool:

  • Validates and debug configuration files.

  • Example:

    ./splunk btool inputs list --debug
    

• diag:

  • Collects diagnostic information for troubleshooting:

    ./splunk diag
    

5. Optimizing Splunk for Performance

Efficient Splunk configurations can significantly improve performance.

Indexing Optimization

• Use multiple indexers in a clustered setup for high-volume environments.
• Define retention policies to manage disk space efficiently.

Search Optimization

• Avoid using wildcards (*) at the start of search terms.
• Use summary indexing to precompute results for recurring searches.

Forwarder Optimization

• Limit the scope of monitored files using whitelists and blacklists in inputs.conf.

• Compress forwarded data to reduce network usage:

  [tcpout]
  compressed = true
  

Real-World Applications of Splunk

Scenario 1: Monitoring Server Logs

A company wants to monitor server logs to identify errors, warnings, and system health metrics.

Steps:

1. Install the Universal Forwarder on each server.

2. Configure the inputs.conf file to monitor server log files:

   [monitor:///var/log/syslog]
   disabled = false
   index = server_logs
   sourcetype = syslog
   

3. On the Indexer, create a new index for server logs in indexes.conf:

   [server_logs]
   homePath = $SPLUNK_DB/server_logs/db
   coldPath = $SPLUNK_DB/server_logs/colddb
   frozenTimePeriodInSecs = 2592000  # Retain for 30 days
   

4. Use the Search Head to create a search for warnings:

   index=server_logs sourcetype=syslog "warning"
   

Outcome:

You can now monitor real-time server warnings and set up alerts for critical events.

Scenario 2: Analyzing Web Traffic

Your organization wants to track website traffic to identify popular pages, response times, and errors.

Steps:

1. Configure the web server to forward access logs to Splunk using a Universal Forwarder.

2. Define an input in inputs.conf:

   [monitor:///var/log/apache/access.log]
   disabled = false
   index = web_traffic
   sourcetype = apache_access
   

3. Create a new index for web logs:

   [web_traffic]
   homePath = $SPLUNK_DB/web_traffic/db
   coldPath = $SPLUNK_DB/web_traffic/colddb
   frozenTimePeriodInSecs = 2592000  # Retain for 30 days
   

4. Build a search to calculate page visit counts:

   index=web_traffic sourcetype=apache_access | stats count by uri_path
   

Outcome:

You can visualize popular pages and response patterns in dashboards.

Hands-On Exercises

Exercise 1: Configure and Test a Forwarder

1. Install a Universal Forwarder on a test server.

2. Configure inputs.conf to monitor a sample log file.

3. Configure outputs.conf to forward data to an Indexer:

   [tcpout]
   defaultGroup = default-autolb-group
   
   [tcpout:default-autolb-group]
   server = 192.168.1.10:9997
   

4. Validate the setup using the CLI:

   ./splunk list forward-server
   

Exercise 2: Create a Custom Dashboard

1. Run a search query to find errors:

   index=server_logs sourcetype=syslog "error"
   

2. Save the search as a report.

3. Add the report to a dashboard:

   • Navigate to Dashboards > Create New Dashboard.
   • Add a panel and link the saved report.

Exercise 3: Optimize a Search

1. Original query:

   index=web_traffic sourcetype=apache_access | stats count by uri_path
   

2. Optimized query using tstats:

   | tstats count where index=web_traffic by uri_path
   

3. Compare performance metrics:

   • Use the Job Inspector to analyze query execution times.

Best Practices for Splunk Administration

Data Management

• Use filters in props.conf and transforms.conf to exclude irrelevant data.
• Implement retention policies to manage disk space.

Search Optimization

• Use summary indexing for frequently used reports.
• Leverage tstats for high-performance searches.

Security

• Enable SSL/TLS for data transport and Splunk Web.
• Regularly update user roles and permissions.

Monitoring

• Configure proactive alerts for resource usage (e.g., CPU, memory).
• Use the Monitoring Console to identify bottlenecks in real-time.

Frequently Asked Questions (FAQs)

Q1: How do I troubleshoot data ingestion issues?

1. Check the forwarder’s splunkd.log for errors.

2. Verify data is reaching the Indexer:

   ./splunk list forward-server
   

3. Ensure inputs.conf and outputs.conf are correctly configured.

Q2: How do I manage large-scale deployments?

• Use Indexer Clustering to handle high data volumes.
• Deploy configurations via the Deployment Server.

Q3: How do I optimize searches?

• Avoid wildcard searches (e.g., index=*).
• Use time range filters and limit unnecessary field extractions.

Distributed Search Environments

A distributed search environment separates the functions of searching and indexing to improve scalability and performance. Here's a detailed look at its components and configurations.

Components of a Distributed Search Environment

1. Search Head:

   • Manages user queries and interfaces with Indexers to retrieve data.
   • Can be scaled horizontally with Search Head Clustering.

2. Indexer:

   • Processes and stores data, making it searchable.
   • Can be part of an Indexer Cluster for redundancy and load balancing.

3. Forwarders:

   • Collect and send data to Indexers.
   • Serve as the primary input point for data ingestion.

4. Deployment Server:

   • Centrally manages configurations for forwarders and other Splunk components.

Configuration Steps

1. Connecting Search Heads to Indexers:

   • Configure the Search Head to recognize Indexers in a distributed environment.

   • Add Indexers using distsearch.conf:

     [distributedSearch]
     servers = 192.168.1.10:8089, 192.168.1.11:8089
     

2. Enabling Distributed Search:

   • In Splunk Web, navigate to Settings > Distributed Search > Search Peers.
   • Add the Indexers by their management port (default: 8089).

3. Validating Search Peers:

   • Ensure all search peers show as Connected in the Search Head’s settings.

4. Testing Distributed Search:

   • Run a query from the Search Head that accesses data stored on the Indexers:

     index=web_logs sourcetype=apache_access | stats count by status
     

Best Practices

• Load Balancing: Distribute searches across multiple Indexers for better performance.
• Replication: Use Indexer Clustering to ensure data availability in case of hardware failure.
• Search Affinity: Assign specific searches to certain Indexers for optimal resource usage.

Indexer Clustering

Indexer Clustering ensures data availability and fault tolerance by replicating data across multiple Indexers.

Key Concepts

1. Replication Factor (RF):

   • The number of copies of data that the cluster maintains.
   • Example: RF=3 means three copies of each piece of data are stored.

2. Search Factor (SF):

   • The number of searchable copies of data.
   • Example: SF=2 means two Indexers hold searchable copies.

3. Cluster Manager:

   • A dedicated node that coordinates replication and monitors the health of the cluster.

Setup Steps

1. Enable Indexer Clustering:

   • On each Indexer, configure server.conf:

     [clustering]
     mode = slave
     master_uri = https://192.168.1.1:8089  # Cluster Manager
     replication_factor = 3
     search_factor = 2
     

2. Configure the Cluster Manager:

   • On the Cluster Manager, configure server.conf:

     [clustering]
     mode = master
     replication_factor = 3
     search_factor = 2
     

   • Restart Splunk services.

3. Monitor Cluster Status:

   • Use the Monitoring Console on the Cluster Manager to view replication and search factors.

Benefits of Clustering

• Ensures high availability of data.
• Supports failover in case of hardware failures.
• Scales horizontally to handle increased data volumes.

Parsing and Transformations

Parsing and transformations allow you to manipulate raw data during ingestion. These steps occur in the Parsing Stage of the data pipeline.

Parsing Concepts

1. Sourcetypes:

   • Define how Splunk processes and tokenizes incoming data.

   • Example:

     [apache_access]
     TIME_FORMAT = %d/%b/%Y:%H:%M:%S %z
     MAX_TIMESTAMP_LOOKAHEAD = 32
     

2. Field Extraction:

   • Splunk uses regular expressions to extract fields from data.
   • Example: Data: 192.168.1.1 - - [01/Jan/2025:12:00:00] "GET /index.html" Fields:
     • IP: 192.168.1.1
     • HTTP Method: GET
     • URI: /index.html

Transformation Concepts

1. props.conf:

   • Controls how data is parsed and indexed.

   • Example:

     [source::/var/log/apache/access.log]
     TRANSFORMS-anonymize = mask_ssn
     

2. transforms.conf:

   • Defines rules for modifying data.

   • Example:

     [mask_ssn]
     REGEX = (\d{3}-\d{2}-\d{4})
     FORMAT = XXX-XX-XXXX
     DEST_KEY = _raw
     

Hands-On Example: Masking Sensitive Data

1. Configure props.conf:

   [source::/var/log/app.log]
   TRANSFORMS-anonymize = mask_credit_card
   

2. Configure transforms.conf:

   [mask_credit_card]
   REGEX = \b\d{16}\b
   FORMAT = XXXX-XXXX-XXXX-XXXX
   DEST_KEY = _raw
   

3. Verify the changes:

   • Search for a credit card number in Splunk; it should appear masked.

Best Practices for Distributed Environments

Data Management

• Use role-based access controls to limit data visibility.
• Configure event routing to indexers based on metadata.

Performance Tuning

• Use lightweight forwarders for data ingestion.
• Limit field extractions to reduce processing overhead.

Security

• Encrypt communication between Search Heads and Indexers using SSL/TLS.
• Regularly audit cluster configurations for compliance.

Splunk Admin Basics (Additional Content)

1. License Enforcement and Warning Mechanism

1.1 What is License Enforcement?

Splunk’s licensing model is based on daily indexed volume. You purchase a license that allows you to ingest a certain amount of data per day (e.g., 10 GB/day).

• Enforcement kicks in when:

  • The indexed data volume exceeds the licensed quota on 3 out of 30 rolling days.

1.2 License Warning and Violation Process

• If you exceed your license limit on any day, Splunk will issue a license warning.

• If you exceed it on 3 separate days in a 30-day window, you will enter a license violation state.

1.3 Effects of a Violation

• Search functionality is disabled for non-admin users.

• Ingesting data still works, but search access is restricted.

• A warning banner is shown in Splunk Web.

1.4 Resolving Violations

• You can resolve a violation by:

  • Purchasing more license capacity.

  • Waiting for the 30-day window to roll forward (older violations expire).

  • Reducing data ingestion.

1.5 Monitoring License Usage

• Splunk Web:

  • Go to Settings > Licensing to view usage, warnings, and violations.

• SPL:

  index=_internal source=*license_usage.log* type="Usage"
  | stats sum(b) AS bytes by idx, sourcetype
  | eval GB=round(bytes/1024/1024/1024, 2)
  

2. Introduction to Knowledge Objects

2.1 What are Knowledge Objects (KOs)?

KOs are user-defined entities that enhance Splunk’s search and visualization capabilities.

2.2 Common Types of KOs

• Saved Searches:

  • Scheduled or manual searches saved with a name.

  • Can be used for dashboards, alerts, or reports.

• Macros:

  • Reusable snippets of SPL, stored with parameters.

  • Used to simplify complex searches.

• Event Types:

  • Tags for events that match certain search conditions.

  • Used for classifying data semantically (e.g., failed_logins, user_logins).

2.3 Where to Manage KOs in Splunk Web

• Go to Settings > Knowledge:

  • You’ll find Search Macros, Saved Searches, Event Types, and others like Tags, Lookups, etc.

2.4 Importance for Admins

While more relevant for Power Users, admins must manage sharing, permissions, and app context for these objects.

3. Configuration Precedence and Directory Priority

3.1 Configuration Layers

Splunk reads configuration files from multiple locations with a defined priority order.

3.2 Directory Hierarchy

1. $SPLUNK_HOME/etc/system/local/

2. $SPLUNK_HOME/etc/apps/<app_name>/local/

3. $SPLUNK_HOME/etc/apps/<app_name>/default/

4. $SPLUNK_HOME/etc/system/default/

• local overrides default

• system overrides apps

3.3 Effective Configuration Precedence

• If the same setting appears in multiple locations, Splunk uses the one with the highest priority.

• Example:

  • A props.conf in system/local will override one in app_name/default.

3.4 Debugging Effective Configuration

Use the following command:

splunk cmd btool props list --debug

It shows the active configuration along with the source file.

4. Splunk Web Admin UI Navigation

Splunk’s Web interface provides intuitive access to most admin functions under the Settings menu.

4.1 Common Admin Entry Points

• Indexes:

  • Location: Settings > Indexes

  • You can create, edit, and delete indexes here.

• Data Inputs:

  • Location: Settings > Data Inputs

  • Used to add new data sources (files, ports, scripts, HEC, etc.)

• Forwarder Management (if using Deployment Server):

  • Location: Settings > Forwarder Management

  • View and manage connected forwarders, server classes, and deployed apps.

• Distributed Search:

  • Location: Settings > Distributed Search

  • Configure search peers and replication settings.

• Users & Authentication:

  • Location: Settings > Access Controls

  • Manage users, roles, and authentication methods (LDAP, SAML, etc.)

4.2 Best Practices for Admins

• Always confirm the scope (App or Global) when editing configurations via Splunk Web.

• Use role-based access controls to limit what each user or role can manage.