# Deploying Splunk ### **1 Splunk Deployment Types** #### **What does "deployment" mean in Splunk?** In Splunk, "deployment" refers to how the system is installed and organized across one or more machines. It determines how data flows through the system, how it's stored, and how users access and analyze it. There are two main types of deployment: standalone and distributed. #### **1.1 Standalone Deployment** A standalone deployment means **all Splunk functions** are installed on a **single machine**. This machine handles: - Collecting data (data input) - Parsing and indexing data (storing it for search) - Running searches and reports - Displaying results in dashboards and visualizations This type of setup is simple and quick to install. **When is standalone deployment suitable?** - Learning and training environments - Personal testing or development use - Small businesses with limited data volume **Advantages of standalone deployment:** - Easy to set up and manage - No need for network configuration between components - Fewer system resources required **Disadvantages of standalone deployment:** - Not designed for large data volumes - Limited performance and scalability - Not suitable for enterprise production use #### **1.2 Distributed Deployment** A distributed deployment separates the different functions of Splunk across **multiple machines**. Each machine has a specific role, and they work together as a system. This model is used in most production environments because it supports **scalability, performance, and fault tolerance**. **Main roles in a distributed deployment:** 1. **Forwarders** – These are lightweight agents installed on data sources (e.g., servers, network devices). They collect and send data to indexers. 2. **Indexers** – These systems receive, parse, and store the data. They also respond to search requests by providing results. 3. **Search Heads** – These machines let users run searches, build dashboards, set alerts, and view reports. They act as the interface for users. **Why use a distributed setup?** - To handle larger data volumes - To improve reliability and performance - To isolate and scale specific functions as needed **When is distributed deployment suitable?** - Medium to large businesses - Environments with high data ingestion rates - Teams with multiple users querying data #### **Comparison of Standalone and Distributed Deployment** | Category | Standalone Deployment | Distributed Deployment | | ------------------ | --------------------- | ------------------------------------ | | Number of machines | One | Multiple | | Complexity | Low | Medium to high | | Performance | Limited | High | | Scalability | Poor | Excellent | | Use case | Testing, learning | Production, large-scale environments | | Data handling | Basic | Handles high volume and concurrency | ### **2. Deployment Components** In a distributed Splunk environment, different components (or roles) are installed on different machines. Each component has a specific responsibility. Understanding these roles is essential before you attempt any Splunk deployment or configuration. #### **2.1 Universal Forwarder (UF)** A Universal Forwarder is a lightweight Splunk agent installed on **data source systems** (such as application servers, web servers, or databases). Its only job is to **collect and forward raw data** to a central Splunk system (usually to Indexers). **Key characteristics:** - Small footprint with minimal resource usage - Cannot parse or transform data - No web interface or dashboards - Typically installed on many servers across an organization **Use case:** Best suited for sending logs or metrics from servers in a production environment to Indexers for further processing. #### **2.2 Heavy Forwarder (HF)** A Heavy Forwarder is a full Splunk instance used to **collect, parse, and forward** data. Unlike the Universal Forwarder, the Heavy Forwarder can: - Perform data transformation using configuration files (`props.conf` and `transforms.conf`) - Filter or route data to different destinations - Support inputs that need scripting or parsing **Use case:** Useful when the data needs to be modified or routed before it reaches the Indexer. For example, masking sensitive data or splitting events. #### **2.3 Deployment Server (DS)** The Deployment Server is a special role used to **manage and distribute configuration files** and apps to multiple Universal Forwarders (and optionally Heavy Forwarders). **Functions include:** - Central management of input configurations (`inputs.conf`, etc.) - Grouping forwarders into "server classes" based on IP, hostnames, etc. - Automatically deploying changes when needed **Use case:** When you have hundreds or thousands of forwarders and you want to update or control them from a single place. #### **2.4 Indexer** An Indexer is the core of the Splunk backend. It performs several vital functions: - Receives data from Forwarders or inputs - Parses and processes the raw data - Creates and stores event indexes for search - Responds to search requests from Search Heads **Key role:** It stores both the raw data and the index metadata that allows fast searching later. **Use case:** All production environments use one or more Indexers depending on data volume and redundancy requirements. #### **2.5 Search Head (SH)** A Search Head provides the **user interface** to Splunk. It is where users: - Write and run searches using SPL (Search Processing Language) - Create dashboards, reports, and alerts - View search results and visualizations **The Search Head does not store data.** It simply queries Indexers (or a cluster of Indexers) to retrieve results. **Use case:** In large environments, Search Heads are separated from Indexers so user searches don't slow down indexing. #### **2.6 Cluster Master (Manager Node)** The Cluster Master, also called the Manager Node, is used when you deploy **Indexer Clustering**. It performs administrative tasks such as: - Managing cluster configurations - Monitoring the health of peer nodes (Indexers) - Coordinating data replication **It does not index or search data.** **Use case:** Required in environments that use clustered Indexers for high availability and data redundancy. #### **2.7 Deployer** A Deployer is used when you set up a **Search Head Cluster**. Its job is to: - Push configuration bundles (apps, settings) to all Search Head cluster members - Ensure consistency across the cluster **Use case:** Ensures all Search Heads in a cluster share the same configurations and apps, avoiding manual updates to each member. #### **2.8 License Master** The License Master manages Splunk’s license usage across all components in a deployment. **Responsibilities:** - Tracks how much data is ingested per day - Issues license warnings or violations if limits are exceeded - Allocates license volumes across environments or departments **Use case:** In medium to large environments, a centralized License Master ensures compliance and better monitoring of license usage. ### **3. Installation Methods** Splunk Enterprise can be installed on various platforms, including Linux, Windows, Docker containers, and Cloud platforms. Each method has its own advantages and is used in different environments depending on scale, automation needs, or operating system preferences. #### **3.1 Installing Splunk on Linux** Linux is the **most common platform** for deploying Splunk in production. It provides better performance and flexibility for configuration. **Installation formats available:** - `.tgz` (tarball file): Best for manual installation and custom directory paths. - `.rpm` (Red Hat-based systems): For package-managed installations using `yum` or `dnf`. **Installation steps using tarball (.tgz):** 1. Download Splunk from the official site. 2. Move the file to your target directory. 3. Extract using: `tar -xvzf splunk---Linux-x86_64.tgz -C /opt` 4. Change directory: `cd /opt/splunk/bin` 5. Start Splunk for the first time and set admin password: `./splunk start --accept-license` **Advantages:** - Offers full control over the installation path and files. - Widely used in production environments. - Easier to integrate with enterprise monitoring tools. #### **3.2 Installing Splunk on Windows** Splunk provides a simple graphical installer for Windows environments. **Installation steps:** 1. Download the `.msi` or `.exe` installer from the Splunk website. 2. Run the installer and follow the GUI wizard. 3. Choose the installation path and set admin credentials. 4. After installation, access Splunk Web at: `http://localhost:8000` **Windows-specific features:** - Native integration with Event Logs, WMI, Active Directory. - Can collect data using prebuilt Windows apps and add-ons. **Use cases:** - Suitable for test environments or for collecting Windows-native logs. #### **3.3 Installing Splunk with Docker and Kubernetes** This method is used for **containerized** or **cloud-native** deployments. It is common in CI/CD pipelines or when you want to automate deployment with infrastructure as code. **Docker Installation:** 1. Pull the official image: `docker pull splunk/splunk:latest` 2. Run a container with environment variables: `docker run -d -p 8000:8000 -e SPLUNK_START_ARGS="--accept-license" -e SPLUNK_PASSWORD="changeme" splunk/splunk` **Kubernetes Installation:** - Use Helm charts provided by Splunk. - Requires knowledge of Kubernetes and configuration files. - Enables rapid scaling and orchestration. **Use cases:** - Development, testing, and cloud-native production environments. - Ideal for organizations using microservices or DevOps. #### **3.4 Splunk Cloud Platform** Splunk Cloud is a **fully managed** Splunk service hosted by Splunk or on major cloud providers (AWS, GCP, Azure). It eliminates the need for infrastructure management. **Key features:** - No installation required. - Managed scaling, backups, and updates by Splunk. - Connect to Splunk Cloud using Forwarders or HEC. **Use cases:** - Organizations that want to use Splunk without managing hardware. - Fast onboarding for analytics teams without DevOps involvement. **Limitations:** - Less control over backend configurations. - Certain custom apps or configurations may not be supported. #### **Comparison of Installation Methods** | Method | Platform | Best for | Notes | | ------------------- | --------------- | ---------------------------------- | ---------------------------------------- | | Linux (.tgz/.rpm) | Server Linux | Production, performance | Most flexible and powerful | | Windows (.exe/.msi) | Windows Server | Small-scale, Windows integration | Easy GUI installation | | Docker | Any (container) | Testing, DevOps, automation | Requires Docker knowledge | | Kubernetes | Cloud-native | Enterprise DevOps teams | Scalable, complex to manage | | Splunk Cloud | Fully managed | Zero-maintenance, rapid deployment | Managed by Splunk, fast but less control | ### **4. Configuration Best Practices** #### **4.1 Use Dedicated Directories for Logs, Configuration Files, and Indexed Data** By default, Splunk stores all of its data and configurations under the main installation directory (`$SPLUNK_HOME`). However, in production environments, it is a best practice to: - Store **indexed data** (hot, warm, cold buckets) on high-performance storage with sufficient capacity. - Place **log files** (such as `_internal` logs) on a separate disk if possible, to avoid performance impact. - Separate **configuration files** (in the `etc/` directory) from data directories to improve clarity and backups. **Benefits:** - Easier disaster recovery - Better performance tuning - Safer during upgrades or migrations #### **4.2 Separate `etc/system/local` and `etc/apps/` for Configuration Hierarchy** Splunk has a **layered configuration system**, meaning configuration files can exist in different locations, each with a specific priority. Two key directories are: - `etc/system/local`: This is the **highest precedence**, but it's only for **critical overrides**. - `etc/apps/`: Preferred location for most custom configurations and knowledge objects (dashboards, searches, inputs, etc.) **Why use apps instead of system/local?** - Apps are **modular and portable** - Easy to version-control and deploy across environments - Compatible with **Deployment Server**, **Search Head Clustering**, and **Deployer** Example: If you want to configure a custom log input, create an app like `TA_custom_inputs`, and place `inputs.conf` inside `etc/apps/TA_custom_inputs/local/`. #### **4.3 Ensure Time Synchronization (NTP) Across All Nodes** Time consistency is **critical** in Splunk deployments because: - **Timestamps** are used to organize, index, and search data. - **Searches over time ranges** can miss data if the time is not aligned across Indexers, Forwarders, and Search Heads. - **Clustering (Indexer/Search Head)** relies on synchronized logs for replication and troubleshooting. **Best practice:** Configure all systems to use a reliable **Network Time Protocol (NTP)** server. #### **4.4 Scale Horizontally by Adding More Indexers or Search Heads** In production environments, you should scale **horizontally**, not vertically. - **Horizontal scaling**: Add more Indexers or Search Heads as data volume or user demand increases. - **Vertical scaling**: Adding more CPU/RAM to a single instance (helpful but has limits). **Examples:** - If indexing becomes slow: Add more Indexers. - If search performance drops during peak hours: Add more Search Heads. - Use **load balancing** between Universal Forwarders and Indexers for better distribution. **Splunk supports both** scaling methods, but horizontal scaling is better for long-term growth. #### **Other Useful Tips for Configuration Management** 1. **Use version control (like Git)** to track and manage configuration changes across teams. 2. **Test changes in staging or dev environments** before applying them to production. 3. Keep a **change log** for all manual edits made to `.conf` files. 4. Validate your configuration changes using Splunk's `btool` command to debug configuration merging: ``` splunk btool inputs list --debug ``` 5. Organize apps clearly: - Use prefixes like `TA_` for Technical Add-ons (data inputs only) - Use `SA_` for shared libraries or utilities - Use `app_` or business names for user-facing apps ### Summary of Best Practices | Practice Area | Best Practice Summary | | ----------------------- | ------------------------------------------------------------- | | Directory Organization | Separate logs, configs, and indexed data | | Configuration Hierarchy | Use `etc/apps/` instead of `system/local` | | Time Management | Sync all nodes with NTP servers | | Scaling | Add Indexers/Search Heads as needed for performance | | Versioning and Testing | Use Git, test in dev, and document all changes | | Validation Tools | Use `btool` and the Monitoring Console to check config status | --- # Monitoring Console ### **1. Purpose and Setup** The **Monitoring Console (MC)** is a built-in application in Splunk that helps administrators monitor the **health, performance, and resource usage** of their entire Splunk environment. #### What does it monitor? - System resources like CPU, memory, and disk usage - Data indexing rates and performance - Search activity and failures - Forwarder connectivity and data throughput - License usage and violations #### Where is it located? - It is **pre-installed** on every Splunk Enterprise instance. - You can access it via the web interface: - Go to "Settings" > "Monitoring Console" #### How is it set up? - **Standalone Splunk Instance**: The Monitoring Console is enabled by default. It monitors itself (the same system it's installed on). - **Distributed Environment**: You must **designate one node** (usually a Search Head) to act as the Monitoring Console. This node then gathers data from other Splunk components such as: - Indexers - Forwarders - Search Heads - Cluster Manager, License Master, etc. The Monitoring Console uses special indexes like `_introspection` and `_internal` to collect its data. ### **2. MC Modes** The Monitoring Console has two main operational modes depending on your deployment type. #### Standalone Mode - This is the **default mode** for single-instance Splunk deployments. - It only monitors the local system it is running on. - No additional setup is needed. #### Distributed Mode - Used in **multi-instance Splunk environments** where different machines handle indexing, searching, and forwarding. - The Monitoring Console node must be configured to collect data from all other nodes in the deployment. - You must define **data inputs and roles** for each instance. ##### How to configure Distributed Mode 1. Go to: "Settings" > "Monitoring Console" > "General Setup" 2. Add all the Splunk instances in your environment. 3. Assign the correct **roles** to each instance: - Indexer - Search Head - Forwarder - Cluster Manager - License Master 4. The console will then start showing dashboards with accurate data from all added instances. ##### Required file for configuration - `distsearch.conf` must be properly set on the Monitoring Console to allow it to search across other instances and collect necessary metrics. ### **3. Key Dashboards** The Monitoring Console contains many prebuilt dashboards that display critical metrics about your Splunk environment. Here are some of the most important ones: #### Resource Usage - Shows CPU, memory, and disk I/O for each Splunk instance - Helps detect overloaded systems or systems with insufficient resources - Can be used to plan hardware upgrades or scaling #### Indexer Performance - Displays indexing throughput (data per second), indexing latency, and the rate of bucket creation - Useful for diagnosing slow indexing performance or backlog issues - Can indicate whether more Indexers are needed #### Search Performance - Monitors search concurrency (how many searches are running at the same time) - Shows skipped searches, which may mean that the system is overloaded - Tracks search latency (how long searches take to run) #### Data Ingestion - Visualizes the amount of data coming in from forwarders - Helps identify if any forwarders have stopped sending data - Lists forwarder connection status and data volume over time #### License Usage - Tracks how much data is indexed each day - Shows license pool usage and any violations (e.g., going over your daily licensed volume) - Alerts when you are nearing or exceeding license limits ### **4. Troubleshooting with Monitoring Console** The Monitoring Console is not just for viewing performance—it is also a powerful tool for **troubleshooting issues** in a Splunk environment. Here’s how it can help in specific problem areas: #### Identifying Bottlenecks in the Search or Indexing Pipeline - The MC can show whether searches are being delayed, queued, or skipped. - Dashboards show whether the system is CPU-bound, memory-limited, or disk-constrained. - Use the "Search Performance" and "Indexer Performance" views to detect whether the bottleneck is due to: - Too many concurrent searches - Slow indexing speed - I/O wait time on the disk - Lack of system resources #### Detecting Skipped Scheduled Searches - Skipped searches mean scheduled reports or alerts didn’t run as expected. - MC lists which searches were skipped and the reason: - System load too high - Scheduler was too busy - Conflicting time windows - This helps you adjust scheduling windows or reduce concurrent jobs. #### Monitoring Replication and Indexing Latency in Clusters - In clustered environments, MC can show: - Replication factor and search factor status - Missing or incomplete bucket copies - Latency in replicating data between Indexers - Use this to verify cluster health and identify whether your cluster is balanced and synchronized. **Example troubleshooting workflow:** 1. A user complains about a slow dashboard. 2. You open the Monitoring Console and check "Search Performance". 3. You find that searches are queued because CPU usage is near 100%. 4. You check the "Resource Usage" dashboard and confirm the Search Head is overloaded. 5. You plan to either optimize the searches or add another Search Head for load distribution. ### **5. MC Configuration Tips** To ensure that your Monitoring Console works effectively, you must configure it properly. Below are essential tips and best practices: #### Add All Cluster Nodes Using General Setup - Navigate to: - "Settings" > "Monitoring Console" > "General Setup" - Here, you can manually **add every Splunk component** in your deployment, including: - Indexers - Search Heads - Cluster Managers - License Master - Deployment Server - This allows the console to **collect and correlate data from all nodes**. #### Assign Server Roles for Accurate Dashboard Metrics - After adding a node, you must assign its **server role**. This tells the MC what type of component it is. - Examples of roles: - Indexer - Search Head - Forwarder - Cluster Master - This ensures that the dashboards show the **correct data for the correct function**. #### Enable Introspection and Telemetry for Deeper Diagnostics - Splunk uses **internal logs** and **metrics logs** to collect detailed system information. - Ensure that the following indexes are enabled and collecting data: - `_introspection`: CPU, memory, and thread usage - `_internal`: Splunk logs, errors, warnings - `_telemetry`: Optional usage and performance data for analytics - These indexes feed the dashboards in MC. Without them, you may see missing or blank panels. #### Tips Summary | Configuration Area | Best Practice | | ------------------------- | ------------------------------------------------ | | Node Registration | Add all components to General Setup | | Role Assignment | Correctly label each node with its function | | Log and Metric Collection | Enable `_internal`, `_introspection` indexes | | Data Access | Ensure the MC has permission to search all nodes | | Security | Use secure credentials and limit MC user access | ### **Final Recap: Monitoring Console (MC)** You’ve now learned that: - The Monitoring Console is essential for observing the health and performance of your Splunk environment. - It works in both standalone and distributed modes. - It includes dashboards for system usage, indexing, searching, ingestion, and licensing. - It is also a powerful tool for diagnosing problems and planning improvements. - Proper setup is critical to get accurate, useful data. --- # Access and Roles ### **1. User Authentication** Authentication is the process by which Splunk verifies a user’s identity before granting access to the system. Splunk supports both internal (native) and external authentication methods. #### Native Splunk Authentication - This is the default method. - Usernames and passwords are stored **locally** in the `passwd` file within the Splunk file system. - Accounts are managed using the Splunk Web UI or the command line. - Suitable for small environments or testing labs where no external identity system is needed. #### External Authentication Methods For large or enterprise environments, external authentication is preferred. Splunk integrates with several identity providers: 1. **LDAP/Active Directory** - Connects to centralized directory services such as Microsoft Active Directory. - Allows user accounts and groups to be imported into Splunk. - Role mapping is based on group membership. - Common in corporate networks. 2. **SAML (Security Assertion Markup Language)** - Enables Single Sign-On (SSO). - Commonly used with Identity Providers (IdPs) like Okta, Azure AD, ADFS. - Users authenticate once and gain access to Splunk and other systems without re-entering credentials. 3. **Scripted Authentication** - Advanced method allowing you to build custom authentication backends. - Requires scripting (Python, shell) and is suitable for environments with non-standard identity systems. - Must follow strict formatting and timeout requirements. ### **2. Role-Based Access Control (RBAC)** Splunk uses a **Role-Based Access Control (RBAC)** model to define **what a user can do** and **what data they can see**. Every user is assigned one or more roles. #### Key Concepts: - **Roles define permissions**, not users. - A user may have **multiple roles**, and roles may **inherit from other roles**. - Permissions are additive. If a user has two roles, the most permissive settings from both will apply. #### What can a role control? 1. Capabilities (UI and API access) 2. Indexes the user can search 3. Restrictions on what the user can query 4. Limits on system resource usage (searches per user, memory usage) ### **3. Role Configuration Options** When creating or modifying a role in Splunk, you configure several types of settings: #### Capabilities - Capabilities are individual permissions that control what a user can do in the UI and through REST APIs. - Examples include: - `admin_all_objects`: Allows full administrative control over Splunk objects - `edit_search_schedule`: Lets users schedule searches - `list_storage_passwords`: Allows access to stored credentials - Capabilities are granular. You can allow a role to view dashboards but prevent it from editing alerts. #### Indexes Allowed - This setting controls **which indexes a role is allowed to search**. - You can define a list of indexes that the role has access to. - If a user searches without specifying an index, Splunk will search only in the indexes allowed by their role. Example: If a user’s role is allowed to search only `index=web_logs`, they cannot run a query on `index=syslog` unless explicitly granted. #### Search Restrictions - This is a powerful tool for **data-level access control**. - You can define a **search filter** that is automatically added to every search by users in a role. Example: ``` index=web_logs host!=test* status!=403 ``` - This ensures that even if the user tries to run a different search, the restriction is always applied. #### Resource Limits These settings help **control system usage** and **prevent performance issues** caused by heavy users. You can limit: - Maximum number of concurrent searches per user - Maximum search time range (e.g., last 30 days) - Maximum CPU usage or disk quota for scheduled reports These values are useful in large organizations where hundreds of users may run searches simultaneously. ### **4. Best Practices** Implementing access control properly is essential to maintain security, performance, and clarity in your Splunk environment. 1. **Follow the Principle of Least Privilege** - Always grant users **only the minimum permissions** they need to do their job. - Avoid giving admin-level capabilities unless absolutely necessary. 2. **Use Custom Roles for Different User Groups** - Define roles based on functional groups such as: - Analyst (can search and view dashboards) - Developer (can schedule searches and manage alerts) - Admin (can manage indexes, users, apps) - This approach makes permissions easier to manage and audit. 3. **Always Test Role Configurations in Non-Production** - Create a test environment where you can safely test new role settings. - Verify that restrictions and capabilities behave as expected. - Avoid making untested changes directly in production environments. ### Summary: Access and Roles | Area | Key Point | | ------------------ | ------------------------------------------------------------------------- | | Authentication | Supports native, LDAP, SAML, and custom scripts | | Roles | Define what users can do and what data they can see | | Capabilities | Control access to features, search scheduling, REST endpoints | | Index Restrictions | Limit users to specific indexes | | Search Filters | Enforce query restrictions by appending filters automatically | | Resource Controls | Prevent overuse of resources through limits on searches and time ranges | | Best Practices | Use least privilege, create role groups, and test in staging environments | --- # Data Collection Splunk is a data analytics platform, and its value begins with the ability to **collect data from a wide range of sources**. This data is ingested through inputs defined in configuration files, processed, and indexed for later searching. ### **1. Data Input Types** Splunk supports multiple input methods, and these are defined primarily using the `inputs.conf` file. Each input method is designed to collect data from different types of systems and formats. #### File and Directory Monitoring - Monitors a specific file or a directory containing files. - Common for collecting **application logs**, **web server logs**, or **custom log files**. - Splunk watches for new content in the file and ingests it line by line. #### TCP/UDP Streams - Accepts data over network ports. - Used in environments where systems send logs over the network, such as **syslog servers**. - TCP provides reliability, while UDP is faster but does not guarantee delivery. #### HTTP Event Collector (HEC) - Accepts data via **REST API POST requests**. - Data must be in **JSON format**. - Uses **token-based authentication** to secure input. - Common for **modern apps and services**, cloud environments, or **IoT devices**. Example: A cloud app can send log events using HTTP POST to Splunk Cloud via HEC. #### Scripted Inputs - Runs a script (written in Python, Bash, etc.) to collect data periodically. - Useful for running API calls or commands on remote systems and collecting the results. - The script's output becomes Splunk events. Example: A script that queries a database and sends the results to Splunk every 10 minutes. #### Modular Inputs - Created using Splunk Add-on Builder or SDK. - More advanced and reusable than simple scripted inputs. - Allows integration with external systems through APIs in a structured, scalable way. Example: A modular input to pull data from ServiceNow or Salesforce. #### Windows Inputs Splunk can natively collect data from Windows systems using built-in features such as: - **WMI (Windows Management Instrumentation)** - **Event Logs** (System, Application, Security) - **Registry Monitoring** - **Performance Counters** (CPU, memory, network usage) #### Cloud and Streaming Inputs (Kafka, AWS S3, Azure Blob) - Splunk integrates with cloud storage and streaming platforms using: - **Splunk Connect for Kafka** - **Splunk Add-on for AWS** - **Splunk Add-on for Microsoft Cloud Services** These allow you to bring in data from cloud-native sources like: - Amazon S3 buckets (logs, CSVs, JSON data) - Kafka topics (real-time event streaming) - Azure Event Hub or Blob Storage ### **2. Forwarder Types** Splunk uses **Forwarders** to send data from source systems to Indexers. #### Universal Forwarder (UF) - A **lightweight** Splunk agent. - Cannot parse, transform, or filter data. - Ideal for **large-scale production environments**. - Low resource usage and stable for 24/7 log collection. Example: Installed on hundreds of servers to send logs to central Indexers. #### Heavy Forwarder (HF) - A **full Splunk instance** with all capabilities, including parsing and indexing (though indexing is typically disabled). - Can apply **props.conf and transforms.conf** to: - Route events to different Indexers - Mask or extract fields - Modify timestamp formats - Used when preprocessing is required **before the data reaches the Indexer**. Example: Used at the edge of a secure network to sanitize and forward logs to the core Splunk environment. ### **3. Parsing Pipeline (Data Processing Flow)** Splunk processes incoming data through a **pipeline** that consists of several key phases. Each phase transforms the data and prepares it for storage and searching. #### Input Phase - Splunk reads raw data from the input defined in `inputs.conf`. - It could be a file, a TCP stream, a script output, or an HEC POST. #### Parsing Phase - Data is **broken into individual events** based on line breaks and timestamps. - Timestamp extraction happens here. - This phase uses configuration from `props.conf`. #### Indexing Phase - Events are written to the index. - Metadata such as **host, source, sourcetype, index, and timestamp** is assigned and stored. - The data is compressed and saved in buckets. #### Search Phase - Once the data is indexed, it becomes searchable using the Splunk Search Processing Language (SPL). - This phase happens when a user or scheduled job runs a query. Understanding this pipeline helps in troubleshooting issues like delayed data, improperly formatted events, or wrong timestamps. ### **4. Timestamps and Time Zone Handling** Correct time handling is essential in Splunk because all search queries are based on time ranges. #### Timestamp Extraction Splunk uses the following settings in `props.conf` to identify the correct timestamp in the raw event: - `TIME_FORMAT`: Defines the exact format of the timestamp. - `TIME_PREFIX`: A string or regex that appears before the timestamp. - `MAX_TIMESTAMP_LOOKAHEAD`: How many characters to scan after the prefix. Example: If your logs look like this: ``` [2024-04-25 10:15:30] User logged in ``` Then you would use: - `TIME_PREFIX = \[` - `TIME_FORMAT = %Y-%m-%d %H:%M:%S` - `MAX_TIMESTAMP_LOOKAHEAD = 20` #### Fallback Timestamp If Splunk cannot extract a timestamp from the event: - It will use the **file modification time** (for file inputs) - Or the **current system time** (for other inputs) #### Disable Timestamp Extraction You can disable timestamp extraction entirely by setting: ``` DATETIME_CONFIG = NONE ``` This might be useful for binary data or metrics that do not require time-based indexing. ### **5. Event Line Breaking** Some logs contain multiple lines per event (e.g., Java stack traces), so Splunk must know where each event starts and ends. This is called **line breaking**, and it is configured in `props.conf`. #### Key Settings - `LINE_BREAKER`: A regular expression that defines the exact point where a new event starts. - `SHOULD_LINEMERGE`: When set to false, tells Splunk to use LINE_BREAKER instead of guessing. - `BREAK_ONLY_BEFORE`: Defines a pattern that marks the **start of a new event**. - `BREAK_ONLY_AFTER`: Defines a pattern that marks the **end of the current event**. Correct configuration ensures that: - One event is not split into multiple parts. - Multiple lines that belong together are not incorrectly separated. Example: If each event starts with a date, like: ``` 2024-04-25 10:10:00 ERROR something happened Details: Line 1 Line 2 ``` Then: - Set `BREAK_ONLY_BEFORE = ^\d{4}-\d{2}-\d{2}` --- # Indexing In Splunk, **indexing** is the process of storing raw data and creating searchable metadata. This process allows Splunk to retrieve data quickly when users run searches, create dashboards, or generate alerts. ### **1. Indexing Basics** When Splunk collects data, it does not just save it as-is. It also processes the data into **events**, extracts metadata (like time, host, source, sourcetype), and stores it in an organized format. Indexing consists of two main activities: - **Storing Raw Event Data** The original content of the log or message is compressed and stored on disk. - **Creating Metadata** Splunk creates a time-based index with fields like: - `_time`: The event timestamp - `host`: The system where the data came from - `source`: The file or input source - `sourcetype`: The format of the data (e.g., json, access_combined) This structured data enables **fast and efficient search**, even across billions of events. Splunk stores data in a special format using **buckets**, which are organized based on the data’s **age and lifecycle stage** (see 5.3 below). ### **2. Index Types** Splunk supports different types of indexes, each optimized for specific data formats or use cases. #### Event Indexes - Default type of index in Splunk. - Used to store **event-based logs**, such as: - Web server logs - Firewall logs - Application logs - Events are timestamped and stored in time order. #### Metrics Indexes - Designed for **numerical time-series data**, such as: - CPU usage - Memory consumption - Disk I/O - Enables efficient storage and faster search for performance metrics. - Used with `mcollect`, `metrics.log`, or `collectd`. #### Summary Indexes - Special indexes that store the **results of scheduled or accelerated searches**. - Useful for: - Generating dashboards quickly - Reducing the need to re-run expensive searches - Example: A nightly search calculates average error rates, and its result is stored in a summary index. #### Internal Indexes These are used by Splunk to store its own logs and internal data. Common internal indexes include: - `_internal`: Scheduler logs, indexing performance, and errors - `_audit`: User login attempts, role changes, and audit events - `_introspection`: System-level performance metrics (CPU, memory, etc.) You typically do not ingest data into these manually, but you **search them for troubleshooting** and system monitoring. ### **3. Bucket Lifecycle** Splunk stores data in units called **buckets**. A bucket is a directory on disk containing raw data and index files. Each bucket belongs to one of several lifecycle stages based on the age of its data. #### Hot Bucket - The bucket **currently receiving new data**. - Stored in memory or fast disk for performance. - Each index can have multiple hot buckets depending on settings. #### Warm Bucket - Once a hot bucket is full or the system restarts, it becomes a **warm bucket**. - It is closed to writing but remains **searchable**. - Stored on disk, usually still fast storage. #### Cold Bucket - Older data is moved from warm to **cold** storage. - This data is still searchable but is placed on **less expensive or slower disk**. #### Frozen Bucket - Once data reaches its **retention limit**, it is frozen. - Frozen data is **deleted** by default. - You can configure Splunk to **archive frozen data** to a backup location. - Frozen data is **no longer searchable** unless manually restored to a **thawed** state. These transitions are configured in `indexes.conf`. **Example settings:** ``` maxHotSpanSecs = 86400 maxTotalDataSizeMB = 500000 frozenTimePeriodInSecs = 31536000 ``` ### **4. Index Configuration Parameters** These settings in the `indexes.conf` file define how each index behaves. #### homePath and coldPath - `homePath`: Location where hot and warm buckets are stored. - `coldPath`: Location where cold buckets are moved after they age out of warm. This allows you to use **different disk volumes** for different lifecycle stages. #### maxDataSize - Controls the maximum size of each hot bucket. - This setting affects how often new buckets are created. - Options include `auto`, `auto_high_volume`, or custom size in MB. #### maxTotalDataSizeMB - Sets the **maximum size of all buckets in the index**. - Once the limit is reached, older data is deleted (frozen) to make space. #### frozenTimePeriodInSecs - Sets the **retention period** for data in seconds. - After this time, the data moves to the frozen state. - For example, to retain data for one year: `frozenTimePeriodInSecs = 31536000` ### **5. Index Cloning and Replication** In **indexer clustering**, Splunk replicates data to multiple peers to provide **high availability and fault tolerance**. #### Replication Factor - The number of copies of raw data that should exist in the cluster. - If replication factor is 3, there will be 3 total copies of each bucket on different indexers. #### Search Factor - The number of **searchable copies** required. - These are copies of buckets with index files that allow searches. - If search factor is 2, two of the bucket copies must be fully searchable. These settings are defined on the **Cluster Manager (Master Node)**, and enforced automatically during indexing. **Benefits:** - Prevents data loss if a peer indexer goes offline. - Maintains search functionality even if some nodes fail. --- # Search In Splunk, "search" refers to the process of querying indexed data to retrieve useful information. Users can perform searches using SPL (Search Processing Language) to analyze logs, monitor activity, detect anomalies, and generate dashboards or alerts. ### **1. Search Types** Splunk supports multiple types of searches, each suited to different use cases. #### Ad-hoc Search - A one-time search entered manually in the Search & Reporting app. - Used for **exploratory data analysis**, quick troubleshooting, or manual investigation. - Not scheduled or saved unless explicitly done by the user. Example: ``` index=web error OR fail | stats count by status ``` #### Scheduled Search - Configured to **run at regular intervals** (e.g., every 5 minutes, hourly, daily). - Often used in: - Reports - Alerts - Dashboards (panel updates) - Results can be stored in: - Lookup files - Summary indexes - Report acceleration cache Example: A search that runs every hour to find failed login attempts and sends an alert if the count exceeds 10. #### Real-time Search - Continuously updates as new events are indexed. - Ideal for **live monitoring**, such as: - Intrusion detection - Application performance - Critical system errors - Uses more system resources than scheduled or ad-hoc searches. - Can be configured with "rolling windows" (e.g., last 30 seconds). ### **2. SPL (Search Processing Language)** SPL is the custom query language used in Splunk. It is inspired by the **UNIX pipe model**, where commands are chained together with the `|` symbol to process data in stages. #### Basic Structure - Start with a **search command** (or default to search). - Pipe the result to one or more **processing commands**. #### Examples 1. Count HTTP 500 errors by host: ``` index=web status=500 | stats count by host ``` 2. Average outgoing bytes per hour from firewall logs: ``` index=firewall | timechart span=1h avg(bytes_out) ``` Common commands include: - `stats`: Perform statistical operations (sum, avg, count, etc.) - `timechart`: Plot time-series data - `table`: Display results in tabular format - `eval`: Create calculated fields - `where`: Filter events using conditions ### **3. Search Modes** Search modes control **how much data is returned and how fast** the search runs. You can choose between three modes in the Splunk UI. #### Fast Mode - Prioritizes speed. - Only essential fields are extracted. - No event highlighting or field discovery. - Best for final dashboards or reports. #### Verbose Mode - Extracts all fields from all events. - Includes full event context. - Useful during investigation or initial search development. - Slower than other modes. #### Smart Mode - Automatically chooses between fast and verbose mode depending on the search. - Offers a balance between performance and detail. ### **4. Search Optimization Tips** Optimizing your searches is critical for performance, especially in large environments with many users. #### Filter by Index and Time Always start your search with **index name** and a **time constraint**: ``` index=app_logs earliest=-1h ``` This dramatically reduces the volume of data Splunk has to scan. #### Use Indexed Fields Early Use fields that are indexed (like `host`, `source`, `sourcetype`) early in the search to limit the dataset as soon as possible. #### Avoid Wildcards Avoid leading wildcards in field names or values: ``` host=*prod* ← Not efficient ``` Better to use exact matches or wildcards only at the end: ``` host=prod* ← More efficient ``` #### Summarize Data Before Further Processing If your dataset is large, use `stats`, `eventstats`, or `tstats` early to reduce the number of events being passed to later parts of the search. ### **5. Accelerations** Splunk provides several acceleration techniques to **improve performance** for commonly used or expensive searches. #### Data Model Acceleration (DMA) - Used with **Pivot** and **CIM-compliant apps** (e.g., Enterprise Security). - Speeds up complex queries by pre-processing and storing results. - Configured in data model settings. #### Report Acceleration - Automatically stores results of scheduled reports. - Only applicable to transforming searches (e.g., those using `stats`, `timechart`). - Great for dashboards that don’t need real-time data. #### Summary Indexing - Stores the output of scheduled searches in a **new index**. - Allows fast lookups and reduced search load. - Requires a separate summary index and scheduled job. Example: ``` | stats avg(cpu_load) by host ``` The result can be stored every 5 minutes in a summary index and reused later. ### **6. Search Job Management** Searches in Splunk are executed as **jobs**. Splunk provides tools to manage and monitor these jobs. #### Activity > Jobs - Lists all current and historical search jobs. - Allows you to: - Inspect job properties - Pause, resume, or cancel jobs - Export results #### Search Quotas - Configured in `limits.conf`. - Controls: - Number of concurrent searches per role - Memory usage - Priority of different search types (e.g., real-time vs scheduled) #### dispatch.ttl (Time to Live) - Determines **how long** completed search results are kept on disk. - After this time, the results are deleted. - Default is 10 minutes for ad-hoc searches, but it can be extended for dashboards or saved searches. Example: To keep search results for one hour: ``` dispatch.ttl = 3600 ``` --- # Configuration Management Splunk is a highly configurable platform. Almost every feature in Splunk—inputs, indexing, search behavior, limits, and user roles—is controlled through **configuration files**. These settings define how Splunk behaves across different components and use cases. ### **1. Splunk Configuration Files** All configuration files in Splunk are located under the **`$SPLUNK_HOME/etc/`** directory. These files are written in **key-value pair format**, similar to: ``` [stanza] setting1 = value1 setting2 = value2 ``` Each `.conf` file controls a specific area of Splunk functionality. These are **not JSON or XML**—they follow Splunk’s own format. Example: ```ini [monitor:///var/log/apache/access.log] sourcetype = access_combined index = web ``` ### **2. Directory Hierarchy** Splunk organizes configuration files into a **layered directory structure**, which helps manage global, app-specific, and user-specific settings separately. #### a. System-Level **Location:** `$SPLUNK_HOME/etc/system/` - `default/`: Contains **factory settings** provided by Splunk. Do not edit this folder. - `local/`: Contains **your overrides** to system-level configurations. This folder has the **highest priority** among system-level files. #### b. App-Level **Location:** `$SPLUNK_HOME/etc/apps//` Each app installed in Splunk has: - `default/`: App-supplied settings - `local/`: Custom settings or overrides created by users or administrators Apps may define inputs, field extractions, dashboards, and more, and are self-contained. #### c. User-Level **Location:** `$SPLUNK_HOME/etc/users///` - These are **user-specific overrides**. - Useful for custom dashboards, saved searches, or UI preferences. ### **3. File Precedence Rules** Splunk must decide which configuration value to use when the same setting is defined in multiple locations. This is controlled by a well-defined **precedence hierarchy**. #### Rule: `local/` always overrides `default/`. #### Complete Precedence Order (from highest to lowest): 1. User’s app `local/` 2. User’s app `default/` 3. App `local/` 4. App `default/` 5. System `local/` 6. System `default/` This hierarchy ensures that **user-level settings override everything else**, and that **defaults** are only used when no override is defined. #### Troubleshooting with `btool` The `btool` command is used to inspect the **merged configuration** and identify the source of a setting. Example: ``` splunk btool inputs list --debug ``` This shows which files contribute to the final input configuration and their file paths. ### **4. Common Configuration Files** Below are some of the most frequently used `.conf` files in Splunk: #### `inputs.conf` - Defines how and where Splunk **ingests data**. - Includes file monitoring, network ports, scripted inputs, etc. Example: ```ini [monitor:///var/log/messages] disabled = false index = os ``` #### `props.conf` - Controls **data parsing**, including: - Field extractions - Timestamp rules - Line breaking behavior Example: ```ini [source::.../apache/access.log] sourcetype = access_combined ``` #### `transforms.conf` - Works with `props.conf` for **event transformation**. - Used for: - Routing events - Masking sensitive fields - Extracting custom fields #### `outputs.conf` - Defines how data is **forwarded** to other Splunk instances (usually Indexers). - Key in Universal Forwarder or Heavy Forwarder configuration. #### `indexes.conf` - Defines **index settings**, such as: - Index names - Retention policies - Storage paths #### `limits.conf` - Sets system-wide **limits**: - Maximum search time - Number of concurrent searches - Memory thresholds ### **5. Deployment Server (for Forwarders)** In large environments, managing hundreds or thousands of Universal Forwarders manually is inefficient. The **Deployment Server** allows centralized configuration management for forwarders. #### Key Concepts - **Server Classes**: Logical groups of forwarders based on criteria like hostnames or IP ranges. - **Deployment Apps**: Configuration packages (e.g., inputs.conf) that get pushed to clients. - **Clients**: Forwarders that are registered to receive updates. #### Key Configuration Files - `deploymentclient.conf`: Placed on the forwarder to define its connection to the deployment server. - `serverclass.conf`: Placed on the Deployment Server to define client groups and app assignments. ### **6. Best Practices** Following best practices for configuration management ensures stability, flexibility, and ease of scaling. #### Never Edit Files in `default/` - These files may be **overwritten** during upgrades. - Always place custom settings in the `local/` directory. #### Use Version Control - Store configuration files in **Git** or another version control system. - Track changes and enable rollbacks when needed. #### Structure Apps Modularly - Build custom apps for each type of function: - One for inputs - One for parsing rules - One for dashboards - Modular apps make migration and upgrades easier. #### Test Before Deploying to Production - Create a **test environment** that mirrors production. - Validate all configurations before pushing them live. ### Summary: Configuration Management | Topic | Key Takeaway | | ------------------- | ------------------------------------------------------------------ | | File Structure | Uses `.conf` files in `$SPLUNK_HOME/etc/` | | Directory Hierarchy | Organized into system, app, and user levels | | Precedence Rules | `local/` overrides `default/`; user settings take highest priority | | Key Config Files | Include `inputs.conf`, `props.conf`, `transforms.conf`, and more | | Deployment Server | Central management for Universal Forwarders via server classes | | Best Practices | Never edit defaults; use Git; test before production | --- # Indexer Clustering Indexer clustering is a Splunk feature used in distributed environments to ensure that **indexed data is always available**, even if one or more Indexers fail. It also helps you **scale horizontally** while maintaining **data redundancy and reliability**. ### **1. Purpose** The primary goals of Indexer Clustering are: - **High Availability**: Ensures data remains available even if one or more indexers go offline. - **Scalability**: Allows you to distribute indexing workload across multiple nodes. - **Data Redundancy**: Maintains multiple copies of indexed data to prevent data loss. This makes clustering ideal for **mission-critical environments** where system uptime and data integrity are key. ### **2. Cluster Roles** An Indexer Cluster consists of several roles, each performing specific functions: #### Cluster Manager (formerly called Master Node) - Manages the entire cluster configuration. - Coordinates bucket replication across peer nodes. - Monitors the health of all indexers (peers). - Does not index or search data itself. This role is defined in `server.conf`. #### Peer Nodes (Indexers) - These are the **workers** in the cluster. - They receive data, index it, and store it in **buckets**. - They also **replicate buckets** to other peers according to cluster policies. Each peer node runs a full Splunk Enterprise instance. #### Search Head - Not technically part of the Indexer Cluster but communicates with it. - Sends search requests to the cluster. - Aware of which peer nodes hold **primary searchable copies** of data. When the Search Head runs a query, it contacts the **Cluster Manager** to determine where the data lives and which peers to query. ### **3. Replication and Search Factors** The two most important settings in an Indexer Cluster are: #### Replication Factor (RF) - Defines **how many copies of raw data** are stored in the cluster. - Default: 3 - Example: With RF = 3, each event is stored on 3 different Indexers. More redundancy improves fault tolerance but consumes more storage. #### Search Factor (SF) - Defines how many **searchable copies** (buckets with index metadata) exist. - Default: 2 - Searchable copies allow Splunk to respond to user queries. To ensure cluster health, the **search factor must always be less than or equal to the replication factor**. #### Summary | Factor | Purpose | Default | | ------------------ | -------------------------------- | ------- | | Replication Factor | Raw data availability | 3 | | Search Factor | Searchability of replicated data | 2 | Increasing RF provides greater protection; increasing SF allows more parallel searches. ### **4. Bucket Replication** When a peer node receives new data, it creates a **hot bucket** and becomes the **origin** of that bucket. The following steps occur: 1. The bucket is created on the origin peer node. 2. The Cluster Manager instructs other peer nodes to **replicate the bucket**. 3. Once replication is complete, **some of those buckets become "primary" copies**, meaning they can respond to search requests. 4. Others are **non-primary** and only become searchable if the primary is lost (e.g., during a node failure). This replication process ensures that: - **No single node failure results in data loss** - **Searches can still run even if a node goes offline** ### **5. Cluster Configuration Files** Several configuration files control how Indexer Clustering operates. These are set differently depending on the role of the node (Cluster Manager, Peer, or Search Head). #### server.conf - Defines whether the node is a **manager**, **peer**, or **search head**. - Contains important settings like: - `clustering` - `multisite` (if applicable) - `pass4SymmKey` (shared cluster secret) #### indexes.conf - Sets the index creation rules, such as: - Retention period - Bucket size - Replication settings (if any overrides are needed) #### outputs.conf - Defines where data should be forwarded if your Indexers also act as forwarders to other tiers or environments. - Common in **tiered architecture setups**. #### distsearch.conf - Used on the Search Head to enable communication with clustered indexers. - Specifies the search peers and their connection settings. All configurations must include a shared `pass4SymmKey` for authentication between cluster nodes. ### **6. Monitoring and Troubleshooting** Proper monitoring of the cluster ensures **bucket replication stays healthy** and **search reliability is maintained**. #### Monitoring via Cluster Manager UI - Navigate to: **Settings > Indexer Clustering** - The UI shows: - Number of peer nodes - Bucket replication status - Fixup tasks - Searchable and non-searchable buckets #### CLI Monitoring You can also check cluster status via command line: ```bash splunk show cluster-status ``` This command displays: - Peer health - Replication and search factors - Buckets that are pending replication #### Watch for Common Issues - **Fixup Tasks**: These occur when buckets need to be replicated again (e.g., after a node comes back online). - **Pending Primaries**: Indicates that some data is not currently searchable. This can impact search completeness and should be addressed immediately. ### Summary: Indexer Clustering | Concept | Description | | ------------------- | ---------------------------------------------------------------- | | Purpose | Provides high availability, scalability, and redundancy | | Cluster Roles | Cluster Manager, Peer Nodes, Search Head | | Replication Factor | Number of raw data copies stored | | Search Factor | Number of searchable bucket copies | | Bucket Replication | Distributes data across nodes, maintains primaries for searches | | Configuration Files | `server.conf`, `indexes.conf`, `outputs.conf`, `distsearch.conf` | | Monitoring Tools | Web UI and `splunk show cluster-status` CLI | | Common Issues | Fixup tasks, pending primaries | --- # Search Head Clustering Search Head Clustering enables multiple Search Heads to work together as a **single logical unit**. It ensures that search configuration, knowledge objects, and scheduled jobs are **consistent and synchronized** across all members, and provides **high availability** in case of failure. ### **1. Purpose** The main goals of Search Head Clustering (SHC) are: - **High Availability**: Ensures search interface availability even if one or more Search Heads go offline. - **Configuration Consistency**: Keeps apps, knowledge objects, and configurations the same across all members. - **Horizontal Scaling**: Allows you to add more Search Heads to support more users and workloads. This is critical for enterprise environments where many users interact with Splunk dashboards, alerts, and searches simultaneously. ### **2. SHC Components** A functional SHC has three primary components: #### Cluster Members - These are the **Search Head nodes** that participate in the cluster. - Must be a **minimum of 3** members for quorum-based captain election to work properly. - Typically deployed behind a **load balancer** to distribute user traffic. #### Captain - One Search Head is elected as the **Captain**. - The Captain: - Coordinates scheduled searches - Manages knowledge object replication - Handles configuration synchronization - If the Captain goes offline, a **new captain is automatically elected**. #### Deployer - The **Deployer** is a **separate Splunk instance** (not part of the SHC itself). - It is used to **push configurations and apps** to all SHC members. - Required for maintaining **centralized app management**. ### **3. SHC Capabilities** SHC introduces several features that improve performance, consistency, and resilience. #### Shared Scheduling - Scheduled searches (like alerts and report updates) are **coordinated by the Captain**. - Ensures that **each search runs only once**, even if defined on multiple members. #### KV Store Replication - The **KV Store** (Key-Value Store) is used for storing: - Lookups - App state - Global variables - KV Store is replicated across all members for consistency. #### User Load Distribution - A **load balancer** can route user traffic across all SHC members. - Reduces load on any single node. - Improves user interface responsiveness and search concurrency. ### **4. Deployment Procedure** To set up an SHC, you must configure each member and bootstrap the cluster. The process involves: #### a. Edit `server.conf` on each Search Head: ```ini [shclustering] disabled = 0 pass4SymmKey = replication_port = ``` - `pass4SymmKey` is the shared secret for secure communication between nodes. - `replication_port` is used for KV Store and configuration replication. #### b. Bootstrap the Captain: Run this on one Search Head only: ``` splunk bootstrap shcluster-captain -servers_list "" ``` - This initializes the SHC and elects the first Captain. #### c. Add Members to the Cluster: Run this command on all other members: ``` splunk add shcluster-member -current_member_uri https://: ``` ### **5. Configuration with Deployer** The Deployer pushes **app bundles** to all SHC members using the following process. #### Push App Bundles: Use this command from the Deployer: ``` splunk apply shcluster-bundle -target https://: -auth admin:pass ``` - This pushes all configurations from the Deployer’s `$SPLUNK_HOME/etc/shcluster/apps/` to all SHC members. #### Important Notes: - The Deployer does **not manage user-specific settings** like saved searches or dashboard changes stored in users’ directories. - It only manages **shared app-level configurations**. ### **6. Troubleshooting SHC** When issues arise, the following commands and practices help diagnose and resolve problems. #### View Cluster Status: ``` splunk show shcluster-status ``` - Displays the captain, replication status, and scheduling sync status. #### List Cluster Members: ``` splunk list shcluster-members ``` - Shows all active members and their roles. #### Synchronize System Clocks: - All SHC members must have **NTP time synchronization**. - Time mismatches can cause **captain election failures** and replication errors. #### Resolve Split-Brain Scenarios: - If two nodes believe they are captain, you may need to: - Restart affected members - Force a captain re-election using CLI tools ### **7. Best Practices** To ensure your SHC is reliable, secure, and maintainable, follow these best practices: #### Use an Odd Number of Members - Always deploy **3, 5, or 7 Search Heads** to ensure a proper voting quorum for captain election. - Even numbers increase the risk of a tie (split-brain). #### Never Edit Configurations Directly on Members - Always use the **Deployer** to manage shared app configurations. - Direct edits can cause **inconsistencies and replication conflicts**. #### Secure Inter-Node Communication - Use **TLS encryption** and secure passwords (`pass4SymmKey`) for all communication. - Restrict access to management ports and ensure firewall policies are in place. ### Summary: Search Head Clustering (SHC) | Topic | Key Point | | ------------------------ | --------------------------------------------------------------- | | Purpose | High availability, scalability, and configuration consistency | | Components | Cluster Members, Captain, Deployer | | Scheduling | Shared across cluster, coordinated by Captain | | KV Store | Replicated across nodes for lookups and app state | | Deployment Process | Configure `server.conf`, bootstrap captain, add members | | Configuration Management | Use Deployer to push shared apps; no direct edits on members | | Troubleshooting Tools | `splunk show shcluster-status`, `splunk list shcluster-members` | | Best Practices | Use odd number of members, secure with TLS, avoid direct edits | --- # Deploying Splunk (Additional Content) ### **1. Boundary Between Search Head Clustering and Indexer Clustering** **Search Head Clustering (SHC)** and **Indexer Clustering** serve fundamentally different purposes, though both are deployed in distributed Splunk architectures. #### Search Head Clustering (SHC): - SHC is a **high-availability solution** designed to support **UI-level redundancy**, load balancing, and search distribution. - It focuses on **coordinating scheduled searches**, **replicating knowledge objects**, and ensuring that **multiple search heads appear as one logical interface** to the user. - SHC uses a **Captain** to coordinate scheduled searches and a **Deployer** to distribute configuration and apps. #### Indexer Clustering: - Indexer Clustering is primarily focused on **data-level redundancy** and **high availability of indexed data**. - It ensures that multiple copies of raw data and index files are maintained across a cluster of **peer indexers**, governed by a **Cluster Manager**. - Key parameters such as **Replication Factor (RF)** and **Search Factor (SF)** control the number of data copies and their searchability. #### Key Comparison (for exam memory aid): | Feature | SHC | Indexer Clustering | | --- | --- | --- | | Focus | Search interface and metadata | Raw data redundancy | | Key Coordinator | Captain | Cluster Manager (Master Node) | | Configuration Tool | Deployer | Configuration files (via CLI or UI) | | Data Stored | None (search head only) | Raw data and indexes | | Primary Benefit | UI redundancy and search scaling | Data resiliency and durability | ### **2. Forwarder Management Security Control** While the Deployment Server (DS) is an effective tool to manage configurations for Universal Forwarders (UFs), **security and control** over which clients are allowed to connect is critical, especially in large-scale environments. #### Access Control Mechanism: The following parameters can be configured in `serverclass.conf` on the Deployment Server to **restrict or permit UF access**: - **`whitelist.`**: Allows matching clients based on `clientName`, `host`, or IP address. - **`blacklist.`**: Explicitly denies access from specified clients. Examples: ```ini [serverClass:windows_clients] whitelist.0 = host::win* [serverClass:restricted] blacklist.0 = ip::10.0.0.50 ``` This access control helps **prevent unauthorized forwarders** from enrolling and receiving apps or configurations. ### **3. License Enforcement Details** The Splunk **License Master** is responsible for monitoring the volume of daily indexed data and **enforcing license limits**. #### License Violations: - If your Splunk environment **exceeds the licensed daily indexing volume** for **five non-consecutive days in a 30-day window**, you will enter a **"License Violation" state**. - In violation mode: - **User searches (non-admin)** are disabled. - Only **admin-level** accounts can run searches for remediation. - Violations do **not delete data**, but will halt scheduled reports and alerting for regular users. #### Monitoring: - Use the **Monitoring Console (MC)** under **License Usage** to visualize indexed volume per sourcetype, index, or forwarder. - Alternatively, monitor the log file: ``` $SPLUNK_HOME/var/log/splunk/license_usage.log ``` ### **4. Splunk Cloud Limitations** While Splunk Cloud provides a scalable and managed solution, it introduces **restrictions** compared to on-prem Splunk Enterprise. #### Common Limitations: - **No direct access to shell or file system**, which restricts: - Running custom Python scripts not validated by Splunk. - Using `btool` for deep configuration inspection. - **Limited access to certain apps or add-ons**, especially those requiring: - File system interaction - Shell scripting or custom binaries - Apps deployed must go through **App Vetting** to be accepted in Splunk Cloud. #### Exam Tip: Questions may ask **“Which feature is not available in Splunk Cloud?”**, so remember that **low-level debugging and certain scripted inputs** are not supported. ### **5. Compatibility of Deployment Server with SHC** A common **exam trap** is the misuse of **Deployment Server** in Search Head Clustering. #### Critical Rule: - **Deployment Server is NOT supported for SHC members**. - You must use a **Deployer** to push configuration bundles (apps, dashboards, saved searches) to SHC nodes. If you try to manage SHC members using a Deployment Server: - Configuration may become **inconsistent** across the cluster. - **Captain election and knowledge object replication may fail**. - **Supportability and upgrade paths may be broken**. ### Summary 1. Clarify the functional separation of SHC and Indexer Clustering under deployment models. 2. Emphasize access control for Universal Forwarders via `serverclass.conf` using whitelist/blacklist. 3. Reinforce licensing consequences, especially the 5-day violation rule and admin-only search fallback. 4. Detail Splunk Cloud limitations, especially around `btool`, unsupported apps, and scripting. 5. Highlight the **incompatibility of Deployment Server with SHC**, and the necessity of using a Deployer. --- # Monitoring Console (Additional Content) ### **1. Which Port Does the Monitoring Console Use?** The **Monitoring Console** (MC) runs on the same web interface as the rest of the Splunk instance it is enabled on. - By **default**, this is **port 8000**, which is the standard Splunk Web UI port. - However, this port is **not exclusive to the Monitoring Console**. It is shared with the broader Splunk Web interface. - Therefore, you **do not need to configure a separate port** specifically for the Monitoring Console. #### Exam Relevance: - Be aware that exam questions may try to mislead by asking whether the MC requires a dedicated listener port. The correct answer is **no**, it uses the existing Splunk Web service port. ### **2. What Is "Autodiscover" in General Setup?** During the **initial configuration of the Monitoring Console**, you can choose to enable **Autodiscover** from the General Setup page. - Autodiscover allows Splunk to **automatically scan for and detect other Splunk instances** (such as indexers, search heads, cluster managers). - Once discovered, these components are added to the MC view, and server roles can be inferred. #### Considerations: - While convenient in **simple or test deployments**, **Autodiscover is often disabled in production** due to the need for **precise control over what components are added**. - In most enterprise environments, **admins prefer to manually add instances** and assign roles explicitly. #### Exam Relevance: - Expect questions that ask whether Autodiscover is mandatory or recommended in production. The best practice is to **manually configure servers** for accuracy and role alignment. ### **3. What Permissions Does MC Rely On for Dashboards?** If Monitoring Console dashboards show **empty panels or missing data**, a likely cause is **search access failure**. This can result from one or more of the following: - **Distributed search misconfiguration**: The MC node may not be set up correctly to query search peers. - **Insufficient user role permissions**: The user accessing MC must have a role that: - Allows access to **necessary indexes**, such as `_introspection`, `_internal`, `_audit`. - Includes **srchIndexesAllowed** for the appropriate indexes. - Has capability to use **rest** and **dispatch** commands (if viewing real-time panels). #### Troubleshooting Tip: - Always check the MC user’s role in **Settings > Access Controls > Roles**, and confirm it has access to the required system indexes. #### Exam Tip: - Questions may ask how to resolve a situation where "the MC dashboard is empty or missing data." Ensure you understand the link between role permissions and dashboard visibility. ### **4. What to Check if the MC Cannot Access Cluster Data?** When the Monitoring Console fails to retrieve cluster-related metrics or data, consider the following areas for investigation: #### a. **Search Peer Configuration**: - Ensure the MC node is **configured as a Search Head** with **peer nodes added** via `distsearch.conf` or through **General Setup**. - Each peer should be reachable and authenticated using a **shared certificate or credential key**, depending on the environment. #### b. **Time Synchronization**: - The MC must be **synchronized with peer nodes** using NTP or another time service. - Time skew across nodes may cause **captain election errors**, replication problems, or **search job mismatches** in dashboards. #### c. **_introspection Index**: - This index contains **performance metrics**, including CPU, memory, and I/O. - If `_introspection` is **disabled** or **inaccessible**: - Dashboards like **Resource Usage** or **Indexer Performance** may be blank. - Check for **index retention policies**, disk space issues, or whether the index is receiving data at all. #### Exam Tip: - You may be asked to identify the cause of "missing data in the Indexer Performance dashboard." Be ready to check **index access**, **role permissions**, and **search peer reachability**. ### Summary 1. **MC uses the same Splunk Web port** (typically 8000); no need for a dedicated port. 2. **Autodiscover is optional** and often disabled in production for control and security. 3. **Permissions and search access** are essential for full dashboard functionality. 4. **Cluster access issues** may involve: - Missing or misconfigured search peers. - Lack of time synchronization. - Inactive or full `_introspection` index. --- # Access and Roles (Additional Content) ### **1. Configuration Files: `authentication.conf` and `authorize.conf`** Splunk controls user access and role-based permissions through a **combination of configuration files**, the most important of which are: #### `authentication.conf` - This file defines the **authentication method** Splunk uses to validate user identities. - It supports configurations for: - **Native authentication** (local password storage) - **LDAP/Active Directory integration** - **SAML (Single Sign-On)** - **Scripted or custom authentication backends** ##### Example: ```ini [authentication] authType = LDAP bindDN = cn=admin,dc=example,dc=com userBaseDN = ou=Users,dc=example,dc=com ``` #### `authorize.conf` - This is the **primary configuration file for role definitions**. - It controls: - Role-to-capability mappings - Index access permissions (`srchIndexesAllowed`, `srchIndexesDefault`) - Search filters (`srchFilter`) - Resource limits (concurrent searches, memory) ##### Example: ```ini [role_analyst] srchIndexesAllowed = main, web_logs srchFilter = host!="test*" schedule_search = enabled ``` ##### Exam Relevance: Questions may ask, “Which configuration file defines roles and their associated capabilities?” The correct answer is **`authorize.conf`**. ### **2. Advanced Role Controls via `restmap.conf` (Optional for Advanced Users)** For more advanced access management, Splunk allows you to **control REST endpoint access** using the **`restmap.conf`** file. - This file maps REST API endpoints to specific capabilities. - By modifying this file, you can **restrict or allow access to REST-based operations** based on user role. - Common use cases include: - Limiting which roles can access endpoints like `/services/search/jobs` - Protecting endpoints from scripted access unless explicitly permitted ##### Example: ```ini [admin:search/jobs] capability.read = list_search_jobs capability.write = edit_search_jobs ``` This mapping ensures that only users with the proper capability can read or write search job data through REST. ##### When to Use: While not required in every environment, this is **important in Splunk deployments involving API clients**, automation scripts, or integrations with external systems. ### **3. Limitations and Bypass Risks of `srchFilter` (Search Restrictions)** In `authorize.conf`, a role can be assigned a **search filter** (`srchFilter`) to enforce data-level restrictions. This filter is **automatically prepended** to all searches run by that role. ##### Example: ```ini srchFilter = host!="prod-secret*" ``` This ensures users with this role cannot search data from hosts starting with `prod-secret`. #### However, this mechanism **is not foolproof**. It can be bypassed in specific scenarios, such as: - **Scheduled Searches or Alerts**: - If a privileged user (admin) creates a scheduled search or alert, its results may be visible to users who would not have been allowed to perform the search directly. - **Macros and Workflow Actions**: - A macro that does not enforce the same restriction might be used in a dashboard. - If the macro is shared globally, a less-privileged user might invoke a search indirectly that returns restricted data. ##### Best Practices to Prevent Bypass: - **Ensure macros and saved searches also implement required filters.** - **Limit sharing of saved objects (alerts, reports, dashboards) that may expose sensitive content.** - **Use `restmap.conf` or app-level restrictions to further control object visibility.** ##### Exam Relevance: Candidates may encounter questions such as: “Why is a user seeing sensitive data despite a `srchFilter` restriction?” Correct responses should include that `srchFilter` only applies at execution time of an **interactive search**, not necessarily to the **output of scheduled or shared objects**. ### Summary | Topic | Explanation | | --- | --- | | `authentication.conf` | Defines how users are authenticated (LDAP, SAML, native) | | `authorize.conf` | Configures roles, capabilities, indexes, search filters | | `restmap.conf` | Maps REST endpoints to capabilities, controls API access | | Search filter limitations | Can be bypassed through saved searches, macros, or alerts if not configured carefully | --- # Data Collection (Additional Content) ### **1. Data Collection Performance Optimization** Optimizing how Splunk collects data from files can significantly impact indexing efficiency and prevent duplication. Several parameters in `inputs.conf` and `props.conf` can be tuned to improve performance: #### **`crcSalt`** - Used to **alter the file signature (checksum)** that Splunk uses to determine if a file has already been indexed. - Prevents **duplicate indexing** when: - Multiple files have the same name and size (e.g., rotated logs) - Files are moved between systems with similar paths - Common usage: append the source path to the checksum ```ini [monitor:///var/log/app.log] crcSalt = ``` Without this, Splunk might skip files it thinks it already indexed. #### **`initCrcLength`** - Sets the number of bytes that Splunk reads during the **initial CRC checksum generation**. - A lower value improves performance on large files when only the beginning is sufficient for unique identification. ```ini [monitor:///data/logs] initCrcLength = 256 ``` This setting helps **accelerate the file discovery and tracking process**. #### **`ignoreOlderThan`** - Tells Splunk to **skip files whose last modification time is older than the specified time window**. - Useful when indexing historical logs is unnecessary or undesirable. ```ini [monitor:///data/archive/] ignoreOlderThan = 30d ``` Prevents Splunk from spending resources reading outdated logs during startup or deployment. ### **2. Throughput Limiting on Universal Forwarder** When deploying Splunk Universal Forwarders (UF) across large environments, you may need to **limit their bandwidth usage** to avoid saturating network links. This is configured in `limits.conf` on the **Universal Forwarder**: ```ini [thruput] maxKBps = 256 ``` - This example limits the UF to **256 KB per second**. - The default is **unlimited (0)**, which may not be suitable for constrained environments. - This setting helps prevent performance bottlenecks on low-bandwidth WANs or remote offices. This is frequently tested in exams via questions about how to “throttle” data ingestion at the source. ### **3. Pre-Indexing Data Cleansing on Heavy Forwarder (HF)** Heavy Forwarders (HF) can **parse and transform data** before it is forwarded to the Indexer. This is useful for **masking sensitive data**, rewriting fields, or extracting new fields. #### Example: Masking sensitive information (e.g., credit card numbers) This is done using a combination of `props.conf` and `transforms.conf`. #### **`props.conf`** ```ini [host::appserver*] TRANSFORMS-mask = mask_cc ``` #### **`transforms.conf`** ```ini [mask_cc] REGEX = \d{4}-\d{4}-\d{4}-\d{4} FORMAT = ****-****-****-**** DEST_KEY = _raw ``` - `REGEX` matches the sensitive pattern (e.g., credit card number) - `FORMAT` replaces it with a masked version - `DEST_KEY = _raw` tells Splunk to rewrite the actual event This method ensures sensitive data is **never indexed**, helping organizations comply with **compliance standards like PCI DSS**. This type of config is commonly tested under the topic of **data preprocessing or field manipulation**. ### Summary | Area | Enhancement | | --- | --- | | File-level performance tuning | `crcSalt`, `initCrcLength`, and `ignoreOlderThan` optimize file monitoring | | Bandwidth throttling | `[thruput] maxKBps` limits data forwarding rate from UF | | Pre-index transformations | Use `props.conf` + `transforms.conf` on HFs to clean or mask data | --- # Indexing (Additional Content) ### **1. Purpose of `thawedBucketDir` (Recovering Archived Data)** In Splunk, data moves through lifecycle stages: **hot**, **warm**, **cold**, and **frozen**. Once data is frozen, it is typically **deleted**—unless an archive location is configured. The `thawedBucketDir` is the designated location used to **restore archived data** back into searchable form. #### Key Points: - Defined in `indexes.conf`: ```ini [my_index] thawedPath = $SPLUNK_DB/my_index/thaweddb ``` - To restore archived data: - Move the **frozen bucket** into the `thawedPath` directory - Restart or reload Splunk for the bucket to be recognized - No re-indexing required; data becomes immediately searchable #### Typical exam context: A question might ask how to “recover expired logs from backup.” The correct approach is to copy the backed-up bucket directory into `thawedBucketDir`. ### **2. Relationship Between Data Model Acceleration and Summary Indexing** Both **Data Model Acceleration (DMA)** and **Summary Indexing** are used to **speed up search performance**, but they differ in implementation. #### Summary Indexing: - Manually configured - Stores **search results** into a regular index - Requires a **scheduled search** that writes to the summary index #### Data Model Acceleration: - Automatically leverages internal **summary-like indexes** to precompute results - Used in **Pivot-based dashboards** and **CIM-compliant apps** (e.g., Enterprise Security) - Stored in: ``` $SPLUNK_HOME/var/lib/splunk/summary ``` #### Linkage: - Both reduce the need to scan raw events - DMA uses hidden summary structures (not a normal summary index, but conceptually similar) - Useful when explaining why DMA enables **fast pivot queries** and **dashboards** DMA-related questions may test your understanding of **search acceleration without user-created summary indexes**. ### **3. Compression Mechanism and Bloom Filter Overview** Splunk’s indexing engine uses several techniques to optimize **disk usage and search performance**. #### Compression: - Raw data is stored in **compressed form** to reduce storage space - Indexes (tsidx files) are also optimized with **tokenization** and **compression** - Compression is transparent to the user and improves storage efficiency without sacrificing performance #### Bloom Filters (used in tsidx files): - A **probabilistic data structure** that quickly checks whether a term **might exist** in a bucket - Used to eliminate non-matching buckets early in search - Improves search speed by avoiding unnecessary reads While Bloom filters are a low-level optimization, understanding them can help answer **advanced performance-oriented questions** about how Splunk narrows search scope quickly. ### **4. How to View Index Size and Bucket Status** To manage indexing efficiently, Splunk provides multiple methods for checking the **size and status of indexes and buckets**. #### From the CLI: - Use the `dbinspect` SPL command: ```spl | dbinspect index=_internal ``` This shows details such as: - Bucket type (hot, warm, cold) - Time range - Size on disk - Use the **REST API**: ``` /services/data/indexes ``` This returns metadata including: - Current size - Event count - Home and cold paths #### From the GUI: - Navigate to: ``` Settings > Indexes ``` This UI displays: - Index size - Event volume - Retention settings - Paths for hot/warm/cold storage This area is often tested through performance troubleshooting or retention configuration questions. ### Summary - `thawedBucketDir`: Used to recover archived (frozen) data for search - DMA vs. Summary Index: Both improve performance; DMA is automated, summary index is manual - Compression and Bloom Filters: Enhance storage and search efficiency - Monitoring Index Size: Use `dbinspect`, REST API, or UI to track bucket state and storage usage --- # Search (Additional Content) ### **1. Transforming vs. Non-Transforming Searches** Understanding the difference between **transforming** and **non-transforming** searches is crucial for questions related to **report acceleration**, **dashboards**, and **search optimization**. #### Transforming Searches: - These **change the shape** of raw events into **summarized results** (usually tables or charts) - Often used in dashboards and scheduled reports - Required for **Report Acceleration** to apply - End with commands like: - `stats` - `timechart` - `top` - `chart` **Example:** ```spl index=web_logs status=500 | stats count by host ``` This **counts events** and transforms them into a table — it’s a transforming search. #### Non-Transforming Searches: - Return **raw event data** - Used for detailed investigation or exploratory searches - Not eligible for report acceleration **Example:** ```spl index=web_logs status=500 host=web01 ``` This simply filters events; it does not transform or summarize them. #### Exam Relevance: Questions may ask: "Which of the following searches qualifies for Report Acceleration?" Only transforming searches like `| stats`, `| timechart`, etc., are valid answers. ### **2. Example Combining `eval` and `where`** Many SPL-based questions test your understanding of **intermediate-level SPL logic** using `eval`, `where`, and conditionals. Here is a **real-world-style example**: **Scenario:** You want to find events where the **calculated response time in milliseconds** exceeds **1000 ms**, and only show those for a specific host. ```spl index=web_logs host=web01 | eval response_ms = duration * 1000 | where response_ms > 1000 ``` #### Breakdown: - `eval response_ms = duration * 1000`: Creates a new field - `where response_ms > 1000`: Filters based on calculated value This combined use of `eval` + `where` often appears in questions that say: "Which search correctly calculates and filters based on a derived field?" ### **3. `tstats` vs. `datamodel` — Differences and Use Cases** In Splunk, **`tstats`** and **Data Models** are tightly linked, especially in CIM-compliant apps like **Enterprise Security (ES)**. #### `datamodel` command: - Used to explore the structure of a data model - Slower than `tstats` because it **does not use pre-acceleration** **Example:** ```spl | datamodel Web Web search ``` #### `tstats` command: - **Faster**, uses pre-accelerated summaries from **Data Model Acceleration** - Preferred in dashboards, alerts, and large-scale environments **Example:** ```spl | tstats count from datamodel=Web.Web where Web.status=500 by Web.host ``` #### Key Comparison: | Feature | `tstats` | `datamodel` | | --- | --- | --- | | Speed | Very fast (uses accelerated summaries) | Slower | | Common in | Dashboards, ES, summary-based reports | Ad hoc data exploration | | Acceleration Used | Yes | No | #### Exam Relevance: Questions might ask: "Which command leverages DMA for faster performance in CIM environments?" Answer: `tstats` ### Summary - **Transforming searches** are required for **report acceleration**; non-transforming are not. - Combining `eval` + `where` shows advanced filtering logic — often used in real troubleshooting. - **`tstats` is faster** and DMA-compatible; **`datamodel` is more exploratory**. --- # Configuration Management (Additional Content) ### **1. Common Fields in `outputs.conf`** The `outputs.conf` file is essential for configuring how data is **forwarded from a Universal or Heavy Forwarder to Indexers**. It’s a commonly tested area, especially in questions about **data routing** and **load balancing**. #### Key Fields to Know: - **`indexer`**: Specifies the address (hostname or IP) of a target indexer. Example: `indexer = 10.1.1.5:9997` - **`server`**: Specifies one or more target indexers (deprecated in favor of `group` and `indexer` in newer versions). - **`group`**: Used to define logical sets of indexers to load balance across. Example: `group = defaultGroup` - **`autoLBFrequency`**: Controls how often (in seconds) the forwarder switches between indexers in a group. Default is `30`. Example: `autoLBFrequency = 60` - **`useACK`** (for reliable delivery with Indexer Acknowledgment): Ensures that events are **acknowledged by the indexer** before being removed from the forwarder's queue. `useACK = true` #### Exam Relevance: You might be asked: "Which setting controls how often a forwarder rotates between indexers?" Correct Answer: `autoLBFrequency` ### **2. Modular App Naming Convention** In larger environments, Splunk encourages a **modular and layered design** for apps. This structure supports **reusability**, **upgradeability**, and **configuration clarity**. #### Naming Standards: - **`TA_app_input`**: Responsible for **input definitions only**, such as `inputs.conf`. Example: Collect logs from a specific source. - **`TA_app_parse`**: Contains **parsing logic**, such as `props.conf` and `transforms.conf`. Used for field extractions, sourcetypes, masking. - **`SA_common`** (Shared App): Common definitions like macros, lookups, and calculated fields that can be used by other apps. These names follow Splunk’s **Technology Add-on (TA)** and **Supporting Add-on (SA)** conventions. #### Exam Relevance: Questions may ask: "Which layer of an app should include props.conf transformations?" Answer: `TA_app_parse` ### **3. `conf.spec` Files and Configuration Precedence** Every `.conf` file in Splunk is governed by an associated **`.spec` file** located in `$SPLUNK_HOME/etc/system/README/`. #### Purpose: - Defines all **legal stanzas and fields** for that `.conf` file. - Serves as the **blueprint** for what the config engine expects. - Helps validate syntax and detect unsupported configurations. Example: `inputs.conf.spec`, `props.conf.spec` #### Advanced Detail: Although `.spec` files are not loaded at runtime, they **define the expected field types, default values, and precedence rules**. This is important for administrators who build **custom apps** or need deep visibility into config resolution. #### Exam Relevance: This may appear as a "select the most accurate statement" type question, such as: "What is the purpose of a `.spec` file in Splunk?" Answer: To define the legal structure and precedence of configuration fields. ### **4. Key Fields in `deploymentclient.conf`** The `deploymentclient.conf` file is placed on **Universal Forwarders (UFs)** to connect them to a **Deployment Server (DS)** for centralized configuration management. #### Common Fields: - **`targetUri`**: The address (host:port) of the Deployment Server. Example: `targetUri = ds01.company.com:8089` - **`clientName`**: A human-readable name for the forwarder (optional). Useful for filtering or targeting in `serverclass.conf`. - **`phoneHomeIntervalInSecs`**: Controls how frequently the forwarder contacts the Deployment Server. Example: `phoneHomeIntervalInSecs = 300` #### Example stanza: ```ini [deployment-client] targetUri = ds01.company.com:8089 clientName = web01_forwarder phoneHomeIntervalInSecs = 600 ``` #### Exam Relevance: Questions may test your understanding with: "Which field in `deploymentclient.conf` specifies how often the forwarder contacts the Deployment Server?" Answer: `phoneHomeIntervalInSecs` ### Summary - **`outputs.conf`**: Know how `group`, `indexer`, and `autoLBFrequency` define data routing and load balancing behavior. - **App modularization**: Recognize and apply naming conventions like `TA_app_input`, `TA_app_parse`, `SA_common`. - **`.spec` files**: Understand they define config structure and legal fields — helpful in identifying misconfigurations. - **`deploymentclient.conf`**: Key to linking forwarders with the DS, especially through fields like `targetUri` and `clientName`. --- # Indexer Clustering (Additional Content) ### 1. Multisite Clustering Overview In more complex or enterprise environments—particularly those involving **disaster recovery**, **geographically distributed data centers**, or **Splunk Enterprise Security (ES)**—you may encounter **multisite indexer clustering**. ##### What is Multisite Clustering? Multisite clustering is an enhancement of traditional indexer clustering that distributes indexing and replication responsibilities across **multiple physical sites**. Each site operates its own set of indexers, while replication policies define how data is stored across and within these sites. ##### Configuration Flag: ```ini [clustering] multisite = true ``` ##### Example Use Case: A company has two data centers: **Site1 (New York)** and **Site2 (San Francisco)**. They want data redundancy both **within a site** and **across sites** for disaster tolerance. ##### Key Configuration Concepts: - **Site-aware Replication Factor (RF):** You can define how many copies are stored **per site** or **in total**. Example: ```ini site_replication_factor = origin:2, total:4 ``` Meaning: - Keep **2 copies** in the origin site where the data was ingested. - Maintain **4 total copies** across all sites. - **Site-aware Search Factor (SF):** Example: ```ini site_search_factor = origin:1, total:2 ``` Meaning: - Ensure **1 searchable copy** is in the origin site. - Ensure **2 searchable copies** exist in total across all sites. ##### Other Key Differences When `multisite = true`: - Nodes must be tagged with their `site` in `server.conf`: ```ini [general] site = site1 ``` - The Cluster Manager (still a single node) must be aware of all sites and replication policies. ##### Why It Matters: - Enables **site isolation** in failure scenarios. - Supports **search affinity** (ensuring searches prefer local site data). - Often required in **regulated industries** with strict DR and uptime requirements. ##### Exam Relevance: Although multisite clustering is an advanced topic, SPLK-3003 may test your **awareness of configuration implications**, such as recognizing valid site_replication_factor syntax or the purpose of the `multisite = true` flag. ### 2. Exam-style Thought Exercises Including test-style prompts throughout your study of indexer clustering improves retention. Here are **section-specific examples**: **After Learning Cluster Roles:** **Question:** Which of the following roles is responsible for coordinating replication but does not index or search data? A. Peer Node B. License Master C. Cluster Manager D. Search Head **Answer:** C. Cluster Manager **After Learning About Replication/Search Factors:** **Question:** If a cluster has `replication_factor = 3` and `search_factor = 2`, how many copies of each bucket will be **searchable**? A. 1 B. 2 C. 3 D. Depends on the number of peer nodes **Answer:** B. 2 **After Multisite Clustering Introduction:** **Question:** Which configuration ensures that at least 2 copies of a bucket exist in the origin site, and 4 total across the cluster? A. `replication_factor = 2` B. `site_replication_factor = origin:2, total:4` C. `site_search_factor = origin:2, total:4` D. `multisite = false` **Answer:** B. `site_replication_factor = origin:2, total:4` ### Summary - **Multisite clustering** allows geographic and logical separation of indexing responsibilities. - It requires setting `multisite = true` and using `site_replication_factor` and `site_search_factor`. - Each peer must identify its site in `server.conf`. - This is especially relevant for **disaster recovery, compliance, and high-availability architectures**. - Including **exam-style questions per section** is an effective method to reinforce memory and simulate real testing scenarios. --- # Search Head Clustering (Additional Content) ### 1. Search Affinity in SHC Search Affinity allows Search Heads in a cluster to prioritize certain Indexers when executing search queries. This helps: - Reduce latency between Search Head and Indexer - Optimize search performance in large or geographically distributed environments **Configuration Location:** - Defined in `distsearch.conf` - Admins can manually control which Search Head connects to which Indexer first by setting search peer priorities or using affinity tagging **Exam Relevance:** You may be asked how to optimize search performance in SHC using indexer proximity or affinity, especially in hybrid or cloud environments. ### 2. Captain Election and Quorum (Raft Protocol) Search Head Clustering relies on a **distributed consensus protocol**, specifically the **Raft protocol**, to elect a captain and maintain consistent cluster behavior. **Key Mechanics:** - A **quorum** is required for any captain to be elected - A quorum is defined as **(N / 2) + 1**, where N is the number of SHC members - If quorum is lost (for example, due to a network split or power outage), SHC loses its ability to schedule searches or replicate knowledge objects **Best Practice:** Always deploy an **odd number of members** (3, 5, 7) to avoid split-vote scenarios. **Exam Tip:** Expect scenario-based questions like: *“What happens when quorum is lost in a 4-node SHC?”* Answer: **Captain cannot be elected; scheduled searches may fail.** ### 3. Purpose of `shclustering.conf` In addition to `server.conf`, the file `shclustering.conf` is used to: - Define the **logical identity** of the SHC - Set labels that help distinguish different SHC deployments in shared environments **Example Configuration:** ```ini [general] shcluster_label = shcluster_prod ``` This file is particularly useful in environments where multiple SHCs exist (such as staging and production) and must be clearly distinguished. **Exam Relevance:** You may be asked where to configure SHC identification or labeling. ### 4. Limitations of the Deployer The Deployer is responsible for pushing shared app configurations to all SHC members from: `$SPLUNK_HOME/etc/shcluster/apps/` However, it **cannot** deploy or manage: - **User-level content**, such as dashboards or saved searches located in `.../users/` directories - **Runtime changes**, such as those made directly in the UI - **Search Head-specific local configurations**, such as `local.meta` on individual SHC members **Best Practices:** - Use **Git or another version control system** to manage deployment content - Test deployments in a **staging SHC** before applying to production **Exam Relevance:** You may face questions like: *“Why didn’t a deployed app update user dashboards?”* Correct answer: **The Deployer does not manage user-level content.** ### 5. Unsupported Features in SHC Search Head Clustering imposes several restrictions: - **Cannot function as a Deployment Server (DS)**: SHC members should never be used to manage forwarders - **Manual commands like `splunk rebuild`** should not be run on individual SHC members - **Early versions (prior to 6.6)** do not support **KV Store clustering** All operations affecting cluster-wide behavior must be coordinated through the Deployer or the SHC management framework. **Common Mistakes to Avoid:** - Assigning Deployment Server roles to SHC members - Editing apps directly on a Search Head instead of through the Deployer **Exam Tip:** You may encounter **multiple-choice exclusion questions** such as: *“Which of the following is not supported in SHC?”* ### 6. Key Logs and Troubleshooting in SHC When problems arise in SHC—such as delayed knowledge object replication or missing dashboards—start by checking key log files: - `shclustering.log`: Tracks SHC-related coordination, bundle replication, and leadership status - `splunkd.log`: General log file that may include election failures, peer sync issues, or KV Store errors **Common Problems and How to Detect:** - **Captain election failure**: Look for Raft election logs or quorum warnings - **Bundle replication delay**: Messages like `bundle push timed out` may appear in `shclustering.log` - **KV Store inconsistency**: Detected by differences in app state or lookup results between members **Exam Relevance:** You may be asked: *“Where can you find evidence that bundle replication has failed?”* Correct answer: **shclustering.log** ### Summary | Area | Enhancement | | --- | --- | | Search Affinity | Optimizes search-to-indexer routing via `distsearch.conf` | | Captain Election | Based on Raft; requires quorum; odd-number SHC member count recommended | | shclustering.conf | Defines cluster label and metadata for multi-cluster identification | | Deployer Limitations | Does not push user content or UI-based changes | | Unsupported Features | DS role, manual low-level commands, unsupported KV Store behavior | | Logs for Troubleshooting | `shclustering.log` and `splunkd.log` provide key insight into cluster health | ---

Shopping cart

Subtotal:

Examination Introduction

Learning Method

Study Plan

Exam Center