SPLK-3003 Indexing

Indexing Detailed Explanation

In Splunk, indexing is the process of storing raw data and creating searchable metadata. This process allows Splunk to retrieve data quickly when users run searches, create dashboards, or generate alerts.

1. Indexing Basics

When Splunk collects data, it does not just save it as-is. It also processes the data into events, extracts metadata (like time, host, source, sourcetype), and stores it in an organized format.

Indexing consists of two main activities:

• Storing Raw Event Data
  The original content of the log or message is compressed and stored on disk.

• Creating Metadata
  Splunk creates a time-based index with fields like:

  • _time: The event timestamp

  • host: The system where the data came from

  • source: The file or input source

  • sourcetype: The format of the data (e.g., json, access_combined)

This structured data enables fast and efficient search, even across billions of events.

Splunk stores data in a special format using buckets, which are organized based on the data’s age and lifecycle stage (see 5.3 below).

2. Index Types

Splunk supports different types of indexes, each optimized for specific data formats or use cases.

Event Indexes

• Default type of index in Splunk.

• Used to store event-based logs, such as:

  • Web server logs

  • Firewall logs

  • Application logs

• Events are timestamped and stored in time order.

Metrics Indexes

• Designed for numerical time-series data, such as:

  • CPU usage

  • Memory consumption

  • Disk I/O

• Enables efficient storage and faster search for performance metrics.

• Used with mcollect, metrics.log, or collectd.

Summary Indexes

• Special indexes that store the results of scheduled or accelerated searches.

• Useful for:

  • Generating dashboards quickly

  • Reducing the need to re-run expensive searches

• Example: A nightly search calculates average error rates, and its result is stored in a summary index.

Internal Indexes

These are used by Splunk to store its own logs and internal data. Common internal indexes include:

• _internal: Scheduler logs, indexing performance, and errors

• _audit: User login attempts, role changes, and audit events

• _introspection: System-level performance metrics (CPU, memory, etc.)

You typically do not ingest data into these manually, but you search them for troubleshooting and system monitoring.

3. Bucket Lifecycle

Splunk stores data in units called buckets. A bucket is a directory on disk containing raw data and index files. Each bucket belongs to one of several lifecycle stages based on the age of its data.

Hot Bucket

• The bucket currently receiving new data.

• Stored in memory or fast disk for performance.

• Each index can have multiple hot buckets depending on settings.

Warm Bucket

• Once a hot bucket is full or the system restarts, it becomes a warm bucket.

• It is closed to writing but remains searchable.

• Stored on disk, usually still fast storage.

Cold Bucket

• Older data is moved from warm to cold storage.

• This data is still searchable but is placed on less expensive or slower disk.

Frozen Bucket

• Once data reaches its retention limit, it is frozen.

• Frozen data is deleted by default.

• You can configure Splunk to archive frozen data to a backup location.

• Frozen data is no longer searchable unless manually restored to a thawed state.

These transitions are configured in indexes.conf.

Example settings:

maxHotSpanSecs = 86400
maxTotalDataSizeMB = 500000
frozenTimePeriodInSecs = 31536000

4. Index Configuration Parameters

These settings in the indexes.conf file define how each index behaves.

homePath and coldPath

• homePath: Location where hot and warm buckets are stored.

• coldPath: Location where cold buckets are moved after they age out of warm.

This allows you to use different disk volumes for different lifecycle stages.

maxDataSize

• Controls the maximum size of each hot bucket.

• This setting affects how often new buckets are created.

• Options include auto, auto_high_volume, or custom size in MB.

maxTotalDataSizeMB

• Sets the maximum size of all buckets in the index.

• Once the limit is reached, older data is deleted (frozen) to make space.

frozenTimePeriodInSecs

• Sets the retention period for data in seconds.

• After this time, the data moves to the frozen state.

• For example, to retain data for one year:
  frozenTimePeriodInSecs = 31536000

5. Index Cloning and Replication

In indexer clustering, Splunk replicates data to multiple peers to provide high availability and fault tolerance.

Replication Factor

• The number of copies of raw data that should exist in the cluster.

• If replication factor is 3, there will be 3 total copies of each bucket on different indexers.

Search Factor

• The number of searchable copies required.

• These are copies of buckets with index files that allow searches.

• If search factor is 2, two of the bucket copies must be fully searchable.

These settings are defined on the Cluster Manager (Master Node), and enforced automatically during indexing.

Benefits:

• Prevents data loss if a peer indexer goes offline.

• Maintains search functionality even if some nodes fail.

Indexing (Additional Content)

1. Purpose of thawedBucketDir (Recovering Archived Data)

In Splunk, data moves through lifecycle stages: hot, warm, cold, and frozen. Once data is frozen, it is typically deleted—unless an archive location is configured. The thawedBucketDir is the designated location used to restore archived data back into searchable form.

Key Points:

• Defined in indexes.conf:

  [my_index]
  thawedPath = $SPLUNK_DB/my_index/thaweddb
  

• To restore archived data:

  • Move the frozen bucket into the thawedPath directory

  • Restart or reload Splunk for the bucket to be recognized

  • No re-indexing required; data becomes immediately searchable

Typical exam context:

A question might ask how to “recover expired logs from backup.” The correct approach is to copy the backed-up bucket directory into thawedBucketDir.

2. Relationship Between Data Model Acceleration and Summary Indexing

Both Data Model Acceleration (DMA) and Summary Indexing are used to speed up search performance, but they differ in implementation.

Summary Indexing:

• Manually configured

• Stores search results into a regular index

• Requires a scheduled search that writes to the summary index

Data Model Acceleration:

• Automatically leverages internal summary-like indexes to precompute results

• Used in Pivot-based dashboards and CIM-compliant apps (e.g., Enterprise Security)

• Stored in:

  $SPLUNK_HOME/var/lib/splunk/summary
  

Linkage:

• Both reduce the need to scan raw events

• DMA uses hidden summary structures (not a normal summary index, but conceptually similar)

• Useful when explaining why DMA enables fast pivot queries and dashboards

DMA-related questions may test your understanding of search acceleration without user-created summary indexes.

3. Compression Mechanism and Bloom Filter Overview

Splunk’s indexing engine uses several techniques to optimize disk usage and search performance.

Compression:

• Raw data is stored in compressed form to reduce storage space

• Indexes (tsidx files) are also optimized with tokenization and compression

• Compression is transparent to the user and improves storage efficiency without sacrificing performance

Bloom Filters (used in tsidx files):

• A probabilistic data structure that quickly checks whether a term might exist in a bucket

• Used to eliminate non-matching buckets early in search

• Improves search speed by avoiding unnecessary reads

While Bloom filters are a low-level optimization, understanding them can help answer advanced performance-oriented questions about how Splunk narrows search scope quickly.

4. How to View Index Size and Bucket Status

To manage indexing efficiently, Splunk provides multiple methods for checking the size and status of indexes and buckets.

From the CLI:

• Use the dbinspect SPL command:

  | dbinspect index=_internal
  

  This shows details such as:

  • Bucket type (hot, warm, cold)

  • Time range

  • Size on disk

• Use the REST API:

  /services/data/indexes
  

  This returns metadata including:

  • Current size

  • Event count

  • Home and cold paths

From the GUI:

• Navigate to:

  Settings > Indexes
  

  This UI displays:

  • Index size

  • Event volume

  • Retention settings

  • Paths for hot/warm/cold storage

This area is often tested through performance troubleshooting or retention configuration questions.

Summary

• thawedBucketDir: Used to recover archived (frozen) data for search

• DMA vs. Summary Index: Both improve performance; DMA is automated, summary index is manual

• Compression and Bloom Filters: Enhance storage and search efficiency

• Monitoring Index Size: Use dbinspect, REST API, or UI to track bucket state and storage usage