SPLK-1004 Using Acceleration Options: Data Models and tsidx Files

Using Acceleration Options: Data Models and tsidx Files Detailed Explanation

1. Data Model Acceleration (DMA)

What is a Data Model?

A data model in Splunk is a structured abstraction layer on top of raw event data. It organizes fields and tags into a meaningful schema that supports:

• Pivot reports

• CIM (Common Information Model) normalization

• tstats searches (high-performance queries)

What is Data Model Acceleration (DMA)?

Data Model Acceleration pre-computes summaries of your data model for faster querying. Instead of scanning raw data, Splunk can read from accelerated summaries, which are optimized for performance.

How to Enable DMA

1. Go to Settings > Data Models

2. Select the data model you want to accelerate (e.g., Web, Authentication)

3. Click Edit > Edit Acceleration

4. Enable acceleration and define a summary range (e.g., 7 days)

5. Save your settings

Once enabled, Splunk will begin building acceleration summaries in the background.

Benefits of DMA

• Speeds up searches using the tstats command

• Greatly improves dashboard performance

• Enables fast reporting over CIM-compliant data

2. tsidx Files and tsidx Reduction

What are tsidx Files?

tsidx stands for time-series index. These files are generated by Splunk and contain indexed metadata that helps locate raw events quickly.

A tsidx file allows Splunk to:

• Search by time

• Filter by indexed fields

• Avoid scanning all raw event data

What is tsidx Reduction?

tsidx reduction is a space-saving technique that keeps only summary metadata in older buckets and removes detailed metadata and raw data. This:

• Reduces disk space usage

• Speeds up metadata-only queries

• May limit full-detail searches for those older events

You can configure tsidxReduction in indexes.conf by specifying:

• How long to retain full detail

• When to switch to reduced mode

3. tstats Command

The tstats command is a high-performance search command that works only on accelerated data models or indexed metadata.

Why is it fast?

• It does not read raw data

• It reads from tsidx files or accelerated summaries

• It’s ideal for aggregations (like counts, sums, averages)

Basic Syntax:

| tstats count where index=web by _time, status

This command:

• Counts events in the web index

• Groups by _time and status

• Runs much faster than equivalent stats on raw data

Advanced Example:

| tstats sum(bytes) as total_bytes from datamodel=Web.Web where (status=200 OR status=404) by _time, http_method

This uses a CIM-compliant data model and returns total bytes by time and HTTP method.

4. Use Cases

a) Dashboards with real-time metrics

• Use tstats to power panels that need to load quickly.

• Especially useful in executive or operations dashboards.

b) Security event analysis (CIM)

• CIM-normalized data models (e.g., Authentication, Intrusion Detection) can be accelerated.

• Security teams use tstats with accelerated models for fast threat hunting.

c) Compliance reporting

• Scheduled compliance reports often scan large timeframes (e.g., last 90 days).

• Using tstats with DMA allows you to generate reports faster and reduce system load.

5. Best Practices

a) Use tstats instead of stats whenever possible

• For aggregated queries, tstats is significantly faster and more efficient.

• Ideal for dashboards, reports, alerts, and large-scale searches.

b) Monitor acceleration summaries regularly

• Check the Data Model Acceleration status for:

  • Summary size

  • Lag (how up-to-date the summaries are)

  • Error messages

• If lag is high, Splunk may not use the summary, causing slow queries.

c) Combine tstats with lookups for enrichment

• tstats returns raw values quickly; you can use lookup to add:

  • User names

  • Department details

  • IP geolocation

Example:

| tstats count where index=firewall by src_ip
| lookup ip_location ip as src_ip OUTPUT city, country

Summary Table: Data Models and tsidx Features

Feature	Description
Data Model Acceleration	Pre-computes summaries of structured data models
tsidx Files	Metadata indexes for efficient search
tsidx Reduction	Space-saving by removing older detailed data
tstats Command	High-speed aggregation command using accelerated data
Key Use Cases	Dashboards, Security Analytics, Compliance Reporting
Best Practice	Use tstats, monitor summaries, enrich with lookups

Using Acceleration Options: Data Models and tsidx Files (Additional Content)

1. Physical Storage of Accelerated Data Models

When a data model is accelerated, Splunk stores the resulting summaries on disk, separate from raw events and traditional indexes.

Storage Location:

By default, the summary data for accelerated data models is stored under:
$SPLUNK_HOME/var/lib/splunk/
within a subdirectory structure that reflects the accelerated model’s name.

Why It Matters:

• This is essential knowledge for system administrators:

  • When investigating acceleration lag or failures

  • When managing disk space usage in high-ingestion environments

• These summary directories can consume significant storage if not managed or rotated properly.

Operational Tip:

• Regularly monitor $SPLUNK_HOME/var/lib/splunk/modinputs/accelerated_datamodels/

• Use the Monitoring Console to inspect summary sizes and rebuild status

2. Enabling tsidx Reduction (Indexes.conf)

tsidx reduction is a feature designed to reduce disk space by trimming down older buckets—removing detailed tsidx metadata while retaining minimal pointers for high-level searches.

Sample Configuration:

[indexname]
enableTsidxReduction = true
minHotIdleSecsBeforeTsidxReduction = 604800  ; 7 days

Key Explanation:

• enableTsidxReduction: Enables the feature for this index

• minHotIdleSecsBeforeTsidxReduction: Time before reduction begins (in seconds)

• After this idle period, hot buckets are marked for reduction to save space

Benefits:

• Reduces disk usage, especially in long-retention environments

• Supports metadata-only querying (e.g., by time, host, sourcetype)

Caveats:

• Once reduced, some search operations like event sampling, preview, _raw access may not work on those buckets

• Full search fidelity is sacrificed for performance and space savings

3. Limitations of tstats Searches

tstats is highly optimized but has specific limitations that are important for both production use and exam preparation.

Not Supported in tstats:

Limitation	Description
Non-indexed fields	Cannot use fields that are not indexed (i.e., extracted at search-time only)
Raw event access	Cannot access _raw, so no rex, eval on raw data
Unmapped fields	Cannot use fields that are not defined in the accelerated data model
Unaccelerated models	tstats only works on accelerated data models; otherwise, it returns nothing

Example of Invalid Usage:

| tstats count where index=web by user_agent  ←  fails if `user_agent` is not indexed or not in data model

Recommendation:

• Only use tstats on fields that are:

  • Indexed OR

  • Included in the accelerated data model structure

Quick Recap of High-Value Deployment Notes

Topic	Details
Data model acceleration storage	Located under $SPLUNK_HOME/var/lib/splunk/...
tsidx reduction configuration	enableTsidxReduction=true in indexes.conf
tstats search limitations	No _raw, no search-time-only fields, works only on indexed or modeled data