SPLK-2002 Infrastructure Planning: Index Design

Infrastructure Planning: Index Design Detailed Explanation

1. Index Design in Splunk

What Is an Index?

An index in Splunk is a logical data store. Think of it like a folder or a container where Splunk keeps the data it collects.

When Splunk receives data (like logs or events), it processes that data and stores it in an index, so that users can later search it.

Splunk Index Structure: Buckets

Splunk stores data inside buckets, and each bucket represents a stage in the data's life cycle. Here's what that means:

Bucket Type	Description
Hot Bucket	Where new data is written (active, fast)
Warm Bucket	Recently indexed, not being written to anymore
Cold Bucket	Older data, moved to slower storage
Frozen Bucket	Very old data; either archived outside Splunk or deleted

Example:

• Logs from today go into Hot.

• After a few days, they move to Warm.

• After a few weeks or months, they go to Cold.

• After reaching the age limit, they go to Frozen, where they can either be archived or deleted.

Index Metadata

Each event stored in an index comes with metadata. This is extra information that helps Splunk organize and search the data.

Key metadata fields include:

• Source: Where the data came from (e.g., /var/log/syslog)

• Sourcetype: Format or type of data (e.g., syslog, access_combined)

• Host: The system that generated the data (e.g., webserver01)

• Timestamp: The time the event occurred

Why it matters: Metadata is what allows you to search with filters like host=web01 sourcetype=access_combined.

2. Index Design in Splunk

Key Considerations in Index Design

Now that you understand what an index is, let’s explore how to design them properly. Good index design keeps your Splunk system organized, fast, and scalable.

a. Retention Planning

What it means: Decide how long to keep data at each bucket stage.

Each index can have its own retention settings, defined in indexes.conf. You can control:

• How long data stays searchable

• When to archive or delete old data

• How to manage disk usage

The key setting is:
frozenTimePeriodInSecs = <number of seconds>

This defines how long data is kept before being moved to Frozen (or deleted).

Example:

• You want to keep web logs searchable for 6 months:

  • frozenTimePeriodInSecs = 15778463 (approx. 6 months)

Why it matters:

• Helps manage storage cost and search performance

• Ensures compliance with data retention policies

b. Segmentation (Index Separation)

What it means: Create different indexes for different types of data or use cases.

Instead of putting all logs into a single index, you divide them by purpose:

• security → For firewall, IDS, authentication logs

• app_logs → For application performance or errors

• infra_logs → For system and infrastructure metrics

• web_logs → For Nginx/Apache access logs

Why it matters:

• Easier access control (restrict who can see which data)

• More efficient searches (target only the indexes you need)

• Simplifies compliance audits (e.g., “only search security data”)

Best Practice:

Use index-level access control to limit who can search which data (via roles).

c. Indexing Volume Estimation

What it means: Estimate how much data each index will receive per day.

You need to know:

• Total volume per day (e.g., 300 GB/day)

• Breakdown by index (e.g., app_logs = 100 GB/day, infra_logs = 50 GB/day)

This helps with:

• Sizing your indexer hardware

• Planning for storage space

• Choosing whether to use clustering

Tip:

Always plan for growth — if your current usage is 300 GB/day, design for 500–700 GB/day in the future.

3. Index Design in Splunk

a. Throttling and Limiting

What it means: Prevent Splunk from indexing too much data too quickly, which could crash the system or overload licensing.

You can throttle or control indexing using:

• limits.conf: Controls system-wide limits, like queue sizes or throughput.

• props.conf and transforms.conf: Can route or drop unwanted data before it gets indexed.

Example:

• You may want to ignore DEBUG-level logs:

  • Use transforms.conf to drop them before indexing.

Why it matters:

• Keeps your license usage under control.

• Protects indexers from being overwhelmed.

• Filters out unnecessary data (saves storage and money).

b. Data Model Acceleration (DMA) Impact

What it means: Some indexes are connected to data models (used in Pivot and certain dashboards). To make them fast, Splunk creates accelerated summaries.

While this improves search speed, it uses:

• More disk space

• Extra CPU resources

Best Practices:

• Only enable acceleration where it’s really needed.

• Use it with smaller or filtered datasets if possible.

• Monitor performance impact in the Monitoring Console.

Infrastructure Planning: Index Design (Additional Content)

1. Index Splitting vs. Summary Indexing

When designing an indexing strategy, it’s important to consider whether to rely solely on raw data indexes or to include summary indexing as part of data optimization and reporting strategies.

Summary Indexing Overview:

• A technique where the results of scheduled searches (e.g., daily aggregates, KPIs) are written into a dedicated index for faster access.

• Often used in conjunction with:

  • Data Model Acceleration (DMA)

  • Report acceleration

When to Use Summary Indexing:

• When reporting across large data volumes causes performance issues

• When real-time performance is required for dashboards

• As a fallback or supplement when DMA is not feasible (e.g., unsupported data types)

Why it matters:
Summary indexing reduces search load on raw data and supports long-term trend analysis with minimal performance impact.

2. Index Naming Convention

Clear and consistent naming of indexes is essential for scalability, maintenance, and access control.

Recommended Naming Convention:

• Use structured, hierarchical patterns such as:

  <team>_<data_type>
  

  Examples:

  • sec_firewall

  • web_access

  • finance_transactions

Benefits:

• Easier to:

  • Assign index-based permissions

  • Implement per-team dashboards

  • Filter searches using index=team_* or index=*_type

• Improves clarity for:

  • Onboarding new admins

  • Writing macros and search constraints

Why it matters:
A logical naming scheme simplifies RBAC (role-based access control), promotes organizational consistency, and aids troubleshooting.

3. Multiple Indexes vs. One Large Index

An important architectural decision is whether to store all data in a single index or to split data across multiple indexes.

Strategy	Advantages	Drawbacks
Single Large Index	- Simpler configuration	- Harder to enforce access control
	- Fewer indexes to manage	- May require field-level filtering
Multiple Smaller Indexes	- Easier to assign roles and permissions	- Slightly more admin overhead
	- Better visibility and isolation by source	- Potential index misalignment in queries

General Best Practice:
Use multiple indexes, especially when different teams, data types, or retention policies are involved.

Why it matters:
Supports security, manageability, and search efficiency as environments grow in size and complexity.

4. Index Cluster Replication Factors

When designing index sizing and storage, don’t forget the impact of clustering, particularly Replication Factor (RF) and Search Factor (SF).

Clustering Factors and Storage Multiplication:

• Replication Factor (RF):

  • Number of total copies of each bucket across indexers.

  • Example: RF=3 triples the raw storage requirement.

• Search Factor (SF):

  • Number of searchable copies of each bucket.

  • Affects how many copies must be fully processed and available for search.

Storage Planning Implication:

• For a 100 GB/day ingestion:

  • RF=3 → 300 GB/day raw storage, before compression.

  • Plus overhead for metadata and indexing.

Why it matters:
You must factor RF/SF into index storage estimation, especially in indexer cluster environments, to avoid undersized or overburdened storage systems.