SPLK-1002 Creating Data Models

Creating Data Models Detailed Explanation

Data models in Splunk provide a structured way to organize and accelerate data for analysis. They are widely used to power dashboards, generate reports, and perform advanced analytics efficiently.

1. What Are Data Models?

Definition

A Data Model is a structured representation of datasets in Splunk, designed to:

• Organize raw data into meaningful categories.
• Apply filters and correlations to refine datasets.
• Enable faster searches and visualizations through acceleration.

Purpose

• Simplify and standardize complex datasets.
• Enhance performance for dashboards and reports.
• Provide a reusable framework for data analysis.

2. Core Components of Data Models

2.1. Datasets

Datasets are the building blocks of a data model. Each dataset represents a portion of the data and can be refined for specific purposes.

1. Event Dataset:

   • Represents raw event data.
   • Example: Logs from index=web_logs.

2. Search Dataset:

   • A subset of events filtered by a search query.
   • Example: Logs with status_code=200.

3. Transaction Dataset:

   • Combines multiple events into transactions based on shared fields or temporal proximity.
   • Example: Grouping events with the same session_id that occurred within 15 minutes.

2.2. Fields

Fields define the attributes of a dataset, making it easier to analyze and visualize data.

1. Auto-Extracted Fields:

   • Fields automatically extracted by Splunk during data ingestion.
   • Example: _time, host, source.

2. Calculated Fields:

   • Fields created within the data model based on expressions or transformations.
   • Example: response_time = end_time - start_time.

2.3. Acceleration

Acceleration improves the performance of data models by precomputing and storing summarized data.

1. How It Works:

   • Splunk precomputes the results of your datasets and stores them as summaries.
   • Accelerated data models are faster but require additional storage.

2. When to Use:

   • Recommended for large datasets frequently queried in dashboards or reports.

3. Configuration:

   • Enable acceleration when creating or editing a data model.
   • Define the summary range (e.g., past 7 days).

3. How to Create a Data Model

3.1. Steps to Create a Data Model

1. Navigate to Data Models:

   • Go to Settings > Data Models.
   • Click New Data Model.

2. Define the Model:

   • Name: Provide a descriptive name (e.g., Web Traffic).
   • Root Dataset: Select the initial dataset (Event, Search, or Transaction).

3. Add Datasets:

   • Add datasets to refine or extend the model.
   • Specify filters for search or transaction datasets.

4. Add Fields:

   • Include auto-extracted fields, calculated fields, or aliases as needed.

5. Enable Acceleration (Optional):

   • Turn on acceleration for faster performance.

6. Save the Data Model.

3.2. Example: Web Traffic Data Model

1. Data Model Name: Web Traffic
2. Root Dataset: Event dataset with index=web_logs.
3. Search Dataset: Refined logs where status_code=200.
4. Transaction Dataset: Group logs by session_id with a maximum span of 15 minutes.

Steps:

1. Create a root dataset:

   index=web_logs
   

2. Add a search dataset:

   status_code=200
   

3. Add a transaction dataset:

   group by session_id maxspan=15m
   

Result: A structured model of web traffic data, ready for use in dashboards or reports.

4. Use Cases for Data Models

1. Website Analytics:

   • Analyze user sessions, page views, and bounce rates.
   • Example: A Web Traffic data model for e-commerce websites.

2. Security Monitoring:

   • Correlate login events with suspicious activities.
   • Example: A User Behavior data model for detecting anomalies.

3. Performance Tracking:

   • Monitor system metrics like CPU and memory usage.
   • Example: A System Health data model for infrastructure monitoring.

5. Best Practices for Data Models

5.1. Optimize for Specific Use Cases

• Define datasets tailored to the business questions you want to answer.
• Example: Use separate datasets for successful and failed transactions.

5.2. Use Acceleration Judiciously

• Enable acceleration only for frequently accessed data models.
• Balance storage requirements with performance needs.

5.3. Simplify Field Selection

• Include only relevant fields to reduce model complexity.
• Example: Focus on user_id, session_id, and status_code for a web traffic model.

5.4. Test Filters and Queries

• Validate dataset filters and queries to ensure accuracy and relevance.

5.5. Regularly Update Models

• Keep data models up to date with changing requirements or new data sources.

6. Practical Exercises

Exercise 1: Create an Event Dataset

1. Create a data model named System Logs.

2. Define an event dataset with:

   index=system_logs
   

3. Add the following fields:

   • host
   • source
   • event_type

4. Save the model.

Task: Verify that the event dataset retrieves logs from index=system_logs.

Exercise 2: Add a Search Dataset

1. Extend the System Logs data model.

2. Add a search dataset with:

   event_type="error"
   

3. Save the changes.

Task: Confirm that the search dataset retrieves only error events.

Exercise 3: Add a Transaction Dataset

1. Add a transaction dataset to the System Logs data model.
2. Group events by:
   • session_id
   • Maximum span: 10 minutes.
3. Save the dataset.

Task: Verify that the transaction dataset groups related events correctly.

7. Advanced Configurations for Data Models

7.1. Adding Calculated Fields

Calculated fields allow you to derive new values from existing data directly within the data model.

Example: Calculate Response Time

1. Field Name: response_time

2. Expression:

   eval response_time = end_time - start_time
   

3. Steps:

   • Open the data model.
   • Add a calculated field under the desired dataset.
   • Define the expression and save.

Result: Adds a response_time field to the dataset for use in analysis.

7.2. Using Aliases for Field Normalization

Aliases let you map inconsistent field names across different sources to a standard name.

Example: Normalize IP Fields

• Source 1 Field: src_ip
• Source 2 Field: source_ip
• Alias Field: ip_address

Steps:

1. Add an alias field named ip_address.
2. Map:
   • src_ip → ip_address
   • source_ip → ip_address

Result: Normalized field names for consistent analysis.

7.3. Advanced Filtering for Search Datasets

Search datasets can include complex queries to filter data more effectively.

Example: Filter Critical Errors

1. Search Query:

   error_level="critical" AND app="web_app"
   

2. Steps:

   • Add a search dataset to your data model.
   • Define the query and save.

Result: Retrieves only critical errors related to the web_app.

7.4. Hierarchical Data Models

You can create hierarchical data models where datasets are nested to represent relationships.

Example: Web Traffic Hierarchy

1. Root Dataset: All web logs (index=web_logs).
2. Child Dataset: Successful requests (status_code=200).
3. Child of Child Dataset: Requests for a specific resource (url="/home").

Steps:

1. Create the root dataset.
2. Add child datasets for each refinement.

Result: A structured hierarchy for detailed analysis.

8. Troubleshooting Data Models

8.1. Missing Fields in Results

Cause

• The field is not included in the dataset.

Solution

1. Open the data model and verify the field is added.
2. Add the field if missing, then save and rebuild the model.

8.2. Data Model Acceleration Fails

Cause

• Insufficient storage or improper configuration.

Solution

1. Check disk space on the Splunk server.
2. Verify acceleration settings:
   • Time range (e.g., last 7 days).
   • Appropriate app context.

8.3. Poor Performance with Accelerated Models

Cause

• Too many fields or overly complex datasets.

Solution

1. Optimize the data model:
   • Remove unnecessary fields.
   • Simplify dataset filters.
2. Monitor system performance to identify bottlenecks.

9. Optimization Strategies

9.1. Limit Fields in Datasets

• Include only the fields required for analysis to improve performance.

9.2. Enable Acceleration Only When Needed

• Use acceleration for high-volume or frequently queried data models.
• Set an appropriate summary range (e.g., 7 days).

9.3. Use Summary Indexing for Historical Data

• Archive older data in summary indexes to reduce the load on accelerated models.

9.4. Test Before Deployment

• Validate the data model’s performance and accuracy in a test environment

10. Practical Exercises

Exercise 1: Add a Calculated Field

1. Open a data model for system logs.

2. Add a calculated field:

   • Name: time_spent

   • Expression:

     eval time_spent = duration * 60
     

3. Save the data model.

Task: Verify that time_spent is available in the dataset.

Exercise 2: Normalize Field Names

1. Open the web traffic data model.
2. Add an alias field:
   • Name: user_ip
   • Map:
     • client_ip → user_ip
     • src_ip → user_ip
3. Save the changes.

Task: Confirm that queries using user_ip return results from all sources.

Exercise 3: Create a Search Dataset with Filters

1. Add a search dataset for critical errors:

   • Query:

     error_level="critical" AND app="payment_service"
     

2. Save the dataset.

Task: Validate that only critical errors related to the payment_service app are included.

Exercise 4: Enable Acceleration

1. Enable acceleration for a high-volume data model:
   • Set a summary range of the past 7 days.
2. Save and rebuild the model.

Task: Measure the performance improvement in dashboard queries.

11. Summary of Key Points

1. Core Components:

   • Datasets: Event, Search, and Transaction datasets structure your data.
   • Fields: Auto-extracted and calculated fields enrich the dataset.

2. Acceleration:

   • Precomputes data for faster searches and dashboards.

3. Advanced Configurations:

   • Add calculated fields, aliases, and filters.
   • Create hierarchical datasets for detailed analysis.

4. Best Practices:

   • Optimize datasets for specific use cases.
   • Use acceleration judiciously to balance performance and storage.
   • Regularly update and test data models to reflect changes in data or requirements.

Creating Data Models (Additional Content)

1. Data Model Permissions and Access Control

Data models in Splunk are objects with scoped access, and their visibility or editability can be restricted at the app level or based on user roles.

Permission Management Options:

• Location: Permissions can be set through Settings > Data Models, then clicking on the specific model and choosing “Permissions”.

• Access Levels:

  • Read: View and use the data model (e.g., for Pivot, tstats).

  • Write: Modify the structure, fields, and acceleration settings.

• Scope:

  • Private: Only the owner can access the model.

  • App-level Sharing: Available to all users within a specific app.

  • Global Sharing: Available across all apps (use with caution).

Best Practice:

For collaborative environments, set read access for analysts and write access only for designated admins or model maintainers.

2. Relationship Between Data Models and Pivot

Data models serve as the foundation for the Pivot interface, allowing users to perform visual, drag-and-drop analysis without writing any SPL (Search Processing Language).

Use Case:

• Users without SPL knowledge (e.g., business analysts, compliance officers) can:

  • Select a data model and dataset,

  • Choose fields to group or filter by,

  • Generate charts, tables, and statistics directly from the UI.

Benefits:

• Encourages self-service analytics.

• Drives consistent use of normalized fields and tagging (especially with CIM).

Example Workflow:

1. Create or select a CIM-compliant data model (e.g., Authentication).

2. Launch Pivot → Choose “Authentication” model.

3. Visually build a report (e.g., “Count of login attempts by user”).

3. Advanced Topic: TSIDX Acceleration Mechanism

When acceleration is enabled on a data model, Splunk creates a set of pre-summarized data files to improve query performance — this process is known as TSIDX (Time Series Index) Acceleration.

How It Works – Under the Hood:

• Splunk generates tsidx files (time-series index files) containing:

  • Precomputed statistics (e.g., counts, sums, averages).

  • Aggregated results over time and specific fields.

• These summaries are stored in:

  $SPLUNK_HOME/var/lib/splunk/summary
  

Query Optimization:

• When a user runs a search against an accelerated data model (via tstats or Pivot), Splunk queries these summarized tsidx files instead of raw data.

• This results in substantially faster query times, especially for large datasets.

Maintenance Considerations:

• Acceleration requires disk space and CPU to build and update the summaries.

• You must configure:

  • Summary range (e.g., last 7 days),

  • Backfill time, if needed.

CLI/Config Reference (Advanced):

• datamodels.conf allows configuration of acceleration parameters:

  acceleration = true
  acceleration.earliest_time = -7d
  acceleration.backfill_time = -30d
  

Summary of Key Enhancements

Topic	Details
Data Model Permissions	Can be configured via Settings > Data Models with role-based read/write access and app-level sharing.
Pivot Integration	Enables users to analyze data without writing SPL by leveraging data models in a visual interface.
TSIDX Acceleration	Behind-the-scenes mechanism that builds and queries precomputed summaries, stored in .tsidx files, to boost performance.