SPLK-3002 Data Audit and Base Searches

Data Audit and Base Searches Detailed Explanation

1. What Are Base Searches?

Simple Definition

A Base Search is a search query written in SPL (Splunk Processing Language) that tells ITSI:

“Here’s how to find the data we need to measure this KPI.”

It’s called a base search because it forms the foundation of:

• KPIs (Key Performance Indicators)

• Service Health Scores

• Dashboards

• Notable Events

Without well-designed base searches, your KPIs won’t work correctly—either they’ll be wrong, too slow, or incomplete.

Example of a Simple Base Search

Let’s say you want to track failed login attempts:

index=auth_logs sourcetype=login | stats count(eval(status="fail")) as failed_logins

This base search will:

• Look in the auth_logs index

• Filter for events where login failed

• Count how many failures happened

This result could then be used as a KPI called "Failed Login Count".

2. Structure of a Base Search

A good base search usually includes these parts:

a. SPL Query

The main search command.
This tells ITSI what data to collect and how to calculate the metric.

b. Time Range Settings

You can define:

• How far back to search (e.g., last 5 minutes, last 1 hour)

• Whether the search runs in real time or on a schedule

c. Split-by Fields (Optional)

This lets you separate the KPI results by dimension, like:

• By host

• By application

• By region

For example:

Show the number of login failures per server.

d. Filters and Thresholds

You can filter out unnecessary data (e.g., test environments).
Some base searches also include logic for setting KPI thresholds.

e. Search Scheduling

You control how often the search runs, using a cron schedule or real-time mode.
For example:

• Every 5 minutes

• Every hour

• Only during business hours

3. Data Audit Tools in ITSI

ITSI includes tools to help monitor the quality and performance of your data. These are helpful if your KPIs don’t look right or if searches are slow.

a. itsi_data_integrity

• This tool helps you check if your KPI data is:

  • Missing (not arriving)

  • Stale (too old)

  • Incomplete

Example:

You expect 100 search results every 5 minutes, but suddenly you get 0. This tool alerts you.

b. Search Inspector

• This is built into Splunk and lets you analyze:

  • How long a search took

  • Where the time was spent (e.g., fetching data vs. calculating stats)

  • Whether there were any errors or skipped results

You can access it from any KPI search window.

c. Audit Dashboards

• ITSI includes prebuilt dashboards that show:

  • Which searches succeeded or failed

  • How long they took

  • Whether they skipped intervals

  • System performance trends

These dashboards help you identify weak points in your setup.

4. Optimization Tips for Base Searches

To make your base searches fast and reliable, follow these best practices:

a. Limit Time Range and Unnecessary Fields

Don’t search more data than needed.
Only pull the fields you’re actually going to use.

Example:

If your KPI updates every 5 minutes, don’t search the last 24 hours of logs.

b. Use Summary Indexing When Possible

Instead of searching raw data over and over, you can:

• Run a scheduled search once

• Save the results into a summary index

• Build your KPIs using this lighter, faster data

It saves time and reduces system load.

c. Schedule Searches at Offset Times

If 100 KPI searches all run at the exact same time (e.g., 12:00), the system will slow down.

Instead:

• Stagger them by a few seconds or minutes

• Use offset cron schedules (e.g., one at :00, one at :02, etc.)

This balances the load across your servers.

Summary: What to Remember About Data Audit and Base Searches

• Base searches are the starting point for building KPIs in ITSI.

• A well-structured base search includes queries, filters, time settings, and more.

• ITSI has built-in tools to help you audit your data quality and search performance.

• Optimize base searches for efficiency, reliability, and accuracy.

Data Audit and Base Searches (Additional Content)

1. How to Validate Your Base Search

Before using a base search to build a KPI, you should verify it returns accurate and efficient results. Here are practical ways to do that:

Basic Debugging Techniques:

• | head 10:
  Quickly check if any results are returned.

• | stats count:
  Confirm that data is actually being matched by your SPL conditions.

• Review timestamps in the result:
  Ensure the search is retrieving events within your expected time range (e.g., last 5 minutes, last 24 hours).

• Avoid complex constructs:
  If possible, reduce the use of join, append, or nested subsearches, which often slow down or fail silently.

• Use the Search Inspector:
  This built-in tool helps you:

  • Analyze search phases

  • Detect "skipped" search errors

  • Understand resource usage or dispatch delays

Best Practice: Always test base searches in the SPL Editor first before assigning them to KPIs or services.

2. Common Mistakes in Base Searches

Even experienced users make common errors. Here are some real-world pitfalls to watch for:

• No time range specified:
  Default “All time” searches can overload indexers and return irrelevant data.

• Incorrect or misspelled fields:
  For example: hostnme=web01 instead of hostname=web01.

• Over-filtered conditions:
  A search like status=200 may exclude errors you should be monitoring.

• Real-time mode misuse:
  Real-time base searches can consume excessive system resources and may not reflect real-time data if indexing is delayed.

• No field extractions:
  If fields like error_code or latency are not extracted properly, the KPI calculation will fail or return null.

3. Base Search Visibility and Permissions

ITSI allows you to manage who can see and use base searches. This improves both data governance and team autonomy.

Visibility Scope:

• Private: Only visible to the creator

• App: Shared within the current app (e.g., ITSI only)

• Global: Available to all apps and users with appropriate roles

Controlling Access:

• Use role-based permissions to allow only specific ITSI teams to use/edit a base search.

• Assign ownership to a team lead or admin to enforce accountability.

Tip: Check the base search’s permissions in the ITSI Search Manager or via Splunk’s standard permissions.conf.

4. Using Base Searches in Templates

Base searches are foundational elements in reusable service templates. Here’s how they enhance efficiency:

a. Reusability:

• A well-designed base search (e.g., index=web sourcetype=access_logs | stats count as request_count) can be reused across:

  • Multiple KPIs

  • Several services

  • Different departments

This improves consistency and reduces duplication.

b. Parameterization with Tokens:

• In service templates, base searches can include tokens such as $host$, $region$, or $app_name$.

• These tokens get replaced at runtime, making the search flexible and adaptable to different services/entities.

Example:

index=infra sourcetype=system_logs host=$host$ | stats avg(cpu_load) as avg_cpu

This makes it easy to apply the same logic to multiple hosts or environments without rewriting the SPL.

Summary

• Validation ensures your base search logic is functional, efficient, and safe for production.

• Debugging and error awareness improve troubleshooting confidence and speed.

• Access control maintains data security and operational clarity.

• Template integration enables scalable monitoring across large environments.