SPLK-1001 Creating Scheduled Reports and Alerts

Creating Scheduled Reports and Alerts Detailed Explanation

8.1 Scheduled Reports

1. Purpose of Scheduled Reports

• Why Schedule Reports?
  1. Automate Report Generation:
     • Save time by having Splunk run recurring reports instead of running them manually.
  2. Ensure Regular Monitoring:
     • Keep track of critical metrics without missing important updates.
     • Example: Monitor weekly sales trends or daily error counts.
  3. Deliver Insights to Stakeholders:
     • Send results automatically to relevant teams or individuals via email.

2. Steps to Schedule a Report

Step 1: Create or Open an Existing Report

• You can schedule any saved report in Splunk.

• To create a report:

  1. Write a search query and execute it.

     • Example:

       index=main error | stats count by host
       

  2. Click Save As > Report.

  3. Provide a name and description for the report.

Step 2: Edit the Schedule

1. Open the saved report from the Reports menu.
2. Click the Edit Schedule button.
3. Define the following:
   • Frequency:
     • Options include:
       • Daily.
       • Weekly.
       • Monthly.
     • Choose a time of day for the report to run.
   • Time Range:
     • Specify what data the report should cover.
     • Example:
       • Last 24 hours for daily reports.
       • Last 7 days for weekly reports.

Step 3: Specify Delivery Options

1. Email Results:
   • Send the report to specific recipients.
   • Configure:
     • Subject: Include a meaningful title.
     • Attachments: Add the report as a CSV, PDF, or inline table.
   • Example:
     • Send a daily error summary to the IT team.
2. Save Results:
   • Automatically save the report to a specific location in CSV format for later use.

3. Use Cases for Scheduled Reports

1. Weekly Error Reports

• Example Query:

  index=errors | stats count by error_type
  

• Purpose:

  • Summarize error types for the week.
  • Identify recurring issues.

2. Daily Performance Summaries

• Example Query:

  index=performance | stats avg(response_time) by server
  

• Purpose:

  • Track the average response time of servers daily.
  • Detect anomalies early.

8.2 Alerts

1. Definition of Alerts

• Alerts notify you when specific conditions in your data are met.
• Why Use Alerts?
  • Proactively respond to issues before they escalate.
  • Automate monitoring without requiring constant manual checks.

2. Types of Alerts

1. Real-Time Alerts

• Triggered Immediately:
  • Splunk evaluates data continuously.
  • The alert triggers as soon as the condition is satisfied.
• Example:
  • Alert when a security breach is detected.

2. Scheduled Alerts

• Evaluated at Intervals:
  • The alert checks for conditions periodically (e.g., every 5 minutes, hourly).
• Example:
  • Alert when error counts exceed a threshold in the last hour.

3. Steps to Create an Alert

Step 1: Perform a Search

• Write a search query to define the condition to monitor.

• Example Query:

  index=main | stats count by error_type | where count > 10
  

  • Finds error types that occur more than 10 times.

Step 2: Save the Query as an Alert

1. Click Save As > Alert.
2. Configure Alert Settings:
   • Trigger Conditions:
     • Example:
       • Trigger when count > 10.
       • Trigger when the same condition is met multiple times.
   • Trigger Frequency:
     • Per result: Triggers for each matching event.
     • Once: Triggers only once until reset.

Step 3: Configure Trigger Actions

1. Send an Email:
   • Notify relevant stakeholders with:
     • Alert details.
     • Search results.
   • Example Email:
     • Subject: "High Error Rate Alert".
     • Body: "The error rate has exceeded the threshold."
2. Run a Script:
   • Execute custom scripts for advanced actions (e.g., restart a server).
3. Create a Ticket:
   • Integrate with ticketing systems (e.g., Jira, ServiceNow) for automated issue tracking.

4. Alert Examples

1. Alert for High CPU Usage

• Example Query:

  index=system_metrics | stats avg(cpu_usage) as avg_cpu by host | where avg_cpu > 90
  

• Condition:

  • CPU usage exceeds 90% for any host.

2. Alert for Failed Logins

• Example Query:

  index=auth_logs status=failed | stats count by user | where count > 5
  

• Condition:

  • More than 5 failed login attempts by any user.

5. Managing Alerts

Monitor Active Alerts

• Navigate to the Alerts menu to view:
  • Active alerts.
  • Triggered alerts and their timestamps.

Edit or Disable Alerts

• Open an alert to modify its conditions or delivery options.
• Disable alerts temporarily when they are not needed.

Key Differences Between Scheduled Reports and Alerts

Feature	Scheduled Reports	Alerts
Purpose	Deliver data summaries regularly.	Notify about specific conditions.
Frequency	Fixed intervals (e.g., daily, weekly).	Real-time or periodic checks.
Trigger Condition	Based on time ranges (e.g., Last 24 hours).	Based on specific data thresholds.
Actions	Email, save results.	Email, scripts, ticketing integration.

Creating Scheduled Reports and Alerts (Additional Content)

1. Throttle (Alert Suppression)

What is Throttling?

Throttling—also called alert suppression—is used to limit how often an alert can fire, even if its triggering condition remains true.

How It Works:

• When an alert condition is met, the alert is triggered.

• After firing, throttling introduces a suppression window.

• During this window, even if the alert condition is met again, the alert will not re-fire.

Example:

• Throttle duration: 60 minutes

• Condition: avg(cpu_usage) > 90

• Outcome: Once the alert fires, it will be suppressed for 60 minutes, regardless of ongoing high CPU usage.

Use Case:

• To prevent alert fatigue or spam, especially in high-frequency environments (e.g., CPU alerts every minute).

2. Alert Severity Levels

Splunk allows you to assign a severity level to each alert to classify its importance and urgency.

Severity	Description	Example Use Case
Info	Informational only; no action typically required	Notification when a report finishes running
Warning	Indicates a potential issue; should be monitored	More than 5 failed login attempts
Critical	Requires immediate attention; serious impact	Unauthorized access detected

Why It Matters:

• Severity levels help prioritize incidents in dashboards, reports, and ticketing systems.

• They may be used by SIEM tools or alert handlers to escalate or route alerts.

3. Permissions for Scheduled Reports and Alerts

When creating or saving a report or alert, permissions must be properly set to ensure that the object:

• Can be accessed by the necessary users.

• Can be executed by the system.

• Can send email or trigger actions.

Visibility Levels:

Visibility Option	Description
Private	Only visible to the user who created it
App-level	Shared within the same Splunk app (e.g., Search & Reporting)
Global	Visible to all users (depending on roles/permissions)

Important Exam Note:

• If a scheduled alert or report is not shared or permissions are too restrictive:

  • The alert may fail to execute.

  • Email delivery may not occur.

  • Other users may not be able to view or edit it.

4. Suppression Window Duration

In addition to standard throttling, Splunk allows you to configure how long to suppress similar alert instances. This adds further control over alert frequency.

Why This Matters:

• Prevents "alert storms" in dynamic environments where the same event occurs repeatedly in a short time.

Example:

• Search condition: avg(cpu_usage) > 90

• Triggered at 10:00 AM

• Suppression: 10 minutes

• Even if CPU stays above 90%, the alert won’t fire again until 10:10 AM.

Exam Tip:

• Look out for scenarios where throttling vs suppression duration is being tested in alert configuration options.

Summary Table for SPLK-1001 Review

Feature	Key Details
Throttle (Suppression)	Limits how often alerts re-fire after initial trigger
Severity Levels	Classifies alerts as Info, Warning, or Critical
Permissions	Impact visibility, email delivery, and execution of reports/alerts
Suppression Duration	Time window during which similar alerts are not re-triggered

Realistic Exam Scenario You Might See

You created a critical alert for failed logins over 5 attempts per user. It keeps triggering every minute and spamming your inbox. What should you do?

Correct Answer: Add throttling or a suppression window to the alert.