SPLK-2003 Configuring External Splunk Search

Configuring External Splunk Search Detailed Explanation

1. Purpose

Splunk SOAR is an automation platform, but it becomes far more powerful when connected to a Splunk instance. This integration allows SOAR to:

• Run real-time searches in Splunk.

• Pull back logs, alerts, or historical data.

• Use those results to enrich, validate, or correlate security events in playbooks.

In short: SOAR asks questions, and Splunk gives the answers.

2. Configuration Steps

To connect SOAR with a Splunk instance, follow these basic setup steps:

Step 1: Install the Splunk App on SOAR

• Go to Apps > App Store in the SOAR interface.

• Search for the Splunk app (developed by Splunk).

• Click Install.

This app includes pre-built actions for querying Splunk using SPL (Search Processing Language).

Step 2: Create an Asset for Splunk

Once the app is installed, configure an Asset to connect to your specific Splunk instance.

Asset Settings:

• Splunk Server URL: The URL of your Splunk search head (e.g., https://splunk.company.com:8089)

• Authentication:

  • Username and password for a Splunk account with search privileges

• Query Timeout:

  • How long SOAR should wait for a search to complete (e.g., 60 seconds)

• Index Scope:

  • Define which indexes the user can search (e.g., main, wineventlog, network_traffic)

You may need to generate an API token from Splunk, depending on how authentication is handled.

Step 3: Test the Connection

• After saving the asset, click Test Connectivity.

• If everything is configured correctly, you should get a success message.

If it fails:

• Check the URL and port.

• Confirm the Splunk user has REST API and search permissions.

• Review firewall rules and network connectivity.

3. Search Usage in Playbooks

Once your Splunk asset is set up, you can start using it inside your playbooks.

Search Block

The Search Block allows you to run SPL queries directly from SOAR.

Example Query:

search index=network_traffic src_ip=10.0.0.1

You can insert variables into the query using data from artifacts or playbook inputs.

This allows your automation to dynamically search for related logs based on the incoming event.

Use Cases

Splunk search integration can be used in many security automation scenarios:

1. Enrichment

• Add context to a suspicious indicator.

• Example: Pull logs showing how many times an IP communicated with internal systems.

2. Validation

• Confirm whether an alert is a false positive or part of a larger pattern.

• Example: Check if a user has logged in from multiple geographies in a short time.

3. Threat Hunting / Correlation

• Use Splunk to find lateral movement, command-and-control activity, or data exfiltration.

• Example: Find all outbound connections from a compromised host in the past 24 hours.

These use cases help turn raw alerts into actionable incidents.

4. Considerations

a. Search Permissions

Make sure the Splunk account used in the asset has:

• Permission to search the correct indexes.

• Access to required SPL commands.

• Ability to use the REST API.

Without proper permissions, searches may fail silently or return incomplete results.

b. Large Result Sets

• Be careful with queries that return huge volumes of data.

• Large datasets can:

  • Slow down playbooks

  • Cause timeouts or memory issues

  • Lead to action failures

Best Practices:

• Use limits in your SPL (e.g., | head 100)

• Filter by host, index, source type, and time range

• Use summary indexing or saved searches for very heavy queries

Summary

Feature	Purpose and Function
Splunk App Installation	Adds the ability to run SPL searches from within SOAR
Asset Configuration	Connects SOAR to your Splunk instance with credentials and settings
Search Block	Executes SPL queries dynamically in playbooks
Enrichment Use Case	Pull related logs to add context to an alert
Validation Use Case	Confirm user behavior or system activity
Correlation Use Case	Identify broader attack patterns like lateral movement
Permissions Requirement	Ensure Splunk user can access indexes and API
Result Set Management	Avoid performance issues with large data pulls

Configuring External Splunk Search (Additional Content)

1. Saved Search Invocation via Splunk App

Purpose:
Instead of writing full SPL queries directly inside the playbook, analysts can use saved searches created and maintained within the Splunk platform. This improves consistency, simplifies complex queries, and enhances maintainability.

How to Use in SOAR:

• The Splunk App for SOAR supports an action called:

  • run saved search (or similar, depending on the app version)

• Required Input Parameters:

  • Saved Search Name: The exact name of the saved search in Splunk.

  • Search Parameters (if any): Some saved searches are configured to accept tokens or arguments, which must be passed in from SOAR.

Example Workflow:

• In Splunk:

  • Saved search named get_recent_logins with SPL:

    search index=auth sourcetype=linux_secure user=$user$
    

• In SOAR:

  • Use run saved search action with:

    • Saved search name: get_recent_logins

    • Parameter user: extracted from an artifact or playbook variable

Advantages:

• Reduces complexity in playbooks.

• Allows central management of search logic.

• Enables version control of core detection logic on Splunk side.

2. Result Handling Techniques for Search Output

Purpose:
After running a search (saved or raw SPL), SOAR playbooks must parse and utilize the returned data. Understanding how to access the correct fields is critical for building reliable logic and formatted outputs.

Common Access Pattern:

• The response from a Splunk search action is typically found in:

  results[0].get("data", [])[0].get("field_name")
  

• results[0]: The first result block from the action.

• .get("data", []): Retrieves the list of events (rows).

• [0]: Selects the first event row.

• .get("field_name"): Gets the desired field value.

Example Use in Format Block:

phantom.format(container=container, template="Last login was from {0}", parameters=[("action_result.data.*.src_ip",)])

Best Practice:

• Use phantom.debug() during testing to log and inspect the actual keys returned from the search.

• Always use .get() with a default to avoid KeyErrors if the field is missing.

3. Permissions Error vs. Network Error – Troubleshooting Scenario

Purpose:
When an external Splunk search fails, it's important to distinguish access issues from connectivity problems. Understanding the typical error signatures helps in rapid diagnosis and resolution.

Case Study Example:

• Scenario 1 – Permission Denied:

  • The Splunk asset is reachable, but the user account does not have permission to search the requested index.

  • Typical error in action result:

    {
      "status": "fail",
      "message": "You do not have permission to access index=wineventlog"
    }
    

  • Resolution:

    • Check Splunk user role.

    • Confirm index permissions in Splunk roles.

• Scenario 2 – Network or Asset Misconfiguration:

  • The Splunk server is not reachable (firewall, DNS, or misconfigured URL/port).

  • Typical error message:

    {
      "status": "fail",
      "message": "Connection refused" or "Unable to connect to Splunk server"
    }
    

  • Resolution:

    • Validate asset configuration (URL, port, auth).

    • Ping the server or use curl to verify access.

    • Check if Splunk REST API port (usually 8089) is open.

Best Practice Tip:

Always run “Test Connectivity” in the asset configuration panel after any changes, and check the phantomd.log or action_result.summary for error root cause clues.

Summary

Topic	Expanded Insight
Saved Search Use	Reuse named searches from Splunk to simplify playbooks
Result Parsing	Use .get("data", [])[0].get("field") to extract and use search results safely
Error Diagnosis	Distinguish permission denial from connectivity issues via error message content