SPLK-2003 System Maintenance

System Maintenance Detailed Explanation

Splunk SOAR is a powerful platform, but like any software system, it needs regular care and maintenance to run smoothly. This includes monitoring system health, backing up data, applying updates, and troubleshooting issues.

1. System Monitoring

Monitoring helps you ensure the SOAR system is healthy and performing well. It also helps identify problems early before they cause downtime.

Health Dashboard

• Found in the web interface under Administration > System Health.

• Shows real-time information about:

  • CPU Usage

  • Memory Consumption

  • Disk Space

  • Service Status (like playbook engine, web server, etc.)

  Green = Good, Yellow = Warning, Red = Critical

Use this to catch issues like: “Disk usage at 95%” or “Engine not responding.”

Log Files

Logs provide detailed records of what’s happening inside the platform.

• Location: /var/log/phantom/

• Common log files:

  • phantomd.log: Main service log.

  • playbook.log: Tracks playbook execution results.

  • api.log: Records all API requests and responses.

  • celery.log: Related to background task execution.

Logs help diagnose issues like:

Why did a playbook fail?

Why didn’t an API call work?

What happened right before a service crashed?

Service Checks

Splunk SOAR runs several internal services. One of the most important is:

• phantomd: The core service that handles web interface, automation logic, and more.

Use systemctl status phantomd (or sudo service phantomd status) to check if it's running.

Other services may include:

• splunkd (if integrated with Splunk)

• postgresql (for the database)

• nginx (for the web server)

Best Practice: Set up automated alerts if services go down.

2. Backup and Restore

Creating regular backups is essential to protect your configuration and data. If something goes wrong (like hardware failure or a corrupted update), you can restore your SOAR system without losing everything.

What to Back Up

• Playbooks: All your automation logic.

• Apps & Assets: Configuration and credentials for integrations.

• Cases: Active and closed investigations.

• System Settings: User roles, custom fields, dashboards.

How to Back Up

• Use the command-line interface (CLI) or automation scripts.

• Example command:

  /opt/phantom/bin/phantom_backup.sh --output /opt/backups/soar_backup.tar.gz
  

• Store backups off the server, ideally in a secure cloud or remote server.

How Often?

• Daily or Weekly backups for production environments.

• Automate this using cron jobs or external scripts.

3. Updates and Patching

Keeping your Splunk SOAR up to date is vital for:

• Fixing security vulnerabilities

• Improving performance

• Gaining access to new features and bug fixes

How to Update

1. Download the latest update package from Splunk.

2. Stop services.

3. Run the update utility or package manager.

4. Restart services and verify system health.

Always back up before running updates!

4. Troubleshooting Tools

When something goes wrong, Splunk SOAR provides tools to help diagnose and fix the issue.

Command Line Utilities

• phantom-syslog: Helps with analyzing system logs and events.

• splunkd: Used if SOAR is connected to Splunk Enterprise.

• phenv: A special shell environment for executing Splunk SOAR commands safely.

Example use:

phenv python3 playbook_test.py

Support Packages

If you need to contact Splunk Support:

• Use the “Generate Support Package” feature from the admin UI or CLI.

• This collects:

  • Logs

  • Configuration files

  • Health stats

The support team uses this to quickly diagnose problems.

Summary

Maintenance Area	Key Purpose and Tools
Health Dashboard	Monitor performance and service status
Log Files	Investigate issues and errors
Service Checks	Ensure core services like phantomd are running
Backups	Protect data and configurations with scheduled backups
Updates & Patches	Fix bugs, close vulnerabilities, and add features
Troubleshooting Tools	Use phantom-syslog, phenv, and support packages for diagnostics

System Maintenance (Additional Content)

1. Service Dependency Hierarchy

Splunk SOAR relies on multiple services running in the background. Among them, phantomd is the main SOAR service that powers automation logic and the web interface. However, phantomd depends on other services to function properly.

Key Services:

• phantomd – Main daemon handling UI and automation logic

• postgresql – Manages SOAR's internal database (containers, artifacts, users)

• nginx – Handles HTTP/S traffic for the web interface

Dependency Chain:

If postgresql or nginx fails, even if phantomd is running, SOAR may become unresponsive or partially functional.

Exam Relevance:

A common exam question may ask:

• If the web UI is unavailable but phantomd is active, which service should you check next?

• Correct answer: nginx

2. Support Package Generation and Storage

When contacting Splunk Support, administrators are often asked to generate a Support Bundle, which contains logs, configuration files, health reports, and other diagnostic information.

Generation Methods:

• Via Web UI:
  Navigate to Administration > System Health > Generate Support Package

• Via Command Line:
  Execute:

  /opt/phantom/bin/generate_support_bundle.sh
  

Default Storage Location:

Unless specified otherwise, the support bundle is stored in the following directory:

/opt/phantom/support/

Exam Relevance:

Questions may ask:

• Where is the support package stored after generation via CLI?

• Correct answer: /opt/phantom/support/

3. Automated Health Monitoring and Alerting

While SOAR has a built-in System Health dashboard, it does not send alerts by default. Organizations must configure external tools to actively monitor SOAR’s status.

Monitoring Options:

• Polling the system health endpoint:

  /rest/system_health
  

• Parsing logs in /var/log/phantom/

• Setting alerts on metrics such as:

  • CPU load

  • Memory usage

  • Disk consumption

  • Service failures

Recommended Tools:

• Splunk (via Universal Forwarder or HEC)

• Prometheus and Grafana

• Nagios

• ELK Stack

Best Practice:

Use external tools to continuously poll SOAR’s health API or monitor logs for critical keywords such as “service failed”, “out of memory”, or “disk full”.

Exam Relevance:

A typical question might ask:

• Which method is most suitable to alert administrators if disk usage exceeds 90 percent?

• Correct answer: Configure an external monitoring tool to query the health API or monitor log output.

Summary

Area	Detail
Service Dependencies	phantomd depends on both postgresql and nginx
Support Bundle Location	Stored at /opt/phantom/support/ by default
Health Monitoring	Best done with external tools using API polling or log analysis