NSE7_SDW-7.2 SD-WAN troubleshooting

SD-WAN Troubleshooting Detailed Explanation

Troubleshooting is a critical skill for maintaining the stability and reliability of an SD-WAN deployment. It involves diagnosing and resolving issues that affect network performance, traffic flow, or device functionality.

5.1 Common Issues

1. Rule Misconfiguration

• What It Is:
  SD-WAN rules define how traffic is routed. Misconfigured rules can lead to unexpected traffic behavior, such as:
  • Traffic being routed to incorrect paths.
  • Certain applications not functioning as intended.
• Troubleshooting Steps:
  1. Verify the rule’s matching conditions (source/destination IP, application type, services).
  2. Check the rule priority to ensure higher-priority rules are not overriding lower-priority ones.
  3. Test the rule with specific traffic to confirm proper matching.

2. SLA Violations

• What It Is:
  SLA (Service Level Agreement) violations occur when a link fails to meet performance thresholds (e.g., latency, jitter, or packet loss).
• Troubleshooting Steps:
  1. Monitor the SLA metrics for each link.
  2. Identify problematic links with consistent SLA breaches.
  3. Adjust path selection rules or reroute traffic to better-performing links.

3. Routing Anomalies

• What It Is:
  Dynamic routing protocols (e.g., OSPF, BGP) may fail due to misconfigurations, causing routing loops or unreachable destinations.
• Troubleshooting Steps:
  1. Check the status of routing neighbors (e.g., OSPF adjacency or BGP peers).
  2. Verify the routing table for missing or incorrect routes.
  3. Look for mismatched network advertisements between devices.

5.2 Diagnostic Tools

CLI Commands

1. diag sys sdwan:

   • Purpose: View the status of SD-WAN members and rules.

   • Usage Example:

     diag sys sdwan
     

     Output shows the status of each WAN link, SLA metrics, and path usage.

2. diag debug application sdwan:

   • Purpose: Analyze rule matching and traffic distribution.

   • Usage Example:

     diag debug application sdwan
     diag debug enable
     

     Output provides real-time insights into how rules are applied to traffic.

Logs and Reports

• Use tools like FortiAnalyzer to review historical logs and performance data:
  • Traffic Logs: Analyze traffic flow and identify bottlenecks.
  • SLA Reports: View link performance trends over time.

5.3 Troubleshooting Workflow

Step 1: Scope the Issue

• Goal: Determine if the issue is related to SD-WAN rules, link performance, or device hardware.
• Example:
  • If traffic is not reaching its destination, check rules first.
  • If SLA metrics are failing, focus on link performance.

Step 2: Apply Temporary Fixes

• Goal: Quickly mitigate the impact of the issue while investigating the root cause.
• Actions:
  • Adjust rule conditions or priorities to reroute traffic.
  • Manually switch traffic to an alternate WAN link if one is degraded.

Step 3: Analyze Root Cause

• Goal: Identify the underlying problem to implement a permanent solution.
• Examples:
  • Misconfigured rules: Correct the matching conditions.
  • Faulty link: Contact the ISP or replace faulty hardware.

Practical Application

Scenario 1: Simulate Link Failures

1. Temporarily disconnect one WAN link to simulate a failure.
2. Observe how SD-WAN reroutes traffic to the backup link.
3. Verify that critical applications remain operational during the failover.

Scenario 2: Analyze Historical Data

1. Use FortiAnalyzer to review logs for the past week.
2. Identify patterns of SLA violations on specific links.
3. Adjust SD-WAN policies to prioritize better-performing links.

Key Takeaways

1. Common Issues:
   Troubleshooting often involves correcting rule misconfigurations, resolving SLA violations, or addressing routing anomalies.
2. Diagnostic Tools:
   Use CLI commands for real-time analysis and tools like FortiAnalyzer for historical insights.
3. Workflow:
   Follow a systematic approach: scope the issue, apply temporary fixes, and identify the root cause.
4. Practical Skills:
   Simulate scenarios like link failures to practice troubleshooting and ensure the network behaves as expected under stress.

SD-WAN Troubleshooting (Additional Content)

Effective SD-WAN troubleshooting requires a structured approach, covering physical connectivity, WAN link performance, SD-WAN rule validation, VPN-related issues, and historical analytics.

1. Physical and WAN Link-Level Troubleshooting

Before diagnosing SD-WAN policy or routing issues, ensure the underlying WAN connection is stable.

1.1 Checking Interface Status

Use the following command to verify WAN interface connectivity:

get system interface

• What to check?
  • The status of WAN interfaces (up/down).
  • The assigned IP address (ensure it is obtained from the ISP).
  • The link speed and duplex mode (mismatches can cause performance degradation).

1.2 Detecting Link Jitter and Packet Loss

Run the following commands to check WAN stability:

• Ping test to check packet loss:

  execute ping -c 50 <ISP_Gateway_IP>
  

• Traceroute test to diagnose high latency:

  execute traceroute <destination_IP>
  

• Key indicators of WAN issues:

  • High packet loss (>5%) indicates ISP issues or faulty WAN links.
  • High latency spikes suggest congestion or route inefficiencies.
  • Multiple unexpected hops in traceroute may indicate routing misconfiguration.

1.3 Verifying MTU/MSS Settings

Incorrect MTU (Maximum Transmission Unit) or MSS (Maximum Segment Size) settings can cause:

• Packet fragmentation, leading to higher retransmissions.
• Performance degradation in VPN tunnels.

To check MTU values, use:

diagnose sys mtu

• If MTU is mismatched, adjust it:

  config system interface
      edit "wan1"
      set mtu 1500
  end
  

• For VPN tunnels, consider setting MSS clamping to avoid fragmentation:

  config vpn ipsec phase1-interface
      edit "vpn1"
      set mtu-override enable
      set mtu 1350
  end
  

2. FortiGate SD-WAN Diagnostic CLI Commands

2.1 Checking Active SD-WAN Routes

To verify how traffic is being routed through SD-WAN, use:

get router info routing-table all

• Ensures traffic is following the intended SD-WAN path.
• Detects missing or incorrect routes that can cause packet drops.

2.2 Checking SD-WAN Health Status

To validate WAN SLA performance metrics, run:

diagnose sys sdwan health-check

• Displays latency, jitter, and packet loss statistics for each WAN link.
• Ensures SD-WAN policies fail over correctly when SLA violations occur.

2.3 Verifying SD-WAN Rule Matching

To check if traffic matches the correct SD-WAN rule, use:

diagnose sys sdwan service

• Identifies which SD-WAN rule is applied to traffic.
• Helps detect overridden or misprioritized rules.

2.4 Debugging Traffic Flow

To trace why traffic is not following expected SD-WAN routes, use:

diagnose debug flow filter addr <destination_IP>
diagnose debug enable
diagnose debug console timestamp enable

• Helps determine if traffic is dropped, misrouted, or NAT-ed incorrectly.
• Useful for troubleshooting firewall rule conflicts.

3. Advanced SD-WAN Troubleshooting

3.1 SD-WAN Rule Not Applied

• Issue: Traffic is not following the expected SD-WAN rule.

• Possible Cause: A higher-priority rule is overriding the intended policy.

• Solution: Run:

  diagnose sys sdwan service
  

  • Identify the rule priority order.
  • Adjust rule weight or priority if necessary.

3.2 SLA Failover Not Triggering

• Issue: SD-WAN is not switching WAN links despite SLA violations.

• Possible Cause: SLA thresholds are set too high, preventing failover.

• Solution: Adjust SLA parameters:

  config system sdwan
      config health-check
          edit "WAN_SLA_Check"
              set latency-threshold 100
              set packetloss-threshold 5
          next
      end
  end
  

• Reducing the latency threshold ensures failover happens sooner.

3.3 SD-WAN Load Balancing Not Working

• Issue: Traffic is only using one WAN link despite load balancing settings.

• Possible Cause: ECMP (Equal-Cost Multi-Path) is not configured.

• Solution: Verify ECMP settings:

  get router info routing-table all
  

  • If only one route exists, enable ECMP:

    config router setting
        set ecmp-max-paths 4
    end
    

4. SD-WAN VPN Troubleshooting

If SD-WAN is integrated with VPN tunnels, additional debugging is required.

4.1 IPsec Tunnel Not Establishing

• Issue: SD-WAN cannot establish an IPsec VPN tunnel.

• Possible Causes:

  • Phase 1/2 configuration mismatch.
  • IKE negotiation failure.

• Solution:

  • Check active VPN tunnels:

    diagnose vpn tunnel list
    

  • Debug IKE negotiation:

    diagnose debug application ike -1
    

4.2 ADVPN Dynamic Tunnels Not Establishing

• Issue: Branch offices cannot dynamically establish ADVPN tunnels.

• Possible Cause: Routing issues in OSPF/BGP.

• Solution: Verify dynamic routing:

  get router info bgp summary
  get router info ospf neighbor
  

  • Ensure OSPF/BGP is advertising correct routes.

  • Restart dynamic VPN discovery:

    diagnose vpn ike gateway
    

5. FortiAnalyzer for Advanced Troubleshooting

FortiAnalyzer helps troubleshoot historical SD-WAN performance issues.

5.1 Analyzing SLA Violations

To check which WAN links frequently fail SLA checks:

• Access FortiAnalyzer SD-WAN reports.
• Identify WAN links exceeding latency or packet loss thresholds.

5.2 Traffic Path Analysis

To trace specific sessions through SD-WAN, use:

• Log replay feature to view past traffic flows.
• Identify incorrect SD-WAN routing decisions.

5.3 ISP-Specific Issue Correlation

• If multiple branch offices report frequent disconnections, compare:
  • Multiple site logs in FortiAnalyzer.
  • Identify common ISP failures across locations.

Conclusion

Key Takeaways

1. Physical and WAN Link Troubleshooting

• Check interface status, run ping/traceroute tests, and verify MTU settings.

2. FortiGate SD-WAN Diagnostic Commands

• Use diagnose sys sdwan health-check to monitor SLA performance.
• Use diagnose sys sdwan service to verify rule matching.
• Use diagnose debug flow filter addr <destination_IP> to trace traffic.

3. Advanced SD-WAN Troubleshooting

• Fix SD-WAN rules by adjusting rule priority.
• Ensure SLA-based failover triggers correctly.
• Enable ECMP for proper load balancing.

4. SD-WAN VPN Troubleshooting

• Use diagnose vpn tunnel list to check active VPN sessions.
• Verify BGP/OSPF routing to resolve ADVPN tunnel failures.

5. FortiAnalyzer for Deep Analysis

• Review SLA reports to identify problematic WAN links.
• Use log replay to analyze historical SD-WAN routing behavior.
• Correlate logs to detect ISP-related issues.