Lesson 7 of 7

Policy Verification and Troubleshooting

Objective

In this lesson you will verify SD‑WAN policy distribution and troubleshoot policy evaluation on edge devices. We focus on using show/debug commands to confirm a policy was pushed from the controller plane, that the control-plane connections are healthy, and that the policy is installed and actively influencing traffic. In production, fast verification and accurate debugging reduce Mean Time To Repair (MTTR) when a path selection or security policy behaves unexpectedly — for example, when SaaS traffic should prefer an MPLS path but is still flowing across Internet links.

Quick Recap

This lesson continues the topology used throughout the lab series. We do not add new devices; we will verify policies on the existing controllers and two edges.

ASCII Topology (exact IPs on every interface)

vManage, vSmart and vBond sit in the management/control cloud. Two edge routers (HQ and Branch) represent production appliances.

                        +-----------------+
                        |   vManage       |
                        | eth0: 10.0.0.10 |
                        +--------+--------+
                                 |
                                 | 10.0.0.0/24
                                 |
       +-----------------+-------+--------+-----------------+
       |                 |                |                 |
+------+------+   +------+-------+  +-----+-----+   +-------+------+
|   vSmart     |   |   vBond     |  |  ISP/Net  |   |  ISP/Net     |
| eth0:10.0.0.11|  | eth0:10.0.0.12| | 192.0.2.2 |   | 203.0.113.2  |
+------+-------+  +------+--------+ +-----+-----+   +-------+------+
       |                 |                  |                 |
       |                 |                  |                 |
+------+-------+  +------+--------+  +------+-------+  +------+-------+
| Edge-HQ      |  | Edge-Branch  |  | Internet     |  | Internet     |
| Catalyst8000 |  | ISR1000       |  | 192.0.2.2    |  | 203.0.113.2  |
| Gi0/0:192.0.2.1/30| Gi0/0:203.0.113.1/30 |            |
+-----------------+ +-----------------+  +--------------+  +--------------+

Device table and primary addresses

DeviceRolePrimary InterfaceIP address
vManageManagement Planeeth010.0.0.10/24
vSmartControl/Policy Planeeth010.0.0.11/24
vBondOrchestrationeth010.0.0.12/24
Edge-HQData-plane EdgeGigabitEthernet0/0192.0.2.1/30
Edge-BranchData-plane EdgeGigabitEthernet0/0203.0.113.1/30

Tip: All management-plane domain names in examples use lab.nhprep.com and passwords use Lab@123. Organization name is NHPREP.

Key Concepts

Before touching the CLI, understand these concepts that drive verification and troubleshooting:

  • Policy distribution and controllers: The management plane (vManage) composes intent and pushes policies to the control plane (vSmart). The control plane then distributes policy to edges. When you verify, you check both the controller and the edge to ensure policy reached its destination.

    • In production: a mis-pushed policy often looks like a local configuration mismatch; verifying controller distribution narrows the problem quickly.
  • Control-plane health influences policy application: Edges keep a secure control connection to controllers. If the control session is down, edges will not receive incremental policy updates and may operate on stale rules.

    • Protocol behavior: edges periodically re-establish control sessions; during an outage, a stale policy may persist until the session is reestablished.
  • Policy evaluation order matters: SD‑WAN policy evaluation follows match → sequence → action ordering. A more specific match or higher-priority policy can override a generic policy; verification needs to confirm which exact rule was installed on the data-plane device.

    • Think of policies like a stack of filters — the first matching filter decides the outcome.
  • Telemetry and AIOps integrations (NWPI, TE, UMTS, ThousandEyes): Visibility tools correlate underlay (TE/UMTS) and overlay (NWPI) metrics to show where the problem occurs and why. Use these to correlate policy events with path anomalies in production.

  • Application-aware routing (AAR) vs. topology-based: Some policies are application-driven (App Route); others select TLOCs by topology. When troubleshooting, confirm whether the policy expects app-detection or simple prefix matching.

Step-by-step configuration

Step 1: Confirm policy exists on the management plane (vManage)

What we are doing: Check that the intended policy was created in the management plane and exported to the control plane. This ensures the source-of-truth contains the policy before we hunt on devices.

# On the vManage CLI or API shell (management node)
show policy list

What just happened: The command enumerates the policy objects that exist in vManage with their names and last-push timestamps. If the policy is absent here, no edge will ever receive it. This is the authoritative check.

Real-world note: In production, vManage is the single pane of intent — always confirm here first before troubleshooting edge behavior.

Verify:

show policy list
Policy Name                       Type          Last Push Time
App-Prefer-MPLS                   AppRoute      2026-03-30 09:12:45
Prefix-Block-Internet             RoutePolicy   2026-03-30 09:13:02
TLOC-Preference-HQ                TLOC          2026-03-30 09:13:20

Step 2: Verify control-plane connectivity from an edge (Edge-HQ)

What we are doing: Ensure Edge-HQ has healthy control connections to the controllers (vSmart/vManage/vBond). Without these, policy updates cannot be delivered.

configure terminal
logging buffered 100000 debugging
end
write memory

# On Edge-HQ CLI
show control connections

What just happened: The configuration enabled buffered logging at debugging level so we can capture policy-evaluation messages during troubleshooting. The show control connections command lists established secure control sessions to controllers including remote IPs, state and time since last message — a direct indicator of control-plane health.

Real-world note: Control reboots or network changes frequently cause edges to lose control connections; logging and control checks are the first triage steps.

Verify:

show control connections
Node-ID: edge-hq.lab.nhprep.com
Local-Controller: 10.0.0.11
Remote-Controller: 10.0.0.11 port 12346 State: UP
Last Received: 00:00:12
Remote-Controller: 10.0.0.10 port 8443 State: UP
Last Received: 00:00:10
Remote-Controller: 10.0.0.12 port 12346 State: UP
Last Received: 00:00:15

Active Tunnels: 2
TLOCs: 2

Step 3: Verify the policy is installed on the edge and view the applied policy entries

What we are doing: Confirm that the edge has the policy pushed and that the specific match/action entries are present. This tells you whether the intended rule is actually enforced by the data plane.

# On the Edge-HQ CLI
show running-config | section policy
show sdwan policy list

What just happened: The first command searches the running configuration for the policy section so you can see the exact policy stanza the device has. The second lists high-level SD‑WAN policies installed on the device. Together they reveal presence and specifics of the rule.

Real-world note: Policy names can be identical but have different sequence numbers; always inspect the full stanza on the edge to understand precedence.

Verify:

show running-config | section policy
policy
 name App-Prefer-MPLS
  sequence 10
   match application-name HTTP-HTTPS
   action prefer tloc-list MPLS_TLOC
  sequence 20
   match prefix 0.0.0.0/0
   action prefer tloc-list INTERNET_TLOC
!
show sdwan policy list
Device: edge-hq.lab.nhprep.com
Policy: App-Prefer-MPLS    Status: Installed    Last Update: 2026-03-30 09:12:46
Policy: Prefix-Block-Internet Status: Installed   Last Update: 2026-03-30 09:13:03
Policy: TLOC-Preference-HQ  Status: Installed    Last Update: 2026-03-30 09:13:21

Step 4: Confirm policy effect on traffic (App-route / policy hit counters)

What we are doing: Check app-route and policy hit counters to confirm live traffic is matching the intended policy and taking the expected TLOC (transport). This verifies not just installation but effectiveness.

# On the Edge-HQ CLI
show app-route statistics
show sdwan policy counters

What just happened: show app-route statistics summarizes application detection and routing decisions for application-aware routing. show sdwan policy counters displays per-policy/per-sequence match counts, which prove whether traffic hits the rule or bypasses it.

Real-world note: In production, these counters help prove that an SLA or routing change actually impacts user traffic; counters persist across restarts only if configured to do so.

Verify:

show app-route statistics
Application: HTTP-HTTPS
Matched Flows: 237
Preferred TLOC: MPLS_TLOC (192.0.2.5)
Last Match Time: 2026-03-30 10:02:14

Application: FACEBOOK
Matched Flows: 0
Preferred TLOC: INTERNET_TLOC (203.0.113.5)
Last Match Time: N/A
show sdwan policy counters
Policy: App-Prefer-MPLS
 Sequence 10 Match Count: 237 Action: prefer MPLS_TLOC
 Sequence 20 Match Count: 12  Action: prefer INTERNET_TLOC

Policy: Prefix-Block-Internet
 Sequence 10 Match Count: 0 Action: drop

Step 5: Collect logs for policy evaluation debugging

What we are doing: If hits are not occurring as expected, collect buffered logs and search for policy-evaluation messages to determine why a flow did or did not match.

# On the Edge-HQ CLI
show logging
show logging | include policy

What just happened: show logging outputs the buffered logs we enabled earlier. Filtering for "policy" surfaces entries where the device evaluated policy matches and actions. These entries reveal mismatches (e.g., application not detected, incorrect prefix, or sequence precedence).

Real-world note: Buffered logs are ephemeral; for sustained troubleshooting integrate with a log sink (syslog/Splunk) and correlate with NWPI/UMTS/TE telemetry for root cause analysis.

Verify:

show logging
*Mar 30 10:00:12: PolicyEngine: Evaluating flow src=10.1.1.10 dst=8.8.8.8 proto=6 dport=80
*Mar 30 10:00:12: PolicyEngine: Matched policy App-Prefer-MPLS sequence 10 (action prefer MPLS_TLOC)
*Mar 30 10:00:12: TLOC selection: MPLS_TLOC (192.0.2.5) selected for flow
*Mar 30 10:00:12: PolicyEngine: Matched policy Prefix-Block-Internet sequence 20 (action prefer INTERNET_TLOC)

Verification Checklist

  • Check 1: Policy exists in vManage — verify with show policy list on vManage and confirm the policy name and last push time.
  • Check 2: Control-plane is UP — verify with show control connections on the edge and check controllers are listed as UP and last-received timers are recent.
  • Check 3: Policy installed on edge — verify with show running-config | section policy and show sdwan policy list on the edge.
  • Check 4: Traffic hits the policy — verify with show app-route statistics and show sdwan policy counters for positive match counters.
  • Check 5: Logs show the evaluation — verify with show logging | include policy and correlate messages with the times of test traffic.

Common Mistakes

SymptomCauseFix
Policy listed in vManage but not on edgeControl-plane (vSmart/vManage) session from edge is down or intermittentCheck show control connections on edge; fix connectivity to controllers, then re-push policy from vManage
App-route counters remain zero despite trafficApplication not being detected (app-detection disabled or DPI not matching)Confirm application detection is enabled and test with traffic that clearly matches signature; inspect show app-route statistics
Policy installed but traffic still uses wrong TLOCA higher-priority rule or explicit route override existsInspect full policy stanza and sequence numbers on the edge to locate a higher-priority match; adjust sequence or rule specificity
Logging shows policy matched but flows not forwardedUnderlay path (TLOC) or next-hop issue on selected transportVerify underlay reachability to TLOC IP and check TE/NWPI telemetry for underlay anomalies; use traceroutes to the TLOC
Policy counters reset after rebootCounters stored in memory onlyFor persistent counters, configure external telemetry/logging and use persistent monitoring (Splunk, NWPI integrations)

Key Takeaways

  • Verify in order: management (vManage) → control-plane (vSmart/vBond sessions) → data-plane (edge configuration and counters). This reduces wasted effort.
  • Policy presence is not enough — confirm the edge installed the same policy stanza and that policy counters or app-route statistics show active matches.
  • Control-plane health is a gating factor for policy distribution; always check show control connections early when a policy appears missing.
  • Use logging and telemetry integrations (NWPI, TE, UMTS, ThousandEyes, Splunk) to correlate policy evaluation with underlay anomalies — in production these integrations dramatically shorten root-cause analysis.

Warning: Buffered debug logging can be verbose and impact performance if left enabled at scale. Use targeted logging for short troubleshooting windows and forward logs to an external collector for long-term analysis.

This completes Lesson 7: Policy Verification and Troubleshooting. Use these verification steps and the checklist to triage and prove policy behavior in your SD‑WAN deployments.