ACI Troubleshooting Policy Based Redirect Deep Dive

Introduction

Imagine you have deployed a service graph with Policy Based Redirect in your ACI fabric, traffic should be flowing through your firewall for inspection, but packets are silently being dropped somewhere in the fabric. Where do you even begin? ACI PBR troubleshooting is one of the most challenging yet critical skills for any engineer working with data center fabrics. Policy Based Redirect (PBR) in ACI introduces a powerful mechanism for service insertion, but its multi-step packet walk across leaves and spines creates numerous points where things can go wrong.

In this deep dive, we will walk through every stage of PBR packet forwarding, from ingress classification on the source leaf all the way through spine COOP lookups and service device delivery. You will learn the exact CLI commands to verify each step, how to use ELAM captures to confirm redirect behavior, and what to look for when troubleshooting common PBR failures. Whether you are preparing for the CCIE Data Center lab or managing a production ACI fabric, this article gives you the systematic methodology you need to isolate and resolve PBR issues quickly.

This article covers:

Why PBR and service graphs simplify service insertion in ACI
The complete five-step PBR packet walk with CLI verification at each stage
ELAM trigger configurations for capturing redirected traffic
Zoning-rule analysis and redirect group inspection
Spine COOP MAC lookup verification
Service leaf endpoint programming validation
Return traffic flow and implicit permit behavior
Common pitfalls and pro tips for reliable PBR deployments

What Is Service Insertion Without PBR in ACI?

Before understanding why PBR matters, it is important to understand what service insertion looks like without it. In a traditional ACI deployment without PBR, inserting a service device such as a firewall between two EPGs requires significant manual configuration effort.

Without PBR, service insertion demands:

More manual contracts between EPGs and the firewall
Complex route configuration to steer traffic to the firewall, typically using VRF stitching
Two-arm firewall deployment as a mandate, since VRF stitching requires separate consumer and provider interfaces

Consider a topology with three leaves: Leaf-1 hosting an APP EPG endpoint (EP1), Leaf-3 hosting a WEB EPG endpoint (EP2), and Leaf-2 hosting the firewall. Without PBR, you need to manually configure contracts to allow the WEB EPG to reach the firewall in VRF1, and separate contracts to allow the APP EPG to reach the firewall in VRF2. This creates operational overhead and increases the chance of misconfiguration.

Aspect	Without PBR	With PBR
Contract configuration	Manual, multiple contracts required	Automated via service graph
VRF stitching	Required	Not required
Firewall mode	Two-arm mandatory	One-arm or two-arm
Forwarding constructs	Manual route manipulation	Attached to contracts via PBR
Implementation	Zoning-rules configured manually	Zoning-rules with redirect action

The complexity of the non-PBR approach is precisely why service graphs with PBR were introduced.

Why Use Service Graphs with ACI PBR Troubleshooting in Mind?

Service graphs with PBR offer a fundamentally cleaner approach to service insertion in ACI. Understanding their architecture is essential for effective ACI PBR troubleshooting, because knowing how the system is supposed to work is the first step toward identifying what has gone wrong.

Service graphs provide several key advantages:

Service graphs automate contract configuration. Instead of manually building contracts between EPGs and the firewall, the service graph handles this automatically.
Easy insertion within a VRF. PBR eliminates the need for VRF stitching, allowing you to insert a service device within a single VRF.
One-arm or two-arm deployment. Unlike the non-PBR approach that mandates two-arm, PBR supports both one-arm and two-arm firewall deployments.
PBR attaches forwarding constructs to contracts. This is the core mechanism: PBR gives you the ability to tie forwarding decisions (redirect actions) directly to the contract subject.
Implementation uses zoning-rules on leaf switches. Just like regular contracts, PBR contracts are enforced through zoning-rules, but with a redir action instead of a simple permit.

The service graph is tied to the subject of the contract between the consumer and provider EPGs, creating a seamless insertion point for the service device.

How Do Shadow and Service EPGs Work in ACI PBR?

When a service graph is deployed in ACI, several constructs are automatically created. Understanding these auto-configured elements is critical for troubleshooting because they appear in CLI output and must be verified during packet walk analysis.

Shadow/Service EPG Fundamentals

Shadow EPGs (also called Service EPGs) are auto-configured when a service graph is deployed. They connect to the service device and serve as the classification mechanism for traffic entering and leaving the service node.

In a two-armed deployment example:

The External Interface is called the Consumer Connector
The Internal Interface is called the Provider Connector
Each connector has its own PcTag (sclass), which is essential for zoning-rule matching

For example, if EP1 belongs to the Web EPG on Leaf-1 and EP2 belongs to the App EPG on Leaf-3, with a firewall on Leaf-2, the service graph creates Shadow EPGs with their own PcTags. The Provider Connector Shadow EPG gets its own unique sclass value that you will see in zoning-rule output.

Pro Tip: When troubleshooting PBR, always verify the sclass values of the Shadow/Service EPGs. These auto-generated PcTags appear in zoning-rules and must match correctly for traffic to flow. If you see unexpected sclass values in your zoning-rule output, check whether the service graph deployed correctly.

How Does ACI PBR Contract Enforcement Work?

Understanding contract enforcement is foundational to ACI PBR troubleshooting. The classification and policy application mechanism in ACI determines where and how redirect decisions are made.

Source and Destination Classification

Every EPG (or ESG) is mapped to its own unique ClassID, also known as a pcTag or sclass/dclass. This classification drives all policy enforcement in the fabric.

The iVXLAN packet header carries critical fields:

Field	Purpose
Source Class-ID	Identifies the source EPG (assigned by ingress leaf)
VRF info (Segment ID)	Identifies the VRF context
SP bit	Indicates Source Policy (ACL) has been applied
DP bit	Indicates Destination Policy (ACL) has been applied

Policy Application Logic

The ingress leaf performs the following sequence:

Assigns the source ClassID (sclass) to the packet based on the ingress VLAN/interface (for EPG traffic) or source IP policy prefix (for L3Out traffic)
Attempts to derive the destination ClassID from either the known destination endpoint or the destination IP
If the destination ClassID can be derived, the leaf applies the policy (ACL/zoning-rule) and sets the Policy Applied bit (SP/DP bit to 1) in the iVXLAN header
If the destination ClassID cannot be derived, the leaf does not set the SP/DP bit (SP/DP bit remains 0)

When the SP/DP bits are not set (value 0), the egress leaf checks the EPG for the local destination and applies the policy there instead.

This behavior has a direct impact on PBR:

If the destination EP is known on the ingress leaf: the redirect happens on the ingress leaf
If the destination EP is unknown on the ingress leaf: the redirect will happen on the egress leaf

Pro Tip: Understanding where the redirect happens is crucial for troubleshooting. If you are running ELAM captures on the wrong leaf, you will not see the redirect action. Always check whether the destination endpoint is known on the ingress leaf first.

ACI PBR Troubleshooting: The Complete Five-Step Packet Walk

This section is the heart of ACI PBR troubleshooting. We will trace a packet through all five steps of the PBR forwarding path, with exact CLI commands to verify each step. The topology uses:

EP1: 192.168.1.100 (source, on Leaf-1)
EP2: 192.168.2.100 (destination, on Leaf-3)
Service Device (Firewall): 172.16.1.100 (on Leaf-2)

Step 1: Ingress Classification on Source Leaf

EP1 sends a packet destined for EP2. The packet arrives at Leaf-1 with the original source and destination IP addresses. Leaf-1 must classify the packet, determine the source and destination sclass values, and perform a policy lookup.

ELAM Trigger on Leaf-1:

To capture this packet on ingress, use the following ELAM trigger in vsh_lc on Leaf-1:

debug platform internal roc elam asic0 trigger init
in-select 6 out-select 1 reset
set outer ipv4 src_ip 192.168.1.100 dst_ip 192.168.2.100
start

The in-select 6 captures the outer packet headers on ingress, while out-select 1 captures the rewrite output. This allows you to see both the original packet and what the leaf does with it after the policy lookup.

Step 2: Policy Lookup and Packet Rewrite

After classification, Leaf-1 performs the policy lookup. Since EP2 is known on Leaf-1 (its destination ClassID can be derived), the redirect happens right here on the ingress leaf.

During the rewrite phase:

Leaf-1 performs a policy lookup and finds a zoning-rule with a redir action
The packet is redirected to the Service BD (Bridge Domain) with the Service MAC of the firewall
The rewritten packet is sent to the MAC Proxy on the spine

The rewritten packet has:

Inner headers: Original SIP/DIP preserved, but the destination MAC is rewritten to the firewall MAC
Outer headers: The DMAC is set to the Anycast MAC, and the packet is encapsulated with the Service BD VNID and sent toward the spine loopback

Step 2 CLI Verification: Confirming PcTags and Zoning-Rules

First, confirm the pcTags of both endpoints:

leaf1# show system internal epm endpoint ip 192.168.1.100 | egrep "VRF vnid|sclass"
BD vnid: 16089032 ::: VRF vnid: 2293762
Flags : 0x80004c04 ::: sclass: 32771

leaf1# show system internal epm endpoint ip 192.168.2.100 | egrep "VRF vnid|sclass"
BD vnid: 15826920 ::: VRF vnid: 2293762
Flags : 0x80004c04 ::: sclass: 16386

The source EP1 has sclass 32771 and the destination EP2 has sclass 16386. Both are in VRF VNID 2293762.

Now verify the zoning-rule that matches this traffic:

leaf1# show zoning-rule scope 2293762 src-epg 32771 dst-epg 16386
+---------+--------+--------+----------+--------+---------+-----------------+
| Rule ID | SrcEPG | DstEPG | FilterID | Dir    | Scope   | Action          |
+---------+--------+--------+----------+--------+---------+-----------------+
| 4308    | 32771  | 16386  | 1        | bi-dir | 2293762 | redir(destgrp-3)|

Key observations from this output:

The Action is redir(destgrp-3) -- this confirms PBR is active and traffic matching this rule will be redirected to destination group 3
The Dir is bi-dir, meaning both directions of the flow are redirected
The FilterID is 1, which we can inspect further

Check the filter to understand what traffic is being matched:

leaf1# show zoning-filter filter 1
+----------+------+--------+-------------+-------------+-------------+-------------+
| FilterId | Name | EtherT | SFromPort   | SToPort     | DFromPort   | DToPort     |
+----------+------+--------+-------------+-------------+-------------+-------------+
| 1        | 1_0  | ip     | unspecified | unspecified | unspecified | unspecified |

This filter matches all IP traffic with unspecified port ranges, meaning all IP traffic between these EPGs will be redirected.

Pro Tip: Avoid using the default filter for PBR deployments. While matching all IP traffic works for basic setups, using specific filters gives you more granular control and makes troubleshooting easier. A filter that matches only specific protocols or ports ensures that only the intended traffic is redirected through the service device.

Step 2 CLI Verification: Checking the Redirect Policy

Next, examine the redirect destination group to understand where packets will be sent:

leaf1# show service redir info group 3
==============================================================
GrpID  Name       destination
3      destgrp-3  dest-[172.16.1.100]-[vxlan-2293762]

This confirms the redirect target is 172.16.1.100 (the firewall) in VRF VNID 2293762.

For more details about the redirect destination:

leaf1# show service redir info destination ip 172.16.1.100 vnid 2293762
=======================================================================================================
Name                                          bdVnid         vMac                vrf
====                                          ======         ====                ====
dest-[172.16.1.100]-[vxlan-2293762]           vxlan-16744311 00:50:56:A8:48:97   mg-cisco-live:v1

This output reveals the parameters used to build the redirected VXLAN packet:

Parameter	Value	Purpose
Redirect IP	172.16.1.100	Service node IP address
Service BD VNID	vxlan-16744311	Bridge Domain VNID for the service network
vMac	00:50:56:A8:48:97	MAC address of the service device
VRF	mg-cisco-live:v1	VRF context for the service node

Step 2 ELAM Verification: Confirming Redirect in Hardware

You can confirm the redirect is happening in hardware by checking the ELAM report:

module-1(DBG-elam-insel6)# report detail | grep service_redir
sug_luc_latch_results_vec.luc3_0.service_redir: 0x1

The service_redir value tells you:

0x1 -- the packet IS being redirected
0x0 -- the packet is NOT being redirected

If you expect a redirect but see 0x0, the zoning-rule is either not installed, not matching, or the redirect policy is not properly programmed.

ACI PBR Troubleshooting: Spine COOP MAC Lookup (Step 3)

After the ingress leaf rewrites the packet and sends it toward the spine, the spine must perform a COOP (Council of Oracles Protocol) MAC lookup to determine which leaf hosts the service device.

Why the Spine Lookup Matters

The ingress leaf sends the redirected packet to the spine with the service device MAC address and the Service BD VNID. The spine must look up this MAC in its COOP database to find the tunnel endpoint (TEP) address of the leaf where the service device is connected.

Warning: A MAC address misconfiguration in the redirect policy will lead to a packet drop on the spine due to a COOP lookup miss. This is one of the most common causes of PBR failure -- the spine simply cannot find the service device MAC in its COOP table, and the packet is silently dropped.

Step 3 CLI Verification: Spine COOP Lookup

On the spine, verify that the service device MAC is present in COOP:

spine1# show coop internal info repo ep key 16744311 00:50:56:A8:48:97 | egrep "Tunnel|EP" | head -n 3
EP bd vnid: 16744311
EP mac : 00:50:56:A8:48:97
Tunnel nh: 10.0.216.68

Key fields in this output:

EP bd vnid: 16744311 -- matches the Service BD VNID from the redirect policy
EP mac: 00:50:56:A8:48:97 -- matches the service device MAC from the redirect policy
Tunnel nh: 10.0.216.68 -- this is the TEP (Tunnel Endpoint) address of the leaf hosting the service device

Mapping the Tunnel Address to a Leaf

To identify which leaf the tunnel address points to, query the APIC:

apic1# moquery -c ipv4Addr -f 'ipv4.Addr.addr=="10.0.216.68"' | grep dn
dn : topology/pod-1/node-102/sys/ipv4/inst/dom-overlay-1/if-[lo0]/addr-[10.0.216.68/32]

This confirms the tunnel points to node-102 (Leaf-2), which is where the service device is connected. If the ELAM on the spine shows the destination TEP as the MAC Proxy address instead of the service leaf TEP, investigate whether the COOP entry is correctly installed.

ACI PBR Troubleshooting: Service Leaf Verification (Step 4)

Once the spine forwards the packet to the correct leaf (Leaf-2 in our example), the service leaf delivers the traffic to the service device. This step requires verifying that the service device endpoint is properly learned and programmed on the service leaf.

ELAM Trigger on Service Leaf

To capture the redirected packet arriving at the service leaf, use an ELAM trigger that matches on the inner headers. Note the difference from Step 1 where we matched on outer headers:

debug platform internal roc elam asic0 trigger init
in-select 14 out-select 1 reset
set inner ipv4 src_ip 192.168.1.100 dst_ip 192.168.2.100
set inner l2 dst_mac <FW-MAC>
start

The in-select 14 is used here to capture inner packet headers on the service leaf, since the packet arrives as a VXLAN-encapsulated frame.

Step 4 CLI Verification: Service Device Endpoint

Verify the service device MAC is learned on Leaf-2:

leaf2# show system internal epm endpoint mac 0050.56a8.4897
MAC : 0050.56a8.4897 ::: Num IPs : 1
IP# 0 : 172.16.1.100 ::: IP# 0 flags : host-tracked
Vlanid : 57 ::: Vlanvnid: 10792 ::: VRF name : mg-cisco-live:v1
BD vnid: 16744311 ::: VRF vnid: 2293762
PhyIf : 0x1a02a000 ::: Tunnel If : 0
Interface : Ethernet1/43
Flags : 0x80004c04 ::: sclass : 16389 ::: Ref count : 5
EP Create Timestamp : 03/28/2023 15:23:44.027077
EP Update Timestamp : 04/14/2023 13:52:41.683129
EP Flags : local|IP|MAC|sclass|timer|

Critical details to verify:

Field	Value	What to Check
MAC	0050.56a8.4897	Must match the vMac in the redirect policy
IP	172.16.1.100	Must match the redirect destination IP
BD vnid	16744311	Must match the Service BD VNID
VRF vnid	2293762	Must match the VRF VNID
Interface	Ethernet1/43	Physical interface where the firewall connects
sclass	16389	PcTag of the Shadow/Service EPG
EP Flags	local	Endpoint must be locally learned

Verifying the Service VLAN

You can also verify the VLAN configuration on the service leaf:

leaf2# show vlan id 57 extended
VLAN Name                          Encap           Ports
---- ----                          ----            -----
57   mg-cisco-live:pbr-one-arm-    vlan-1417       Eth1/43
     CL2023ctxv1:provider:

This shows:

Internal VLAN 57 (FD_VLAN) is mapped to access encap VLAN 1417
The service device is connected on Eth1/43
The VLAN name includes the service graph name and indicates this is the provider connector

Pro Tip: You can also look up the VLAN by its access encapsulation using the command show vlan encap-id 1417. This is useful when you know the access VLAN but not the internal FD_VLAN ID.

ACI PBR Troubleshooting: Return Traffic Flow (Step 5)

After the firewall inspects the traffic, it sends the packet back into the fabric. Understanding the return traffic flow is essential for complete ACI PBR troubleshooting, as issues can occur in this direction as well.

How Return Traffic Works

When the service device completes its inspection, it sends the traffic back to the ACI fabric with:

Destination MAC: The ACI router MAC (the well-known 00:22:BD:F8:19:FF anycast gateway MAC)
Destination IP: EP2 (192.168.2.100)
Source MAC: The firewall MAC (0050.56a8.4897)

Leaf-2 receives this return traffic on the Service EPG VLAN and performs a policy lookup. The critical point here is that the traffic from the Shadow EPG to the provider EPG is implicitly allowed with a default filter. This implicit permit is automatically created by the service graph deployment.

ELAM Trigger for Return Traffic

To capture the return traffic from the firewall on Leaf-2:

debug platform internal roc elam asic0 trigger init
in-select 6 out-select 1 reset
set outer ipv4 src_ip 192.168.1.100 dst_ip 192.168.2.100
set outer l2 src_mac <FW-MAC>
start

Note that for return traffic, we use in-select 6 (outer headers) and match on the source MAC being the firewall MAC, since the firewall is sending the packet back into the fabric.

Step 5 CLI Verification: Implicit Permit Zoning-Rule

Verify the zoning-rule that permits traffic from the Shadow EPG to the destination EPG:

leaf2# show system internal epm endpoint mac 0050.56a8.4897 | egrep "VRF vnid|sclass"
BD vnid: 16744311 ::: VRF vnid: 2293762
Flags : 0x80004c04 ::: sclass: 16389

leaf1# show system internal epm endpoint ip 192.168.2.100 | egrep "VRF vnid|sclass"
BD vnid: 15826920 ::: VRF vnid: 2293762
Flags : 0x80000c80 ::: sclass: 16386

The firewall has sclass 16389 (Shadow EPG) and the destination EP2 has sclass 16386. Now check the zoning-rule:

leaf2# show zoning-rule scope 2293762 src-epg 16389 dst-epg 16386
+---------+--------+--------+----------+---------+---------+---------+------+--------+----------------+
| Rule ID | SrcEPG | DstEPG | FilterID | Dir     | operSt  | Scope   | Name | Action | Priority       |
+---------+--------+--------+----------+---------+---------+---------+------+--------+----------------+
| 4152    | 16389  | 16386  | default  | uni-dir | enabled | 2293762 |      | permit | src_dst_any (9)|
+---------+--------+--------+----------+---------+---------+---------+------+--------+----------------+

Key observations:

Action is permit -- not redirect. The return traffic from the Shadow EPG to the provider is simply permitted.
FilterID is default -- this is the implicit permit created by the service graph.
Dir is uni-dir -- this rule applies in one direction only (from Shadow EPG to provider).
Priority is src_dst_any (9) -- this is the priority level for service graph implicit rules.

Using contract_parser for Verification

You can also use the contract_parser.py utility to get a human-readable view of the contract:

leaf2# contract_parser.py --vrf mg-cisco-live:v1 --dst epg tn-mg-cisco-live/ap-a1/epg-e2
[9:4152] [vrf:mg-cisco-live:v1] permit any
  tn-mg-cisco-live/G-pbr-one-arm-CL2023ctxv1/C-provider(16389)
  tn-mg-cisco-live/ap-a1/epg-e2(16386)
  [contract:uni/tn-mg-cisco-live/brc-ip] [hit=172]

This output confirms:

The rule permits traffic from the provider connector (sclass 16389) of the service graph to EPG e2 (sclass 16386)
The contract has been hit 172 times, confirming traffic is flowing
The service graph name is visible in the path (G-pbr-one-arm-CL2023ctxv1)

Pro Tip: The contract_parser.py tool is invaluable for PBR troubleshooting. It translates raw zoning-rule output into readable contract information, showing you the EPG names, contract names, and hit counts. A hit count of zero on a rule you expect to be active indicates traffic is not matching that rule.

Return Traffic for the Reverse Direction

An important note from the reference: for return traffic in the reverse direction (EP2 to EP1), the PBR flow follows the same pattern. The same service graph and redirect policy handles both directions when the zoning-rule direction is bi-dir.

Common ACI PBR Troubleshooting Scenarios and Solutions

Based on the packet walk analysis above, here are the most common failure points and how to diagnose them:

Scenario 1: Packet Drop on Spine (COOP Miss)

Symptom: Traffic from EP1 never reaches the firewall.

Root Cause: MAC address misconfiguration in the redirect policy. The spine cannot find the service device MAC in its COOP table.

Diagnosis:

Check show service redir info destination on the ingress leaf to confirm the vMac value
On the spine, verify COOP has the entry using show coop internal info repo ep key <BD-VNID> <MAC>
If the COOP entry is missing, check whether the service device endpoint is learned on the service leaf

Scenario 2: Redirect Not Happening

Symptom: Traffic flows directly between EPGs without going through the firewall.

Diagnosis:

Check the ELAM report for service_redir: 0x0 -- this confirms no redirect
Verify the zoning-rule has a redir action using show zoning-rule scope <VRF-VNID> src-epg <sclass> dst-epg <dclass>
Verify the service graph deployed successfully and Shadow EPGs are created

Scenario 3: Traffic Redirected but Firewall Not Receiving

Symptom: ELAM on ingress leaf shows service_redir: 0x1, but the firewall never sees the packet.

Diagnosis:

Verify spine COOP has the correct MAC and tunnel next-hop
Confirm the tunnel next-hop points to the correct leaf node
On the service leaf, verify the endpoint is locally learned with correct interface and VLAN

Scenario 4: Return Traffic Dropped After Firewall

Symptom: Traffic reaches the firewall, but the response is dropped.

Diagnosis:

Check that the implicit permit zoning-rule exists between the Shadow EPG sclass and the destination EPG sclass
Verify the firewall is sending traffic back to the ACI router MAC
Confirm the return traffic is classified into the correct Shadow EPG on the service leaf

ELAM Trigger Reference for ACI PBR Troubleshooting

ELAM (Memory Logic Analyzer) captures are the most powerful tool for ACI PBR troubleshooting. Here is a summary of the ELAM triggers used at each step of the packet walk:

Step	Leaf	in-select	Match Criteria	Purpose
1 (Ingress)	Leaf-1	6 (outer)	outer src_ip, dst_ip	Capture original packet on ingress
4 (Service Leaf)	Leaf-2	14 (inner)	inner src_ip, dst_ip, dst_mac	Capture redirected packet arriving at service leaf
5 (Return)	Leaf-2	6 (outer)	outer src_ip, dst_ip, src_mac (FW)	Capture return traffic from firewall

Key points about ELAM usage:

in-select 6 captures outer packet headers and is used when the packet originates locally or when you need to see the outer encapsulation
in-select 14 captures inner packet headers and is used when the packet arrives as a VXLAN-encapsulated frame from another leaf via the spine
out-select 1 is consistently used across all triggers to capture the rewrite output
Always check the service_redir field in the ELAM report detail to confirm whether redirect is occurring

Key CLI Commands Summary for ACI PBR Troubleshooting

For quick reference, here are all the essential commands organized by verification stage:

Endpoint and Classification Commands

show system internal epm endpoint ip <IP> | egrep "VRF vnid|sclass"
show system internal epm endpoint mac <MAC>

Zoning-Rule and Policy Commands

show zoning-rule scope <VRF-VNID> src-epg <sclass> dst-epg <dclass>
show zoning-filter filter <filter-id>

Redirect Policy Commands

show service redir info group <group-id>
show service redir info destination ip <IP> vnid <VRF-VNID>

Spine COOP Commands

show coop internal info repo ep key <BD-VNID> <MAC> | egrep "Tunnel|EP"

APIC Queries

moquery -c ipv4Addr -f 'ipv4.Addr.addr=="<TEP-IP>"' | grep dn

VLAN Verification Commands

show vlan id <vlan-id> extended
show vlan encap-id <access-vlan>

Contract Parser

contract_parser.py --vrf <vrf-name> --dst epg <epg-dn>

Frequently Asked Questions

Where does the PBR redirect happen -- on the ingress or egress leaf?

It depends on whether the destination endpoint is known on the ingress leaf. If the destination EP is known (its ClassID can be derived), the redirect happens on the ingress leaf. If the destination EP is unknown, the redirect happens on the egress leaf. You can determine this by checking whether the destination endpoint appears in the EPM table on the ingress leaf using show system internal epm endpoint ip <destination-IP>.

What causes a packet drop on the spine during PBR?

The most common cause is a MAC address misconfiguration in the redirect policy. When the ingress leaf redirects the packet, it uses the service device MAC from the redirect policy. The spine performs a COOP MAC lookup using this MAC and the Service BD VNID. If the MAC is incorrect or the service device has not been learned on any leaf, the COOP lookup misses and the packet is dropped silently on the spine. Verify the MAC using show service redir info destination and cross-reference with the actual firewall MAC address.

How can I confirm that a packet is actually being redirected by PBR?

Use an ELAM capture on the leaf where the redirect should occur. After triggering the ELAM and generating traffic, check the report detail for the service_redir field. A value of 0x1 confirms the packet is being redirected, while 0x0 means no redirect is happening. Additionally, verify the zoning-rule output shows a redir(destgrp-X) action rather than a simple permit action.

What is the difference between in-select 6 and in-select 14 in ELAM?

in-select 6 is used to match on outer packet headers. Use this when capturing traffic that originates locally on the leaf or when you need to see the outer VXLAN encapsulation headers. in-select 14 is used to match on inner packet headers. Use this when capturing traffic that arrives at a leaf from the spine as a VXLAN-encapsulated frame, such as redirected traffic arriving at the service leaf.

How does return traffic from the firewall get permitted?

When a service graph is deployed, ACI automatically creates an implicit permit zoning-rule between the Shadow/Service EPG and the provider (or consumer) EPG. This rule uses the default filter and has a uni-dir direction. You can verify it exists using show zoning-rule scope <VRF-VNID> src-epg <shadow-sclass> dst-epg <provider-sclass>. The action should show permit with a priority of src_dst_any (9).

Why should I avoid using the default filter for PBR contracts?

While the default filter (matching all IP traffic) works, it provides no granularity in controlling which traffic gets redirected through the service device. Using specific filters that match only the desired protocols or port ranges gives you better control over the redirect behavior and makes troubleshooting easier. When all traffic is redirected with a default filter, it becomes harder to isolate specific traffic flows during troubleshooting.

Conclusion

ACI PBR troubleshooting requires a systematic approach that follows the packet through every stage of its journey across the fabric. By understanding the five-step packet walk -- from ingress classification, through policy lookup and packet rewrite, spine COOP MAC lookup, service leaf delivery, and finally the return traffic flow -- you can quickly isolate where a failure occurs and determine the root cause.

The key takeaways from this deep dive are:

Service graphs automate the complexity of service insertion, but you must understand the auto-created Shadow EPGs and their PcTags to troubleshoot effectively
ELAM captures are your most powerful tool -- use in-select 6 for outer headers and in-select 14 for inner headers, and always check the service_redir field
Zoning-rules with redir actions are the mechanism that drives PBR, and verifying these rules with the correct sclass/dclass values is step one of any troubleshooting workflow
Spine COOP MAC lookups are a critical forwarding step -- a misconfigured MAC in the redirect policy silently drops packets on the spine
Return traffic relies on implicit permit rules created by the service graph, and verifying these rules exist is essential for bidirectional traffic flow

Mastering these verification techniques will save you significant time in both lab environments and production data center fabrics. Whether you are designing a new ACI PBR deployment or troubleshooting an existing one, following this structured packet walk methodology ensures you never miss a step.

For more hands-on practice with data center technologies and CCIE-level troubleshooting techniques, explore the training resources available at NHPREP.