Q: What is the difference between L2VNI and L3VNI, and how do they interact in symmetric IRB?

**L2VNI (Layer 2 VNI)** extends a VLAN across the fabric. It carries bridged traffic for a specific broadcast domain. Each VLAN maps to one L2VNI (e.g., VLAN 10 → VNI 10010). Type-2 routes in L2VNI advertise MAC addresses and MAC/IP bindings. Hosts in the same L2VNI can communicate at Layer 2 regardless of which leaf they are on. **L3VNI (Layer 3 VNI)** is used for inter-subnet routing within the EVPN fabric. It is a single VNI shared across all VRFs (or one per VRF) that carries routed traffic between subnets. In **symmetric IRB**, when Host-A (VLAN 10, L2VNI 10010) wants to reach Host-B (VLAN 20, L2VNI 10020), the routing happens at the ingress leaf: (1) Leaf-1 receives the packet on VLAN 10, performs Layer 3 lookup, and determines Host-B is in VLAN 20. (2) Leaf-1 encapsulates the packet in VXLAN with **L3VNI 20000** (the shared routing VNI) as the VNI, not L2VNI 10020. (3) The original Ethernet frame is preserved as the inner payload, but the outer VNI indicates "this is a routed packet." (4) Leaf-2 receives the VXLAN packet, sees L3VNI 20000, decapsulates, and bridges the packet onto VLAN 20. This is symmetric because **both ingress and egress leaves perform both routing and bridging**. In asymmetric IRB, the ingress leaf routes directly to the destination L2VNI, which requires the ingress leaf to know all remote host /32 routes — this does not scale. L3VNI is advertised via **EVPN Type-5 routes (IP prefix routes)**. I verify with `show nve vni` and `show bgp l2vpn evpn route-type 5`.

Q: What underlay multicast configuration is required for BUM traffic in VXLAN, and how does ingress replication compare?

For multicast underlay BUM forwarding, three components must align: (1) **PIM-SM or PIM-SSM** on spines and leaves. PIM-SSM is preferred because it eliminates the need for an RP — leaves simply join the SSM group (S,G) for each VNI. (2) **IGMP snooping** on leaf switchports to track which hosts need which multicast groups. Without IGMP snooping, BUM traffic is flooded to all ports in the VLAN, wasting bandwidth. (3) **Multicast group per VNI**: each L2VNI maps to a unique multicast group IP (e.g., VNI 10010 → 239.1.1.10). All leaves interested in that VNI join that group. When a leaf receives a BUM frame, it encapsulates it in VXLAN and sends it to the multicast group; PIM routes it to all joined leaves. If DHCP offers are missing, I check: `show ip igmp groups` (is the leaf joined?), `show ip pim neighbor` (is PIM up?), `show ip mroute` (is the (S,G) state present?), and `show nve vni 10010 | include mcast-group`. **Ingress replication** is the alternative: instead of multicast, the leaf unicasts BUM frames to every known remote VTEP. Pros: no multicast in underlay, works over Layer 3 DCI. Cons: replication burden on the source leaf scales O(N) with VTEP count; for 64 leaves, each BUM packet is replicated 63 times. Cisco recommends **multicast for BUM within the DC** and **ingress replication for DCI (Data Center Interconnect)** where multicast is not available. I verify ingress replication with `show nve vni 10010 | include replication`.

Q: What EVPN multi-homing mode prevents duplicate frames, and what is the role of the Ethernet Segment Identifier (ESI)?

The issue is likely **All-Active multi-homing without proper DF (Designated Forwarder) election**. In EVPN multi-homing, an Ethernet Segment (ES) is a set of links from a host to multiple leaves, identified by a unique **ESI** (10-byte identifier, e.g., 0000:0000:0000:0000:0001). The ESI is advertised in **EVPN Type-1 routes (Ethernet Auto-Discovery)**. For BUM traffic, only the **DF** forwards to the ES; non-DF leaves drop BUM traffic to prevent duplicates. DF election uses a hash of the ESI and VNI modulo the number of PEs. If both leaves think they are DF, or if DF election is not configured, duplicates occur. To fix: (1) Ensure both leaves have the same ESI on the port-channel: `evpn ethernet-segment 0000:0000:0000:0000:0001`. (2) Configure **DF election preference** if needed: `df-election preference 100` on the preferred leaf. (3) Use **Single-Active multi-homing** instead of All-Active if the host does not support LACP or if the fabric cannot handle All-Active complexity. In Single-Active, only one leaf is active at a time; the backup takes over on failure. For LACP-based All-Active, the host must use **LACP in fast mode** and the leaves must use the **same system MAC** for the port-channel (via `lacp system-priority` and `lacp system-mac`). I verify with `show port-channel summary`, `show evpn es`, and `show evpn df-election`.

Q: What are the three key architectural differences, and when is each the right choice?

**Cisco ACI**: A policy-driven SDN controller (APIC) that abstracts VXLAN/EVPN behind application profiles, EPGs, and contracts. Pros: integrated microsegmentation, centralized policy, strong multi-tenant support, built-in telemetry. Cons: proprietary, expensive licensing, steep learning curve, lock-in to Cisco hardware. Best for: enterprises with existing Cisco investments, heavy multi-tenancy, and teams that prefer GUI-driven policy over CLI. **Arista CloudVision (CVP)**: Network management and automation platform that configures EOS switches via configlets and streaming telemetry. CVP does not replace the control plane — BGP-EVPN runs natively on switches. Pros: lean, Linux-based EOS, excellent programmability (eAPI, Netconf), strong telemetry, open standards. Cons: less integrated security policy than ACI; multi-tenancy requires manual VRF design. Best for: cloud-scale data centers, DevOps-friendly teams, and environments prioritizing automation and observability. **SONiC (Software for Open Networking in the Cloud)**: Open-source NOS from Microsoft/OCP, runs on white-box switches. Pros: fully open, containerized architecture (FPMSyncd, BGP container, SWSS), multi-vendor hardware support, no licensing costs. Cons: smaller commercial support ecosystem, requires strong in-house engineering, some features lag behind commercial NOS. Best for: hyperscalers, service providers, and organizations with custom automation needs. For a 128-leaf fabric with strong automation requirements and budget constraints, I recommend **Arista + CVP** for balance of capability and support. If the organization is cloud-native and has strong Linux engineering, **SONiC + Ansible/Custom controller** is compelling.

Q: What VXLAN QoS feature copies the inner DSCP to the outer header, and what is the default behavior?

By default, VXLAN encapsulation does **not** copy the inner DSCP to the outer UDP/IP header. The outer header is set to DSCP 0 (best effort), which means voice traffic (DSCP EF) loses its priority in the underlay. The fix is **VXLAN QoS tunnel mode**. On Cisco NX-OS, use `qos map dscp-tunnel` or `vpc bind vpc_id dscp` to copy the inner DSCP to the outer header. On Arista EOS, use `vxlan qos dscp propagation` or `vxlan qos map`. On Cumulus Linux, set `nv set nve vxlan dscp propagation on`. There are three tunnel modes: (1) **Uniform mode**: copy inner DSCP to outer DSCP (recommended for end-to-end QoS). (2) **Pipe mode**: set outer DSCP to a fixed value based on switch QoS policy, ignoring inner DSCP. (3) **Short-pipe mode**: similar to pipe, but the egress switch uses the inner DSCP for scheduling. For voice/video fabrics, uniform mode is standard. I also verify that the **underlay switches trust DSCP** on all inter-switch links: `mls qos trust dscp` (Cisco) or `qos trust dscp` (Arista). If the underlay resets DSCP at any hop, propagation is meaningless. I verify with `show policy-map interface` and packet captures on spine-to-leaf links. For ECN (Explicit Congestion Notification), VXLAN should also propagate ECN bits so that end hosts can react to underlay congestion.

Question 1

What three EVPN-specific misconfigurations cause intra-VLAN communication failure despite a healthy underlay?

Accepted Answer

First, **VNI mismatch or missing VLAN-to-VNI mapping**. On Cisco, `vlan 10` must map to `vn-segment 10010` under `vlan configuration`. If one leaf maps VLAN 10 to VNI 10010 and another maps it to VNI 10020, EVPN treats them as different broadcast domains. I verify with `show vlan` and `show nve vni`. Second, **missing EVPN RT (Route Target) import/export**. EVPN uses RTs for VPN membership. If Leaf-1 exports RT 65000:10010 but Leaf-2 only imports 65000:10020, the MAC/IP routes are rejected. I check `show bgp l2vpn evpn` and `show running-config | section evpn`. Third, **NVE interface down or wrong source IP**. The NVE (Network Virtualization Endpoint) interface must use a loopback IP that is reachable via the underlay. If the loopback is not advertised in IS-IS/BGP, remote VTEPs cannot establish VXLAN tunnels. I verify with `show nve peers` — the peer list should show all remote leaf loopbacks. If peers are missing, I check `show ip route` for the remote loopback and `ping source `. A subtle issue is **ingress replication missing for BUM traffic**: if multicast underlay is not configured and ingress replication is not enabled, ARP broadcasts never reach the remote leaf. I check `show nve vni 10010 | include replication`.

Question 2

What are the trade-offs of iBGP RR vs eBGP leaf-spine in EVPN, and which does Cisco recommend for large-scale fabrics?

Accepted Answer

**iBGP with spine route reflectors**: Leaf switches peer iBGP to both spines, which act as route reflectors (RR) for EVPN address family. Pros: single AS simplifies policy; BGP path attributes (LocalPref, MED) are preserved end-to-end; RR on spines is standard in Cisco's DMZ-Border design. Cons: all leaves must be in the same AS, which can be operationally limiting if some leaves are managed by a different team or tenant; iBGP requires full-mesh or RR, and if both spines fail, the overlay control plane is disrupted. **eBGP leaf-spine**: Each leaf is in its own AS (or leaves share an AS, spines share another). Pros: natural path diversity via AS_PATH; no RR needed — spines simply forward eBGP routes; supports multi-tenant designs where each tenant brings their own AS; loop prevention is automatic via AS_PATH. Cons: AS_PATH length affects best-path selection, which can interfere with EVPN aliasing and multi-homing; eBGP requires more careful next-hop handling — spines must not change the next-hop (use `next-hop-unchanged` for EVPN). Cisco's current recommendation for large-scale fabrics (>64 leaves) is **eBGP leaf-spine with separate AS per leaf** because it scales better, provides better tenant isolation, and aligns with modern disaggregated NOS (Cumulus, SONiC, Aruba CX). For Cisco NX-OS specifically, iBGP RR is still common in enterprise deployments. I verify with `show bgp l2vpn evpn summary` and check the AS numbers.

Question 3

How does anycast gateway work in EVPN, and what prevents duplicate ARP replies from causing issues?

Accepted Answer

In a distributed anycast gateway design, all leaf switches that host a given VLAN share the same virtual gateway IP and MAC (e.g., IP 10.1.1.1, MAC 0000.0c9f.f001). This is configured using **Virtual ARP (VARP)** on Arista, **Distributed Anycast Gateway** on Cisco, or **Anycast Gateway** on Cumulus. When a host ARPs for the gateway, all local leaves reply with the same MAC address. This is intentional and does not cause issues because: (1) The host receives multiple identical ARP replies (same IP-MAC binding) and caches one. (2) EVPN **Type-2 routes (MAC/IP)** advertise the host MAC and IP to all leaves, so every leaf knows which hosts are locally attached and which are remote. (3) Traffic to the gateway is handled locally on the leaf where the host is attached — this is **symmetric IRB (Integrated Routing and Bridging)**. For asymmetric IRB, the leaf routes the packet to the destination VNI, which may be on a different leaf; for symmetric IRB, the leaf routes to a shared L3VNI. The key EVPN route for anycast gateway is the **Type-2 route with MAC-only advertisement** for the gateway MAC, ensuring all leaves know it. Duplicate ARP replies are suppressed at the host level because the binding is identical. Issues only arise if the anycast gateway MAC differs between leaves (configuration drift) or if **proxy-ARP** is misconfigured. I verify with `show ip arp` on the host and `show bgp l2vpn evpn route-type 2` on the leaf.

Question 4

Why do spine switches learn MAC addresses in a VXLAN fabric, and what two design changes eliminate MAC learning on spines?

Accepted Answer

In a basic VXLAN/EVPN design, spines should **not** learn MAC addresses — they are pure Layer 3 underlay routers. However, if spines are configured as **BGP route reflectors for EVPN** and also participate in the overlay (e.g., running VTEP or proxy functions), they may cache EVPN routes. The real issue is likely that the spines are not spines at all — they may be **collapsed core** switches acting as both L3 underlay and L2 gateway. In a proper spine-leaf design, spines only forward VXLAN-encapsulated packets based on the outer IP header (VTEP source/destination). The inner Ethernet frame is never decapsulated on the spine. To eliminate MAC learning on spines: (1) **Ensure spines are not VTEPs**: only leaf switches should have `interface nve` configured. I verify with `show nve peers` on spines — it should be empty. (2) **Use BGP-EVPN RR without VTEP functionality**: configure spines as pure BGP RRs with `address-family l2vpn evpn` but no VNI membership. On Cisco, use `neighbor x.x.x.x route-reflector-client` under the EVPN AF. If the fabric uses **multicast for BUM traffic** (PIM-SSM in the underlay), spines run PIM but do not join multicast groups for VNIs — they simply route multicast. Another optimization is **MAC address reduction at the leaf**: enable **local-proxy-ARP** and **suppress duplicate MACs** so that only one MAC/IP route is advertised per host, not per VLAN. I also check for **MAC mobility flapping** — if a host moves frequently, EVPN generates multiple Type-2 routes with sequence numbers, increasing table size. I verify with `show bgp l2vpn evpn` and `show system resource`.

Question 5

What EVPN feature enables seamless VM mobility, and what three timers/messages ensure rapid convergence?

Accepted Answer

The feature is **EVPN MAC Mobility (Type-2 route with sequence number)**. When a VM moves, Leaf-2 detects the MAC on a new local interface and advertises a Type-2 route with a higher sequence number than Leaf-1's advertisement. All remote leaves replace the old route with the new one. Three convergence mechanisms: (1) **GRATUITOUS ARP / RARP**: the VM sends a GARP after vMotion, which Leaf-2 detects immediately. On Cisco, `fabric forwarding mode anycast-gateway` enables GARP processing. (2) **MAC mobility sequence number**: Leaf-2 increments the sequence number in the Type-2 route. Default sequence number starts at 0 and increments per move. If a MAC moves more than 5 times in 180 seconds (default), Cisco places it in **move-freeze** to prevent flapping. I tune this with `l2vpn evpn mac-move-bulk-processing` and `mac-move interval 5 sequence 10`. (3) **Remote leaf timer**: remote leaves use the **BGP withdrawal time** to remove the old route. With BGP RR, this is typically <1 second if `advertise-pip` is enabled. For faster convergence, I enable **BGP fast external fallover** and reduce the **EVPN neighbor holdtime** to 9 seconds. I verify mobility with `show l2route evpn mac ip` and `show bgp l2vpn evpn route-type 2`. I also check `show mac address-table moves` for excessive flapping. If the old leaf still receives traffic, it may be because **remote VTEPs have not processed the update** — I check `show bgp l2vpn evpn update-group` for pending updates.

Question 6

What is the difference between L2VNI and L3VNI, and how do they interact in symmetric IRB?

Accepted Answer

**L2VNI (Layer 2 VNI)** extends a VLAN across the fabric. It carries bridged traffic for a specific broadcast domain. Each VLAN maps to one L2VNI (e.g., VLAN 10 → VNI 10010). Type-2 routes in L2VNI advertise MAC addresses and MAC/IP bindings. Hosts in the same L2VNI can communicate at Layer 2 regardless of which leaf they are on. **L3VNI (Layer 3 VNI)** is used for inter-subnet routing within the EVPN fabric. It is a single VNI shared across all VRFs (or one per VRF) that carries routed traffic between subnets. In **symmetric IRB**, when Host-A (VLAN 10, L2VNI 10010) wants to reach Host-B (VLAN 20, L2VNI 10020), the routing happens at the ingress leaf: (1) Leaf-1 receives the packet on VLAN 10, performs Layer 3 lookup, and determines Host-B is in VLAN 20. (2) Leaf-1 encapsulates the packet in VXLAN with **L3VNI 20000** (the shared routing VNI) as the VNI, not L2VNI 10020. (3) The original Ethernet frame is preserved as the inner payload, but the outer VNI indicates "this is a routed packet." (4) Leaf-2 receives the VXLAN packet, sees L3VNI 20000, decapsulates, and bridges the packet onto VLAN 20. This is symmetric because **both ingress and egress leaves perform both routing and bridging**. In asymmetric IRB, the ingress leaf routes directly to the destination L2VNI, which requires the ingress leaf to know all remote host /32 routes — this does not scale. L3VNI is advertised via **EVPN Type-5 routes (IP prefix routes)**. I verify with `show nve vni` and `show bgp l2vpn evpn route-type 5`.

Question 7

What underlay multicast configuration is required for BUM traffic in VXLAN, and how does ingress replication compare?

Accepted Answer

For multicast underlay BUM forwarding, three components must align: (1) **PIM-SM or PIM-SSM** on spines and leaves. PIM-SSM is preferred because it eliminates the need for an RP — leaves simply join the SSM group (S,G) for each VNI. (2) **IGMP snooping** on leaf switchports to track which hosts need which multicast groups. Without IGMP snooping, BUM traffic is flooded to all ports in the VLAN, wasting bandwidth. (3) **Multicast group per VNI**: each L2VNI maps to a unique multicast group IP (e.g., VNI 10010 → 239.1.1.10). All leaves interested in that VNI join that group. When a leaf receives a BUM frame, it encapsulates it in VXLAN and sends it to the multicast group; PIM routes it to all joined leaves. If DHCP offers are missing, I check: `show ip igmp groups` (is the leaf joined?), `show ip pim neighbor` (is PIM up?), `show ip mroute` (is the (S,G) state present?), and `show nve vni 10010 | include mcast-group`. **Ingress replication** is the alternative: instead of multicast, the leaf unicasts BUM frames to every known remote VTEP. Pros: no multicast in underlay, works over Layer 3 DCI. Cons: replication burden on the source leaf scales O(N) with VTEP count; for 64 leaves, each BUM packet is replicated 63 times. Cisco recommends **multicast for BUM within the DC** and **ingress replication for DCI (Data Center Interconnect)** where multicast is not available. I verify ingress replication with `show nve vni 10010 | include replication`.

Question 8

What EVPN multi-homing mode prevents duplicate frames, and what is the role of the Ethernet Segment Identifier (ESI)?

Accepted Answer

The issue is likely **All-Active multi-homing without proper DF (Designated Forwarder) election**. In EVPN multi-homing, an Ethernet Segment (ES) is a set of links from a host to multiple leaves, identified by a unique **ESI** (10-byte identifier, e.g., 0000:0000:0000:0000:0001). The ESI is advertised in **EVPN Type-1 routes (Ethernet Auto-Discovery)**. For BUM traffic, only the **DF** forwards to the ES; non-DF leaves drop BUM traffic to prevent duplicates. DF election uses a hash of the ESI and VNI modulo the number of PEs. If both leaves think they are DF, or if DF election is not configured, duplicates occur. To fix: (1) Ensure both leaves have the same ESI on the port-channel: `evpn ethernet-segment 0000:0000:0000:0000:0001`. (2) Configure **DF election preference** if needed: `df-election preference 100` on the preferred leaf. (3) Use **Single-Active multi-homing** instead of All-Active if the host does not support LACP or if the fabric cannot handle All-Active complexity. In Single-Active, only one leaf is active at a time; the backup takes over on failure. For LACP-based All-Active, the host must use **LACP in fast mode** and the leaves must use the **same system MAC** for the port-channel (via `lacp system-priority` and `lacp system-mac`). I verify with `show port-channel summary`, `show evpn es`, and `show evpn df-election`.

Question 9

What are the three key architectural differences, and when is each the right choice?

Accepted Answer

**Cisco ACI**: A policy-driven SDN controller (APIC) that abstracts VXLAN/EVPN behind application profiles, EPGs, and contracts. Pros: integrated microsegmentation, centralized policy, strong multi-tenant support, built-in telemetry. Cons: proprietary, expensive licensing, steep learning curve, lock-in to Cisco hardware. Best for: enterprises with existing Cisco investments, heavy multi-tenancy, and teams that prefer GUI-driven policy over CLI. **Arista CloudVision (CVP)**: Network management and automation platform that configures EOS switches via configlets and streaming telemetry. CVP does not replace the control plane — BGP-EVPN runs natively on switches. Pros: lean, Linux-based EOS, excellent programmability (eAPI, Netconf), strong telemetry, open standards. Cons: less integrated security policy than ACI; multi-tenancy requires manual VRF design. Best for: cloud-scale data centers, DevOps-friendly teams, and environments prioritizing automation and observability. **SONiC (Software for Open Networking in the Cloud)**: Open-source NOS from Microsoft/OCP, runs on white-box switches. Pros: fully open, containerized architecture (FPMSyncd, BGP container, SWSS), multi-vendor hardware support, no licensing costs. Cons: smaller commercial support ecosystem, requires strong in-house engineering, some features lag behind commercial NOS. Best for: hyperscalers, service providers, and organizations with custom automation needs. For a 128-leaf fabric with strong automation requirements and budget constraints, I recommend **Arista + CVP** for balance of capability and support. If the organization is cloud-native and has strong Linux engineering, **SONiC + Ansible/Custom controller** is compelling.

Question 10

What VXLAN QoS feature copies the inner DSCP to the outer header, and what is the default behavior?

Accepted Answer

By default, VXLAN encapsulation does **not** copy the inner DSCP to the outer UDP/IP header. The outer header is set to DSCP 0 (best effort), which means voice traffic (DSCP EF) loses its priority in the underlay. The fix is **VXLAN QoS tunnel mode**. On Cisco NX-OS, use `qos map dscp-tunnel` or `vpc bind vpc_id <id> dscp` to copy the inner DSCP to the outer header. On Arista EOS, use `vxlan qos dscp propagation` or `vxlan qos map`. On Cumulus Linux, set `nv set nve vxlan dscp propagation on`. There are three tunnel modes: (1) **Uniform mode**: copy inner DSCP to outer DSCP (recommended for end-to-end QoS). (2) **Pipe mode**: set outer DSCP to a fixed value based on switch QoS policy, ignoring inner DSCP. (3) **Short-pipe mode**: similar to pipe, but the egress switch uses the inner DSCP for scheduling. For voice/video fabrics, uniform mode is standard. I also verify that the **underlay switches trust DSCP** on all inter-switch links: `mls qos trust dscp` (Cisco) or `qos trust dscp` (Arista). If the underlay resets DSCP at any hop, propagation is meaningless. I verify with `show policy-map interface` and packet captures on spine-to-leaf links. For ECN (Explicit Congestion Notification), VXLAN should also propagate ECN bits so that end hosts can react to underlay congestion.

Question 11

Why does the Type-5 route exist but forwarding fail, and what is the role of the BGP next-hop in EVPN prefix routes?

Accepted Answer

A Type-5 route advertises an IP prefix (e.g., 10.0.0.0/8) with a next-hop of the advertising VTEP (border leaf). Remote leaves install this route in their VRF routing table. However, forwarding fails if the **next-hop VTEP IP is not reachable** in the underlay. I verify with `ping source ` and `show ip route `. If the underlay IS-IS/OSPF does not advertise the border leaf loopback, remote leaves cannot build the VXLAN tunnel. Another cause: **missing L3VNI on the border leaf**. Type-5 routes require an associated L3VNI for the VRF. If the border leaf advertises the prefix but does not configure `vrf TENANT` with an L3VNI, the remote leaf cannot determine which VNI to use for VXLAN encapsulation. I verify with `show bgp l2vpn evpn route-type 5` and check the `Route Distinguisher` and `Extended Community` fields. A third cause: **host route more specific than Type-5**. If a remote leaf has a Type-2 host route for 10.1.1.1/32 (from a local VM), it preferentially uses the /32 over the Type-5 /8. If the /32 is stale (VM moved), traffic is blackholed. I check `show ip route vrf TENANT` for overlapping prefixes and clear stale routes. For external connectivity, the border leaf must also perform **NAT or BGP peering with the WAN router**, which is outside EVPN but critical for end-to-end reachability.

Question 12

What is the oversubscription ratio, and how many 100 Gbps spine-facing ports does each leaf need for a 3:1 oversubscribed design?

Accepted Answer

Each server has 2 × 25 Gbps = 50 Gbps of theoretical NIC bandwidth. With 10,000 servers, total server-facing bandwidth is 500 Tbps. However, not all servers transmit at line rate simultaneously. **Oversubscription ratio** = total server-facing bandwidth / total uplink bandwidth. For 3:1 oversubscription, total uplink bandwidth needed = 500 Tbps / 3 ≈ 167 Tbps. Each leaf has 10,000 servers / 32 leaves = 312.5 servers per leaf. Server-facing bandwidth per leaf = 312.5 × 50 Gbps = 15.6 Tbps. Wait — that is wrong. Let me recalculate: 10,000 servers / 32 leaves = 312 servers per leaf. Each leaf has 312 × 2 × 25 Gbps = 15,625 Gbps = 15.6 Tbps of server-facing bandwidth. For 3:1 oversubscription, uplink bandwidth per leaf = 15.6 Tbps / 3 = 5.2 Tbps. At 100 Gbps per spine port, each leaf needs 5,200 Gbps / 100 Gbps = **52 × 100 Gbps uplinks**. With 4 spines, that is 52 / 4 = **13 ports to each spine**. This is clearly unrealistic — no leaf has 52 × 100G ports. The real-world answer: you do **not** design for all 10,000 servers at line rate. You design for **traffic demand**: if average utilization is 5 Gbps per server (10%), total fabric demand is 50 Tbps. At 3:1, uplinks = 16.7 Tbps, or ~52 × 100G across all leaves, meaning **2 × 100G per leaf** (8 total to 4 spines). For a more practical design with modern hardware: each leaf is a 64 × 100G switch. Use **48 ports for servers** (24 servers at 2×25G with breakout cables) and **16 × 100G for spines**. Oversubscription = (48 × 25G) / (16 × 100G) = 1,200 / 1,600 = **0.75:1 (actually under-subscribed)**. This is common for AI/ML fabrics where east-west bandwidth is critical. For general compute, **3:1 is typical**: 48 × 25G server downlinks = 1,200 Gbps; 4 × 100G spine uplinks = 400 Gbps; ratio = 3:1.

Question 13

How do you safely extend Layer 2 into VXLAN without allowing STP BPDUs to propagate, and what is the BPDU Tunnel alternative?

Accepted Answer

The root cause is **STP BPDU propagation across the VXLAN boundary**. VXLAN itself does not carry BPDUs — they are Layer 2 control frames with destination MAC 0180.c200.0000, and VXLAN typically only carries data traffic. However, if the border leaf bridges the legacy VLAN directly into the VXLAN VNI without filtering, BPDUs may leak. The fix: (1) **BPDU Guard on the legacy-facing port**: `spanning-tree bpdufilter enable` or `spanning-tree bpduguard enable` on the border leaf port-channel to the legacy domain. This blocks BPDUs from entering the VXLAN fabric. (2) **STP isolation**: run a separate STP domain on the legacy side and do not extend it. The border leaf acts as a **Layer 2 gateway**, not a transparent bridge. (3) **BPDU Tunnel (802.1Q tunneling)**: if the legacy domain must remain a single STP domain across the VXLAN fabric, use BPDU Tunneling. The border leaf encapsulates BPDUs in a separate VLAN (P-tagged) and transports them across the VXLAN fabric to the remote border leaf, which decapsulates them. This preserves the legacy STP topology while isolating it from the VXLAN underlay STP. However, BPDU Tunneling is risky — a loop in the legacy domain can still storm through the VXLAN tunnel. The modern best practice is **Layer 2 termination at the border leaf**: convert the legacy VLAN to VXLAN at the border, and run a separate STP instance on the legacy side. I verify with `show spanning-tree` on the border leaf and ensure the VXLAN fabric uses a different STP priority/root than the legacy domain.

Question 14

What three causes put BGP-EVPN neighbors in admin idle, and what is the correct upgrade rollback procedure?

Accepted Answer

**Idle (Admin)** means BGP is administratively shut down on the neighbor. Causes: (1) **Neighbor shutdown command**: `neighbor x.x.x.x shutdown` was added accidentally or during maintenance. Fix: `no neighbor x.x.x.x shutdown`. (2) **VRF or address-family misconfiguration after upgrade**: the upgrade may have changed the default VRF behavior or reset the EVPN address-family config. I check `show running-config bgp` and compare to pre-upgrade backup. (3) **License change**: some NX-OS versions require specific licenses (e.g., LAN_ENTERPRISE_SERVICES_PKG) for EVPN. If the upgrade expired a temporary license, EVPN features are disabled and BGP neighbors go admin idle. I check `show license usage` and `show feature`. Rollback procedure: (1) **Pre-upgrade**: save config, take `show tech-support`, and verify `show install all impact` output. (2) **During upgrade**: use ISSU (In-Service Software Upgrade) if available to minimize downtime. (3) **Post-issue**: if EVPN breaks, first check `show bgp l2vpn evpn summary` and `show nve peers`. If neighbors are admin idle, check config drift. (4) **Rollback**: `install all rollback` on NX-OS reverts to the previous image and config. I verify with `show install all status` and `show version`. For critical fabrics, I maintain a **maintenance window with a physical console** because if EVPN breaks, remote SSH via management VRF may also fail if the management network depends on EVPN routes.

Question 15

What streaming telemetry and API technologies enable real-time EVPN monitoring, and what key metrics define fabric health?

Accepted Answer

**gNMI (gRPC Network Management Interface)** is the modern standard for streaming telemetry. On Cisco, enable `telemetry grpc` and subscribe to EVPN-specific YANG paths: `/ Cisco-IOS-XR-evpn-oper:evpn/active/Ethernet-Segments`, `/ Cisco-IOS-XR-bgp-oper:bgp/evpn/routes`. On Arista, use `streaming-gnmi` with OpenConfig paths: `/network-instances/network-instance/protocols/protocol/bgp/neighbors`. For non-gNMI devices, **SNMP** with custom MIBs or **NETCONF** with periodic polling are fallbacks. Key metrics: (1) **BGP-EVPN neighbor state**: 0=down, 1=up. Alert if <1 for >30s. (2) **MAC mobility sequence number delta**: if a MAC's sequence number increments >3 times in 5 minutes, flag flapping. (3) **VTEP peer count**: alert if the count drops by >10% (indicates a leaf failure). (4) **EVPN route churn rate**: BGP updates per second for EVPN address family. Normal is <10/sec; >100/sec indicates instability. (5) **VXLAN encapsulation/decapsulation errors**: `show nve internal error-stats`. For programmatic monitoring, I use **Telegraf + InfluxDB/Prometheus**: Telegraf's `gnmi` input plugin streams YANG data to Prometheus, and Grafana dashboards visualize fabric topology. I also use **pyATS/Genie** for automated config validation and **Batfish** for offline intent verification. For alerting, I integrate Prometheus Alertmanager with PagerDuty. The ultimate goal is **closed-loop automation**: if a VTEP fails, automatically reroute traffic and open a Jira ticket.

VXLAN & EVPN Spine-Leaf: 15 Data Center Interview Questions

More CCNP Enterprise Interview Prep