Firewall Data Path Troubleshooting

Introduction

When traffic stops flowing through your Cisco Secure Firewall, the pressure to restore connectivity can be intense. The key to resolving data path issues quickly is knowing exactly which tools to use, in what order, and how to interpret their output. Rather than guessing or rebooting the device, a structured troubleshooting approach lets you isolate the problem to a specific phase of packet processing and apply a targeted fix.

In this lesson, you will learn the systematic methodology for troubleshooting data path problems on Cisco Secure Firewall Threat Defense (FTD). We will cover the primary diagnostic tools available to you — Packet Tracer, captures (including Lina, Snort, ASP drop, and internal switch captures), and System Support Trace — along with the correct syntax and interpretation of their output. By the end of this lesson, you will be able to confidently diagnose why traffic is being dropped, identify which policy or NAT rule is responsible, and verify that your fix resolves the issue.

Key Concepts

Before diving into the tools themselves, it is important to understand the foundational troubleshooting methodology. When you encounter a connectivity issue on Secure Firewall, follow these steps in order:

Understand the topology — Know the interfaces, zones, and network segments involved.
Understand the packet flow — Identify the source, destination, ports, and expected NAT translations.
Collect diagnostics simultaneously at the time of the issue — This is critical. Gather multiple data sources at once rather than one at a time.

The diagnostics you should collect simultaneously include:

Tool	Purpose
Packet Tracer	Simulates a packet through the firewall and shows every processing phase
Captures with Trace	Captures real traffic and traces it through the engine
ASP Drop Captures	Shows packets dropped by the accelerated security path
System Support Trace	Provides firewall engine (Snort) debug-level visibility into rule evaluation
Connection Events	Shows completed or blocked connections in the management center
Syslogs	Real-time log messages for session creation, teardown, and denials

Important: Always collect your troubleshooting file and show tech output before rebooting the device. A reboot clears runtime state that may be essential for diagnosis.

Understanding where in the data path a packet is processed — and where it fails — is the core skill this lesson builds. The Secure Firewall processes packets through a series of ordered phases, from Layer 2 MAC access lists, through NAT untranslation, access list evaluation, inspection, and finally flow creation. Each tool gives you visibility into different parts of this pipeline.

How It Works

Packet Processing Phases

When a packet enters the Secure Firewall, it passes through multiple processing phases in a specific order. Packet Tracer reveals these phases explicitly. A typical packet traverses the following sequence:

CAPTURE — The packet is evaluated against any active capture filters.
ACCESS-LIST (MAC) — Implicit MAC-level access list check.
UN-NAT — If destination NAT is configured, the firewall performs untranslation to determine the real destination address.
ACCESS-LIST (Security Policy) — The packet is evaluated against the access control list applied to the ingress interface.
Inspection and Additional Phases — Connection settings, application inspection, intrusion policy evaluation, and other checks occur.
FLOW-CREATION — If all phases allow the packet, a new flow is created in the connection table and the packet is dispatched for forwarding.

The final result tells you the ingress interface, the egress interface, and whether the action is allow or drop. If any phase results in a drop, the output tells you exactly which phase denied the packet and which configuration rule was responsible.

Capture Architecture

The Secure Firewall has multiple capture points that operate at different layers of the architecture:

Capture Type	Location	Use Case
Lina Captures	Data plane (ASA engine)	Standard packet captures on named interfaces
Snort Captures	Snort inspection engine	Captures traffic as seen by the intrusion and application engines
ASP Drop Captures	Accelerated security path	Shows packets silently dropped by the hardware/fast path
Internal Switch Captures	Internal switch (41xx, 93xx, 31xx, 42xx)	Captures at the chassis internal switch level before packets reach the security module
Backplane Captures	Between internal switch and module	Captures traffic traversing the backplane interface

For platforms with an internal switch (Firepower 4100, 9300, 3100, and 4200 series), understanding the internal switch architecture is essential. Traffic flows from the physical port through the internal switch, across the backplane via the in_data_uplink1 interface to the security module running FTD or ASA, and back. The in_data_uplink1 interface connects the internal switch to the module, while in_mgmt_uplink1 connects the chassis management interface to the module.

Configuration Example

Using Packet Tracer

Packet Tracer simulates a packet entering a specified interface and shows you every processing phase. Here is how to trace a TCP packet arriving on the outside interface destined for RDP:

firepower# packet-tracer input outside tcp 10.1.1.1 5555 10.1.10.10 3389 detailed

The command breaks down as follows:

input outside — The ingress interface where the simulated packet enters
tcp — The IP protocol
10.1.1.1 5555 — Source IP and source port
10.1.10.10 3389 — Destination IP and destination port (RDP)
detailed — Includes internal flow and policy structure information

The output shows each phase. For example, the UN-NAT phase reveals the destination translation:

Phase: 3
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (outside,dmz) source dynamic any interface destination static interface Win7-vm service rdp-outside rdp-outside
Additional Information:
NAT divert to egress interface dmz
Untranslate 10.1.10.10/3389 to 192.168.103.221/3389

The access list phase shows which rule matched:

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_in in interface outside
access-list outside_in extended permit tcp any any eq 3389

At the end, the result summary tells you the final action:

Phase: 12
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 16538274, packet dispatched to next module

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: dmz
output-status: up
output-line-status: up
Action: allow

Tip: Focus on the end result first. If the action is allow, the issue may not be on this firewall. If it is drop, scroll up to find which phase denied the packet.

You can also transmit the simulated packet out of the firewall to test end-to-end reachability:

firepower# packet-tracer input inside tcp 10.1.1.20 10000 10.1.2.100 80 transmit detailed

The transmit keyword causes the simulated packet to actually egress the firewall toward the destination.

Configuring Lina Captures

To capture real traffic on an interface, use the capture command. For high-traffic environments, use the headers-only option and increase the buffer size:

firepower# capture capin interface inside headers-only buffer 10000000

To capture decrypted traffic across a VPN tunnel where the Secure Firewall is the VPN endpoint, use the include-decrypted option:

firepower# capture OUT interface outside trace include-decrypted match tcp any any

To view the captured packets:

firepower# show cap capout
1 packet captured
1: 12:08:30.837709 10.1.1.20.10000 > 10.1.2.100.80: S 1119191062:1119191062(0) win

Snort-Side Captures

To capture traffic as seen by the Snort engine, enter the FTD CLI and use the capture-traffic command:

> capture-traffic
Please choose domain to capture traffic from:
0 - br1
1 - Router
Selection? 1

Please specify tcpdump options desired.
(or enter '?' for a list of supported options)
Options: -w SNORTCAP.pcap -c 1000 host 192.168.1.2 and port 80

The -w flag writes the capture to a file, -c 1000 limits the capture to 1000 packets, and the filter uses standard tcpdump BPF syntax. Capture files are written to /ngfw/var/common/.

Internal Switch Captures (3100 and 4200 Series)

On the Secure Firewall 3100 and 4200 platforms, internal switch captures use the standard CLI with the switch keyword. To see available interfaces:

> capture capsw switch interface ?
Available interfaces to listen:
  in_data_uplink1    Capture packets on internal data uplink 1 interface
  in_mgmt_uplink1    Capture packets on internal mgmt uplink1 interface
  inside             Name of interface Ethernet 1/1.205
  outside            Name of interface Ethernet 1/1.206
  diagnostic         Name of interface Management 1/1

To collect switch capture files, use the diagnostic CLI:

firepower# copy flash:/packet-capture/sess-1-capsw-ethernet-1-1-0.pcap tftp://198.51.100.10/

Alternatively, copy the capture file to /ngfw/var/common/ and download it from FMC via Devices > File Download:

> expert
admin@firepower:~$ sudo cp /mnt/disk0/packet-capture/sess-1-capsw-ethernet-1-1-0.pcap /ngfw/var/common/

System Support Trace

System Support Trace provides Snort-level visibility into how the firewall engine evaluates rules for a specific flow:

> system support trace
Please specify an IP protocol: tcp
Please specify a client IP address: 192.168.1.40
Please specify a client port:
Please specify a server IP address: 192.168.2.40
Please specify a server port:
Enable firewall-engine-debug too? [n]: y

When enabled with firewall-engine-debug, the output shows the rule evaluation process, including which access control rule is being matched, the zones being evaluated, and the final verdict. This is invaluable when you suspect a specific access control policy rule is not matching as expected.

Real-World Application

In production environments, data path troubleshooting is a daily activity for security engineers. Here are the most common scenarios where these tools prove essential:

NAT issues — Packet Tracer quickly reveals whether destination untranslation is occurring correctly and which NAT rule is matching. This is especially useful in environments with overlapping NAT rules.
Access policy denials — When users report they cannot reach a service, Packet Tracer shows exactly which access list or access control policy phase is denying the traffic.
VPN traffic inspection — Using captures with the include-decrypted option lets you see the actual payload after tunnel decryption, which is essential for troubleshooting inspection issues on VPN traffic.
Platform-level drops — On chassis-based platforms (4100, 9300) and newer models (3100, 4200), internal switch captures reveal whether packets are being dropped before they even reach the security module. Application-level captures alone do not provide full visibility to the packet path within the chassis, so taking simultaneous chassis and application-level captures is recommended.

Best Practice: When analyzing internal switch captures in Wireshark, use the filter ! vntag to hide VN-tagged duplicate packets in front interface captures. For backplane captures, use the Wireshark filter frame.number & 1 to remove duplicates.

In FMC version 7.1 and later, Packet Tracer is available directly in the FMC GUI under the Device menu. You can define the simulated packet, view results, run multiple traces in separate tabs, review trace history, and even replay traces from a PCAP file across managed devices. REST API support is also available for automated troubleshooting workflows.

Summary

Follow a structured approach: understand the topology, understand the packet flow, then collect Packet Tracer, captures, system support trace, connection events, and syslogs simultaneously at the time of the issue.
Packet Tracer simulates a packet through every processing phase and reveals exactly which policy, NAT rule, or inspection engine allows or drops the traffic. Use the detailed keyword for full internal flow information and the transmit keyword to send the simulated packet to the destination.
Multiple capture points exist on Secure Firewall: Lina captures for data plane traffic, Snort captures for inspection engine traffic, ASP drop captures for silently dropped packets, and internal switch captures for chassis-level visibility on 4100, 9300, 3100, and 4200 platforms.
System Support Trace with firewall-engine-debug enabled provides Snort rule evaluation details, showing you exactly how access control policies are being matched against a specific flow.
For your next step, practice combining these tools on a lab firewall — run Packet Tracer while simultaneously collecting captures with trace to build confidence in correlating output from multiple diagnostic sources.