Question 1

What is missing in the routing table, and how do you add a persistent route for 10.1.0.0/16 via eth1?

Accepted Answer

The routing table only has a connected route for 10.0.0.0/24 on eth1. Since there is no route for 10.1.0.0/16, Linux uses the default route via eth0 (192.168.1.1), which does not have a path to 10.1.0.0/16. The fix is to add a static route: `ip route add 10.1.0.0/16 via 10.0.0.1 dev eth1`. To make it persistent across reboots on modern systems (Ubuntu 18.04+, RHEL 8+), I edit `/etc/netplan/01-netcfg.yaml` for Netplan or create `/etc/sysconfig/network-scripts/route-eth1` on RHEL with `10.1.0.0/16 via 10.0.0.1`. I verify with `ip route show` and `ip route get 10.1.0.1`. For policy-based routing (if the server needs to source traffic from eth1 for specific destinations), I use `ip rule add from 10.0.0.10 table 100` and `ip route add 10.1.0.0/16 via 10.0.0.1 table 100`. This ensures that even if the default route exists, traffic sourced from eth1 uses eth1's routing table.

Question 2

How do you capture only the TCP handshake, HTTP headers, and connection teardown for a specific client IP, while rotating files every 100 MB?

Accepted Answer

I use a targeted tcpdump command with BPF (Berkeley Packet Filter) syntax and rotation: `tcpdump -i eth0 -nn -s 0 -A -w /var/capture/http_debug.pcap -C 100 -W 5 host 203.0.113.50 and port 443 and '(tcp[tcpflags] & tcp-syn != 0 or tcp[tcpflags] & tcp-fin != 0 or tcp[tcpflags] & tcp-rst != 0 or tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450)'`. Let me break this down: `-nn` prevents DNS resolution; `-s 0` captures full packets; `-A` prints ASCII (for headers); `-C 100 -W 5` rotates at 100MB and keeps 5 files. The BPF filter `host 203.0.113.50 and port 443` isolates the client. The TCP flags filter captures SYN, FIN, RST. The `tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450` magic detects HTTP by looking for "HTTP" at the TCP payload start. For TLS, I capture only the **Client Hello** and **Server Hello** by matching the TLS record type (0x16 = handshake) and content type. For 502 debugging, I prefer to use `tshark` with display filters: `tshark -i eth0 -Y "http.response.code == 502" -T fields -e ip.src -e ip.dst -e http.host`.

Question 3

What is the most likely cause, and what is the correct FRR daemon startup sequence?

Accepted Answer

In FRR, `bgpd` depends on `zebra` but also requires the **vtysh** configuration to have `hostname` and `password` set in `/etc/frr/vtysh.conf`, and the daemons must be enabled in `/etc/frr/daemons`. If `bgpd=no` in `/etc/frr/daemons`, the systemd service will not start bgpd. The correct startup sequence is: (1) `zebra` — manages the kernel routing table; (2) `bgpd` — BGP daemon; (3) `ospfd`/`ospf6d` — if needed; (4) `staticd` — for static routes. I verify with `systemctl status frr` and `vtysh -c "show bgp summary"`. If bgpd is enabled but not starting, I check `journalctl -u frr` for errors like "bind failed" (port 179 in use) or "configuration file error". A common issue is that the previous admin ran `vtysh -b` (boot configuration) but the config was not saved to `/etc/frr/frr.conf`. I ensure the running config is saved with `write memory` or `copy running-config startup-config` in vtysh. For production routers, I also enable **watchfrr** which monitors daemon health and restarts crashed daemons automatically.

Question 4

What two Linux networking features must be enabled for the DHCP relay to work across VLANs, and how do you verify DHCP relay forwarding?

Accepted Answer

First, the **VLAN interface** must be properly created and up: `ip link add link eth0 name eth0.100 type vlan id 100` and `ip link set eth0.100 up`. Second, the **DHCP relay agent (isc-dhcp-relay or dhcrelay)** must be configured to listen on both the client-facing interface (eth0.100) and the server-facing interface (eth1). The command is `dhcrelay -i eth0.100 -i eth1 10.0.0.10` where 10.0.0.10 is the DHCP server. The most common misconfiguration is forgetting to enable **IP forwarding** (`sysctl net.ipv4.ip_forward=1`) or a firewall rule blocking UDP 67/68 on the VLAN interface. I verify with `tcpdump -i eth0.100 port 67 or port 68` to see if DISCOVER packets arrive from clients. If they arrive but no OFFER is returned, I check `tcpdump -i eth1 port 67 or port 68` to confirm the relay is forwarding to the server. I also check `journalctl -u isc-dhcp-relay` for errors like "no subnet declaration for eth0.100". If the DHCP server is on a different subnet, the server must have a **helper-address** or **subnet declaration** for the 10.0.100.0/24 range. On the Linux relay itself, I ensure the `iptables` allow rule exists: `iptables -A INPUT -p udp --dport 67:68 -j ACCEPT`.

Question 5

What are the three ways to solve this, and which is the most secure for production network automation?

Accepted Answer

Option 1: **Host networking** (`--network host`). The container shares the host network namespace and can reach 10.255.0.0/24 directly. This is the least secure because the container has full network access and may conflict with host ports. Option 2: **Custom bridge network with static routes**. I create a custom bridge `docker network create --subnet 172.30.0.0/16 automation-net` and add a route on the host: `ip route add 10.255.0.0/24 via 172.30.0.1`. Then I add a route on the management network router pointing 172.30.0.0/16 back to the Docker host. This isolates automation traffic but requires route management. Option 3 (most secure): **Macvlan network**. I create a macvlan interface on the host's eth0: `docker network create -d macvlan --subnet 10.255.0.0/24 --gateway 10.255.0.1 -o parent=eth0 mgmt-net`. The container gets its own MAC address and IP on the 10.255.0.0/24 network, appearing as a physical host to the management network. This provides Layer 2 isolation and allows firewall rules per container IP. For extra security, I use **IPvlan L3** which removes the MAC address visibility but still provides routable IPs. I verify with `docker network inspect mgmt-net` and `ping 10.255.0.10` from inside the container.

Question 6

What LACP negotiation mismatch causes this, and what is the fix on both Linux and the switch?

Accepted Answer

The mismatch is almost certainly **LACP rate** (slow vs fast). Linux bonding defaults to **slow LACP** (30-second transmit interval), while Cisco switches default to **fast LACP** (1-second). If one side is fast and the other is slow, they still form a bundle, but some switch platforms mark the slow partner as `I` (inactive) or display warnings. The real issue may be that the switch is using **passive LACP** and Linux is also passive, so no LACPDU exchange occurs. The fix: on Linux, set `lacp_rate=fast` in `/etc/sysconfig/network-scripts/ifcfg-bond0` or `bond0` config: `echo fast > /sys/class/net/bond0/bonding/lacp_rate`. On the Cisco switch, configure `lacp rate fast` on the port-channel member interfaces. I also verify that the **port-channel mode is active** (`channel-group 1 mode active`) so the switch initiates LACPDU. Another cause is **MTU mismatch** — if eth0 has MTU 1500 and eth1 has MTU 9000, LACP may form but higher-layer traffic is blackholed. I check `ip link show` and `cat /sys/class/net/bond0/bonding/miimon`. For production, I also enable `xmit_hash_policy=layer3+4` so traffic is distributed across both links based on src/dst IP and port, not just MAC address.

Question 7

How does Linux bridging differ from switch ASIC-based VLAN handling, and what command verifies VLAN membership on a bridge port?

Accepted Answer

Linux bridging (`bridge-utils` or `iproute2`) handles VLANs in software unless the NIC supports hardware offloading (SR-IOV, switchdev). In a Linux NOS like Cumulus or SONiC, VLANs are typically implemented using **bridge VLAN filtering** (`bridge vlan add`) rather than traditional `vconfig` or `ip link add type vlan`. The bridge itself is the VLAN-aware entity, and ports are assigned to VLANs with PVID (native) and tagged memberships. To verify, I use `bridge vlan show` which displays the VLAN table per port, similar to `show vlan brief` on Cisco. If a trunk port is missing a VLAN, I add it with `bridge vlan add dev swp1 vid 10-20` for tagged VLANs 10–20. The key difference from switch ASICs is that Linux bridges use the **kernel forwarding path** unless `switchdev` offloading is enabled. On Mellanox/Cumulus, I enable `hw-offload` to push VLAN filtering to the Spectrum ASIC. I verify offloading with `ethtool -k swp1 | grep hw-tc-offload`. For SONiC, VLAN handling is abstracted through `config vlan add` and `config vlan member add`, which translates to Linux bridge commands under the hood. If traffic is still not flowing, I check `bridge link show` for STP state — Linux bridges run STP by default, and a port may be `disabled` or `blocking`.

Question 8

What are the three ways to implement this, and which is the most reversible for temporary blocks?

Accepted Answer

Option 1: **iptables** (legacy). `iptables -A INPUT -s 192.0.2.100 -j DROP`. To survive reboot, I save with `iptables-save > /etc/iptables/rules.v4` (on Debian) or `service iptables save` (on RHEL). Reversal: `iptables -D INPUT -s 192.0.2.100 -j DROP`. Option 2: **nftables** (modern replacement). `nft add rule inet filter input ip saddr 192.0.2.100 drop`. nftables uses atomic rule replacement and is more efficient. Reversal requires knowing the handle: `nft -a list chain inet filter input` then `nft delete rule inet filter input handle <N>`. Option 3 (most reversible): **ipset with a timeout**. I create an ipset: `ipset create tempblock hash:ip timeout 3600` and add the IP: `ipset add tempblock 192.0.2.100`. Then I add one iptables/nftables rule that references the ipset: `iptables -A INPUT -m set --match-set tempblock src -j DROP`. The IP is automatically removed after 1 hour. This is the most reversible because it self-expires. For immediate removal, `ipset del tempblock 192.0.2.100`. For the interview, I also mention **fail2ban** as the operational standard for dynamic blocks — it watches logs and auto-manages ipsets based on patterns.

Question 9

What is the most likely cause, and what is the correct namespace setup sequence?

Accepted Answer

By default, when a network namespace is created, the loopback interface is present but **not brought up**. `ping 127.0.0.1` fails because `lo` is in the `DOWN` state. The correct sequence is: `ip netns add router` → `ip netns exec router ip link set lo up` → `ip netns exec router ip addr add 127.0.0.1/8 dev lo`. After this, local loopback works. To connect the namespace to the host or other namespaces, I create a **veth pair**: `ip link add veth-host type veth peer name veth-router netns router`. Then I assign IPs and bring both ends up. For routing between namespaces, I enable IP forwarding on the host and add routes pointing to the veth interfaces. I verify with `ip netns exec router ip addr show` and `ip netns exec router ip route show`. A common mistake is trying to use `ping` from the host to the namespace without a veth or without assigning a routable IP to the namespace. Another issue is that **iptables rules in the host** may block traffic to the namespace's IP range if the `FORWARD` chain has a default DROP policy.

Question 10

What ethtool diagnostics reveal the root cause, and what are the three most common fixes?

Accepted Answer

I start with `ethtool -S eth0` to get detailed NIC statistics: `rx_missed_errors`, `rx_length_errors`, `rx_crc_errors`, `rx_fifo_errors`. If `rx_crc_errors` is high, it is a Layer 1 issue (cable, SFP, or speed/duplex mismatch). If `rx_fifo_errors` or `rx_missed_errors` is high, the NIC ring buffer is overflowing because the CPU cannot drain packets fast enough. I check `ethtool -g eth0` for ring buffer sizes and increase with `ethtool -G eth0 rx 4096 tx 4096`. I also check `ethtool -c eth0` for interrupt coalescing — if too aggressive, it batches packets and causes latency; if too relaxed, it burns CPU. The three most common fixes are: (1) **Speed/duplex mismatch** — force both sides to 1000/full with `ethtool -s eth0 speed 1000 duplex full autoneg off`. (2) **Ring buffer overflow** — increase RX ring size and enable **RSS (Receive Side Scaling)** with `ethtool -L eth0 combined 8` to distribute interrupts across CPU cores. (3) **NIC driver bug or firmware issue** — update the driver (`modinfo igb` / `ethtool -i eth0`) and firmware. I also check `dmesg | grep eth0` for hardware errors. For Intel NICs, `lspci -vv | grep -A 20 Ethernet` shows PCIe link speed — if the NIC is in a x1 slot instead of x4, bandwidth is insufficient for 10Gbps.

Question 11

What Linux kernel and Syslog-NG tuning parameters prevent log drops at scale?

Accepted Answer

Log drops occur because the UDP socket buffer overflows or Syslog-NG cannot write to disk fast enough. First, I increase the kernel socket buffers: `sysctl net.core.rmem_max=134217728` and `net.core.rmem_default=134217728` (128MB). This prevents UDP drops at the kernel level. Second, I tune Syslog-NG in `/etc/syslog-ng/syslog-ng.conf`: increase `log_fifo_size(1000000)` for the source, and use `flush_lines(1000)` with `flush_timeout(1000)` for the destination file to batch disk writes. Third, I enable **disk buffering** (`disk-buffer()`) so if the destination is slow, Syslog-NG writes to a reliable disk queue instead of dropping messages. I also consider using **multiple worker threads** in Syslog-NG 4.0+: `flags(no-multi-line) worker-threads(4)`. For very high scale (> 1000 devices), I switch the source from UDP to **TCP or RELP** (Reliable Event Logging Protocol) which provides acknowledgment and retransmission. I verify drops with `netstat -su | grep RcvbufErrors` and Syslog-NG's internal stats: `syslog-ng-ctl stats | grep dropped`. If disk I/O is the bottleneck, I move the log partition to SSD or use **log aggregation** (Kafka, Fluentd) instead of flat files.

Question 12

What does this output mean, and how do you determine the actual path MTU to the destination?

Accepted Answer

`ping -M do -s 1472` sends an ICMP echo request with DF (Don't Fragment) set and a payload of 1472 bytes. Adding the 8-byte ICMP header and 20-byte IP header, the total IP packet is 1500 bytes. The `Frag needed` response means a router on the path has a link MTU smaller than 1500 bytes and cannot forward the packet without fragmenting. To find the actual path MTU, I use `tracepath 8.8.8.8` which automatically discovers the MTU by sending probes of increasing size with DF set and interpreting the ICMP unreachable messages. Alternatively, I use `ping -M do -s <size>` with a binary search: try 1472 (1500 total), if it fails try 1000 (1028 total), then narrow down. For TCP connections, Linux has **Path MTU Discovery (PMTUD)** built in, but if ICMP is blocked by a firewall, the sending host never receives the "Frag needed" message and experiences **PMTUD blackhole** — TCP hangs. The fix for PMTUD blackholes is **TCP MSS clamping** on the router or `iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1360` to force a safe MSS regardless of path MTU. I verify the path MTU with `ip route get 8.8.8.8` and check `cache` entries.

Question 13

What netfilter/iptables concept fixes asymmetric routing, and what is the conntrack role?

Accepted Answer

Asymmetric routing breaks TCP because packets from A→B take one path and B→A take another, but the Linux router in the middle may not see both directions of the flow and cannot maintain conntrack state. The fix is to ensure the Linux router is in the **forwarding path for both directions** or to disable reverse path filtering with `sysctl net.ipv4.conf.all.rp_filter=0`. However, the correct architectural fix is to make the Linux router the default gateway for both subnets so all inter-subnet traffic passes through it symmetrically. For NAT, I use **iptables conntrack (connection tracking)** which maintains a state table of all flows. When a packet leaves via SNAT (`iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE`), conntrack records the original src/dst. When the return packet arrives, conntrack reverses the NAT and forwards it to the original source. If asymmetric routing bypasses the router, conntrack never sees the return packet and the connection hangs. I verify conntrack with `conntrack -L` and `iptables -t nat -L -n -v`. For high-performance routers, I consider using **nftables** with **flowtables** which offload established connections to the NIC hardware, bypassing the kernel netfilter stack entirely.

Question 14

What Linux feature dynamically persists routes learned from routing protocols, and how does it interact with the kernel routing table?

Accepted Answer

Routing protocols (BGP, OSPF) run in user-space daemons (FRR, BIRD, Quagga) and inject routes into the kernel routing table via **netlink sockets**. The kernel does not natively persist these routes — they are lost on reboot. The daemon must re-converge after boot. FRR solves this by reading its config file (`/etc/frr/frr.conf`) at startup and re-establishing all routing sessions. The config file is the persistence mechanism, not the kernel. However, for routes learned dynamically that need to be preserved (e.g., BGP customer routes during a control plane restart), FRR uses the **Linux Routing Daemon (zebra)** which maintains the RIB (Routing Information Base) in memory and syncs it to the kernel FIB (Forwarding Information Base). For graceful restart, FRR supports **NSF (Non-Stop Forwarding)** where the data plane (kernel FIB) keeps forwarding using stale routes while the control plane (bgpd) restarts and reconverges. For static routes that must be kernel-persistent but managed dynamically, I use **`ip route save` and `ip route restore`** which serialize the routing table to a binary netlink format. Some distributions also support `/etc/network/interfaces` or Netplan for static routes, but these are not dynamic. The modern approach is **systemd-networkd** with `[Route]` sections in `.network` files, which supports dynamic route management via `networkctl`.

Question 15

What tool captures traffic per process, and how do you decrypt the TLS for analysis without the server private key?

Accepted Answer

For per-process capture, I use **`nsenter`** to enter the process network namespace and run tcpdump there, or use **`ss -tp` and `lsof -i`** to map ports to PIDs, then capture on the specific port. The best tool is **`tcpdump` with BPF** filtering by the port the process listens on, but if the process uses dynamic ports, I use **`strace -e trace=network -p 1234`** to monitor syscalls. For TLS decryption without the private key, I use **SSLKEYLOGFILE** if the application supports it (browsers, curl, OpenSSL-based apps). I set `SSLKEYLOGFILE=/tmp/ssl-keys.log` in the process environment, capture the traffic with tcpdump, and load both the pcap and the key log into Wireshark for decryption. If the application does not support SSLKEYLOGFILE, I use **`sslsplit`** or **`mitmproxy`** as a transparent proxy to intercept and decrypt traffic, though this requires installing a trusted CA on the client. For Java applications, I use **`-Djavax.net.debug=ssl,keygen`** to dump key material. Another approach is **`eBPF/bpftrace`** to trace the OpenSSL `SSL_write` and `SSL_read` functions and extract plaintext directly from memory. This is the most powerful method but requires root and kernel headers.

Linux for Network Engineers: 15 Interview Questions on CLI, Networking & Troubleshooting

More Network Automation Interview Prep