Question 1

Why did it fail, and how do you rewrite the task to be idempotent across the entire fleet?

Accepted Answer

The playbook failed because `ios_config` with `lines` is not fully idempotent when the existing configuration has overlapping but non-identical lines. If the router already has `ntp server 10.0.0.1`, adding `ntp server 10.0.0.2` with `lines` may trigger a change every time the playbook runs, or it may conflict if the `replace` parameter is not set correctly. The fix is to use `ios_config` with `parents` and `lines` together, or better, use **`ios_ntp`** module which is purpose-built and idempotent. Even better, I use **`netconf_config`** or **`restconf`** with a YANG data model because they enforce atomic configuration changes. For maximum safety, I first run the playbook in **check mode** (`--check --diff`) to see what would change. I also use **`backup: yes`** to snapshot the running-config before any change. In production, I wrap the playbook in a **Molecule test** with a virtual IOS node to validate idempotency before touching real hardware. The key interview phrase: "I never run `ios_config` in production without `--check` first, and I always validate in a lab pipeline."

Question 2

What is the robust replacement for screen-scraping `show` commands, and how do you handle devices that do not support modern APIs?

Accepted Answer

The robust replacement is **YANG models via NETCONF or RESTCONF**. Instead of parsing human-readable output, I query the standardized YANG model (e.g., `ietf-interfaces` or `Cisco-IOS-XE-native`) using `ncclient` in Python. The XML/JSON response is structured and unaffected by CLI formatting changes. For devices that do not support NETCONF (legacy switches, older ASA models), I maintain a **fallback strategy**: first try NETCONF, then RESTCONF, then `pyATS/Genie` parsing, then TextFSM as the last resort. I implement this as an **abstraction layer** in my Python library so the calling code does not care which transport is used. Specifically, I use **NAPALM** which handles this multi-backend abstraction natively — `napalm.get_interfaces_ip()` works across IOS, JunOS, and NXOS with the same Python call. For the interview, I also mention **grpc/gNMI** for telemetry and configuration on modern IOS-XE and NXOS devices, as it is even faster than NETCONF for high-frequency operations.

Question 3

What three pipeline safeguards prevent this, and what is the rollback mechanism?

Accepted Answer

First, **pre-commit hooks and CI linting**. I use `yamllint` for Ansible playbooks, `ansible-lint` for best practices, and `batfish` or `cisco.nxos.nxos_command` validation in a staging environment before any merge to `main` is allowed. Second, **canary deployment**. Instead of pushing to all 12 routers at once, the pipeline targets a `canary` group of 2 routers first, runs connectivity tests for 15 minutes, and only then proceeds to the remaining 10. Third, **automated rollback via configuration archive**. Before any change, the pipeline triggers `archive config` or `copy running-config startup-config` with a timestamp, and stores the backup in Git or TFTP. If the post-change health check (ping, BGP neighbor count, management SSH) fails, the pipeline automatically runs a `configure replace` with the backup config. For the 12 locked-out routers, I use **out-of-band console servers** (Opengear, Lantronix) that the pipeline can access via a separate management network to push the rollback config even when in-band SSH is blocked. I also implement a **kill switch** — a Jenkins parameter `EMERGENCY_STOP` that any on-call engineer can set to halt all pipeline executions immediately.

Question 4

How do you enforce that NetBox is always the source of truth and prevent manual out-of-band changes?

Accepted Answer

The root cause is that the router allows manual CLI changes that are not synchronized back to NetBox. I implement three controls: (1) **Configuration Drift Detection** — a daily Ansible playbook runs `show ip interface brief` on all routers, compares the output with NetBox using `netbox.netbox.netbox_ip_address` facts, and opens a ServiceNow ticket for any mismatch. (2) **GitOps-only provisioning** — engineers no longer log into routers directly. All changes are made as NetBox IP/VLAN requests that trigger a pipeline. The pipeline generates the config fragment, pushes it via NETCONF, and updates the router's intended state in Git. (3) **Event-driven synchronization** using **Nautobot Apps** or **NetBox webhooks**. When an IP is created in NetBox, a webhook triggers the provisioning pipeline automatically. When a device is updated via Ansible, the playbook calls the NetBox REST API to sync the change back. For devices that absolutely require emergency CLI access, I enable **AAA logging** to a SIEM and alert the automation team within 5 minutes of any manual login. The key phrase: "The router is a puppet; NetBox is the puppeteer."

Question 5

Why is the subnet missing the association despite Terraform reporting success, and how do you debug it?

Accepted Answer

This is likely a **race condition** or **implicit dependency bug**. In Terraform, `aws_route_table_association` depends on both the subnet and the route table existing. If the association resource does not explicitly declare `depends_on = [aws_route_table.main]`, Terraform may try to create the association before the route table is fully propagated by the AWS API, causing a silent failure. AWS returns a 200 OK but the association is not actually established. To debug, I run `terraform show` and `terraform state list` to verify the association resource ID. Then I use the AWS CLI to check: `aws ec2 describe-route-tables --route-table-ids rtb-xxxxx` and look for the `Association` block. If missing, I taint the association resource (`terraform taint aws_route_table_association.example`) and re-apply. For prevention, I always use explicit `depends_on` for resources that have eventual consistency APIs, and I enable **`terraform plan -refresh-only`** as a pre-apply check in CI. I also use **`check` blocks** (Terraform 1.5+) to assert that every subnet has a route table association after apply, causing the pipeline to fail if the assertion is violated.

Question 6

Design an Ansible playbook that standardizes all three items safely, with rollback capability, for a mixed fleet of IOS, NXOS, and ASA devices.

Accepted Answer

I use a **multi-platform playbook** with `ansible_network_os` conditionals. The inventory groups devices by `ansible_network_os`: `ios`, `nxos`, `asa`. The playbook has three tasks: (1) **Banner** — use `cisco.ios.ios_banner` for IOS, `cisco.nxos.nxos_banner` for NXOS, and `cisco.asa.asa_banner` for ASA, with `text: "Authorized Access Only"`. (2) **SNMP** — use `ios_snmp_server`, `nxos_snmp_server`, and `asa_snmp_server` modules with a loop over `snmp_communities` and `snmp_hosts` variables. (3) **AAA users** — use `ios_user`, `nxos_user`, and `asa_user` with `privilege 15` and `ssh_key` for key-based auth. For safety, I run the playbook with `serial: 5` so only 5 devices are touched at a time. Before each task, I use `ios_command` to snapshot the config: `show running-config | section snmp` and save it to a local file with `backup: yes`. For rollback, I create a `rollback.yml` playbook that reads the backup files and uses `configure replace` (IOS), `rollback` (NXOS), or `clear configure` + restore (ASA). I also use **`block/rescue/always`** in Ansible so that if any device fails, the rescue block runs the rollback task for that specific host, and the `always` block verifies SSH reachability.

Question 7

How do you reduce the runtime to under 2 minutes without adding more CPU cores?

Accepted Answer

The bottleneck is sequential SSH — Paramiko opens one connection at a time. I refactor the script to use **asyncio with asyncssh** or **concurrent.futures.ThreadPoolExecutor** with a pool size of 20–30. With ThreadPoolExecutor, I map the `get_version()` function across all 100 IPs simultaneously (limited by the pool size), reducing wall-clock time to roughly the duration of the slowest device plus connection overhead. For even better performance, I replace SSH with **gNMI or NETCONF** because these protocols reuse persistent connections and support request pipelining. If the devices support it, I use a single gNMI subscription to all 100 devices via a **gNMI collector** (like Telegraf or gNMIc), which reduces the runtime to under 30 seconds. For the interview, I also mention **connection caching** with `netmiko` using `ConnectHandler` in a context manager pool, or using **Nornir** which is purpose-built for parallel network automation. Nornir with 20 workers typically completes 100 devices in 45–90 seconds. The key optimization is: parallelize I/O, not CPU.

Question 8

What are the immediate remediation steps, and what three Git/tooling changes prevent this permanently?

Accepted Answer

Immediate steps: (1) Rotate all exposed passwords within 1 hour; (2) Use `git filter-repo` or BFG Repo-Cleaner to purge the commit from Git history entirely — simply deleting the file in a new commit leaves it in history forever; (3) Audit GitHub access logs to see if the repository was cloned or forked externally. Permanent prevention: (1) **pre-commit hooks** with `git-secrets` or `truffleHog` that scan for password patterns, private keys, and API tokens before any commit is allowed; (2) **Ansible Vault** or **HashiCorp Vault** integration — all secrets are encrypted with `ansible-vault encrypt` and the vault password is stored in a CI/CD secret manager, never in Git; (3) **Branch protection rules** on `main` that require 2 code reviews, passing CI checks (including secret scanning), and status checks before merge. I also implement **GitHub Advanced Security** or **GitLab Secret Detection** in the pipeline. For extra defense, I use **conftest/OPA** policies that reject any playbook containing unencrypted `password:` or `community:` strings during the CI plan stage.

Question 9

What modern telemetry protocol replaces SNMP polling, and how does the architecture differ?

Accepted Answer

The replacement is **gNMI (gRPC Network Management Interface)** or **model-driven telemetry (MDT)** using NETCONF notifications. Unlike SNMP polling where the server asks each switch "what is your counter?" every 30 seconds, gNMI uses a **subscription model**: the switch pushes data to a collector (Telegraf, Prometheus, or a custom gNMIc server) when the counter changes or at a configured cadence. This eliminates the monitoring server bottleneck because the server is no longer opening 1,000 TCP sessions every 30 seconds — it maintains persistent gRPC streams and receives data passively. The architecture uses a **dial-out** model: each switch dials out to a centralized gNMI collector. For Cisco IOS-XE, I enable `telemetry ietf subscription 100` with a 30-second interval and stream to a Kafka or InfluxDB backend. I also mention that **SNMP traps** are push-based but lossy and unidirectional, while gNMI is reliable (over HTTP/2), bidirectional, and supports ON_CHANGE subscriptions so data is only sent when the interface state actually changes. For non-Cisco devices, I use **OpenConfig YANG models** with gNMI to achieve vendor-agnostic telemetry.

Question 10

What Jinja2 filter and Ansible strategy prevent undefined variables from breaking configuration rendering?

Accepted Answer

The fix is the **`default` filter** combined with **`trim_blocks` and `lstrip_blocks`** in Jinja2. In the template, I write `{{ ntp_server | default('10.0.0.1') }}` so undefined variables fall back to a safe default. For optional configuration blocks, I use `{% if ntp_server is defined %}` guards so the block is only rendered when the variable exists. To prevent empty lines, I enable `trim_blocks: True` and `lstrip_blocks: True` in the Ansible `template` module, which removes whitespace around Jinja2 tags. For stricter safety, I set `error_on_undefined_vars = True` in `ansible.cfg` so the playbook fails fast during template rendering rather than pushing a broken config to the device. I also use **Ansible `validate`** parameter on the `template` task: `validate: '/usr/bin/visudo -cf %s'` for sudoers, or for Cisco configs, I copy the rendered config to flash and run `show archive config diff` before applying. The best practice is a **JSON Schema validation** step in CI that validates the variable file against a schema before Ansible even starts.

Question 11

What is the correct Terraform backend setup for team-based network infrastructure, and how do you secure it?

Accepted Answer

The correct backend is **remote state with locking**, typically Amazon S3 for state storage and DynamoDB for state locking. I configure the backend in `backend.tf`: `terraform { backend "s3" { bucket = "nhprep-terraform-state", key = "network/firewall.tfstate", region = "ap-south-1", dynamodb_table = "terraform-locks", encrypt = true } }`. The S3 bucket has **versioning enabled** so previous state files are recoverable, and **server-side encryption** with KMS so the state is encrypted at rest. The DynamoDB table ensures that only one engineer can run `terraform apply` at a time, preventing race conditions. For security, I enforce **least-privilege IAM roles**: engineers have read access to the state but only the CI/CD pipeline role has write access. I also enable **Terraform Cloud** or **Terraform Enterprise** for RBAC, run history, and cost estimation. For network-specific secrets (API keys, router passwords), I use **HashiCorp Vault** as the secrets backend, not Terraform variables. I never store credentials in `terraform.tfvars` or state files. I also run **`terraform plan` in CI on every pull request** so the team reviews infrastructure changes before they are applied.

Question 12

What are the three ways to fix this, ranked from best to worst practice?

Accepted Answer

Best practice: (1) **Add the controller's CA certificate to the system trust store** or the Python `certifi` bundle. I export the controller's root CA cert, place it in `/etc/ssl/certs/` or `C:\Python\Lib\site-packages\certifi\cacert.pem`, and the script verifies the chain without code changes. I can also pass `verify=/path/to/ca-bundle.crt` in the `requests.get()` call. (2) **Use a dedicated certificate bundle** for the application — I create `controller-ca.crt` in the project directory and set `verify='./controller-ca.crt'`. This is isolated from the system and works in containers. Worst practice: (3) **`verify=False`** — this disables TLS verification entirely, exposes the script to MITM attacks, and should only be used for 30 seconds during emergency debugging. For the interview, I also mention **certificate pinning** as an additional security layer: the script hashes the controller's leaf certificate and refuses to connect if the fingerprint changes, even if the CA is valid. This prevents rogue CA attacks. I also use `urllib3` warnings suppression carefully: `urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)` should never be committed to production code.

Question 13

Design a Python + SNMP/gNMI solution that detects, verifies, and remediates without causing a loop or outage.

Accepted Answer

I build an **event-driven remediation pipeline** using Python, Telegraf, and Ansible. Step 1: **Detection** — Telegraf collects interface counters every 60 seconds via gNMI or SNMP and writes to InfluxDB. A Kapacitor or custom Python daemon queries InfluxDB and calculates `delta(input_errors) / delta(input_packets)` over a 5-minute window. If > 1%, it publishes a message to a Redis queue. Step 2: **Verification** — The remediation worker consumes the message and runs a Python script that checks three things via NETCONF: (a) the interface is an access port (not a trunk to another switch), (b) the interface is not a port-channel member, (c) no critical device is connected (I maintain a MAC allowlist of servers/phones). Step 3: **Remediation** — If all checks pass, the worker calls Ansible to run `ios_command` with `shutdown` followed by `no shutdown` 10 seconds later. Step 4: **Validation** — After the bounce, the worker polls the interface for 3 minutes and verifies the error counter has reset. If errors persist, it opens a P1 ticket in ServiceNow and leaves the port shut down. To prevent loops, I never auto-remediate trunk ports or ports with BPDU Guard disabled. I also implement a **circuit breaker**: if more than 5 ports on the same switch auto-bounce within 10 minutes, the worker stops and alerts the NOC because the issue is likely a bad cable or SFP, not individual port flakiness.

Question 14

How do you consolidate inventory management at scale, and what tool replaces flat inventory files?

Accepted Answer

Flat inventory files do not scale because they create variable precedence conflicts and duplication. I replace them with **NetBox or Nautobot as the dynamic inventory source**. I install `ansible-netbox` or `netbox.netbox` collection and configure `ansible-inventory --list` to query NetBox's REST API at runtime. NetBox becomes the single source of truth: devices, sites, racks, IP addresses, and custom fields (like `ansible_network_os`, `dc_tier`, `maintenance_window`) all live in one database. The Ansible inventory script groups devices by site, role, platform, and tags dynamically. For the 20-DC scale, I also implement **inventory caching** with `cache: yes` and `cache_plugin: jsonfile` so the API is not hammered on every playbook run. For offline execution, I use `ansible-inventory --export` to snapshot the dynamic inventory to a JSON file. I also mention **Ansible Execution Environments** (AWX/Tower) where the inventory is synced from NetBox on a schedule, and job templates reference the synced inventory. This eliminates local inventory files entirely. For variable precedence, I rely on NetBox config contexts which are hierarchical: global > region > site > device, matching how network variables actually cascade in production.

Question 15

What three metrics prove the value of network automation to executive leadership?

Accepted Answer

Metric 1: **Change Failure Rate (CFR)**. Before automation, 15% of manual config changes caused incidents. After automation with Molecule testing and canary deployments, CFR drops to < 2%. I collect this from the ITSM tool (ServiceNow) by tagging incidents with `root_cause: config_change`. Metric 2: **Deployment Lead Time**. A firewall rule change used to take 14 days (ticket → review → change window → implementation). With GitOps, it takes 4 hours (PR → CI validation → auto-deploy). I measure this from Git merge timestamp to production apply timestamp. Metric 3: **Configuration Drift Percentage**. Using nightly Ansible/NetBox reconciliation scans, I report what percentage of the fleet matches the intended state. Before automation, drift was 35% due to manual changes. After enforcing GitOps-only, drift is < 3%. For the CISO dashboard, I normalize these into a **Network Automation Maturity Score** (0–100) weighted by business impact. I also track **Mean Time to Rollback (MTTRb)** — how quickly we can undo a bad automated change. A low MTTRb (under 5 minutes) proves automation is safely reversible. Finally, I correlate automation coverage with **audit findings**: networks under full automation have 80% fewer security compliance findings than manually managed networks.

Network Automation Interview Questions: Ansible, Python & NetDevOps Scenarios

More Network Automation Interview Prep