Question 1

What causes asymmetric routing in VPC peering, and how do you fix it while maintaining internet egress for both VPCs?

Accepted Answer

Asymmetric routing occurs because the partner VPC has a default route (0.0.0.0/0) pointing to an Internet Gateway or NAT Gateway, and the return traffic follows that default route instead of the peering connection. In AWS, VPC peering is not transitive and does not override existing routes unless the route table explicitly prefers the peering connection prefix. To fix it, I ensure that the partner's route table has a **specific route** for the company's /16 pointing to the peering connection (`pcx-xxxx`), and this route must have a longer prefix than 0.0.0.0/0. Since both are /16s, the specific route wins by being more specific only if it is an exact match or summary. The real issue is that the partner may have multiple route tables — one for public subnets with IGW and one for private subnets without. The peering route must be in the **private subnet route tables** where the application lives, not just the public route table. I also verify that **Security Groups** and **Network ACLs** allow return traffic on the ephemeral ports. For internet egress, both VPCs keep their 0.0.0.0/0 → NAT Gateway routes in the public route tables, but the private route tables use the peering connection for cross-VPC traffic and NAT Gateway for internet-bound traffic.

Question 2

What Azure feature enforces this, and what is the default behavior of VNet peering that you must override?

Accepted Answer

By default, Azure VNet peering allows direct spoke-to-spoke traffic if both spokes are peered to the same hub and **Use remote gateways** or **Allow forwarded traffic** is enabled. This bypasses the hub NVA. To force traffic through the NVA, I must disable direct spoke peering and instead use **User-Defined Routes (UDRs)** on every spoke subnet. The UDR sets the next-hop for other spoke prefixes to the **internal IP of the NVA** in the hub VNet. I also enable **Allow forwarded traffic** on the hub-to-spoke peering so the NVA can forward traffic between spokes. For scale, I use **Azure Virtual WAN (vWAN)** with a secured hub and Azure Firewall Manager, which eliminates manual UDR management by injecting routes automatically. However, if a third-party NVA (Palo Alto, FortiGate) is required, vWAN secured hub supports **Network Virtual Appliance in the hub** with automated route injection via BGP. I verify with `Get-AzEffectiveRouteTable` on a spoke VM and check the `NextHopType` is `VirtualAppliance`. The critical gotcha: if the NVA fails, spoke-to-spoke traffic blackholes unless I deploy the NVA in an availability set with a **load balancer front-end** and use BGP health probes to withdraw routes when the NVA is unhealthy.

Question 3

What are the three most likely causes, and how does BGP path selection on the VGW differ from a router?

Accepted Answer

First, the **VGW (Virtual Private Gateway)** may not be propagating the BGP route into the VPC route table because **Route Propagation** is disabled on the subnet's route table. I enable it with `EnableVgwRoutePropagation`. Second, the on-prem firewall or router may have an ACL blocking the return traffic from the specific /24. I verify with `traceroute` from the EC2 and check the Direct Connect CloudWatch metrics for `ConnectionError`. Third, the **Direct Connect VIF** may be advertising a more specific prefix (e.g., 10.0.1.0/24) back to on-prem, causing asymmetric routing if the on-prem router prefers the more specific over the summary. AWS VGW does not support BGP path manipulation like AS_PATH prepending or MED — it simply propagates what it receives. To influence routing, I use **Longest Prefix Match** by advertising more specific /24s from on-prem and summaries from AWS, or vice versa. I also verify that the **VGW is attached to the correct VPC** and that the Direct Connect Gateway (if used) is associated with the correct VGW. A common mistake is using a public VIF instead of a private VIF, which would route traffic over the internet for public IPs.

Question 4

What are the two architecture options, and which is more cost-effective at scale?

Accepted Answer

Option 1: **Direct Connect + ExpressRoute with a colocation exchange**. I provision an AWS Direct Connect in Equinix, cross-connect to an Azure ExpressRoute in the same Equinix facility, and route between them using a **multi-cloud router** (Aviatrix, Megaport MCR, or a physical router in the colo). This provides private connectivity with sub-2ms latency but has high upfront costs (cross-connects, port fees, router). Option 2: **Site-to-Site VPN between AWS VGW and Azure VPN Gateway** over the internet. This is cheaper but less reliable and higher latency. For scale (> 500 Mbps sustained), Option 1 is more cost-effective because Direct Connect and ExpressRoute offer per-GB pricing that is 40–70% cheaper than VPN data transfer. For the interview, I also mention **AWS Transit Gateway (TGW) inter-region peering** + **Azure Virtual WAN** as a modern alternative: if both clouds have a presence in the same region, I can use a **cloud exchange provider** like Equinix Fabric or Megaport to create a virtual cross-connect between TGW and vWAN without physical hardware. This is the most scalable option because it is API-driven and supports 10+ Gbps. I also implement **multi-cloud DNS resolution** using Route 53 Resolver endpoints and Azure Private DNS Resolver so instances in AWS can resolve Azure private endpoints by name.

Question 5

What is the most likely cause, and how does Calico network policy differ from AWS Security Groups in this scenario?

Accepted Answer

The most likely cause is that a **default-deny NetworkPolicy** exists in the `monitoring` namespace. In Kubernetes, if any NetworkPolicy selects a pod, all traffic not explicitly allowed is denied. Even if there is no deny rule, the presence of an ingress-allow policy on `monitoring` pods implicitly blocks all other ingress. I check with `kubectl get networkpolicies -n monitoring`. Calico network policies are **L3-L7 aware** and support **global network sets** (IP pools), **host endpoints**, and **egress gateways**. AWS Security Groups are L4 stateful firewalls attached to ENIs (Elastic Network Interfaces). They operate below Kubernetes and cannot see pod identities — they only see node IPs. If the EKS nodes are in a private subnet and the Security Group allows inter-node traffic on all ports, Security Groups are not the issue. However, if **Source/Destination Check** is disabled on the ENI and Calico is running in **VXLAN overlay mode**, pod-to-pod traffic is encapsulated and may bypass Security Group rules that expect node IPs. The fix is to ensure Calico is running in **Amazon VPC CNI mode** (IP-per-pod) so each pod gets an ENI and Security Group directly, or to add explicit Calico NetworkPolicy rules allowing `production` namespace labels to reach `monitoring` namespace labels on the required ports. I verify with `calicoctl get networkpolicy -n monitoring` and `calicoctl get wep` (workload endpoints).

Question 6

What Azure-specific NSG behavior causes intermittent failures, and what is the fix?

Accepted Answer

The intermittent failures are caused by **NSG flow asymmetry** or **Azure Load Balancer health probes** interfering, but the most common cause is that the App subnet NSG allows outbound TCP 1433, but the DB subnet NSG does not allow **return traffic on ephemeral ports** (1024–65535). Azure NSGs are stateful, so return traffic should be allowed automatically. However, if there is a **Network Virtual Appliance (NVA)** between App and DB, or if the traffic is hairpinning through an Azure Firewall, the stateful tracking may break because the NVA sees asymmetric flows. The fix is to ensure that the DB subnet NSG has an explicit inbound rule: `Source: VirtualNetwork`, `Destination: 10.0.3.0/24`, `Port: 1433`, `Protocol: TCP`, `Action: Allow`. I also verify that there is no **deny rule with higher priority** (lower number) blocking the App subnet. Azure NSGs process rules by priority number — a deny rule with priority 100 will override an allow rule with priority 110. Intermittency suggests that some connections work (when they hit the allow rule first) and others fail (when a higher-priority deny rule matches). I use **Network Watcher** `NSG Flow Logs` and `IP Flow Verify` to trace exactly which rule is matching the failed connections. Another cause is **SNAT port exhaustion** on the App tier if many connections are sourced from the same IP — I enable **Azure NAT Gateway** on the App subnet to provide scalable outbound SNAT.

Question 7

How do you retrofit TGW to send all traffic through an inspection VPC without re-architecting every VPC?

Accepted Answer

I use **AWS Transit Gateway inter-VPC traffic inspection** via a dedicated **inspection VPC** attached to the TGW. The architecture is: (1) Create an inspection VPC with a third-party firewall cluster (Palo Alto VM-Series, FortiGate-VM, or AWS Network Firewall) across multiple AZs. (2) Create a new TGW **Route Table** for the inspection VPC and associate all 100 VPC attachments with it. (3) In the TGW route table, set the default route (0.0.0.0/0) to point to the inspection VPC attachment. (4) Within the inspection VPC, the firewall appliances inspect traffic and route it back to the TGW via a separate TGW attachment in a **different route table** (egress table). This creates a **service VPC sandwich**: VPCs → TGW → Inspection VPC → TGW → Destination VPC. AWS calls this **Transit Gateway Appliance Mode** (enabled on the inspection VPC attachment). The key benefit is that no changes are needed in the 100 VPCs — only the TGW route table is modified. For logging, I enable **VPC Flow Logs** on the TGW and **AWS Network Firewall logging** to S3/CloudWatch. I also use **AWS Firewall Manager** to enforce that all TGW route tables in the organization use the inspection VPC as the next hop. For high availability, the firewall cluster uses **Gateway Load Balancer (GWLB)** as the TGW attachment target, allowing automatic failover between firewall instances.

Question 8

What is the correct hybrid DNS architecture, and how do you configure conditional forwarders in both directions?

Accepted Answer

The correct architecture is a **bi-directional DNS resolver chain**. On the Azure side, I deploy **Azure Private DNS Resolver** in a dedicated `dns-resolver` subnet with inbound and outbound endpoints. The inbound endpoint receives DNS queries from on-prem via ExpressRoute or VPN. The outbound endpoint forwards queries to on-prem DNS servers using **DNS forwarding rulesets**. On the on-prem side, I configure the AD DNS servers with **conditional forwarders** for the Azure private DNS zones (e.g., `prod.azure.nhprep.com`) pointing to the Azure Private DNS Resolver inbound endpoint IPs. For Azure VMs to resolve on-prem names, the Azure VNet DNS settings point to the Private DNS Resolver outbound endpoint, which forwards to the on-prem DNS servers via the hybrid connection. I also configure **Private Endpoint DNS integration** so that Azure PaaS services (Storage, SQL, Key Vault) resolve to their private IPs within the VNet using Azure Private DNS Zones (`privatelink.blob.core.windows.net`). The critical detail: **never use the same DNS server IP for both inbound and outbound** — this creates loops. I use separate NICs or at least separate IPs for the resolver endpoints. I verify resolution with `nslookup` from both sides and use `Resolve-DnsName` in PowerShell. For troubleshooting, I enable DNS query logging on both the on-prem DNS servers and the Azure DNS Resolver diagnostics.

Question 9

What ALB/target group setting causes 502s during deployments, and what is the graceful shutdown fix?

Accepted Answer

The 502s occur because the ALB is still sending requests to ECS tasks that are in the process of shutting down but have not yet deregistered from the target group. By default, ECS stops the task immediately when a new deployment starts, and the ALB takes 30–60 seconds to notice the target is unhealthy. The fix is to implement **graceful shutdown** with proper **deregistration delay** and **connection draining**. First, I set the target group **deregistration delay** to 60 seconds so the ALB stops sending new connections to the task immediately upon deregistration, while allowing existing connections to complete. Second, I configure the ECS task with a **stopTimeout** of 60 seconds and a **healthCheckGracePeriod** of 120 seconds. Third, in the application code, I implement a **SIGTERM handler** that sets the task to unhealthy (returning 503 on `/health`) upon receiving the termination signal, then waits 30 seconds before shutting down. This signals the ALB to drain connections early. I also enable **ALB access logs** to confirm the 502s are coming from specific target IPs during deployment windows. If the application uses WebSockets, I ensure the ALB **stickiness** is disabled for the target group so new connections are not pinned to dying tasks.

Question 10

What Azure networking feature guarantees private global transit, and how does BGP communities control the path?

Accepted Answer

The feature is **Azure ExpressRoute Global Reach**. It connects two ExpressRoute circuits in different regions via Microsoft's private backbone, ensuring traffic never leaves the Microsoft network. Global Reach uses **BGP peering** between the two ExpressRoute circuits, with Microsoft managing the private transit. To control the path and prioritize one region over another, I use **BGP communities**. I tag routes advertised from West Europe with community `12076:51001` (premium local) and routes from Southeast Asia with `12076:51002`. On the on-prem BGP router, I apply a route-map that prefers the community associated with the primary region. During failover, if the primary region's ExpressRoute circuit goes down, the BGP session drops, the route is withdrawn, and traffic automatically follows the secondary region's community-tagged path. For additional resilience, I deploy **Azure Virtual WAN** with a global transit architecture — vWAN provides built-in any-to-any connectivity across regions without requiring ExpressRoute Global Reach, but it uses the Microsoft backbone similarly. I verify the private path with `tracert` from a VM in West Europe to a VM in Southeast Asia — the hops should all be within `10.x.x.x` (Microsoft internal) and never hit public IPs.

Question 11

What three optimizations reduce plan time and state file size for large VPC deployments?

Accepted Answer

First, **modularize the Terraform code**. Instead of one monolithic root module, I create a `vpc` module that uses `for_each` over a map of subnet definitions. This reduces resource count in the root state and allows parallel evaluation. Second, **use `-refresh=false`** during plan in CI/CD if the infrastructure has not changed externally. Refresh is the slowest part of Terraform for large VPCs because it queries every subnet, route table, and ENI. I run a scheduled `terraform refresh` job nightly and use `terraform plan -refresh=false` for PR validation. Third, **migrate to Terraform 1.5+ with `terraform test`** and **state partitioning**. I split the 18 subnets into separate state files by environment or tier using `terraform_remote_state` data sources. For example, `web-tier.tfstate`, `app-tier.tfstate`, and `db-tier.tfstate` each manage 6 subnets. This reduces individual state files to ~15MB and plan times to under 2 minutes. I also use **`parallelism=10`** (default is 10, but I increase to 30 for read-only plans) to speed up API calls. For AWS specifically, I enable **AWS SDK retry and backoff tuning** in the provider block to handle throttling gracefully. Finally, I consider **AWS CloudFormation or CDK** for the initial VPC creation and use Terraform only for workload resources, since CloudFormation handles large VPCs more efficiently due to native AWS optimization.

Question 12

What is the missing AWS component, and why does a public subnet not solve this for Lambda?

Accepted Answer

The missing component is a **NAT Gateway** in the public subnet, with the Lambda placed in a **private subnet** whose route table has 0.0.0.0/0 → NAT Gateway. Lambda functions in a VPC do not receive public IPs, even if the subnet is public (has an IGW route). This is a fundamental Lambda behavior: VPC-attached Lambdas use **Elastic Network Interfaces (ENIs)** in the subnet, and ENIs in a public subnet without public IPs cannot reach the internet. A public subnet only works for resources with public IPs (EC2 with Elastic IP, ALB). The fix is: (1) Place the Lambda in a private subnet; (2) Create a NAT Gateway in a public subnet; (3) Update the private subnet route table with 0.0.0.0/0 → NAT Gateway; (4) Ensure the Lambda's Security Group allows outbound HTTPS to 0.0.0.0/0. For higher availability, I deploy NAT Gateways in multiple AZs and use one per AZ. For cost optimization at low scale, I use a **NAT Instance** instead of NAT Gateway, though it requires manual HA configuration. An alternative is **AWS PrivateLink** or **VPC Endpoints** for the third-party API if they support it, which avoids the internet entirely. For S3 and DynamoDB, I use **Gateway VPC Endpoints** which are free and do not require NAT.

Question 13

What is the root cause, and how do you design the VNet CIDR to support 10,000 pods?

Accepted Answer

The root cause is **IP address exhaustion** in the AKS node subnet. Azure CNI assigns a real VNet IP address to every pod (not an overlay IP). By default, each node reserves a /24 (256 IPs) from the node subnet for its pods. If the node subnet is a /23 (512 IPs), it can only support ~2 nodes (512 / 256 = 2), which is clearly insufficient. To support 10,000 pods, I need at least a /17 (32,768 IPs) for the pod subnet, or better, I redesign using **Azure CNI Overlay** or **Cilium** which uses an overlay network and only allocates node IPs from the VNet. For Azure CNI without overlay, the formula is: `Pod subnet size = max_pods_per_node × number_of_nodes + buffer`. For 200 nodes × 30 pods/node = 6,000 pods, I need a /17 or larger. I also enable **node subnet autoscaling** (preview feature) so AKS can automatically expand the subnet by delegating additional CIDRs. For the interview, I mention that **Azure CNI Powered by Cilium** (available in AKS 1.27+) combines the performance of native CNI with the scalability of overlay, and it supports **eBPF-based network policies** which are more efficient than iptables. I verify the IP allocation with `kubectl get nodes -o json | jq '.items[].spec.podCIDR'` and check Azure subnet usage with `az network vnet subnet show`.

Question 14

What AWS service enforces this at scale, and how do you handle legitimate bastion host exceptions?

Accepted Answer

The service is **AWS Firewall Manager** integrated with **AWS Config Rules** (specifically the `restricted-ssh` managed rule). I create a Firewall Manager policy that applies the Config rule to all Organization accounts and all VPC Security Groups. The rule detects any ingress rule with 0.0.0.0/0 on TCP 22 and marks it non-compliant. For legitimate bastion hosts, I use **tag-based exemptions**. I tag the approved bastion Security Groups with `Exception: Bastion-Approved` and configure the Firewall Manager policy to ignore resources with that tag. I also require that bastion hosts are in a dedicated **bastion VPC** peered to workload VPCs, and they use **Session Manager** (AWS Systems Manager) instead of direct SSH, eliminating the need for open port 22 entirely. For enforcement, I set the Config rule to **auto-remediate** by invoking a Lambda function that removes the offending ingress rule and notifies the account owner via SNS. The remediation Lambda also checks CloudTrail to identify who created the rule. For audit purposes, I export AWS Config compliance data to **Security Hub** and generate monthly reports for the CISO.

Question 15

What is the network architecture, and what BGP/feature behavior ensures sub-5-minute failover?

Accepted Answer

The architecture is **active-active multi-region with Route 53 health checks and BGP anycast**. In Region A (primary), I deploy the application across 3 AZs with an ALB and AWS Global Accelerator. In Region B (standby), I deploy an identical stack. Both regions advertise the same **anycast IP ranges** via BGP to AWS Global Accelerator or to Direct Connect partners. AWS Global Accelerator uses **BGP health checks** and anycast steering to route users to the healthy region automatically. For private connectivity between regions, I use **Transit Gateway inter-region peering** with BGP enabled. Both TGWs advertise the same summarized CIDRs to on-prem via Direct Connect, but with different BGP attributes: Region A uses AS_PATH prepend `65001 65001` (shorter = preferred), while Region B uses `65001 65001 65001 65001` (longer = backup). If Region A fails, the BGP session to its TGW drops, the route is withdrawn, and on-prem routers converge to Region B within 2–3 BGP scan cycles (~30–60 seconds). For RPO < 1 minute, I use **DynamoDB Global Tables** and **Aurora Global Database** for data replication. The network layer itself does not affect RPO — it only needs to converge fast enough to meet RTO. I also enable **Health-based routing in Route 53** with failover records that switch DNS in under 60 seconds. For the interview, the key phrase is: "Network convergence is bounded by BGP timers; I tune hold-time to 9 seconds and use BFD on Direct Connect for sub-second failure detection."

Cloud Networking Interview Questions: AWS, Azure & Hybrid Connectivity Scenarios

More Cloud Networking Interview Prep