Network Troubleshooting
The layered "site is slow" methodology, port and connectivity checks, tcpdump capture, and DHCP, VLAN, ICMP and proxy fundamentals for diagnosis.
8 questions
JuniorTheoryVery commonWhat does ping test, and why doesn't a failed ping prove the host is down?
What does ping test, and why doesn't a failed ping prove the host is down?
ping sends ICMP echo-requests and waits for replies, so it tests IP-layer reachability and round-trip time, not whether a TCP service is up. A failed ping does not prove the host is down: firewalls and security groups often drop or rate-limit ICMP while TCP ports stay open, so a host can serve curl fine yet never answer ping.
Common mistakes
- ✗Reading a failed
pingas proof the host is down when ICMP is just filtered - ✗Thinking a successful
pingproves a TCP service is listening - ✗Assuming ICMP and TCP are always allowed or blocked together
Follow-up questions
- →How would you confirm a host is up when ICMP is filtered end-to-end?
- →Why do many security groups rate-limit ICMP rather than block it outright?
MiddleDebuggingVery commonRequests hit the wrong host after a migration. Diagnose the DNS output below.
Requests hit the wrong host after a migration. Diagnose the DNS output below.
The recursive resolver returns the old record 10.20.4.9 with a large remaining TTL, while the authoritative server already answers 10.20.4.11 — so it is a stale cache, not routing or the app. Fix now: flush the resolver cache or wait out the TTL; prevent it by lowering the record's TTL before a migration.
Common mistakes
- ✗Blaming the authoritative server when it already returns the correct record
- ✗Ignoring the large TTL that keeps the stale answer alive in the resolver
- ✗Reaching for routing or app config before comparing resolver and authoritative
Follow-up questions
- →Why does querying the authoritative server directly bypass the stale answer?
- →How does lowering TTL ahead of a change shrink the propagation window?
JuniorTheoryCommonHow do you diagnose an unreachable service by working up the network layers?
How do you diagnose an unreachable service by working up the network layers?
Work the stack bottom-up, one layer at a time: link (interface up?), IP (address and subnet), routing (default gateway, ip route), DNS (does the name resolve?), transport (is the port open?), then the app (does it answer, check logs). Isolating each layer shows where the break is instead of guessing.
Common mistakes
- ✗Jumping straight to the application without ruling out DNS, routing, or the port
- ✗Believing a successful
pingproves every lower layer and the port are fine - ✗Changing several things at once so you never learn which layer was broken
Follow-up questions
- →Why check DNS before the TCP port when a connection by name fails?
- →How does isolating one layer at a time speed up finding the root cause?
MiddleDebuggingCommonAn app slowly stops accepting connections. ss shows this — what is wrong?
An app slowly stops accepting connections. ss shows this — what is wrong?
Sockets pile up in CLOSE_WAIT on the app's own :8080: the peer sent FIN but the app never called close() to send its own FIN back. That is an app bug — it leaks sockets and descriptors until it hits the fd limit and stops accepting. Many TIME_WAIT, by contrast, is normal on whoever closed first.
Common mistakes
- ✗Confusing
CLOSE_WAIT(local app hasn't closed) withTIME_WAIT(normal) - ✗Treating it as kernel tuning instead of a missing
close()in the app - ✗Raising the fd
ulimitto mask a leak that keeps growing regardless
Follow-up questions
- →Why is a large
TIME_WAITcount usually fine on the side that closed first? - →How would you find the code path leaking sockets from the fd list?
MiddleDebuggingCommonClients cannot reach a service. From the capture, name the fault and the next step.
Clients cannot reach a service. From the capture, name the fault and the next step.
The SYN is retransmitted with no SYN-ACK and no RST, so it is being silently dropped — a firewall or security group blocks 8443, or nothing is listening and a device swallows the packet. A returning RST would instead mean a listener that refuses. Next: check the firewall/security-group rules and run ss -ltn to confirm the server listens.
Common mistakes
- ✗Reading retransmitted SYNs as a RST (refused) rather than a silent drop
- ✗Assuming DNS is at fault when the name already resolved to an address
- ✗Not distinguishing no-reply (dropped) from a RST (listening but refusing)
Follow-up questions
- →How would the capture look different if the server sent a RST instead?
- →Why does a dropped SYN cause a hang while a RST causes an instant failure?
JuniorTheoryOccasionalA host boots with no IP (or a 169.254.x.x address) — what is DHCP and the DORA exchange?
A host boots with no IP (or a 169.254.x.x address) — what is DHCP and the DORA exchange?
DHCP hands out IP configuration. A client with no lease runs the DORA exchange — broadcast DISCOVER, server OFFER, client REQUEST, server ACK — leasing an address plus gateway, DNS, and subnet mask. A self-assigned 169.254.x.x address means DORA got no reply: no reachable server, or a broken DHCP relay.
Common mistakes
- ✗Getting the DORA order or direction wrong — the client initiates DISCOVER
- ✗Reading a
169.254.x.xself-assigned address as a valid working lease - ✗Forgetting a relay (ip-helper) is needed for DHCP across a subnet boundary
Follow-up questions
- →Why does DHCP across subnets need a relay agent or
ip helper-address? - →What does a
169.254.x.xaddress tell you versus getting no address at all?
MiddleTheoryOccasionalWhy is round-robin DNS not a real load balancer?
Why is round-robin DNS not a real load balancer?
Round-robin DNS returns a rotated list of A records, spreading new lookups evenly — but it is not a real balancer. It has no health checks: a dead IP stays in rotation until the record is edited and its TTL expires. It ignores load, and client and resolver caching makes the split lumpy. Real balancing needs an L4/L7 balancer with health checks.
Common mistakes
- ✗Assuming round-robin DNS health-checks and removes dead backends
- ✗Thinking DNS returns only one address and cannot rotate a record set
- ✗Ignoring client and resolver caching that makes the split uneven
Follow-up questions
- →How long can a dead IP keep receiving traffic after you pull it from the record?
- →How does an L4 or L7 balancer's health check improve on round-robin DNS?
MiddleDebuggingRareSSH works but large transfers hang intermittently. Diagnose the probes below.
SSH works but large transfers hang intermittently. Diagnose the probes below.
Small requests pass but large transfers hang, and ping -M do -s 1472 fails with mtu=1400 — a PMTU black hole. A VPN lowered the path MTU to 1400; DF-set packets are dropped, but the ICMP fragmentation needed is filtered, so the sender never learns and stalls. Fix: allow that ICMP, or clamp TCP MSS to the PMTU.
Common mistakes
- ✗Reading a clean small
pingas proof the whole path is healthy - ✗Blaming random packet loss when only large, DF-set packets are dropped
- ✗Missing that filtered ICMP
fragmentation neededis what creates the black hole
Follow-up questions
- →Why do small requests succeed while only large transfers hit the black hole?
- →How does clamping TCP MSS avoid relying on ICMP getting through?
