The packet that never comes back

It is not a monitoring tool and it is not a fix. It is a flashlight you point at the path and hope the path is honest enough to reflect something back.

The mechanism underneath is almost insultingly simple. Every IP packet carries a Time To Live field — a counter that starts at some number and drops by one at every router it passes. When the counter hits zero, the router discards the packet and sends back an ICMP Time Exceeded message to whoever sent it.

That field exists to stop loops. A packet circling between two misconfigured routers forever is a small denial-of-service attack you inflict on yourself. TTL is the fuse that burns out before the loop becomes permanent.

Traceroute turns that safety mechanism into a measuring instrument. Send a packet with TTL 1. The first router decrements it to zero, kills it, and reports back. You now know the first hop. Send TTL 2, get the second. Repeat until the destination answers or you give up.

You are, in effect, deliberately breaking your own packets in sequence and reading the complaint notes.

Three probes, three numbers

Each hop gets three probes by default, and you get three round-trip times in milliseconds. That is not decoration. Three numbers that agree tell you the link is stable. Three numbers spread across a factor of ten tell you something on that hop is congested, rate-limited, or taking the slow path through a queue.

A hop that shows ` *` means no answer came back within the timeout. This is where most people draw the wrong conclusion.

An asterisk does not mean the router is broken. It very often means the router answered you the first time, decided you were not worth the CPU cycles, and stopped. Core routers carry a control-plane policing policy that deprioritizes ICMP generation precisely because tools like yours can be pointed at them by the thousand. The router is fine. The router is forwarding your actual traffic at line rate while ignoring your diagnostic pings. It has better things to do.

If the hops after the asterisk respond normally, the path is intact and only that one node is being coy. If everything after the asterisk is also silent, you have found something real.

UDP, ICMP, and why the first version broke

The original traceroute did not use UDP. It started with ICMP Echo Request — the same packet ping sends — and got burned almost immediately. On the first test night, more than half of the router vendors in the path declined to send back a Time Exceeded when the triggering packet was an ICMP Echo. The tool was blind on more than half the internet.

The fix was to switch to UDP datagrams aimed at high, almost certainly unused port numbers. The TTL trick still works, but now the destination itself, once reached, replies with ICMP Port Unreachable — which is your signal that the path is complete. Different protocol, same idea, and a much higher response rate.

This is why `traceroute` on Linux and BSD defaults to UDP, and why Windows `tracert` defaults to ICMP. Two implementations, two histories, two sets of assumptions about what routers will bother to answer. Neither is more correct. They just fail differently.

What you are actually looking at

The output is a list of hops. Each line has an IP address, sometimes a hostname if reverse DNS resolves, and the three RTTs.

Read it as a shape, not a list of facts. A jump from 12 ms to 180 ms between two consecutive hops is a long haul — probably a submarine cable or a congested peering point. A steady 200 ms across every hop from the fourth onward usually means the latency was introduced early and everything downstream inherited it.

The hostnames are the most useful part and the least reliable. `ae-1-3501.edge4.Amsterdam1.Level3.net` tells you the city, the provider, and roughly where in their topology you are. It also tells you nothing about the physical path, because that name was assigned by a human who may have been tired. Reverse DNS on backbone routers is a courtesy, not a contract.

The path is not a path

Here is the part that trips people up. A traceroute is not the route your traffic takes. It is one sample of the route your traffic took, at that moment, from that source, with those probe packets.

BGP decides the path. BGP is a negotiation between autonomous systems that changes constantly based on policy, cost, congestion, and the occasional fat-fingered prefix announcement. Run the same traceroute twice an hour apart and you can get two completely different paths. This is not a bug. This is the internet working as designed.

It also means traceroute from your laptop and traceroute from your server in a different datacenter can disagree entirely about how far away the same destination is. Both are correct. Neither is the truth.

Choosing your tool

`traceroute` is the baseline. It is installed almost everywhere, it is small, and it tells you what you need in about four seconds. It is also stateless — you run it, you read it, you run it again. If you want to know whether a hop is consistently slow or just slow right now, you run it a dozen times and squint at the output.

`mtr` — My Traceroute — solves exactly that problem. It runs traceroute continuously and refreshes the display, so you get a live picture of packet loss and latency per hop over time. For diagnosing an intermittent problem, `mtr` is the better tool by a wide margin. It is also less useful for a quick one-off check, because now you have to watch it for a minute instead of reading a screen.

`tcptraceroute` and `traceroute -T` send TCP SYN packets instead of UDP or ICMP. This matters when your target sits behind a firewall that drops ICMP and UDP but has to let TCP through on port 443, because that is the port customers use. The probe looks more like real traffic, so it gets treated more like real traffic. The cost is that you need raw socket privileges, and some networks still rate-limit you into uselessness.

`paris-traceroute` tries to eliminate load balancing artifacts — the phenomenon where different probes of the same hop hit different physical routers and produce inconsistent RTTs that look like jitter but are not. It is more accurate and more work to set up. Reach for it when you are debugging something subtle and the normal tools keep giving you answers that do not quite make sense.

What traceroute cannot tell you

It cannot tell you why a specific application is slow. It measures the network path, not the server. A 15 ms path to a host that takes four seconds to respond is a server problem, and no amount of traceroute will find it.

It cannot tell you about the return path. Traceroute measures outbound hops. The reply comes back over a route that may be entirely different, because BGP does not have to be symmetric. Half of all "the latency is on hop seven" conclusions are wrong for this reason. If you need the return path, you need a traceroute from the other end, which means access to the other end.

It cannot see past a firewall that drops Time Exceeded entirely. You will get a clean run of asterisks and no information. That is not a failure of the tool. That is a network that has decided not to participate.

Reading the silence

The most useful skill with traceroute is not running it. It is knowing when the output is telling you something and when it is telling you nothing.

A path that completes with a few asterisks in the middle and clean RTTs on either side is healthy. A path that dies at hop four and stays dead is a real routing problem, and you now have an IP address to hand to whoever owns that network. A path with 40 percent packet loss from hop six onward, confirmed by `mtr` over five minutes, is a congested link you can name.

Everything else is noise dressed up as data. The internet is a few hundred thousand routers making independent decisions several times a second, and you are looking at one photograph of that. Useful photograph. Not the whole album.

Where this lands depends on what you are actually chasing. A one-off "is it them or is it us" question gets answered by plain `traceroute` in ten seconds. An intermittent complaint that only shows up under load needs `mtr` left running while you do something else. And a problem that lives behind a firewall on port 443 needs a TCP probe, raw socket privileges, and the patience to remember that the answer you get is only ever about the path you probed, from where you probed it, at the moment you asked.