modem

Debugging a modem that only failed in a room

Every test passed. Two devices on a desk would not talk

The two-device mode puts a real Bell 103 connection across a room, speaker to microphone. It worked on one machine, it worked between two processes, and it passed every test in the repository. Put a laptop and a phone on a desk and it connected, sat there looking healthy, and moved no data at all.

What follows is how that was found, including the three answers that were confident and wrong. The wrong answers are the useful part: each one was plausible, each one produced a real code change, and none of them was the bug.

A wire is not a room

Every test in this project fed one end's output straight into the other end's input. That proves the protocol agrees with itself, which is worth proving, and it quietly assumes something no room provides: that a device hears only the far end.

Cross-wired in software, your own transmission never reaches your own receiver. Over a room, it is the loudest thing there.

A device's own loudspeaker is a few centimetres from its own microphone. The other device is across the desk. Measured on real hardware, that puts a device's own tone six to nine times louder in its own microphone than the far device's. Nothing in the test suite had ever contained that fact, so nothing in the test suite could fail because of it.

Three wrong answers

Browser noise suppression

The first theory was the most attractive one. Noise suppressors work by learning a steady noise floor and subtracting it, a Bell 103 mark tone is about as steady as a signal gets, and this modem holds one continuously whenever an end is listening rather than talking. A suppressor gets seconds of training on precisely the signal it then has to carry.

It was modelled, measured, and it is false at desk levels. Noise suppression and automatic gain control both left the link at zero errors. The page still asks browsers to switch them off and still reports a device that refuses, because failing to measure harm in one model is not proof there is none - but it was not the fault, and believing it cost a day.

The idle tone is too loud

The second theory was right about the mechanism and wrong about the cure. An end that is listening holds a continuous tone, and that tone is what deafens it. Turning it down is the obvious lever, so it was turned down - and turn handover broke at every level tried.

That turned out to be a second, real defect standing behind the first. A burst does not end at its last data bit: the far end's deframer needs to see the tone held past the final stop bit, and the window deciding that bit spans about a symbol either side. Dropping the level immediately afterwards landed inside the last byte of the handover packet, corrupted it, failed its checksum, and the turn never changed hands. Both fixes were needed, and neither works without the other.

Pick a better number

The third answer was the worst, because it looked like progress. The idle level was measured and set three times - 0.2, then 0.07, then 0.05 - each from a fresh reading of one pair of devices, each an improvement on the last, and all three beside the point.

Both ends run the same constant, so turning it down turns the far end down by exactly as much. The ratio between them never moves.

That ratio is set by where the two devices are sitting. No constant chosen on one desk can know it about another, and a fourth attempt would have been the same mistake with a fourth number.

Building the lab

The common thread in all three wrong answers is that they were reasoning against a model. The model's central figure - how much louder a device's own speaker is in its own microphone - had been calculated from the inverse square law and never once measured.

So two things got built. The first is a simulated channel in the test suite that composes the whole path a desk-to-desk call goes through: band limiting, a desk reflection, room noise, a device's own loudspeaker leaking back in, and the browser's own audio processing. It reports a byte error rate, so "does it work" became a number.

The second is a lab that drives two real devices. Both load a page, register, poll for an instruction and post back what they measured. One holds a tone while the other listens; then they swap; then both run the actual modem and stream every status change and decoded byte back with timestamps. A failing call can be read from both ends at once against a common clock, rather than reconstructed afterwards from what somebody saw on a screen.

The first run of the simulated channel reported total failure for every condition, including the unimpaired control. That was the harness, not the modem: a receiver tunes to the far end's band, so configuring both sides identically listens to the wrong pair of tones entirely. It would have read as proof that the acoustic path was hopeless. The control existed, ran first, and caught it.

A measurement with no control is a number with no meaning.

The bug

With two real devices reporting, the failure appeared in a single run. The laptop dialled, performed the whole overture, reached a connected state - and reported no carrier for the entire call. The phone, answering, reported carrier immediately. One direction worked and the other never had.

Carrier detection asks whether the far end's tone is forty times louder than the total energy on the line. Over a wire that is exactly the right question. Over a room, the total includes the device's own loudspeaker.

The receiver was asking whether the far end was forty times louder than its own voice.

It never is. At six to nine times the other way round, the answer is always no, at any volume, because that ratio is geometry rather than a setting. Every earlier fix had been adjusting levels on both sides of a comparison that could not come out right.

The correction is that a device's own tones are known, not noise. They are measured and subtracted before the comparison is made.

The first attempt at that inverted the bug. Subtract everything and a moment containing nothing but your own tone leaves nothing to compare against, so that tone's own faint leakage reads as an overwhelmingly strong far end. A receiver that hears itself and calls it a carrier is worse than one that hears nothing. An existing test caught it on the first run, and the fix is a floor on the comparison, set from a measurement of how much a tone actually leaks into a neighbouring band.

Then both devices decoded each other, twice, across a desk.

That is where this account stopped for a day, and it was half the story. Turning the idle tone down, which had been done for the sake of anyone in the room, turned out to be carrying the other half. Measured afterwards: at the self-jam the two devices actually produced, with the idle tone back at its old level, both directions fail at ninety-nine per cent byte errors. Fixing the comparison is what lets a receiver notice a far end at all. Being quiet while listening is what leaves it something to notice.

A fix you can explain is not always the whole fix.

Neither half was found by reasoning. Both were found by measuring, and the second one only because a review asked what a test was really proving.

What it cost, and what it taught

Four rounds of changes to fix one fault. Two of them turned out to be necessary - the handover defect found along the way was real, and the link needs that fix too - and two were pure detour. The useful ratio is not four to one but the shape of it: most of the effort went into answers that were plausible, testable-sounding and wrong.

Three things are worth taking from it.

A test that passes on a wire says nothing about a room. It still proves plenty - the protocol agreed with itself throughout - but the suite was thorough about the wrong thing. Every assumption a test makes silently is an assumption nothing will ever check.

A plausible theory is not evidence. Noise suppression explained the symptom, matched the mechanism, and was wrong. It survived as long as it did because it was never measured, only agreed with.

Build the instrument. The bug was found in one run of a lab that took an afternoon to build, after several days of reasoning had produced three wrong answers. The instrument is still there, and the numbers it produced are now tests.

The harness, in full

Try the demo