Tractor Beam: Watching What the Internet Deletes

An adaptive polling engine for public feeds, and the rate limit problem that made it interesting.

Tractor Beam started with a slightly paranoid question: what happens to a post after it goes up? Almost every tool shows you the internet in the present tense, whatever exists right now. None of them told me when something got edited, or the moment it was quietly pulled. So I built the one that watches for the deletion instead of the post.

It's a diff engine

The core loop is deliberately dull. Fetch a public JSON or HTML endpoint, normalize it, diff it against the last snapshot. A changed title, an altered body, an item missing from the feed — all of those become events. Then you try to run that loop against thousands of sources without getting blocked, and without missing a post that lived for thirty seconds before it vanished. The diff took an afternoon. Staying unblocked took the rest of the project.

Adaptive polling is a control loop

A fixed interval loses both ways. Poll slowly and you miss short-lived deletions. Poll fast and you burn your rate limit hammering sources that never change. So the interval isn't fixed. Each source carries an activity score, and the poll interval scales inversely to it. A source that's moving right now gets checked in seconds. A quiet one relaxes back toward a slow baseline. The request budget steers itself toward wherever change is actually happening.

# activity_score is an EWMA of recent changes, updated each poll
interval  = base_interval / (1 + activity_score.clamp(0, 8))
interval *= (1 + rand * 0.2)   # jitter so sources do not sync up
# hot source   -> checked in ~2s
# quiet source -> relaxes toward base_interval

The score is an exponentially weighted moving average, so a burst of activity ramps polling up fast and a lull lets it decay back down on its own. Every interval also gets a little random jitter. Without it, a thousand sources that started together fall into lockstep and stampede the same endpoints on the same tick.

Cutting its teeth on r/UFO

The proving ground was, fittingly, r/UFO. Posts about sightings and leaks get pulled there constantly, sometimes by moderators, sometimes by a poster with second thoughts. That's a steady supply of real deletions to measure against. Tractor Beam caught them inside its polling window reliably enough that I stopped refreshing threads by hand.

The internet ships without an undo history. This is a way of keeping one.

The plumbing decides

None of this is specific to one subreddit. Adaptive change detection over public endpoints works the same whether you're watching a competitor's pricing page, a regulatory filing feed, or a changelog someone forgot was public. What decides whether a watcher is worth trusting isn't the diff. It's backoff, jitter, rate limit behavior, and snapshotting you can't fool. Get those wrong and you don't end up with a broken watcher. You end up with a quiet one.