I had a quick squiz in the logs and it's pretty clear that "tencent cloud" is a hive of villainy

We're dropping the user agents that are obviously bots, and now what's left is bursts of clients that are clearly not people but which have exactly the user agent string a real browser would have

So I foresee some amount of IP range blocks in the near future haha

do the bots normally follow redirects? could have some fun if they did :)

jbk: internet wisdom says they don't.

jbk: that said we might get the blocks wrong so it would be "nice" to respond with a redirect to a "your isp has been blocked for abuse" page.

yeah, I suppose that would be the nice thing to do....

some of those bots are stupid enough to get stuck all by themselves. The facebook ai crawler made multiple requests per second fetching the same page for three months before I noticed and manually blocked it :|

i was just wondering what would happen if you redirected them to a hostname that resolved to 127.0.0.1 or the like :)

There are people who give AI user-agents many gigs of gzip 0 or whatever

one case someone crafted something of a few hundred KB that expands to a few TB through multiple layers of decompression.

[illumos-gate] 16757 fnvpair functions for uint8 are misnamed -- Robert Mustacchi <rm⊙fo>