Yeah the IP blocking is difficult with AI crawlers. They're run out of large networks so the crude ban hammer blast radius is gonna touch some innocent bystanders

Also not convinced the (free version of the) Anubis solution amounts to much more than security theater

These crawlers are a serious issue with no easy solution right now

No idea about the infra involved but you could give Anubis a try to see if it alleviates some of the load

hrm...

It’s so bad even on my small stafix

*static blog too

Traffic has gone 10x in the past year and it’s all AI crowlers and most just ignore robots.txt or worse, use it to crawl hidden stuff

it is very much "or worse"

I've got a nasty one....

connect() is getting EINPROGRESS and doesn't seem to be sending out to the switch. did a packet capture switch side and didn't see anything coming in fronm dig on 2 CNs

the process just loops

Smithx10: maybe check `connstat` and see what state the TCP connection is in, also run snoop on the host and see if SYN packets are going out or not.

very strange that snoop on the Server said we sent them, but switch doesn't see them github.com/Smithx10/debug-images/blob/main/pcap.png. The dig command intermittently gets stuck

rzezeski: yeah Fri Sep 12 20:08:34 UTC 2025 gist.github.com/Smithx10/740b9d61d2…9b30e94a98815#file-dns-command-L140 when it times out tcp 10 seconds, this is what happens in the capture client side gist.github.com/Smithx10/740b9d61d2…0e94a98815#file-gistfile1-txt-L1058

rzezeski: when it hangs it looks like the latest one that was hung is SYN_SENT from what I can tell gist.github.com/Smithx10/e07c3de9dd5cb5915febac1de182da3d

Yep, SYN_SENT is the state I would expect if the host sent a SYN and is still waiting for its ACK. If you see it hitting snoop it should technically have hit the wire, but snoop intercepts before the driver, so it could be the driver or the device rejected to send it for some reason. What is the link it's going out on?

i40e in aggr.

I have at least 2 CN in this state, behavior is happening in bhyve guests and os zones

I am gonna takethe aggr out of the scenario in one of them

okay, one thing you might do is take two snapshots of kstat for any mac, aggr, and i40e stats. And look for any counters that are changing that look suspicious, like errors or drops.

are other connections on the same host working on that link? like other TCP connections going out over the same aggr?

yeah there a bunch of virtual machines

internal customers

so other VMs are getting new connections established over the link okay?

no, everyone is having intermittent issues

tcp, udp

what kstat dumps you think have the most value?

hmmm, okay, I mean it kind of sounds like the i40e Tx freeze issues I fixed years ago, but those would grind the i40e device to a halt (no traffic for anyone) until a reset.

I can't think of them off the top of my head, but anything with 'mac', 'i40e', or 'aggr' in the name

connect(4, 0x005EAC28, 16, SOV_DEFAULT) Err#150 EINPROGRESS

errr

whhos

whoops*

I have to step away for a bit

rzezeski: kstats gist.github.com/Smithx10/822e28e4568bb67866939ae51bacf036 gist.github.com/Smithx10/9475909a1dd0b1eb90920caa7d950a21

Smithx10: i40e kstats might also be informative (there are a bunch of different reasons for a tx drop)

sommerfeld: I think they are in there on that gist should be 3 files i40e, mac, and aggr

each gist is a server, I haven't bounced if you'd like to see anything more