Website crashing because of bots

This morning when I woke up I was greeted by several messages from readers of Irregular Webcomic! that my entire domain irregularwebcomic.net appeared to have been removed and replaced with a domain parking page. I also had an email from my webhost, explaining what had happened. They’d detected an unacceptably high server load from my site and acted by disabling it.

I’ve been having problems with the site for months now. The phpBB forums have a setting to disable the boards if the server load gets above a certain amount, which I’ve been tweaking to try to get them to be stable. But there’s enough stuff hitting my site that the forums have been offline more than online for some time now. Well, apparently overnight the webhost got sick of the server load and pulled the plug.

I checked the access logs and noticed that a handful of IP subnets were responsible for thousands of hits a day, whereas most IPs were only logging a few hits. So I banned the offending subnets in the site’s .htaccess file. I looked at the live server log updates to check the effectiveness and confirmed they were now being served HTTP 403 Forbidden responses. But I noticed that these subnets were hammering the server, with sudden spurts of 30 or 40 HTTP requests within a second, then stopping for a few seconds, then doing it again. And again, and again. It’s no wonder the server load was unusually high.

It seems these may be relatively new webcrawler bots that are trawling sites looking for text to use to train Large Language Model “AI”s. Lots of sites have been complaining about these recently, and they seem to be causing major headaches for many site owners.

Anyway, my IP blocks seem to be working, and the forums have been stable and online since I made the changes this morning. I may look at a more drastic solution as well, and investigate getting a free Cloudflare account and changing my DNS to route requests via Cloudflare, which can detect and block bot-like behaviour. I didn’t have time to do that today, but might check it out when I do have some time.

Because today I had to finish repping my science class lesson, which I managed just before the class at midday (after spending an hour or so in the morning with the web server issue). Then I took Scully out for a walk after lunch, and then had another class in the afternoon, and three more this evening. So it’s been a very full day.

In critical thinking we started a topic on bikeshedding, or thinking about the wrong thing. I think this is a good one for teaching the kids specific critical thinking tips. It seems to be going well so far.

In one class we had a sad moment as one of the students said goodbye to the classmates. She’d been participating for several weeks from the UK while on a trip there, but is now moving back home to Bermuda, and unfortunately the time zone there is really bad for the class – she would have to get up at 5am to continue. So she said goodbye, and hoped to be back in the UK and rejoin around Christmas time. I hope so!