What is the anti bot impact for you?

We're doing STUPID amounts of traffic to these bots and they eventually bring the site down. I've added more memory (we were already close to capacity before the bots) and I've better tuned MySQL, PHP and nginx. But I've recently added more security and rate checks with Cloudflare. My desire is for this to be as close to unnoticeable as possible but I can only test for myself.

So, please just leave a post if these measures are ok or if they are driving you mental.

Christian

Were config changes done earlier this morning? I noted SAU went down and I'm pretty sure it was this morning, so that may have been while redoing config.

It's amazing that bots crawling is causing such high demand on the server. Do you know what countries most of the bots are coming from, or what type of companies they're from in the analytics?

My first thought process is as much stuff to stop the boys in the first place, BUT, that has the potential to kill us in search engines :/

I wonder if a lot of this bot activity is from everyone trying to make better and better AI so they're scraping the internet even more :/

I know there are Configs you can do to reject AI specifically from hitting the servers, but don't ask me what it is exactly, I just know that it's possible.

I've noticed nothing so far.

9 hours ago, MBS206 said:

Were config changes done earlier this morning? I noted SAU went down and I'm pretty sure it was this morning, so that may have been while redoing config.

It's amazing that bots crawling is causing such high demand on the server. Do you know what countries most of the bots are coming from, or what type of companies they're from in the analytics?

My first thought process is as much stuff to stop the boys in the first place, BUT, that has the potential to kill us in search engines :/

I wonder if a lot of this bot activity is from everyone trying to make better and better AI so they're scraping the internet even more :/

I know there are Configs you can do to reject AI specifically from hitting the servers, but don't ask me what it is exactly, I just know that it's possible.

All our traffic goes through Cloudflare so I have a few rules now in place. All Vietnamese, Chinese, Singapore and Czech traffic is blocked. I'll undo these if I find the rate limiting working.

Nothing touches the top search crawlers; Bing, Google, Duck Duck Go, they still have full access and generally are FAR lower in numbers than the other bots.

Some of the barrages were insane, like several thousand requests per second.

I tried more aggressive caching to just let the bots do their thing, but had mixed results.

This is the graph of rate limit hits since I reduced it earlier today.

Anyway, we've been good since I implemented this so, for the love of god, I hope we've made some progress as this has been killing me.

On the laptop (Mac), the site works perfectly (except notifications bug out for some reason), but on the phone (iPhone) I get smashed with Cloudflare requests, the icons are non-existent and I couldn't actually reply to any threads (wheel of death).

I'm not running a VPN, so shouldn't locate me elsewhere. Not a big deal, as I can still view the content, so worth it if it saves the bot wars.

On that note, given I'll be trawling this treasure trove for a while yet, any way to contribute to running costs? No doubt there's a bit of cost involved running the joint.

Hmm, just added a new post to my ongoing build thread, and can't upload photos there. Not sure if I've reached an upload limit, or whether related to these changes? I was able to upload those screenshots above.

Hi @PranK ,

I'm replying on my laptop, Win 11 Firefox 148.0.2, with no extensions that break JS or anything like that.

But on my desktop, Win 11 Firefox 152.0.6 and NoScript, but with all the individual things that can be blocked by NoScript at least temporarily allowed (I usually block Google analytics, Cloudflare type stuff, bookface, etc), I get either the spinning wheel of death in reply or compose boxes, or they do not respond to being clicked on at all. And some page elements, such as the "Reply to this topic" that sits in the reply box until you click on it, and some others around the page, do not load at all. Just little broken image/element boxes shown.

I also get the same behaviour on the desktop using Edge, with absolutely no NoScript type stuff installed.

I will update this laptop's Firefox to 152 soon. So if you do not hear from me again, it will be because of the FF version, I guess.

Further update. It's probably my desktop computer as a whole, not the anti-bot stuff you've done. But I cannot, for the life of me, work out what is going on. Edge just gives the spinny wheel of death. Chrome won't respond to clicking on the reply box. FF will do either, depending on whether I enable google tracking and so on.

I have not updated this laptop's FF yet. Will do that next.

On the subject of the anti-bot stuff directly, I find that the rate limiting is a real issue, as is the non-bot verification stuff. My usage pattern is usually to hold down ctrl and click a number of links to open up pages as new tabs. Say, going down the list of new content. But these things you've done mean that I can get rate limited and shut out, or many/all of the new tabs fail the non-bot check or sit there waiting for me to do something human on them, meaning that they are not ready for me to look at when I get to them. But that's just me and I will adapt if I have to.

What wpould be good would be if there was a cookie or some means of labelling a given broswer as "already verified as not a bot".

Latest FF update on this lappy works. So it's not that, fer sher. Farkin.

@GTSBoy Ok, I've disabled the rate limiting on Cloudflare for the moment, can you try again on the machine that wasn't working?

3 hours ago, PranK said:

@GTSBoy Ok, I've disabled the rate limiting on Cloudflare for the moment, can you try again on the machine that wasn't working?

Well, that's interesting. I'm posting this in the reply box of the machine, that, last time I was sitting at it, was giving me a different shitty outcome on every tab, but certainly none of them was allowing me to do this!

What does that tell you? I'm hella confuzzled, myself.

Yep those DDoS and bot attacks will be daily occurrences now, since it's very easy to spin up a crawler and blast any site. Definitely rate limit (will need some tuning as we get more traffic data in Cloudflare). Legit crawlers don't smash your site, they should be making like 1 req/s.

1 hour ago, GTSBoy said:

Well, that's interesting. I'm posting this in the reply box of the machine, that, last time I was sitting at it, was giving me a different shitty outcome on every tab, but certainly none of them was allowing me to do this!

What does that tell you? I'm hella confuzzled, myself.

So, my gut says this ....

Your browser will make as many requests as it needs to to get the HTML, CSS and Javascript from the server. I'm not sure why it would be too different between machines though. 🤔

Thankfully I think we're over a bit of a hump now. No downtime in a few days. Hopefully its ok for a bit.

On 20/07/2026 at 11:00 PM, niZmO_Man said:

Yep those DDoS and bot attacks will be daily occurrences now, since it's very easy to spin up a crawler and blast any site. Definitely rate limit (will need some tuning as we get more traffic data in Cloudflare). Legit crawlers don't smash your site, they should be making like 1 req/s.

Yep, I'm target blocking only dodgy ones now. I think I just need to keep up with my blocklist. All legit and SEO etc are still allowed and we don't see any issues from them.

On 18/07/2026 at 11:57 PM, MYC35 said:

On that note, given I'll be trawling this treasure trove for a while yet, any way to contribute to running costs? No doubt there's a bit of cost involved running the joint.

See the SAU web supporter subscriptions in the store. Benefits being:

1. No ads!

2. No flood control

3. Edit own posts/titles

4. New Supporter bar under your name (Currently broken, resolving soon)

5. Lasts one year. 

I think it should also mention the warm and fuzzy feeling of supporting something worthwhile.

On the bot measures:

• When you first set it up I had a few minutes of the cloudflare "checking you are not a bot" page coming up with every single click. Hasn't happened since.

• I have seen the comment box not responding twice I think. Seemed to resolve itself after refreshing.

• Might be unrelated, but the links to latest comments seem to not work anymore in the discover/unread feed. Both the xx replies and xx hours ago just link to the thread. It used to be a magic link to go to the latest unread post in the thread. Now it always just opens the first post in the thread and need to manually go to the newest comments.

11 minutes ago, soviet_merlin said:

1. When you first set it up I had a few minutes of the cloudflare "checking you are not a bot" page coming up with every single click. Hasn't happened since.

2. I have seen the comment box not responding twice I think. Seemed to resolve itself after refreshing.

I think this is because he disabled it after my report of it appearing to cripple my shit. The cloudflare bot checks disappeared at the exact same time as the events in our posts from Sunday evening.

3 hours ago, soviet_merlin said:

On the bot measures:

• When you first set it up I had a few minutes of the cloudflare "checking you are not a bot" page coming up with every single click. Hasn't happened since.

• I have seen the comment box not responding twice I think. Seemed to resolve itself after refreshing.

• Might be unrelated, but the links to latest comments seem to not work anymore in the discover/unread feed. Both the xx replies and xx hours ago just link to the thread. It used to be a magic link to go to the latest unread post in the thread. Now it always just opens the first post in the thread and need to manually go to the newest comments.

Links to latest works for me.

I've had bugs before where it's land me in a random spot in the thread, but I think that was it attempting to say "this was where you last read up to, everything after is new".

10 minutes ago, MBS206 said:

Links to latest works for me.

I've had bugs before where it's land me in a random spot in the thread, but I think that was it attempting to say "this was where you last read up to, everything after is new".

Odd, I would have thought it's a server side thing. See this screenshot where I'm hovering on the "16 replies link". It is just the thread URL and has no special url to go to first new comment. I see the same on mobile and desktop.

And yes, it still keeps track of where I last read. It shows a blue line with "Unread replies" before the first new post in the thread.

Hmm.

First up, I never knew the "16 replies" was a link. I just ctrl-click the thread title link and am always taken direct to the first post I haven't yet read.

Second, when I hover over that link, I see a url that ends with what you would expect for going to the first unread post - viz - https://www.sau.com.au/forums/topic/483562-whoretown-xiv-back-to-basics/?do=getNewComment

So.... it's working for me, even though I don't use it.

10 minutes ago, GTSBoy said:

Second, when I hover over that link, I see a url that ends with what you would expect for going to the first unread post - viz - https://www.sau.com.au/forums/topic/483562-whoretown-xiv-back-to-basics/?do=getNewComment

So.... it's working for me, even though I don't use it.

Interesting! Turns out for me the thread title is also showing the getNewComment link and that works still. I'm happy with that and will just use that in the future. Problem solved.