• News
• New
• Index
• About

One down side of adding additional translations to the site is: web spiders, bots, and crawlers end up having index entire copies of the site. This has been pushing daily page requests near 3,000,000. while Google Analytics reported requests have remained around 100,000.

This works out to for every 1 request by a member, there are 30 requests by bots.

Our web server Nginx allows for rate limiting either at an IP level or a server level.

ACDB makes a lot of use of redirects so I can't really rate limit people to 1r/s without

1) Breaking the lounge
2) Making redirects take 2 seconds for the page to respond

I also can't give members more than 3r/s without risking race conditions.

This left me wanting a solution where I could have more generous rate limits for members and stricter rate limits for everyone else.

Unfortunately Nginx has some serious hatred problems for simple conditionals. They live under a mantra of ＂IfIsEvil＂.

Luckily, the world of programing is one where if you have a will, there is always a way.

Pushing aside useless AI summaries, I eventually landed on https://nginx.nginx.narkive.com/tRRv3Xv8/limit-req-inside-a-if-block-not-allowed

While this page's solution made use of a user agent check; that conditional could easily be replaced with a cookie test. A few tweaks later, the setup is live on ACDB. Members will enjoy a fast and responsive site as it should be while the bots will be limited to an amount of requests that shouldn't gum up the site again for a while.

--
Rei