Welcome to Anime Characters Database

Visual Search

Series Finder

The visual search engine for animated characters. We index characters by eye color, hair color, hair length, age, gender, and animal ears.

Increase your fun by becoming a member today!

Take part in Dragon Battles. Wish your favorite characters a Happy Birthday. Build a Family Tree. And so much more.

Login | Register

News | Apr 19, 2025 | Era of the Bots

More News | RSS

2025 has been a year of great challenge keeping websites online and responsive. ACDB has roughly 20k to 30k ＂human＂ daily visitors, this has been consistent for the past few years.

But this year, we have had wave after wave of new AI discovering ACDB.

The first and worst was Alibaba, which at one point used more than 100k servers to make 200 requests per second for over 4 million pages in a day. Contrast this with the 20k regular uses that make 100k requests in a day.

Initially I thought it was great seeing so much traffic. It was stimulating looking for ways to optimize the site to serve even more pages. Optimizing code. Adding indexes to databases. Dropping pages that were out dated. Removing pages that should not have been publicly accessible in the first place. The fun eventually ran out. I am not running a website for bots, I am running a website for the anime community. Real people were getting error pages, and slow pages. While we could buy more server capacity, the bots showed me the site clearly has room to grow 10x before that would be necessary. I choose to block the largest data centers owned by Alibaba. Not one regular user was affected, and the load on the site returned to normal.

On FreeBSD this was accomplished with

ipfw add 669 deny ip from 47.0.0.0/8 to me
ipfw add 670 deny ip from 106.0.0.0/8 to me 443
ipfw add 671 deny ip from 139.224.0.0/11 to me 443
ipfw add 672 deny ip from 140.205.0.0/16 to me 443
ipfw add 673 deny ip from 152.32.0.0/14 to me 443
ipfw add 674 deny ip from 163.177.0.0/16 to me 443
ipfw add 675 deny ip from 180.97.0.0/16 to me 443
ipfw add 676 deny ip from 182.92.0.0/15 to me 443

This did not last for long, as yet another AI discovered ACDB. This time one with more than 400k servers at their disposal. I dread the day IPv6 gets involved.

One important point I'd like to make is that Web Spiders like Bing and Google are not the same as the current AI bots.

Web spiders show at least some level of respect, reading robots.txt and following Crawl delay directives. This allows one to request Google crawl only 1 page per second. Bots on the other hand, use multiple servers to requests pages as fast as possible - no delay between requests.

I can deal with a few bots requesting 1 page per second, whereas the AI bots are requesting 100x as many pages. the long term solution would be to force AI bots to follow the same rules web spiders do. Until then, they are a menace on the web indistinguishable from a hostile DDoS attack.

~~~

I've also been using nginx to limit each IP to one request every 2 seconds.

limit_req_zone $binary_remote_addr zone=one:10m rate=30r/m;

As well as targeted limits for members and non members.

For higher priority, speedier access to ACDB, it helps to log in.

--
Rei

Happy Birthday To

View All Characters born on April 22

Latest Up Voted Quotes

More Quotes

Family Tree

Latest Family Tree Relations