Welcome to Anime Characters Database

The visual search engine for animated characters. We index characters by eye color, hair color, hair length, age, gender, and animal ears.

News | 02:34 pm | Era of the Bots

2025 has been a year of great challenge keeping websites online and responsive. ACDB has roughly 20k to 30k ＂human＂ visitors, this has been consistent for the past few years.

But this year, we have had wave after wave of new AI discovering ACDB.

The first and worst was Alibaba, which at one point used more than 100k servers to make 200 requests per second for over 4 million pages in a day. Contrast this with the 20k regular uses that make 100k requests in a day.

Initially I thought it was great seeing so much traffic. It was stimulating looking for ways to optimize the site to serve even more pages. Optimizing code. Adding indexes to databases. Dropping pages that were out dated. Removing pages that should not have been publicly accessible in the first place. The fun eventually ran out. I am not running a website for bots, I am running a website for the anime community. Real people were getting error pages, and slow pages. While we could buy more server capacity, the bots showed me the site clearly has room to grow 10x before that would be necessary. I choose to block the largest data centers owned by Alibaba. Not one regular user was affected, and the load on the site returned to normal.

On FreeBSD this was accomplished with

ipfw add 669 deny ip from 47.0.0.0/8 to me
ipfw add 670 deny ip from 106.0.0.0/8 to me 443
ipfw add 671 deny ip from 139.224.0.0/11 to me 443
ipfw add 672 deny ip from 140.205.0.0/16 to me 443
ipfw add 673 deny ip from 152.32.0.0/14 to me 443
ipfw add 674 deny ip from 163.177.0.0/16 to me 443
ipfw add 675 deny ip from 180.97.0.0/16 to me 443
ipfw add 676 deny ip from 182.92.0.0/15 to me 443

This did not last for long, as yet another AI discovered ACDB. This time one with more than 400k servers at their disposal. I dread the day IPv6 gets involved.

One important point I'd like to make is that Web Spiders like Bing and Google are not the same as the current AI bots.

Web spiders show at least some level of respect, reading robots.txt and following Crawl delay directives. This allows one to request Google crawl only 1 page per second. Bots on the other hand, use multiple servers to requests pages as fast as possible - no delay between requests.

I can deal with a few bots requesting 1 page per second, whereas the AI bots are requesting 100x as many pages. the long term solution would be to force AI bots to follow the same rules web spiders do. Until then, they are a menace on the web indistinguishable from a hostile DDoS attack.

~~~

I've also been using nginx to limit each IP to one request every 2 seconds.

limit_req_zone $binary_remote_addr zone=one:10m rate=30r/m;

As well as targeted limits for members and non members.

For higher priority, speedier access to ACDB, it helps to log in.


--
Rei

Happy Birthday To

Latest Up Voted Quotes

Family Tree

Latest Family Tree Relations

Logs

04:31 pm RinaK added an image 04:30 pm RinaK added an image 04:18 pm Zero_FX added a relation 04:17 pm Zero_FX added a relation 03:04 pm NatsumiSawada modified a character 03:03 pm NatsumiSawada modified a character 03:03 pm NatsumiSawada modified a character 03:03 pm NatsumiSawada modified a character 03:03 pm NatsumiSawada new character Miyoshi Hayashi Reward: 3 67 30 03:03 pm NatsumiSawada uploaded a character Reward: 3 84 10 03:03 pm NatsumiSawada new character Jouji Ema Reward: 1 43 11 03:03 pm NatsumiSawada uploaded a character Reward: 17 15 03:02 pm NatsumiSawada new character Akira Nikaidou Reward: 8 81 86 03:02 pm NatsumiSawada uploaded a character Reward: 2 80 83 03:02 pm NatsumiSawada new character Yuuichi Medaka Reward: 4 31 53 03:02 pm NatsumiSawada uploaded a character Reward: 66 2 02:57 pm NatsumiSawada modified a title 02:42 pm NatsumiSawada added an image 02:41 pm NatsumiSawada added an image 02:41 pm NatsumiSawada added an image 02:37 pm NatsumiSawada modified a character 02:37 pm NatsumiSawada modified a character 02:37 pm NatsumiSawada new character Renji Kusakabe Reward: 1 62 62 02:37 pm NatsumiSawada uploaded a character Reward: 8 60 67 02:37 pm NatsumiSawada new character Lily Baker Reward: 85 44 02:37 pm NatsumiSawada uploaded a character Reward: 1 96 82 02:34 pm NatsumiSawada modified a title 02:28 pm Moon_Moon new character Nigou Reward: 4 40 13 02:28 pm Moon_Moon uploaded a character Reward: 6 18 80 02:27 pm Moon_Moon new character Sirei Reward: 3 47 37

Characters	144501	220 this month
Titles	14173	43 this month
Members	81770	205 this month
Quotes	356303	3203 this month
Voice Actors	10588	4 this month