This is a very small Django app that's purpose is to block requests to the site where the User Agent header contains a string that is associated with known AI Scrapers and indexers that will scrape the site for the purpose of training LLMs. If you don't want the content of your website to be used to train LLMs, then this app will help you block AI scraping bots.

This app isn't foolproof as at some point in the future, the AI scraping bots may change their User Agent headers. That said, this app is intended to be easily configurable so that if you discover a user agent string that is not currently in the list, you can easily add it.

How does it work?

This app adds a simple middleware file that checks the user agent header of every request to the site. If the header contains a string known to be associated with an AI scraper, then the middleware will immediately return an HTTP 403 status code.

Installation

TBD

Usage

# settings.py

INSTALLED_APPS = [
    # Other installed apps...
    'ai_blocker',
]

MIDDLEWARE = [
    # Other middleware...
    'ai_blocker.middleware.AIBlockerMiddleware',
]

The precise location in the middleware chain where you add this middleware is up to you.

Configuration

TBD