42 lines
1.2 KiB
Markdown
42 lines
1.2 KiB
Markdown
# Django AI Blocker
|
|
|
|
This is a very small Django app that's purpose is to block requests to the site where the User Agent header contains a
|
|
string that is associated with known AI Scrapers and indexers that will scrape the site for the purpose of training
|
|
LLMs. If you don't want the content of your website to be used to train LLMs, then this app will help you block AI
|
|
scraping bots.
|
|
|
|
This app isn't foolproof as at some point in the future, the AI scraping bots may change their User Agent headers. That
|
|
said, this app is intended to be easily configurable so that if you discover a user agent string that is not currently
|
|
in the list, you can easily add it.
|
|
|
|
## How does it work?
|
|
|
|
This app adds a simple middleware file that checks the user agent header of every request to the site. If the header
|
|
contains a string known to be associated with an AI scraper, then the middleware will immediately return an HTTP 403
|
|
status code.
|
|
|
|
## Installation
|
|
|
|
TBD
|
|
|
|
## Usage
|
|
|
|
```py
|
|
# settings.py
|
|
|
|
INSTALLED_APPS = [
|
|
# Other installed apps...
|
|
'ai_blocker',
|
|
]
|
|
|
|
MIDDLEWARE = [
|
|
# Other middleware...
|
|
'ai_blocker.middleware.AIBlockerMiddleware',
|
|
]
|
|
```
|
|
|
|
The precise location in the middleware chain where you add this middleware is up to you.
|
|
|
|
## Configuration
|
|
|
|
TBD |