Add initial readme
This commit is contained in:
42
readme.md
Normal file
42
readme.md
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
# Django AI Blocker
|
||||||
|
|
||||||
|
This is a very small Django app that's purpose is to block requests to the site where the User Agent header contains a
|
||||||
|
string that is associated with known AI Scrapers and indexers that will scrape the site for the purpose of training
|
||||||
|
LLMs. If you don't want the content of your website to be used to train LLMs, then this app will help you block AI
|
||||||
|
scraping bots.
|
||||||
|
|
||||||
|
This app isn't foolproof as at some point in the future, the AI scraping bots may change their User Agent headers. That
|
||||||
|
said, this app is intended to be easily configurable so that if you discover a user agent string that is not currently
|
||||||
|
in the list, you can easily add it.
|
||||||
|
|
||||||
|
## How does it work?
|
||||||
|
|
||||||
|
This app adds a simple middleware file that checks the user agent header of every request to the site. If the header
|
||||||
|
contains a string known to be associated with an AI scraper, then the middleware will immediately return an HTTP 403
|
||||||
|
status code.
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
TBD
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```py
|
||||||
|
# settings.py
|
||||||
|
|
||||||
|
INSTALLED_APPS = [
|
||||||
|
# Other installed apps...
|
||||||
|
'ai_blocker',
|
||||||
|
]
|
||||||
|
|
||||||
|
MIDDLEWARE = [
|
||||||
|
# Other middleware...
|
||||||
|
'ai_blocker.middleware.AIBlockerMiddleware',
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
The precise location in the middleware chain where you add this middleware is up to you.
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
TBD
|
||||||
Reference in New Issue
Block a user