From ffbbe86446fa6a2c885e2ce4709263a55a412315 Mon Sep 17 00:00:00 2001 From: Andrew Luchuk Date: Sat, 4 Oct 2025 17:43:48 -0400 Subject: [PATCH] Add initial readme --- readme.md | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 readme.md diff --git a/readme.md b/readme.md new file mode 100644 index 0000000..326624b --- /dev/null +++ b/readme.md @@ -0,0 +1,42 @@ +# Django AI Blocker + +This is a very small Django app that's purpose is to block requests to the site where the User Agent header contains a +string that is associated with known AI Scrapers and indexers that will scrape the site for the purpose of training +LLMs. If you don't want the content of your website to be used to train LLMs, then this app will help you block AI +scraping bots. + +This app isn't foolproof as at some point in the future, the AI scraping bots may change their User Agent headers. That +said, this app is intended to be easily configurable so that if you discover a user agent string that is not currently +in the list, you can easily add it. + +## How does it work? + +This app adds a simple middleware file that checks the user agent header of every request to the site. If the header +contains a string known to be associated with an AI scraper, then the middleware will immediately return an HTTP 403 +status code. + +## Installation + +TBD + +## Usage + +```py +# settings.py + +INSTALLED_APPS = [ + # Other installed apps... + 'ai_blocker', +] + +MIDDLEWARE = [ + # Other middleware... + 'ai_blocker.middleware.AIBlockerMiddleware', +] +``` + +The precise location in the middleware chain where you add this middleware is up to you. + +## Configuration + +TBD \ No newline at end of file