TORinator

A Tor IP address indexer. A local database is maintained and regularly updated from multiple sources. By default, the sources are scraped every 45 minutes in order to prevent rate limiting / blocking. You can override this using an enviornment variable - details below.

API Docs

# Get All IPs
curl -s http://127.0.0.1:5000/api/v1/ips | jq '.'

# Get All Exclusions
curl -s http://127.0.0.1:5000/api/v1/exclusions | jq '.'

# Add Exclusion IP
curl -X POST http://127.0.0.1:5000/api/v1/exclusions \
  -H 'Content-Type: application/json' \
  -d '{ "ip": "202.59.9.238" }' | jq '.'

# Remove Exclusion IP
curl -X DELETE http://127.0.0.1:5000/api/v1/exclusions/202.59.9.238

Configuration

The application can be configured using environment variables. The following vars are available:

Environment Variable	Description	Default
TORINATOR_DB_TYPE	Database type (memory / sqlite)	sqlite
TORINATOR_DATA_PATH	Path to store the database	./
TORINATOR_UPDATE_SOURCE	Update source (web / file )	web
TORINATOR_UPDATE_CADENCE	Update cadence in seconds	2700

You can find examples in the docker-compose.yml file.

Running (Production)

# Be sure image is built
docker-compose build

# Run compose in background. Service accessible via localhost:5000
docker-compose up -d

# Alternatively, in virtualenv
torinator run

Running (Development)

Once you have installed the setup.py and torinator[dev] dependencies, you can run pre-commit to lint all server files:

pre-commit run --all-files

To run:

# Setup virtual env
python3 -m venv torinator_venv
. torinator_venv/bin/activate

# Link torinator to working directory & install dev dependencies
python setup.py develop easy_install torinator[dev]

# Can use to run the server
torinator run

# Or with env vars
TORINATOR_UPDATE_SOURCE=file torinator run

Adding Sources

While a little cumbersome, you can add additional sources in the ./src/torinator/list_updater.py file. In the __init__, add additional sources with their URL's, parsing function, and filepath (if desired). Using a filepath allows you to reference a local file for parsing instead of hitting the web everytime. This is useful during development to prevent rate limiting.

Local files should be stored in ./src/torinator/resources/* and are copied by the MANIFEST.in during build.

2.5 KiB Raw Blame History