84 lines
2.5 KiB
Markdown
84 lines
2.5 KiB
Markdown
|
# TORinator
|
||
|
|
A Tor IP address indexer. A local database is maintained and regularly updated
|
||
|
|
from multiple sources. By default, the sources are scraped every 45 minutes in
|
||
|
|
order to prevent rate limiting / blocking. You can override this using an
|
||
|
|
enviornment variable - details below.
|
||
|
|
|
||
|
|
|
||
|
|
## API Docs
|
||
|
|
|
||
|
|
# Get All IPs
|
||
|
|
curl -s http://127.0.0.1:5000/api/v1/ips | jq '.'
|
||
|
|
|
||
|
|
# Get All Exclusions
|
||
|
|
curl -s http://127.0.0.1:5000/api/v1/exclusions | jq '.'
|
||
|
|
|
||
|
|
# Add Exclusion IP
|
||
|
|
curl -X POST http://127.0.0.1:5000/api/v1/exclusions \
|
||
|
|
-H 'Content-Type: application/json' \
|
||
|
|
-d '{ "ip": "202.59.9.238" }' | jq '.'
|
||
|
|
|
||
|
|
# Remove Exclusion IP
|
||
|
|
curl -X DELETE http://127.0.0.1:5000/api/v1/exclusions/202.59.9.238
|
||
|
|
|
||
|
|
|
||
|
|
## Configuration
|
||
|
|
The application can be configured using environment variables. The following
|
||
|
|
vars are available:
|
||
|
|
|
||
|
|
| Environment Variable | Description | Default |
|
||
|
|
|--------------------------|---------------------------------|---------|
|
||
|
|
| TORINATOR_DB_TYPE | Database type (memory / sqlite) | sqlite |
|
||
|
|
| TORINATOR_DATA_PATH | Path to store the database | ./ |
|
||
|
|
| TORINATOR_UPDATE_SOURCE | Update source (web / file ) | web |
|
||
|
|
| TORINATOR_UPDATE_CADENCE | Update cadence in seconds | 2700 |
|
||
|
|
|
||
|
|
You can find examples in the `docker-compose.yml` file.
|
||
|
|
|
||
|
|
|
||
|
|
## Running (Production)
|
||
|
|
```
|
||
|
|
# Be sure image is built
|
||
|
|
docker-compose build
|
||
|
|
|
||
|
|
# Run compose in background. Service accessible via localhost:5000
|
||
|
|
docker-compose up -d
|
||
|
|
|
||
|
|
# Alternatively, in virtualenv
|
||
|
|
torinator run
|
||
|
|
```
|
||
|
|
|
||
|
|
## Running (Development)
|
||
|
|
Once you have installed the setup.py and torinator[dev] dependencies, you can
|
||
|
|
run `pre-commit` to lint all server files:
|
||
|
|
|
||
|
|
```
|
||
|
|
pre-commit run --all-files
|
||
|
|
```
|
||
|
|
|
||
|
|
To run:
|
||
|
|
```
|
||
|
|
# Setup virtual env
|
||
|
|
python3 -m venv torinator_venv
|
||
|
|
. torinator_venv/bin/activate
|
||
|
|
|
||
|
|
# Link torinator to working directory & install dev dependencies
|
||
|
|
python setup.py develop easy_install torinator[dev]
|
||
|
|
|
||
|
|
# Can use to run the server
|
||
|
|
torinator run
|
||
|
|
|
||
|
|
# Or with env vars
|
||
|
|
TORINATOR_UPDATE_SOURCE=file torinator run
|
||
|
|
```
|
||
|
|
|
||
|
|
### Adding Sources
|
||
|
|
While a little cumbersome, you can add additional sources in the
|
||
|
|
`./src/torinator/list_updater.py` file. In the `__init__`, add additional
|
||
|
|
sources with their URL's, parsing function, and filepath (if desired). Using
|
||
|
|
a filepath allows you to reference a local file for parsing instead of hitting
|
||
|
|
the web everytime. This is useful during development to prevent rate limiting.
|
||
|
|
|
||
|
|
Local files should be stored in `./src/torinator/resources/*` and are copied by
|
||
|
|
the `MANIFEST.in` during build.
|