overseer/README.md

<p align="center">
    <img src="https://gitea.va.reichard.io/evan/overseer/raw/branch/master/resources/overseer.png" width="600">
</p>

<p align="center">
    <a href="https://gitea.va.reichard.io/evan/overseer/raw/branch/master/resources/screenshots/home.png">
        <img src="https://gitea.va.reichard.io/evan/overseer/raw/branch/master/resources/screenshots/home.png" width="24%">
    </a>
    <a href="https://gitea.va.reichard.io/evan/overseer/raw/branch/master/resources/screenshots/ongoing_scan.png">
        <img src="https://gitea.va.reichard.io/evan/overseer/raw/branch/master/resources/screenshots/ongoing_scan.png" width="24%">
    </a>
    <a href="https://gitea.va.reichard.io/evan/overseer/raw/branch/master/resources/screenshots/differences.png">
        <img src="https://gitea.va.reichard.io/evan/overseer/raw/branch/master/resources/screenshots/differences.png" width="24%">
    </a>
    <a href="https://gitea.va.reichard.io/evan/overseer/raw/branch/master/resources/screenshots/no_differences.png">
        <img src="https://gitea.va.reichard.io/evan/overseer/raw/branch/master/resources/screenshots/no_differences.png" width="24%">
    </a>
</p>

---


A port scanning web interface that allows you to compare results while storing
them to a database.

## Running
```
# Be sure image is built
docker-compose build

# Run compose in background. Service accessible via localhost:5000
docker-compose up -d

# Alternatively, in virtualenv
overseer run
```

### Configuration
The application can be configured using environment variables. The following
vars are available:

| Environment Variable | Description                     | Default |
|----------------------|---------------------------------|---------|
| DB_TYPE              | Database type (memory / sqlite) | sqlite  |
| DATA_PATH            | Path to store the database      | ./      |

You can find a commented out example in the `docker-compose.yml` file.

## Developing
Once you have installed the setup.py and overseer[dev] dependencies, you can
run `pre-commit` to lint all server and client files:

```
pre-commit run --all-files
```

### Server
```
# Setup virtual env
python3 -m venv overseer_venv
. overseer_venv/bin/activate

# Link overseer to working directory & install dev dependencies
python setup.py develop easy_install overseer[dev]

# Can use to run the server
overseer run
```

### Client
```
# Change directory
cd ./src/overseer_client

# On the fly reloading (no server access)
yarn serve

# Build resources and auto populate in server static / template resources
yarn build

# Lint (this is covered by the pre-commit)
yarn lint
```


## Testing
Once the above development steps are completed, the pytest dependency should
be installed:
```
pytest
```

If you want to quickly test the scanners ability, you can spin up some TCP
listeners:
```
# Listens on ports 60 -> 70
for i in {60..70}; do (nc -l "$i" &) ; done
```

## Design
### Server
The server is written in Python and uses Flask and SQLAlchemy. I wanted to have
the capability to extend to another database fairly easily if so. This gave me
that. I also used Flask's Blueprint to allow easy versioning of the API.

In addition to regular POST / GET API endpoints, I added WebSocket support.
This allows us to see the progress of the scan without polling, and makes the
overrall user experience better.

For the scanner, we have the ScanManager and the Scanner class itself. The
manager is responsible initiating new scans, publishing the status to the
websocket, and committing status changes to the Database.

The scanner itself threads the TCP and UDP scan. It also parses the
`nmap-payloads` file to build a dictionary of UDP payloads to check for
responses to.

### Client
The client is writting in Vue.js framework. This makes maintaining easier and
provides us with a very reactive interface.

Some capabilities:

- Notification drawer with live progress and clickable notifications
- Live TCP / UDP / Total progress bars
- View all scans for a capability and see port results come in on the fly
- Compare any scan with its preceding scan
- Clicking the target (host / ip) brings you to all scans