<!-- markdownlint-disable-file MD013 MD043 -->

# topology-ip-intel-downloader

`topology-ip-intel-downloader` builds the Netdata IP intelligence databases used by
topology and netflow enrichment.

It always writes a fixed Netdata output set:

- ASN MMDB: `topology-ip-asn.mmdb`
- GEO MMDB: `topology-ip-geo.mmdb`
- metadata JSON: `topology-ip-intel.json`

The GEO MMDB is not country-only. It can contain:

- `country.iso_code`
- `city.names.en`
- `region`
- `subdivisions`
- `location.latitude`
- `location.longitude`

Both MMDB outputs also include Netdata classification metadata under `netdata.*`
for CIDRs that must be tracked individually.

## Stock databases and refreshed databases

Netdata packages ship a stock payload under:

- `${NETDATA_STOCK_DATA_DIR}/topology-ip-intel/topology-ip-asn.mmdb`
- `${NETDATA_STOCK_DATA_DIR}/topology-ip-intel/topology-ip-geo.mmdb`
- `${NETDATA_STOCK_DATA_DIR}/topology-ip-intel/topology-ip-intel.json`

Refreshed databases generated by this downloader are written to:

- `${NETDATA_CACHE_DIR}/topology-ip-intel`

The netflow plugin auto-detects the cache copy first and then falls back to the
stock copy. This means:

- packaged stock files provide out-of-box enrichment
- downloader runs update the cache copy
- the refreshed cache copy overrides the packaged stock copy automatically

Important:

- source installs from a Git checkout do not carry the packaged stock MMDBs
- packaged builds stage the stock payload from CI/release tooling
- local/source installs should run the downloader if they want a local cache copy
- 32-bit packaged builds still ship the staged stock payload, but do not build or install the downloader binary

## Config lookup

If `--config` is not provided, the downloader loads the first existing file from:

1. `/etc/netdata/topology-ip-intel.yaml`
2. `/usr/lib/netdata/conf.d/topology-ip-intel.yaml`

These paths are resolved from the compiled Netdata install layout, so prefixed
installs use their own prefix automatically.

## Common CLI usage

The CLI is optimized for the common case:

- `--asn provider:artifact[@format]`
- `--geo provider:artifact[@format]`
- `--no-asn`
- `--no-geo`

Rules:

- `--asn` affects only ASN sources
- `--geo` affects only GEO sources
- repeated flags are ordered by precedence
- first source wins when address ranges overlap
- `--no-asn` or `--no-geo` removes that output file if it exists

Examples:

Use the built-in defaults:

```bash
topology-ip-intel-downloader
```

Refresh both families explicitly from DB-IP:

```bash
topology-ip-intel-downloader \
  --asn dbip:asn-lite \
  --geo dbip:city-lite
```

Use public ASN and country providers:

```bash
topology-ip-intel-downloader \
  --asn iptoasn:combined \
  --geo ip2location:country-lite
```

Use MaxMind GeoLite2 sources when `MAXMIND_LICENSE_KEY` is available in the
environment:

```bash
MAXMIND_LICENSE_KEY="..." topology-ip-intel-downloader \
  --asn maxmind:geolite2-asn \
  --geo maxmind:geolite2-country
```

Use CAIDA RouteViews prefix2as for ASN-only attribution:

```bash
topology-ip-intel-downloader \
  --asn caida:prefix2as \
  --geo dbip:country-lite
```

Prefer a custom GEO source, then fall back to DB-IP:

```bash
topology-ip-intel-downloader \
  --geo iptoasn:combined \
  --geo dbip:city-lite
```

Disable GEO output entirely for one run:

```bash
topology-ip-intel-downloader --no-geo
```

The downloader prints the effective execution plan before downloading, including:

- effective ASN source order
- effective GEO source order
- which output files will be written
- which output files will be removed

## Advanced config

The config file uses ordered `sources[]` entries. Each entry is explicit about:

- `family`: `asn` or `geo`
- `provider`
- `artifact`
- `format`
- optional `url` or `path`

Built-in DB-IP sources can omit `url` and `path`. The downloader resolves the
current monthly download URL from the official DB-IP landing page.

Built-in MaxMind sources require `MAXMIND_LICENSE_KEY` in the environment.
The generated metadata redacts URL query strings so license keys are not written
to `topology-ip-intel.json`.

User-provided databases can point directly to:

- a local file with `path`
- a direct URL with `url`

Example:

```yaml
sources:
  - name: custom-asn
    family: asn
    provider: dbip
    artifact: asn-lite
    path: /srv/ip-intel/custom-asn.mmdb

  - name: fallback-asn
    family: asn
    provider: dbip
    artifact: asn-lite
    format: mmdb

  - name: custom-geo
    family: geo
    provider: dbip
    artifact: city-lite
    url: https://example.internal/geo.mmdb.gz
    format: mmdb

output:
  directory: /var/cache/netdata/topology-ip-intel
  asn_file: topology-ip-asn.mmdb
  geo_file: topology-ip-geo.mmdb
  metadata_file: topology-ip-intel.json
```

Important:

- source order is per family
- earlier entries win on overlap
- removing every source of a family disables that family
- when a family is disabled, its stale output file is deleted on the next run

## Supported built-in sources

ASN sources:

- `dbip:asn-lite` (`mmdb`, `csv`)
- `iptoasn:combined` (`tsv`)
- `caida:prefix2as` (`tsv`)
- `maxmind:geolite2-asn` (`mmdb`, requires `MAXMIND_LICENSE_KEY`)

GEO sources:

- `dbip:country-lite` (`mmdb`, `csv`)
- `dbip:city-lite` (`mmdb`, `csv`)
- `iptoasn:combined` (`tsv`, country only)
- `maxmind:geolite2-country` (`csv`, requires `MAXMIND_LICENSE_KEY`, country only)
- `ip2location:country-lite` (`csv`, country only)
- `ipdeny:country-zones` (`cidr`, country only)
- `ipip:country` (`txt`, country only)

Supported formats:

- `mmdb`
- `csv`
- `tsv` for `iptoasn:combined` and `caida:prefix2as`
- `cidr` for `ipdeny:country-zones`
- `txt` for `ipip:country`

The downloader accepts direct file/URL config for advanced cases, but the CLI
source tokens are intentionally focused on built-in source families.

## Output behavior

Every successful run stages all files privately and then publishes each final
output atomically with `rename()`.

This gives two guarantees:

- readers never see a half-written MMDB file
- the final visible output set matches the currently enabled families

If a family is disabled:

- its final MMDB file is removed
- metadata is rewritten to describe the new output set

## Preparing a local stock payload staging directory

From the repository root:

```bash
./src/go/tools/topology-ip-intel-downloader/refresh-stock.sh
```

This generates a local staging directory at:

- `./artifacts/topology-ip-intel-stock/README.md`
- `./artifacts/topology-ip-intel-stock/topology-ip-asn.mmdb`
- `./artifacts/topology-ip-intel-stock/topology-ip-geo.mmdb`
- `./artifacts/topology-ip-intel-stock/topology-ip-intel.json`

This is useful for:

- checking the current packaged stock payload locally
- preparing a staged payload for package/release work
- verifying DB-IP downloads without storing generated binaries in Git
