<!--startmeta
custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/generic_json-over-http_ipam.md"
meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml"
sidebar_label: "Generic JSON-over-HTTP IPAM"
learn_status: "Published"
learn_rel_path: "Network Performance Monitoring/Network Flows/Enrichment Methods"
keywords: ['ipam', 'cmdb', 'infoblox', 'bluecat', 'phpipam', 'servicenow', 'custom', 'prefix list', 'network identity']
message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE"
endmeta-->

<!-- markdownlint-disable-file -->

# Generic JSON-over-HTTP IPAM


<img src="https://netdata.cloud/img/network-wired.svg" width="150"/>


Plugin: netflow-plugin
Module: generic-ipam

<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />

## Overview

Annotate network flows with network-identity labels from any JSON-over-HTTP IPAM
or CMDB endpoint. The plugin does not care WHO produces the JSON -- it cares
about the JSON shape after your `transform` (a jq expression) runs over it.
Examples: Infoblox WAPI, BlueCat REST API, phpIPAM, ServiceNow CMDB queries, an
internal aggregator, a Lambda function, a static file served from S3, a
hand-rolled CMDB.

This card is also the **reference description** of how all the other Network
Identity cards work under the hood. AWS IP Ranges, GCP IP Ranges, Azure IP Ranges
and NetBox are all special-cased instances of this same mechanism: each is just
a known URL plus a recommended jq `transform`. Read this card to understand the
generic shape, then read the per-source cards for the URL and transform that fit
that specific provider.

For the full network-identity concept (merge order vs GeoIP and static, what
fields you can populate, why TLS verification cannot be disabled), see
[Network Identity](https://github.com/netdata/netdata/blob/master/docs/npm/network-flows/enrichment.md).


The plugin issues a periodic HTTP request (GET by default, POST optionally) to
your configured URL with the headers you specify, parses the response as JSON,
runs your `transform` jq expression (compiled by the
[jaq](https://github.com/01mf02/jaq) library) over the parsed body, and merges
the resulting per-prefix rows into the network-attributes trie.

The same flow applies to every JSON-over-HTTP source; the integration-specific
part is the URL and the `transform` expression.


This integration is supported on all platforms.

This integration supports multiple instances configured side-by-side.


### Default Behavior

#### Auto-Detection

Disabled by default. Add a named entry under `enrichment.network_sources` for each IPAM/CMDB/endpoint you want to import.

#### Limits

Resource use scales with response size, transform complexity, refresh interval, and the number of emitted prefixes. Empty transform output is treated as a fetch failure.

#### Performance Impact

One HTTP request per refresh interval plus a jq transform over the response. Runtime enrichment does prefix matching for source and destination IPs, and cost scales with the number of loaded network-source records.

## Setup


### Prerequisites

#### An HTTP/HTTPS endpoint returning JSON

The endpoint must respond with a parseable JSON document (the plugin sets
`Accept: application/json`). Only GET and POST are accepted as request
methods. There is no pagination, no
cursor following, no `Link: rel=next` handling -- the fetch is one-shot
per cycle. If your IPAM paginates, expose a bulk endpoint or wrap it in a
server-side aggregator that returns the full list at one URL.


#### Authentication via headers

The plugin has no built-in OAuth flow, basic-auth helper, or token
refresh. Whatever the API needs (bearer tokens, API keys, custom header
names, basic-auth realms) goes into the `headers:` map. The header map is
an arbitrary string-to-string mapping, so any single-shot scheme works.
For short-lived tokens, refresh them outside Netdata and reload the
plugin config.


#### A POST endpoint must accept an empty body

When `method: POST` is configured, the plugin sends the request with the
configured headers but **no request body**. If your CMDB requires a JSON query body to return prefixes,
wrap it server-side with an endpoint that accepts GET (or POST with no
body) and returns the full prefix set.



### Configuration

#### Options

Add a named entry under `enrichment.network_sources`. Unknown keys cause a
config error.


<details open><summary>Config options</summary>



| Option | Description | Default | Required |
|:-----|:------------|:--------|:---------:|
| url | HTTP/HTTPS endpoint. Required (a non-empty URL is enforced by validation). |  | yes |
| method | HTTP method. Only `GET` and `POST` are accepted; anything else fails validation. Note that `POST` is sent with no request body. | GET | no |
| headers | Map of additional HTTP request headers. Use this for any authentication scheme (`Authorization: Bearer ...`, `token: ...`, custom API-key headers, basic-auth realms encoded explicitly, etc.). Values are passed through verbatim. | {} | no |
| interval | Refresh cadence. Values below 60 seconds are accepted, but refreshes still run no faster than once per minute. Pick the value that matches how often your IPAM actually changes (5-15 minutes for a curated CMDB; daily for slow-moving prefix lists). | 60s | no |
| timeout | Per-request timeout. Must be greater than 0. | 60s | no |
| proxy | Whether to honor the system HTTP/HTTPS proxy environment variables. Set to `false` to bypass the proxy for this source (useful when the IPAM is on the internal network and the proxy is for outbound traffic only). | true | no |
| tls.enable | Enables custom TLS settings (custom CA bundle, mTLS client certificate). Must be `true` whenever any of `tls.ca_file`, `tls.cert_file`, `tls.key_file` is set. | false | no |
| tls.verify | Certificate verification toggle. **Setting this to `false` is rejected by validation** -- network-identity data flows directly into security-relevant enrichment, so MITM-able responses are not allowed. Use `tls.ca_file` to trust internal CAs instead. | true | no |
| tls.skip_verify | Legacy alias for the same intent as `tls.verify: false`. **Setting this to `true` is rejected by validation.** Use `tls.ca_file`. | false | no |
| tls.ca_file | PEM file with the CA bundle to trust for this endpoint (instead of, or in addition to, the system roots). The recommended way to talk to an internal IPAM behind your own PKI. |  | no |
| tls.cert_file | PEM file with the client certificate, for mTLS-protected endpoints. |  | no |
| tls.key_file | PEM file with the client private key. Required only when the key is in a separate file from the certificate; if omitted, the cert file is reused. `tls.cert_file` must also be set whenever `tls.key_file` is set. |  | no |
| transform | jq expression compiled by [jaq](https://github.com/01mf02/jaq). Receives the entire parsed JSON body once and must produce a **stream of objects**, where each object has at minimum a `prefix` field (a CIDR string -- `IPv4/len` or `IPv6/len`) plus any of the optional attribute fields: `name`, `role`, `site`, `region`, `country`, `state`, `city`, `tenant`, `asn`, `asn_name`. Any field not produced is treated as empty. The transform compiles at startup -- a syntax error fails the config load. An empty stream at runtime is treated as a fetch failure and triggers backoff. | . | yes |


</details>



#### via File

The configuration file name for this integration is `netflow.yaml`.


You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the
Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory).

```bash
cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config netflow.yaml
```

##### Examples

###### Flat list with `{prefix, name, env}`

The simplest IPAM shape: a top-level array of subnets, each with a CIDR
string and a few labels. Maps `env` -> tenant and the `name` -> NET_NAME.


```yaml
enrichment:
  network_sources:
    custom_ipam:
      url: "https://ipam.corp.example/api/v1/prefixes"
      headers:
        Authorization: "Bearer abcdef0123456789"
      interval: 5m
      transform: |
        .[] | {
          prefix: .prefix,
          name: .name,
          tenant: .env
        }

```
###### phpIPAM with API token

phpIPAM exposes `/api/<APP>/subnets/`. Replace `<APP>` with your phpIPAM
application name. phpIPAM splits the address into a base and a mask --
the transform reassembles them into a CIDR string.


<details open><summary>Config</summary>

```yaml
enrichment:
  network_sources:
    phpipam:
      url: "https://ipam.example/api/netdata/subnets/"
      headers:
        token: "abcdef..."
      interval: 10m
      transform: |
        .data[] | {
          prefix: (.subnet + "/" + (.mask|tostring)),
          name: .description,
          tenant: (.custom_tenant // ""),
          site: (.location.name // "")
        }

```
</details>

###### Hierarchical CMDB export

A CMDB that returns nested objects -- top-level sites, each with a list
of vlans, each with a list of prefixes. Flattens to one row per prefix,
inheriting site and role labels from the parent.


<details open><summary>Config</summary>

```yaml
enrichment:
  network_sources:
    cmdb:
      url: "https://cmdb.example/export/networks.json"
      headers:
        Authorization: "Bearer <CMDB_TOKEN>"
      interval: 15m
      transform: |
        .sites[] as $site
        | $site.vlans[] as $vlan
        | $vlan.prefixes[]
        | {
            prefix: .cidr,
            name: .label,
            site: $site.name,
            region: $site.region,
            role: $vlan.role,
            tenant: $vlan.tenant
          }

```
</details>

###### Internal IPAM behind mTLS

When the IPAM is fronted by your internal PKI and requires a client
certificate. `tls.enable: true` activates the custom TLS path; `tls.verify`
must remain true (cannot be disabled).


<details open><summary>Config</summary>

```yaml
enrichment:
  network_sources:
    corp_ipam:
      url: "https://ipam.corp/api/networks"
      tls:
        enable: true
        ca_file: /etc/netdata/ssl/corp-ca.pem
        cert_file: /etc/netdata/ssl/netdata.crt
        key_file: /etc/netdata/ssl/netdata.key
      interval: 10m
      transform: |
        .[] | {
          prefix: .cidr,
          name: .label,
          tenant: .tenant
        }

```
</details>

###### Source with the system proxy bypassed

When the agent host has an outbound HTTP proxy configured but the IPAM
lives on the internal network and should be reached directly.


<details open><summary>Config</summary>

```yaml
enrichment:
  network_sources:
    internal_ipam:
      url: "https://ipam.internal/api/networks"
      proxy: false
      headers:
        Authorization: "Bearer ..."
      interval: 5m
      transform: |
        .[] | { prefix: .cidr, name: .label }

```
</details>



### Endpoint requires pagination

The plugin does not paginate. Either raise the page size in the URL to
cover your full inventory, or wrap the endpoint with a server-side
aggregator that returns all results at one URL. There is no built-in
`Link: rel=next` follower.


### POST endpoint requires a request body

The plugin's POST request is sent with no body. If your CMDB requires a
JSON query body to return prefixes, wrap it server-side with an endpoint
that accepts GET (or accepts POST with no body) and returns the full prefix
set.


### TLS verification cannot be disabled

`tls.verify: false` and `tls.skip_verify: true` are both rejected during
configuration validation. Use `tls.ca_file` to trust internal CAs. This is
deliberate -- network-identity rows feed enrichment used in security
investigations and capacity decisions, where silently accepting MITM-able
responses would corrupt every downstream analysis.


### Empty result back-off

An empty stream from the jq transform is treated as a fetch failure.
The source then backs
off exponentially -- starting at `interval / 10` (floor 1s), doubling on
each consecutive failure, and capped at the regular `interval`. On the
next successful non-empty fetch the cadence resets to `interval`. If your
IPAM legitimately has no prefixes (a quiet state), have the upstream
return at least one synthetic prefix so the source does not back off.


### Refresh appears slower than configured

The fetch loop floors the configured `interval` at 60 seconds. Configuring
`interval: 5s` does not produce a 5-second loop -- it produces a 60-second
loop. Pick a value at or above 60s that matches how often your IPAM
actually changes (5-15 minutes is typical for a curated CMDB).


### Unknown config keys cause errors

The config struct uses `deny_unknown_fields`. Typos like `headres:` or a
non-existent option fail config load with a parse error rather than being
silently ignored. Check the `enrichment.network_sources.<name>` schema
listed under "Config options".


### JSON parse errors are silent in the dashboard

Decode failures (HTTP error, JSON parse error, jq runtime error, schema
mismatch on `prefix`) are logged but do not surface in the dashboard.
Watch the Netdata journal for warnings:
`journalctl --namespace netdata | grep network_sources`.


### Prefer explicit authorization headers over URL credentials

URLs with embedded credentials (`https://user:pass@host`) are converted to
HTTP Basic authentication by the HTTP client. Prefer `headers:` for clarity
and to avoid storing credentials in URLs -- e.g.
`headers: { Authorization: "Basic dXNlcjpwYXNz" }` for HTTP basic-auth.
