<!--startmeta
custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/go/plugin/go.d/discovery/sdext/discoverer/httpsd/README.md"
meta_yaml: "https://github.com/netdata/netdata/edit/master/src/go/plugin/go.d/discovery/sdext/discoverer/httpsd/metadata.yaml"
sidebar_label: "HTTP endpoint"
learn_status: "Published"
learn_rel_path: "Collecting Metrics/Service Discovery/Discoverer"
keywords: ['service discovery', 'sd', 'http', 'rest', 'discovery', 'cmdb']
message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE SERVICE DISCOVERY DISCOVERER'S metadata.yaml FILE"
endmeta-->

# HTTP endpoint discovery


<img src="https://netdata.cloud/img/http.svg" width="150"/>


Kind: `http`

<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />

## Overview

Netdata can pull a list of monitorable targets from any HTTP endpoint you control — a CMDB API, an internal asset registry, a static file served by nginx, or a Prometheus-style file_sd export. The discoverer fetches the endpoint, decodes JSON or YAML, and feeds each item into the `services:` rule engine. This is the "bring your own source-of-truth" discoverer.

This page covers HTTP-specific setup. For the broader Service Discovery model and the shared template-helper reference, see [Service Discovery](https://github.com/netdata/netdata/blob/master/src/collectors/SERVICE-DISCOVERY.md).


### How it works

Each discovery cycle, the discoverer:

1. **Fetches** the configured `url` over HTTP/HTTPS, honouring all standard go.d collector HTTP options (auth, headers, TLS, proxy, timeout).
2. **Decodes** the response as either JSON or YAML according to `format` (auto / json / yaml). With `format: auto`, the decoder uses `Content-Type` if it is unambiguous, otherwise tries JSON first then YAML.
3. **Accepts two shapes** at the top level: a bare array (`[ item, item, … ]`) **or** an envelope (`{ "items": [ … ] }`). Anything else is rejected.
4. **Builds one target per array element**, exposing `.Item` (the decoded element — could be a string, a map, a number, …), `.TUID`, and `.Hash`.
5. **Runs the `services:` rules** against each target. The default stock rule passes the item through unchanged via the `toYaml` helper, so an endpoint that already serves go.d job configurations works with zero rule authoring.
6. **Reconciles** disappeared items — when a target is no longer in the response, the corresponding job stops on the next reconcile.


### Limitations

- Only **one URL per pipeline**. To pull from multiple sources, configure multiple HTTP discovery pipelines (each as its own UI entry, or split the file into one job per source).
- **Response size** is capped at 10 MiB.
- A DynCfg **test** performs one complete fetch only when `method` is empty/default or exactly `GET`. It uses the configured authentication, headers, TLS, proxy, redirects, and timeout; requires HTTP 200; and parses every returned item. Redirects can make up to 10 HTTP requests. The test discards the targets without evaluating `services:` rules or creating jobs. Other configured methods are intentionally validation-only.
- **One-shot mode** (`interval: 0`) fetches a single time when the pipeline starts. It does **not** refetch on SD reload — recreate the pipeline to refresh.
- A missing **`bearer_token_file`** under `/var/run/secrets/` is treated as optional when Netdata is **not** running in Kubernetes (so the same config can be used in a Helm deployment without erroring out on dev hosts).
- Configured bearer-token and TLS files have [file-safety requirements](#option-headers-username-password-bearer-token-file-proxy-url-tls-skip-verify-etc).
- The discoverer does not introspect the items it received — anything beyond what the upstream endpoint provides must be inferred via service rules.


## Setup

You can configure the `http` discoverer in two ways:

| Method | Best for | How to |
|:--|:--|:--|
| [**UI**](#via-ui) | Fast setup without editing files | Go to `Collectors -> go.d -> ServiceDiscovery -> http`, then add a discovery pipeline. |
| [**File**](#via-file) | File-based configuration or automation | Edit `/etc/netdata/go.d/sd/http.conf` and define the `discoverer:` and `services:` blocks. |

### Prerequisites

#### Endpoint that returns JSON or YAML

Stand up an HTTP endpoint that returns either a top-level array (`[ "https://a/health", "https://b/health" ]`) or an envelope (`{ "items": [...] }`). Items can be primitives (strings, numbers), maps, or any nestable value the rule engine knows how to consume.


#### Choose a pass-through vs. curated approach

- **Pass-through**: have your endpoint emit ready-made go.d job configurations and use the stock rule, which renders each item directly via `toYaml`. Zero rule authoring on the Netdata side.
- **Curated**: have your endpoint emit raw data (URLs, hostnames, tags) and write `services:` rules that map the data to the right collector module. More work, more flexibility.


### Configuration

#### Options

The configuration file has two top-level blocks: `discoverer:` (the options below) and `services:` (rules that turn fetched items into collector jobs — see [Service Rules](#service-rules)).

After editing the file, restart the Netdata Agent to load the updated discovery pipeline.



| Option | Description | Default | Required |
|:-----|:------------|:--------|:---------:|
| [url](#option-url) | HTTP/HTTPS endpoint that returns the items. |  | yes |
| [interval](#option-interval) | How often to refetch the endpoint. | 1m | no |
| [format](#option-format) | Response format. One of `auto`, `json`, `yaml`. | auto | no |
| timeout | Per-request HTTP timeout. | 2s | no |
| [headers / username / password / bearer_token_file / proxy_url / tls_skip_verify / etc.](#option-headers-username-password-bearer-token-file-proxy-url-tls-skip-verify-etc) | All standard go.d HTTP options are accepted (basic auth, bearer tokens, custom headers, HTTP proxy, TLS options). |  | no |

<a id="option-url"></a>
##### url

Must be a fully-qualified `http://` or `https://` URL. The endpoint is expected to return either a bare array or an `{"items": [...]}` envelope (see [Service Rules](#service-rules) for the input model).


<a id="option-interval"></a>
##### interval

Set to `0` for one-shot mode — the endpoint is fetched once when the pipeline starts and never again. SD reload does not retrigger; recreate the pipeline to refresh.


<a id="option-format"></a>
##### format

With `auto`, the decoder uses `Content-Type` when it is unambiguous (`application/json`, `application/yaml`, `*+json`, `*+yaml`), otherwise tries JSON first then YAML.


<a id="option-headers-username-password-bearer-token-file-proxy-url-tls-skip-verify-etc"></a>
##### headers / username / password / bearer_token_file / proxy_url / tls_skip_verify / etc.

See any go.d HTTP-based collector (`httpcheck`, `prometheus`, `nginx`, …) for the full set. Bearer-token and TLS CA paths must resolve to regular files no larger than 1 MiB. When both `tls_cert` and `tls_key` are configured, each path has the same requirement. Symlinks to regular files are supported. When `bearer_token_file` points under `/var/run/secrets/` and Netdata is **not** running inside Kubernetes, a missing token file is silently ignored.




#### via UI

1. Open the Netdata Dynamic Configuration UI.
2. Go to `Collectors -> go.d -> ServiceDiscovery -> http`.
3. Add a new discovery pipeline and give it a name.
4. Fill in the discoverer-specific settings and the service rules.
5. Save the discovery pipeline.

#### via File

Define the discovery pipeline in `/etc/netdata/go.d/sd/http.conf`.

The file has two top-level blocks: `discoverer:` (the options above) and `services:` (rules that turn discovered targets into collector jobs — see [Service Rules](#service-rules)).

After editing the file, restart the Netdata Agent to load the updated discovery pipeline.

##### Examples

###### Pass-through go.d jobs (stock rule)

The endpoint serves go.d job configurations directly. Each item must include a `module` field. The stock rule pipes the item through `toYaml` unchanged.


```yaml
disabled: no
discoverer:
  http:
    url: https://cmdb.example.com/netdata/jobs.yaml
    interval: 5m
    format: auto
services:
  - id: passthrough
    match: '{{ true }}'
    config_template: |
      {{ .Item | toYaml }}

```
###### Array of bare URLs → httpcheck

The endpoint returns `[ "https://a/health", "https://b/health" ]`. Map each URL to an `httpcheck` job.


```yaml
disabled: no
discoverer:
  http:
    url: https://cmdb.example.com/netdata/health-urls.json
    interval: 1m
services:
  - id: httpcheck
    match: '{{ kindIs "string" .Item }}'
    config_template: |
      name: {{ .TUID }}
      url: {{ .Item }}

```
###### Array of objects with custom shape

The endpoint returns `[ { "name": "api", "url": "https://api.example.com/health" }, … ]`.


```yaml
disabled: no
discoverer:
  http:
    url: https://cmdb.example.com/netdata/services.json
services:
  - id: httpcheck
    match: '{{ and (kindIs "map" .Item) (hasKey .Item "url") }}'
    config_template: |
      name: {{ .Item.name }}
      url: {{ .Item.url }}

```
###### Bearer-token authentication

Authenticate against the source-of-truth endpoint using a bearer token from a file.

```yaml
disabled: no
discoverer:
  http:
    url: https://cmdb.example.com/api/v1/netdata/jobs
    bearer_token_file: /etc/netdata/secrets/cmdb-token
    headers:
      Accept: application/yaml
services:
  - id: passthrough
    match: '{{ true }}'
    config_template: |
      {{ .Item | toYaml }}

```


## Service Rules

A `services:` rule turns each fetched item into one or more collector jobs. The HTTP discoverer is unique among SD discoverers in that the target's data shape is **defined by the upstream endpoint**, not by this discoverer — `.Item` is whatever JSON/YAML element the endpoint returned.

The shared rule model — function reference (`match`, sprig including `kindIs`/`hasKey`/`toYaml`), `config_template` rendering rules, and the `missingkey=error` failure semantics — lives on the [Service Discovery](https://github.com/netdata/netdata/blob/master/src/collectors/SERVICE-DISCOVERY.md) hub page. The notes below are HTTP-specific.


### How rules are evaluated

Quick reference — see [Rule evaluation semantics](https://github.com/netdata/netdata/blob/master/src/collectors/SERVICE-DISCOVERY.md#rule-evaluation-semantics) on the hub page for the full model.



- **Type-check `.Item` first** — Because `.Item` is whatever the endpoint serves, write defensive rules that check the type before reading sub-fields. `kindIs "string" .Item`, `kindIs "map" .Item`, and `hasKey .Item "<key>"` are the workhorses. A rule that does `{{ .Item.url }}` on a non-map item will fail at template-render time (`missingkey=error`) and the rule will be skipped.
- **Pass-through requires `module:` in the upstream payload** — The pass-through rule (`{{ .Item / toYaml }}`) forwards the item unchanged to the collector subsystem. The collector subsystem requires every job to have `name:` and `module:`. If your endpoint omits `module:`, the resulting job has no module and the agent rejects it. Either include `module:` upstream or wrap with a curated rule that adds it.
- **Module inference from rule id** — When you write a curated rule and the rendered job omits `module:`, the rule `id` is used as the module name. So `id: httpcheck` is enough to produce httpcheck jobs without writing `module: httpcheck` in every template.

### Template Variables

Available inside both `match` expressions and `config_template` bodies for HTTP targets.


| Variable | Type | Description |
|:---------|:-----|:------------|
| `.Item` | any | The decoded array element. Type depends on the upstream endpoint — could be a string, a number, a bool, a map, or a nested structure. Always type-check before reading sub-fields. |
| `.TUID` | string | Stable per-target ID (`http_<endpoint-label>_<hash>`). Useful as a job `name:` when the upstream payload does not provide one. |
| `.Hash` | uint64 | Hash of the item content. Used internally for change detection. |

### Examples

Each example shows one or more entries from the `services:` array. Order matters — see [How rules are evaluated](#how-rules-are-evaluated).

#### Pass-through (default stock rule)

The endpoint already returns valid go.d job configurations. Forward each item unchanged via `toYaml`. Each item **must** include a `module` field (and `name`).


```yaml
- id: passthrough
  match: '{{ true }}'
  config_template: |
    {{ .Item | toYaml }}

```

#### Array of strings → httpcheck (curated)

Endpoint serves `[ "https://a/health", "https://b/health" ]`. Use `kindIs "string"` to gate the rule, then map each string to an `httpcheck` job. `id: httpcheck` makes the module infer automatically.


```yaml
- id: httpcheck
  match: '{{ kindIs "string" .Item }}'
  config_template: |
    name: {{ .TUID }}
    url: {{ .Item }}

```

#### Array of objects → httpcheck (curated)

Endpoint serves `[ { "name": "api", "url": "https://api/health" }, … ]`. Type-check that the item is a map and has a `url` key, then map fields into the rendered job.


```yaml
- id: httpcheck
  match: '{{ and (kindIs "map" .Item) (hasKey .Item "url") }}'
  config_template: |
    name: {{ .Item.name }}
    url: {{ .Item.url }}

```

#### Multiple modules from one endpoint

Your endpoint mixes shapes — some items target `httpcheck`, some target `prometheus`. Use `hasKey` to discriminate, with each rule producing its own module's jobs.


```yaml
- id: prometheus
  match: '{{ and (kindIs "map" .Item) (hasKey .Item "metrics_url") }}'
  config_template: |
    name: {{ .Item.name }}
    url: {{ .Item.metrics_url }}

- id: httpcheck
  match: '{{ and (kindIs "map" .Item) (hasKey .Item "health_url") }}'
  config_template: |
    name: {{ .Item.name }}
    url: {{ .Item.health_url }}

```



## Verify discovery worked

After enabling the discoverer, confirm the endpoint is reachable and items are being parsed.

### Test a UI-managed configuration

DynCfg test performs one complete production fetch for an empty/default or exact `GET` method. Success proves that the request could be built, the configured auth/TLS/proxy/redirect path worked at that instant, the endpoint returned HTTP 200 with at most 10 MiB, and every item parsed into a target.

The fetched targets are discarded: the test does not evaluate `services:` rules, render or validate collector jobs, publish targets, or install jobs. Non-GET methods are intentionally reported as validation-only because executing them could mutate the endpoint.


### Confirm the endpoint is being fetched

Watch the agent log for `discoverer=http` messages. With systemd:

```bash
journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep "discoverer=http"
```

A successful fetch logs the number of items decoded. Failures (DNS, TLS, auth, parse) appear at warn level.


### Reproduce the fetch with curl

When the discoverer log shows a parse error, hit the endpoint with `curl` to inspect what it returned:

```bash
curl -sS -H "Accept: application/yaml" https://cmdb.example.com/netdata/jobs.yaml | head -40
```

The response **must** be a top-level array or `{"items": [...]}` envelope.


### Confirm jobs are being created

In the Netdata UI go to `Collectors -> go.d -> <module>`. Pass-through jobs use the `name` your endpoint provided; curated rules use whatever you set in the `config_template`.




## Troubleshooting

### parse response as json: ...; parse response as yaml: ...

The response is neither valid JSON nor valid YAML. Common causes: the endpoint returned an HTML error page (check status code and `Content-Type`), the JSON has trailing garbage, or YAML indentation is wrong. Reproduce with `curl -i` to see the headers + body.


### Items decoded but no jobs created

Your `services:` rules are not matching, or they match but the rendered template is empty. With pass-through (`{{ .Item | toYaml }}`), make sure each upstream item includes `module:` and `name:`. With curated rules, double-check the type checks (`kindIs`, `hasKey`).


### TLS/certificate errors against an internal endpoint

Use `tls_skip_verify: yes` to bypass for testing, then mount the issuing CA and set `tls_ca: /path/to/ca.crt` for production.


### Bearer token file not found

When Netdata runs **outside** Kubernetes and the configured `bearer_token_file` points under `/var/run/secrets/`, missing tokens are silently ignored — this is intentional so the same config works in dev and in Helm. If you are inside k8s, the file must exist.
