# Recipe: add a new go.d collector integration

This recipe assumes you are adding a brand-new go.d module
called `<name>`. For modifying an existing collector, see
`update-collector.md`.

## 0. Read first

- `<repo>/.agents/skills/project-writing-collectors/SKILL.md`
  -- the broader "how to write a collector" context (NIDL
  contexts, dashboard shaping, plugin landscape).
- `<repo>/src/go/plugin/go.d/docs/how-to-write-a-collector.md`
  -- the canonical framework V2 code/layout guide for new go.d collectors.
- `../SKILL.md` -- this skill's overview.
- `../schema-reference.md` -- the `collector.json` schema
  fields you will be filling in.

## 1. Create the module skeleton

New go.d collectors MUST use framework V2. Code layout details live in
`src/go/plugin/go.d/docs/how-to-write-a-collector.md`; this recipe covers the
integration artifact side. A normal V2 collector directory includes:

```
src/go/plugin/go.d/collector/<name>/
|-- collector.go       # Register/CreateV2/New/public lifecycle
|-- config.go          # Config struct
|-- collect.go         # Collect orchestration
|-- metrix.go          # Typed metrix instruments
|-- write_metrics.go   # Metric writes
|-- charts.yaml        # V2 chart template
|-- config_schema.json # DYNCFG schema
|-- metadata.yaml      # Integration metadata (this skill's territory)
|-- taxonomy.yaml      # Dashboard TOC placement
|-- README.md          # Will become a symlink to integrations/<slug>.md once gen runs
|-- testdata/          # Fixtures
`-- ...other .go files
```

Plus stock conf:

```
src/go/plugin/go.d/config/go.d/<name>.conf
```

Plus alerts (if any):

```
src/health/health.d/<name>.conf
```

## 2. Author `metadata.yaml`

Use an existing rich collector as a template:
`src/go/plugin/go.d/collector/postgres/metadata.yaml`.

Required top-level fields per `collector.json`:

```yaml
plugin_name: go.d.plugin
modules:
  - meta:
      plugin_name: go.d.plugin
      module_name: <name>
      monitored_instance:
        name: "<Display Name>"
        link: "https://upstream-site.example/"
        categories:
          - data-collection.<category>     # see categories.yaml for valid ids
        icon_filename: "<name>.svg"
      keywords: [<keywords>]
      related_resources:
        integrations:
          list: []
      info_provided_to_referring_integrations:
        description: ""
    overview:
      data_collection:
        metrics_description: |
          First sentence: Monitor <thing> or collect <data> from <thing>.
          Add more detail after that only if it is useful on the full page.
        method_description: |
          One paragraph: how we collect it.
      supported_platforms:
        include: []
        exclude: []
      multi_instance: true
      additional_permissions:
        description: ""
      default_behavior:
        auto_detection:
          description: ""
        limits:
          description: ""
        performance_impact:
          description: ""
    setup:
      prerequisites:
        list: []
      configuration:
        file:
          name: "go.d/<name>.conf"
        options:
          description: ""
          folding:
            title: "Config options"
            enabled: true
          list: []
        examples:
          folding:
            title: "Config"
            enabled: true
          list:
            - name: "Basic"
              description: "Basic configuration."
              config: |
                jobs:
                  - name: local
                    url: http://localhost:1234
    troubleshooting:
      problems:
        list: []
    alerts: []
    metrics:
      folding:
        title: "Metrics"
        enabled: false
      description: ""
      availability: []
      scopes:
        - name: global   # will be auto-rewritten to "<Display Name> instance"
          description: ""
          labels: []
          metrics:
            - name: <name>.<context>
              description: <Chart title>
              unit: <unit>
              chart_type: line   # one of: line, area, stacked, heatmap
              dimensions:
                - name: <dim>
```

The first sentence of `metrics_description` is also used as the
description in generated catalog-style pages such as
`src/collectors/COLLECTORS.md`. It SHOULD stay product-facing and stable:
start with an action phrase, describe the integration, and MUST NOT
describe configuration variables, defaults, limits, or setup steps.
Put those details in the setup, default-behavior, examples, or
troubleshooting fields.

Hit every REQUIRED field. The validator is strict (fatal on
warnings). Refer to `../schema-reference.md` for the
exhaustive field list.

## 3. Make sure `categories.yaml` has your category

If your `monitored_instance.categories` references a category
that doesn't exist in `integrations/categories.yaml`, the
validator will warn (fatal). Either pick an existing category
or add a new one under the appropriate parent (typically
`data-collection`).

## 4. Taxonomy, stock `.conf`, `config_schema.json`, alerts, README

These files are the rest of the collector consistency rule and MUST stay
synchronized with the collector code:

- `src/go/plugin/go.d/collector/<name>/taxonomy.yaml` --
  dashboard TOC placement for chart contexts. Static collectors
  use ordered `items:` trees; plain strings in structural `items:`
  own chart contexts. Dynamic collectors use `type: selector` with
  `context_prefix:` or `collect_plugin:` and matching
	  `metadata.yaml.metrics.dynamic_*` declarations. Display widgets
	  use `type: context` with `contexts:` and `chart_library`; those
	  referenced contexts MUST also be owned by structural items.
	  Pick `--section-id` from
	  `integrations/taxonomy/sections.yaml`; `section_id` is a stable
	  registry ID, not a path to invent in the collector file.
	  Seed the initial explicit context list with:
	  ```bash
	  python3 integrations/gen_taxonomy_seed.py src/go/plugin/go.d/collector/<name>/metadata.yaml --module-name <name> --section-id <section.id> --placement-id <name> --icon <icon>
	  ```
	  For a rich recent example with groups, context ownership, and
	  generated integration docs, read
	  `src/go/plugin/go.d/collector/cato_networks/taxonomy.yaml`.
- `src/go/plugin/go.d/config/go.d/<name>.conf` -- the stock
  config users will see at
  `/etc/netdata/go.d/<name>.conf`. It SHOULD stay minimal but
  representative. Show every common option with a comment.
- `src/go/plugin/go.d/collector/<name>/config_schema.json` --
  the DYNCFG schema. Each option in the stock `.conf` SHOULD
  have a corresponding entry here, with the same default.
- `src/health/health.d/<name>.conf` -- alerts on the metrics
  declared in `metadata.yaml`. Each alert in this file SHOULD
  have a matching entry under `metadata.yaml.modules[0].alerts[]`.
- `src/go/plugin/go.d/collector/<name>/README.md` -- this is the
  USER-FACING documentation. After step 5, this file will be
  REPLACED with a symlink to
  `integrations/<slug>.md`. You MUST NOT hand-write the
  README; the generator does. Stub it as empty initially.

## 5. Run the pipeline locally

From the repo root:

```bash
./integrations/pip.sh   # once
python3 integrations/gen_integrations.py
python3 integrations/gen_taxonomy.py --check-only
python3 integrations/check_collector_taxonomy.py --pr-diff master...HEAD
python3 -m unittest integrations.tests.test_taxonomy
python3 integrations/gen_docs_integrations.py -c go.d.plugin/<name>
python3 integrations/gen_doc_collector_page.py
python3 integrations/gen_doc_secrets_page.py
# If service-discovery rules or sdext metadata changed:
python3 integrations/gen_doc_service_discovery_page.py
```

Expected outputs:

- `integrations/integrations.js` and `integrations/integrations.json`
  regenerated (gitignored, MUST NOT be committed).
- Collector taxonomy validated. If `gen_taxonomy.py` fails, fix
  `taxonomy.yaml` or the matching `metadata.yaml.metrics.dynamic_*`
  declaration before continuing.
- Touched-collector taxonomy coverage validated in the same `--pr-diff` mode
  used by CI. If your local base branch is not `master`, adjust the diff range
  to the PR base.
- `src/go/plugin/go.d/collector/<name>/integrations/<slug>.md`
  CREATED. Inspect: it SHOULD contain the `<!--startmeta`
  banner with your `sidebar_label` and `learn_rel_path`, then
  the rendered overview / setup / metrics / alerts /
  troubleshooting sections.
- `src/go/plugin/go.d/collector/<name>/README.md` becomes a
  symlink to `integrations/<slug>.md` (because there is
  exactly one integration in this directory).
- `src/collectors/COLLECTORS.md` updated to include your new
  collector in its category section.
- If service-discovery rules or `sdext` metadata changed,
  `src/collectors/SERVICE-DISCOVERY.md` updated.

If `gen_integrations.py` exits non-zero, read the warning
output -- a schema validation failed. Fix `metadata.yaml` and
re-run.

## 6. Verify locally

- Open the generated `integrations/<slug>.md` and make sure
  every section reads correctly.
- Open `src/collectors/COLLECTORS.md` and find your collector
  in the table.
- Run `python3 integrations/check_collector_taxonomy.py --pr-diff master...HEAD`
  before opening the PR. If your local base branch is not `master`, adjust the
  diff range to the PR base.
- From `src/go`, run `timeout 15s go run ./cmd/godplugin -m <name> -d` to
  confirm go.d can load the module after the `init.go`, `go.d.conf`, stock
  config, and README wiring changes. Success means the module is registered, a
  job starts, and the command keeps running until timeout stops it; `unknown
  module`, `no jobs started`, config-load errors, or immediate exit are
  failures. Use `-c <config-dir>` when testing a non-standard config path.
- If service-discovery rules or `sdext` metadata changed, run
  `python3 integrations/gen_doc_service_discovery_page.py` and commit
  `src/collectors/SERVICE-DISCOVERY.md`.
- Run
  `git status --porcelain | rg '^(\?\?|!!| M|M |A |AM) integrations/(integrations\.(js|json)|taxonomy\.json)$' || true`
  and make sure it prints no output; these generated runtime catalogs are
  gitignored and MUST NOT be committed.
- Run `git diff` and confirm the only changes are in:
  - `src/go/plugin/go.d/collector/<name>/...` (your new module
    files).
  - `src/go/plugin/go.d/collector/init.go` (registration import).
  - `src/go/plugin/go.d/config/go.d.conf` (module toggle).
  - `src/go/plugin/go.d/config/go.d/<name>.conf` (stock config).
  - `src/go/plugin/go.d/README.md` (collector list).
  - `src/go/plugin/go.d/collector/<name>/integrations/<slug>.md`
    (the generated integration page).
  - `src/go/plugin/go.d/collector/<name>/README.md` (now a
    symlink).
  - `src/collectors/COLLECTORS.md` (umbrella page updated).
  - `src/health/health.d/<name>.conf` (alerts file).
  - `src/collectors/SERVICE-DISCOVERY.md` if service-discovery changed.
  - Possibly `integrations/categories.yaml` if you added a
    category.
  - NOT `integrations/integrations.js` or
    `integrations/integrations.json` (gitignored).
  - NOT `integrations/taxonomy.json` (gitignored).

## 7. Commit and push

Single PR, single commit (or a few logical commits) covering
the collector consistency rule plus the generated integration
page and umbrella update. Reviewers will check that affected
artifacts were updated together.

## 8. CI

- `check-markdown.yml` will run on the PR. It runs the same
  pipeline scripts, validates taxonomy, and validates Learn ingest. If your
  committed integration page diverges from CI's regen, the
  workflow fails -- fix locally and re-push.
- After merge, `generate-integrations.yml` triggers on master.
  Since you already committed the regen, this SHOULD NOT
  produce changes. If it does, the auto-PR
  (`Regenerate integrations docs`) catches the drift -- merge
  it.

## 9. Surface arrival timing

- `src/collectors/COLLECTORS.md` is live in the repo
  immediately after merge.
- The cloud-frontend dashboard's Integrations page rebuilds
  on its own schedule (when the cloud-frontend CI re-runs
  `gen_integrations.py` against master). Coordinate with
  the dashboard team if you need to know the exact next
  build.
- The Learn site's per-integration page lands within a few
  hours -- Learn's `ingest.yml` workflow runs every 3 hours
  (see the `learn-site-structure` skill for details).

## Common mistakes

- **Forgetting one collector-consistency artifact.** The most common
  cause of review feedback. Use `git status` after step 5 to
  confirm every affected source/generated artifact is staged.
- **Hand-editing `integrations/<slug>.md` after generation.**
  Never. It is regenerated each time. Edit `metadata.yaml`
  and re-run.
- **Skipping `gen_doc_collector_page.py`.** This forgets to
  update `src/collectors/COLLECTORS.md`, leaving your
  collector invisible in the umbrella table even though the
  per-integration page exists.
- **Categories typo.** A category id that doesn't match
  `categories.yaml` causes validation to fail (warnings are
  fatal). The renderer would silently fall back to
  `data-collection.applications` if your only declared
  category is bogus -- that fallback is itself the symptom
  of a typo, not the desired outcome.
- **Slug collision.** If `clean_string(meta.name)` produces
  the same slug as an existing collector in the same
  directory, one overwrites the other silently. Pick a
  unique enough display name.
