# chartengine

`chartengine` compiles chart templates and builds deterministic chart plans (`create`, `update`, `remove`) from `metrix.Reader` snapshots.

**Audience**: `ModuleV2` collector authors and framework contributors.

**See also**: [charttpl](/src/go/plugin/framework/charttpl/README.md) (template DSL),
[metrix](/src/go/pkg/metrix/README.md) (metrics storage and read API).

## Purpose

| Stage                 | Responsibility                                           |
|-----------------------|----------------------------------------------------------|
| `charttpl`            | Template decode/defaults/validation                      |
| `chartengine.Compile` | Build immutable program IR                               |
| `Engine.PreparePlan`  | Prepare plan actions plus explicit commit/abort boundary |
| `chartemit.ApplyPlan` | Emit plan to Netdata wire protocol                       |

## Collector-Facing Contract

For `ModuleV2` collectors, the runtime integration expects:

| Requirement                                                               | Why it matters                                                      |
|---------------------------------------------------------------------------|---------------------------------------------------------------------|
| `MetricStore()` returns `metrix.CollectorStore` (cycle-managed)           | Job runtime controls cycle boundaries and success/failure semantics |
| `ChartTemplateYAML()` returns valid `charttpl` YAML                       | Loaded once at autodetection/post-check                             |
| Collector writes metrics during `Collect()` only                          | Planner runs after a successful cycle commit                        |
| Metric names used in template selectors are present in group metric scope | Compile/validate consistency                                        |

## Public API Surface

| API                                                 | Purpose                                                                                                                                                                 |
|-----------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `New(opts...)`                                      | Create engine with policy/runtime options                                                                                                                               |
| `Load(spec, revision)` / `LoadYAML(data, revision)` | Compile and publish program revision                                                                                                                                    |
| `PreparePlan(reader)`                               | Build deterministic action plan from reader snapshot and return an explicit attempt                                                                                     |
| `RuntimeStore()`                                    | Access chartengine internal runtime metrics store                                                                                                                       |
| `WithEnginePolicy(...)`                             | Configure selector + autogen behavior                                                                                                                                   |
| `WithRuntimeStore(...)`                             | Override/disable self-metrics store                                                                                                                                     |
| `WithSeriesSelectionAllVisible()`                   | Process all visible series instead of filtering to latest successful collect cycle. Intended for runtime/internal stores that commit immediately (no cycle boundaries). |
| `WithEmitTypeIDBudgetPrefix(...)`                   | Set the effective type-id prefix used by autogen budget checks                                                                                                          |
| `WithRuntimePlannerMode(...)`                       | Enable runtime planner mode with no-write-tick semantics, for jobs/tests that drive planning directly from runtime metrics instead of collect-cycle boundaries.         |

## End-to-End Example (Single Flow)

```go
// 1) Collector writes metrics.
store := metrix.NewCollectorStore()
meter := store.Write().SnapshotMeter("app")
meter.Counter("requests_total").ObserveTotal(100)

// 2) Engine loads chart template.
engine, err := chartengine.New(
	chartengine.WithEnginePolicy(chartengine.EnginePolicy{
		Autogen: &chartengine.AutogenPolicy{Enabled: false},
	}),
)
// handle err

err = engine.LoadYAML([]byte(`
version: v1
groups:
  - family: App
    metrics: [app.requests_total]
    charts:
      - id: requests
        title: Requests
        context: requests
        units: requests/s
        dimensions:
          - selector: app.requests_total
            name: requests
`), 1)
// handle err

// 3) Prepare plan from flattened+raw reader, emit it, then commit.
// ReadFlatten() is included even for templates with static dimensions
// because it is required for inferred dimensions, structured-family autogen,
// and is the standard pattern.
attempt, err := engine.PreparePlan(store.Read(metrix.ReadRaw(), metrix.ReadFlatten()))
// handle err
plan := attempt.Plan()
defer attempt.Abort()

err = chartemit.ApplyPlan(api, plan, chartemit.EmitEnv{
TypeID:      "plugin.job",
UpdateEvery: 1,
Plugin:      "example",
Module:      "example",
JobName:     "example",
})
// handle err
err = attempt.Commit()
// handle err

```

## PreparePlan Lifecycle

`PreparePlan` executes a deterministic phase pipeline.
Terms like "materialized state" and "route cache" are defined in the Engine State section below.

| Phase           | Summary                                                                                        |
|-----------------|------------------------------------------------------------------------------------------------|
| Prepare         | Resolve program/index/cache/materialized state                                                 |
| Validate reader | Ensure flattened metadata is available when inferred dimensions are used                       |
| Scan            | Iterate series, filter by success sequence and selector, route to chart/dimension accumulators |
| Cache retain    | Prune route cache entries not seen in the latest successful sequence                           |
| Lifecycle caps  | Enforce chart/dimension cap policy                                                             |
| Materialize     | Emit create/update actions from accumulated state                                              |
| Expiry          | Emit removals for stale charts/dimensions                                                      |
| Sort            | Deterministically sort inferred dimension output                                               |

## Reader Requirements

| Scenario                                                                       | Required reader mode                                                       |
|--------------------------------------------------------------------------------|----------------------------------------------------------------------------|
| Static named dimensions only                                                   | `Read(...)` is sufficient (no flatten needed)                              |
| Inferred dimensions (`name` and `name_from_label` omitted)                     | Must use flattened reader metadata (`ReadFlatten`)                         |
| Structured autogen families (`Histogram`, `Summary`, `StateSet`, `MeasureSet`) | Must use flattened reader metadata (`ReadFlatten`) or they are not visible |
| Runtime/default `ModuleV2` path                                                | `Read(ReadRaw(), ReadFlatten())`                                           |

If inferred dimensions are present without flattened reader metadata, `PreparePlan` returns an explicit error.

## Action Semantics

| Action                  | Meaning                                                                |
|-------------------------|------------------------------------------------------------------------|
| `CreateChartAction`     | Materialize chart instance (with chart metadata and labels)            |
| `CreateDimensionAction` | Materialize dimension for a chart                                      |
| `UpdateChartAction`     | Emit chart values for current cycle; unseen dims become `IsEmpty=true` |
| `RemoveDimensionAction` | Obsolete one dimension                                                 |
| `RemoveChartAction`     | Obsolete one chart                                                     |

`chartemit` normalizes emitted action order by phase:

1. create chart/dimensions
2. update values
3. remove dimensions/charts

## Routing and Collision Rules

Each metric series is routed to a chart and dimension based on template selectors.
The following rules apply when routing conflicts arise:

| Rule                                          | Behavior                                                                                       |
|-----------------------------------------------|------------------------------------------------------------------------------------------------|
| Template vs autogen chart ID collision        | Template wins; autogen chart is replaced                                                       |
| Cross-template chart ID collision             | Existing owner keeps ownership; subsequent series are **silently ignored** (see warning below) |
| Duplicate dimension observations within build | First observed dimension metadata wins; values are reduced (summed)                            |

> [!WARNING]
> Cross-template chart ID collisions cause silent data loss — conflicting series are dropped with no error and no log entry. If metrics are missing, check for duplicate rendered chart IDs across template groups.

## Lifecycle Defaults and Policy

Default lifecycle policy when template omits lifecycle:

| Policy                           | Default        |
|----------------------------------|----------------|
| `max_instances`                  | `0` (disabled) |
| `expire_after_cycles`            | `5`            |
| `dimensions.max_dims`            | `0` (disabled) |
| `dimensions.expire_after_cycles` | `0`            |

## Autogen Notes

| Topic                 | Behavior                                                                                                                                       |
|-----------------------|------------------------------------------------------------------------------------------------------------------------------------------------|
| Trigger               | Unmatched series only when autogen is enabled                                                                                                  |
| Structured families   | Autogen has dedicated source builders for flattened `Histogram`, `Summary`, `StateSet`, and `MeasureSet` families                              |
| Metric metadata usage | Uses `metrix.MetricMeta` hints for title/family/unit where allowed                                                                             |
| Type ID budget        | Enforced via `AutogenPolicy.MaxTypeIDLen` + effective emit type-id prefix (`WithEmitTypeIDBudgetPrefix(...)`)                                  |
| Lifecycle             | Autogen applies `ExpireAfterSuccessCycles` to **both** chart and dimension expiry (unlike template lifecycle where they default independently) |

`MeasureSet` autogen specifics:

- chartengine treats `MeasureSet` as a structured family, similar to `StateSet`, not as grouped scalar coincidence
- flattened `MeasureSet` inputs are expected to carry:
    - `SourceKind = MetricKindMeasureSet`
    - `FlattenRole = FlattenRoleMeasureSetField`
    - per-field metric names like `<name>_<field>`
    - a synthetic reserved field label (`measure_field=<field>`)
- the synthetic `measure_field` label is the authoritative field-identity channel; the per-field metric-name suffix remains for `MetricMeta(name)` compatibility
- gauge-like `MeasureSet` fields autogen with absolute algorithm behavior; counter-like `MeasureSet` fields autogen with incremental algorithm behavior

### Reserved Flattened Label Keys

These label keys are treated specially by chartengine when consuming flattened structured-family or distribution inputs:

| Key / Pattern   | Meaning                                                                                                                |
|-----------------|------------------------------------------------------------------------------------------------------------------------|
| `le`            | Histogram bucket bound label                                                                                           |
| `quantile`      | Summary quantile label                                                                                                 |
| `measure_field` | `MeasureSet` field identity label                                                                                      |
| `<metric-name>` | `StateSet` special case: the flattened state name is carried under a synthetic label whose key is the base metric name |

Notes:

- `le`, `quantile`, and `measure_field` are static reserved flattened-label keys in chartengine.
- `StateSet` is different: it does not use a global static key; it uses the base metric name itself as the synthetic flattened label key.
- These keys are part of the flatten contract between `metrix` and chartengine. Reusing them as ordinary user labels on those flattened inputs is not supported.

## Runtime Metrics

`chartengine` self-instruments to a runtime store by default (disable with `WithRuntimeStore(nil)`).

| Family                    | Examples                                                       |
|---------------------------|----------------------------------------------------------------|
| `ChartEngine/Build`       | build success/error/skipped counters, build duration summaries |
| `ChartEngine/Actions`     | action counters by kind                                        |
| `ChartEngine/Series`      | scanned/matched/filtered series counters                       |
| `ChartEngine/Route Cache` | hit/miss/entries/prune counters                                |
| `ChartEngine/Lifecycle`   | removal counters by scope/reason                               |
| `ChartEngine/Plan`        | gauges for chart instances/inferred dimensions                 |

## Engine State

| Area               | Design                                                                                                                                      |
|--------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
| Program            | Immutable compiled IR per revision                                                                                                          |
| Engine state       | Serialized under `Engine.mu` for load/build transitions                                                                                     |
| Route cache        | Series identity + revision keyed cache; retained by successful sequence, pruned on each build                                               |
| Materialized state | Tracks existing chart/dimension instances for incremental create/update/remove decisions; persists across cycles, resets on template reload |
| Determinism        | Sorted chart IDs and inferred dimensions provide stable action ordering                                                                     |
