# Overriding Stock Alerts

This guide explains how to customize Netdata's stock alerts. User configurations survive upgrades, making this the recommended approach.

## Quick Reference

| Goal                                | Method                                                                                             |
|-------------------------------------|----------------------------------------------------------------------------------------------------|
| Change thresholds for ALL instances | Create a template with the same name                                                               |
| Change thresholds for ONE instance  | Create an alarm with the same name                                                                 |
| Disable an alert completely         | Use `enabled alarms` in netdata.conf                                                               |
| Enable only specific alerts         | Use [`enabled alarms` without `*`](/src/health/REFERENCE.md#enable-only-specific-alerts-whitelist) |
| Silence notifications only          | Set `to: silent`                                                                                   |

## Understanding Overrides

Netdata's alerting uses **templates** (match all instances of a context) and **alarms** (match one specific instance). Stock alerts are mostly templates—they apply to all disks, all CPUs, etc.

To override, create an alert with the **same name**. User definitions are processed before stock definitions, so yours wins.

See [Alert Configuration Ordering](/src/health/alert-configuration-ordering.md) for the full conceptual explanation.

## Where to Put Your Overrides

Put your overrides in your [Netdata config directory](/docs/netdata-agent/configuration/README.md) under `health.d/` — files there survive upgrades. The **stock config directory** holds Netdata's built-in alert definitions and is replaced during updates.

:::note
Config paths vary by install prefix. Run `sudo ./edit-config health.d/<file>` from your Netdata config directory to resolve the correct user path automatically, or check the `[directories]` section of `netdata.conf` (keys `health config` and `stock health config`) for exact locations.
:::

## Method 1: Override All Instances (Template)

Create a template with the same name to change thresholds for ALL instances.

**Example: Raise CPU steal thresholds globally**

Stock alert from `cpu.conf` (in the stock config directory):

```yaml
   template: 20min_steal_cpu
         on: system.cpu
      class: Latency
       type: System
  component: CPU
host labels: _os=linux
     lookup: average -20m unaligned of steal
      units: %
      every: 5m
       warn: $this > (($status >= $WARNING) ? (5) : (10))
      delay: down 1h multiplier 1.5 max 2h
    summary: System CPU steal time
       info: Average CPU steal time over the last 20 minutes
         to: silent
```

Your override (create it with `sudo ./edit-config health.d/my-overrides.conf`):

```yaml
   template: 20min_steal_cpu
         on: system.cpu
      class: Latency
       type: System
  component: CPU
host labels: _os=linux
     lookup: average -20m unaligned of steal
      units: %
      every: 5m
       warn: $this > (($status >= $WARNING) ? (10) : (20))
      delay: down 1h multiplier 1.5 max 2h
    summary: System CPU steal time
       info: Average CPU steal time over the last 20 minutes
         to: silent
```

**Why it works:** Same name + same context. Your template is processed first, creating the alert. The stock template is then skipped.

> **Note:** Most stock alerts are templates. If a stock alert is an alarm (rare), you must override it with an alarm, not a template—alarms are always processed before templates.

## Method 2: Override One Instance (Alarm)

Create an alarm to override thresholds for ONE specific instance while keeping stock thresholds for others.

**Example: Different disk space threshold for `/mnt/data`**

Stock template (applies to all disks):

```yaml
template: disk_space_usage
      on: disk.space
   lookup: max -1m percentage of avail
     warn: $this < 20
     crit: $this < 10
```

Your override (create it with `sudo ./edit-config health.d/my-overrides.conf`):

```yaml
alarm: disk_space_usage
   on: disk_space._mnt_data
   lookup: max -1m percentage of avail
     warn: $this < 5
     crit: $this < 2
```

**Why it works:**

- Both have the **same name** (`disk_space_usage`)
- Your **alarm** targets the specific chart ID `disk_space._mnt_data`
- Alarms are processed before templates (when names match)
- For `/mnt/data`: your alarm creates the alert, stock template is skipped
- For all other disks: stock template creates alerts normally

**Key difference:** Templates use `on:` with a **context** (`disk.space`). Alarms use `on:` with a **chart ID** (`disk_space._mnt_data`).

### Finding Chart IDs

To find the exact chart ID for an instance:

```bash
curl -s "http://localhost:19999/api/v1/charts" | grep -o '"id":"disk_space[^"]*"'
```

Or check the chart title in the Netdata dashboard—the chart ID is shown in the URL when you click on a chart.

### Alternative: Using Chart Labels

Instead of chart IDs, you can match by labels:

```yaml
template: disk_space_usage
      on: disk.space
chart labels: mount_point=/mnt/data
   lookup: max -1m percentage of avail
     warn: $this < 5
     crit: $this < 2
```

Use labels when:

- You want to target multiple instances sharing a label
- Chart IDs are dynamic or unpredictable

## Method 3: Copy Entire Stock File

If you want to modify many alerts in one stock file, copy it entirely:

```bash
# from your Netdata config directory:
sudo ./edit-config health.d/cpu.conf
```

**Important:** When a file with the same name exists in both directories, Netdata loads **only** the user file. The stock file is completely ignored.

This means your copy must include ALL alerts you want—not just the ones you're changing.

## Disabling Alerts

### Option A: Global Disable

In your `netdata.conf` (edit it with `sudo ./edit-config netdata.conf`):

```ini
[health]
    enabled alarms = !20min_steal_cpu !disk_space_usage *
```

This disables `20min_steal_cpu` and `disk_space_usage` while keeping all other alerts (`*`).

This is a `netdata.conf` change, so restart your Netdata Agent to apply it — `netdatacli reload-health` (see [Applying Changes](#applying-changes) below) only reloads `health.d/*.conf` files, not `netdata.conf`.

### Option B: Per-Alert Disable

Create an override that never matches:

```yaml
template: 20min_steal_cpu
      on: system.cpu
host labels: _hostname=!*
```

The pattern `!*` is a special disable shortcut—the health config parser recognizes it and disables the alert.

### Option C: Silence Notifications Only

Keep the alert monitoring but stop notifications:

```yaml
template: 20min_steal_cpu
      on: system.cpu
  lookup: average -20m unaligned of steal
   units: %
   every: 5m
    warn: $this > (($status >= $WARNING) ? (5) : (10))
      to: silent
```

The alert still appears in the dashboard but sends no notifications.

## Applying Changes

Reload the health configuration:

```bash
sudo netdatacli reload-health
```

If `netdatacli` isn't available, send `SIGUSR2` to the Netdata process.

### Verify Your Override

Check via API:

```bash
curl -s "http://localhost:19999/api/v1/alarms?all" | jq '.alarms | to_entries[] | select(.value.name == "20min_steal_cpu") | .value'
```

Key fields to check:

- `source`: confirms which config file is active (user override vs stock)
- `lookup_*`: data query parameters
- `warn`, `crit`: threshold expressions

Or navigate to the Alerts tab in the dashboard and verify thresholds match your override.

### Check for Errors

If your override isn't working, check the logs:

```bash
# systemd journal (most Linux distributions):
journalctl --namespace netdata -g health --no-pager | tail -20

# Log files (if journal not available):
grep -i health /var/log/netdata/error.log | tail -20
```

Common issues:

- Syntax errors in configuration
- Alert name doesn't match exactly (case-sensitive)
- File permissions prevent Netdata from reading your config

## Troubleshooting

### Override Not Taking Effect

1. **Reload configuration**: `sudo netdatacli reload-health`
2. **Check file permissions**: Netdata must be able to read your file
3. **Verify exact name match**: Alert names are case-sensitive
4. **Check for syntax errors**: Look in error.log

### Both Stock and Override Alerts Appear

This happens when matching criteria don't overlap. For example:

- Your override has `host labels: production`
- Stock alert has no host labels restriction

Both can create alerts on different hosts. Ensure your override matches at least the same scope as the stock alert.

### UI Edit Replaced My Config

Editing an alert through the dashboard UI creates a dynamic configuration that **replaces** any file-based definition. To restore file-based behavior, remove the dynamic config through the UI or API.

## FAQ

### Do I need to copy all fields when overriding an alert?

Yes. Your override is a complete alert definition, not a "patch" on the stock alert. Include all fields: `lookup`, `calc`, `warn`, `crit`, `units`, etc.

If you omit a field, the alert uses its default value—not the stock alert's value.

### How do I override the same alert differently on different hosts?

Use `host labels` to create host-specific overrides:

```yaml
# Production servers: stricter thresholds
template: cpu_usage
      on: system.cpu
host labels: environment=production
     warn: $this > 70

# Development servers: relaxed thresholds
template: cpu_usage
      on: system.cpu
host labels: environment=development
     warn: $this > 90
```

Both can coexist because they match different hosts.

### How do I find what stock alerts exist?

List the stock alert files (the command below uses the default stock directory):

```bash
ls /usr/lib/netdata/conf.d/health.d/
```

View a specific stock alert:

```bash
cat /usr/lib/netdata/conf.d/health.d/cpu.conf
```

Or use the API to list all alert names without dealing with paths:

```bash
curl -s "http://localhost:19999/api/v1/alarms?all" | jq '.alarms | to_entries[].value.name' | sort -u
```

### Can I add new alerts without affecting stock alerts?

Yes. Create alerts with **different names** than stock alerts. They'll coexist independently.

```yaml
# This is a NEW alert, not an override
template: my_custom_disk_alert
      on: disk.space
   lookup: max -5m percentage of avail
     warn: $this < 15
```

### What happens to my overrides after a Netdata upgrade?

User config files are preserved across upgrades. Stock files are replaced.

Your overrides continue working. However, if a stock alert is renamed or removed in a new version, your override may become orphaned (still works, but no longer overriding anything).

### How do I override for multiple specific instances?

Option 1: Create multiple alarms (one per instance):

```yaml
alarm: disk_space_usage
   on: disk_space._mnt_data
     warn: $this < 5

alarm: disk_space_usage
   on: disk_space._mnt_backup
     warn: $this < 5
```

Option 2: Use chart labels if instances share a label:

```yaml
template: disk_space_usage
      on: disk.space
chart labels: storage_tier=bulk
     warn: $this < 5
```

### Can I see what overrides are currently active?

Check which config files Netdata loaded:

```bash
# systemd journal:
journalctl --namespace netdata -g "health.*load\|health.*read" --no-pager

# Log files:
grep -iE "health.*(load|read)" /var/log/netdata/error.log
```

Compare your active alert config vs stock (default locations shown):

```bash
# Your override
cat /etc/netdata/health.d/my-overrides.conf

# Stock definition
cat /usr/lib/netdata/conf.d/health.d/disks.conf
```

### Why does editing an alert in the UI override my file-based config?

UI edits create dynamic configurations that take precedence over all file-based configs. This is by design—it allows quick adjustments without SSH access.

To restore file-based control, remove the dynamic config through the UI (reset to default) or via the API.

## Related Documentation

- [Health Configuration Reference](/src/health/REFERENCE.md)
- [Alert Configuration Ordering](/src/health/alert-configuration-ordering.md)
