# IBM i (AS/400) collector

## Overview

Monitors IBM i (AS/400) systems using SQL services and CL commands to
expose CPU, memory, storage, job, and subsystem activity.

**Dependencies:**
- unixODBC 2.3+ with IBM i Access ODBC driver
- IBM i 7.2 or later with SQL services enabled

**Required Libraries:**
- libodbc.so (provided by unixODBC)
- IBM i Access Client Solutions

**Collection paths**

The collector executes queries in multiple tracks:

- **Fast path (5s)**: lightweight system status queries remain sequential on the main plugin thread.
- **Slow path (10s beat)**: heavier queries (per-queue metrics, subsystems, plan cache, etc.) run in a background worker with bounded concurrency.
- **Batch path (≥60s beat)**: optional long-period worker used for expensive aggregate queries such as queue totals. Disabled by default unless queue totals are explicitly enabled.

**CPU Collection Methods:**

The collector uses a hybrid approach for CPU utilization metrics to handle IBM i 7.4+ where
`AVERAGE_CPU_*` columns were deprecated:

1. **Primary Method - TOTAL_CPU_TIME**: Uses the monotonic `TOTAL_CPU_TIME` counter from
   `QSYS2.SYSTEM_STATUS()` to calculate CPU utilization via delta-based calculation. This is
   the most accurate method but requires `*JOBCTL` special authority. TOTAL_CPU_TIME is a
   cumulative counter in nanoseconds representing CPU-seconds consumed, naturally in per-core
   scale.

2. **Fallback Method - ELAPSED_CPU_USED**: If `*JOBCTL` authority is not available, falls back
   to `ELAPSED_CPU_USED` with automatic reset detection. This method tracks when IBM i statistics
   are reset (either manually or via `reset_statistics` configuration) and re-establishes a
   baseline after detecting resets. The values are already in per-core scale.

3. **Legacy Method - AVERAGE_CPU_UTILIZATION**: For IBM i versions before 7.4, uses the now-
   deprecated `AVERAGE_CPU_UTILIZATION` column, which IBM reports in the same per-core scale.

The collector automatically selects the appropriate method based on available permissions and
logs which method is being used.

**CPU Metric Scale:**

CPU utilization is reported using the "100% = 1 CPU core" semantic. This means:
- 100% indicates one CPU core is fully utilized
- 400% indicates four CPU cores are fully utilized
- Values are limited to 100% × ConfiguredCPUs, matching the partition's configured capacity

For shared LPARs, the metrics show absolute CPU consumption in per-core scale, not relative to
entitled capacity. For example, a shared LPAR entitled to 0.20 cores can show 150% utilization
when bursting above entitlement.

**Statistics Reset Behavior:**

The `reset_statistics` configuration option controls whether the collector resets IBM i system
statistics on each query via `SYSTEM_STATUS(RESET_STATISTICS=>'YES')`. When enabled:

- System-level statistics (CPU, memory pools, etc.) are reset after each collection cycle
- Matches legacy behavior but clears global statistics that other tools may rely on
- The ELAPSED_CPU_USED fallback method will detect and handle these resets automatically
- **Caution**: Enabling this affects all users and applications on the IBM i system

Default: `false` (statistics are not reset, using `RESET_STATISTICS=>'NO'`)

**Chart Gaps During Baseline Resets:**

The `as400.system_activity_cpu_rate` and `as400.system_activity_cpu_utilization` charts rely on
delta calculations. When the collector detects that IBM i reset these statistics—or when it is
still establishing the initial baseline—it intentionally skips a sample instead of emitting a zero
or spike. Netdata renders those skipped samples as small gaps, which is expected behaviour.

**Cardinality Management:**

To prevent performance issues from excessive metric creation, the collector enforces cardinality
limits on per-instance metrics (disks, subsystems, job queues, message queues, output queues,
active jobs, network interfaces, HTTP servers).

**How Limits Work:**
- The collector counts instances before collecting metrics
- If count exceeds the configured `max_*` limit, **collection is skipped entirely** for that category
- The collector logs a warning: `"[category] count (X) exceeds limit (Y), skipping collection"`
- No metrics are collected for that category until you adjust the configuration

**Configuration Options:**

Use **both** limit and selector options together to manage high-cardinality environments:

| Option | Purpose | Default |
|--------|---------|---------|
| `max_disks` | Maximum disk units to monitor | 100 |
| `max_subsystems` | Maximum subsystems to monitor | 100 |
| `max_job_queues` | Maximum job queues to monitor | 100 |
| `max_message_queues` | Maximum message queues to monitor | 100 |
| `max_output_queues` | Maximum output queues to monitor | 100 |
| `active_jobs` | Fully qualified active jobs to monitor (`JOB_NUMBER/USER/JOB_NAME`) | `[]` |
| `collect_disks_matching` | Glob pattern to filter disks (e.g., `"001* 002*"`) | `""` (match all) |
| `collect_subsystems_matching` | Glob pattern to filter subsystems (e.g., `"QINTER QBATCH"`) | `""` (match all) |
| `collect_job_queues_matching` | Glob pattern to filter job queues (e.g., `"QSYS/*"`) | `""` (match all) |

Optional batch-path controls:

| Option | Purpose | Default |
|--------|---------|---------|
| `batch_path` | Enables the long-period batch worker for aggregate queries | `false` |
| `batch_path_update_every` | Batch worker cadence (minimum 60s, recommend ≥600s in production) | `60s` |
| `batch_path_max_connections` | Maximum concurrent connections for batch queries | `1` |
| `collect_message_queue_totals` | Enables full-scan counting of all message queues and messages | `auto` (off) |
| `collect_job_queue_totals` | Enables aggregate counting of job queues and queued jobs | `auto` (off) |
| `collect_output_queue_totals` | Enables aggregate counting of output queues and spooled files | `auto` (off) |

> **Warning:** queue totals require scanning IBM i catalog views and can be very expensive on large systems. Leave these options disabled unless aggregate counts are absolutely necessary.


**Example Workflow:**

1. System has 500 disks, collector skips disk metrics (exceeds default limit of 100)
2. Check logs: `"disk count (500) exceeds limit (100), skipping per-disk metrics"`
3. Two options:
   - **Option A**: Increase limit: `max_disks: 500` (collects all 500 disks)
   - **Option B**: Use selector: `collect_disks_matching: "00[1-5]*"` (cherry-pick specific disks)

**Best Practices:**
- Use selectors to monitor only business-critical objects in large environments
- Set limits based on your Netdata server's capacity (each instance = multiple charts)
- Start with defaults and adjust based on actual usage patterns

**IBM i 7.2–7.3 Behavior Note (Message Queues):**

IBM i 7.4 introduced a message-queue table function that returns only the live backlog. On
7.2–7.3 systems we fall back to the `QSYS2.MESSAGE_QUEUE_INFO` view, which includes *all*
recorded messages (even those already processed/cleared from the queue). Aggregations—especially
`MAX(SEVERITY)`—therefore reflect the historical log, not just the outstanding backlog. This
behaviour is inherent to the IBM SQL service and can lead to higher-than-expected max severity
values on pre-7.4 systems.

Network interface metrics have a fixed internal limit of 50 instances, and HTTP server metrics are capped at 200 instances; these limits are currently not configurable.


This collector is part of the [Netdata](https://github.com/netdata/netdata) monitoring solution.

## Collected metrics

Metrics grouped by scope.

The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.

### Per IBM i (AS/400) instance


These metrics refer to the entire monitored IBM i (AS/400) instance.

This scope has no labels.

Metrics:

| Metric | Dimensions | Unit |
|:-------|:-----------|:-----|
| netdata.plugin_ibm.as400_query_latency_fast | count_disks, count_http_servers, count_network_interfaces, detect_ibmi_version_primary, detect_ibmi_version_fallback, disk_instances, disk_instances_enhanced, disk_status, http_server_info, job_info, memory_pools, network_connections, network_interfaces, serial_number, system_name, system_activity, system_model, system_status, temp_storage_named, temp_storage_total, technology_refresh_level, active_job | ms |
| netdata.plugin_ibm.as400_query_latency_slow | analyze_plan_cache, count_subsystems, subsystems, message_queue_aggregates, job_queues, output_queue_info, plan_cache_summary | ms |
| netdata.plugin_ibm.as400_query_latency_batch | message_queue_totals, job_queue_totals, output_queue_totals | ms |

These metrics refer to the entire monitored IBM i (AS/400) instance.

This scope has no labels.

Metrics:

| Metric | Dimensions | Unit |
|:-------|:-----------|:-----|
| as400.cpu_utilization | utilization | percentage |
| as400.cpu_utilization_entitled | utilization | percentage |
| as400.cpu_configuration | configured | cpus |
| as400.cpu_capacity | capacity | percentage |
| as400.total_jobs | total | jobs |
| as400.active_jobs_by_type | batch, interactive, active | jobs |
| as400.job_queue_length | waiting | jobs |
| as400.main_storage_size | total | bytes |
| as400.temporary_storage | current, maximum | MiB |
| as400.memory_pool_usage | machine, base, interactive, spool | bytes |
| as400.memory_pool_defined | machine, base | bytes |
| as400.memory_pool_reserved | machine, base | bytes |
| as400.memory_pool_threads | machine, base | threads |
| as400.memory_pool_max_threads | machine, base | threads |
| as400.disk_busy_average | busy | percentage |
| as400.system_asp_usage | used | percentage |
| as400.system_asp_storage | total | MiB |
| as400.total_auxiliary_storage | total | MiB |
| as400.system_threads | active, per_processor | threads |
| as400.network_connections | remote, total | connections |
| as400.network_connection_states | listen, close_wait | connections |
| as400.temp_storage_total | current, peak | bytes |
| as400.system_activity_cpu_rate | average | percentage |
| as400.system_activity_cpu_utilization | average, minimum, maximum | percentage |



### Per activejob

These metrics refer to individual activejob instances.

Labels:

| Label | Description |
|:------|:------------|
| job_name | Job_name identifier |
| job_status | Job_status identifier |
| subsystem | Subsystem identifier |
| job_type | Job_type identifier |

Metrics:

| Metric | Dimensions | Unit |
|:-------|:-----------|:-----|
| as400.activejob_cpu | cpu | percentage |
| as400.activejob_resources | temp_storage | MiB |
| as400.activejob_time | cpu_time, total_time | seconds |
| as400.activejob_activity | disk_io, interactive_transactions | operations/s |
| as400.activejob_threads | threads | threads |

### Per disk

These metrics refer to individual disk instances.

Labels:

| Label | Description |
|:------|:------------|
| disk_unit | Disk_unit identifier |
| disk_type | Disk_type identifier |
| disk_model | Disk_model identifier |
| hardware_status | Hardware_status identifier |
| disk_serial_number | Disk_serial_number identifier |

Metrics:

| Metric | Dimensions | Unit |
|:-------|:-----------|:-----|
| as400.disk_busy | busy | percentage |
| as400.disk_io_requests | read, write | requests/s |
| as400.disk_space_usage | used | percentage |
| as400.disk_capacity | available, used | gigabytes |
| as400.disk_blocks | read, write | blocks/s |
| as400.disk_ssd_health | life_remaining | percentage |
| as400.disk_ssd_age | power_on_days | days |

### Per httpserver

These metrics refer to individual httpserver instances.

Labels:

| Label | Description |
|:------|:------------|
| server | Server identifier |
| function | Function identifier |

Metrics:

| Metric | Dimensions | Unit |
|:-------|:-----------|:-----|
| as400.http_server_connections | normal, ssl | connections |
| as400.http_server_threads | active, idle | threads |
| as400.http_server_requests | requests, responses, rejected | requests/s |
| as400.http_server_bytes | received, sent | bytes/s |

### Per jobqueue

These metrics refer to individual jobqueue instances.

Labels:

| Label | Description |
|:------|:------------|
| job_queue | Job_queue identifier |
| library | Library identifier |
| status | Status identifier |

Metrics:

| Metric | Dimensions | Unit |
|:-------|:-----------|:-----|
| as400.jobqueue_length | jobs | jobs |

### Per messagequeue

These metrics refer to individual messagequeue instances.

Labels:

| Label | Description |
|:------|:------------|
| library | Library identifier |
| queue | Queue identifier |

Metrics:

| Metric | Dimensions | Unit |
|:-------|:-----------|:-----|
| as400.message_queue_messages | total, informational, inquiry, diagnostic, escape, notify, sender_copy | messages |
| as400.message_queue_severity | max | severity |

### Per networkinterface

These metrics refer to individual networkinterface instances.

Labels:

| Label | Description |
|:------|:------------|
| interface | Interface identifier |
| interface_type | Interface_type identifier |
| connection_type | Connection_type identifier |
| internet_address | Internet_address identifier |
| network_address | Network_address identifier |
| subnet_mask | Subnet_mask identifier |

Metrics:

| Metric | Dimensions | Unit |
|:-------|:-----------|:-----|
| as400.network_interface_status | active | status |
| as400.network_interface_mtu | mtu | bytes |

### Per outputqueue

These metrics refer to individual outputqueue instances.

Labels:

| Label | Description |
|:------|:------------|
| library | Library identifier |
| queue | Queue identifier |
| status | Status identifier |

Metrics:

| Metric | Dimensions | Unit |
|:-------|:-----------|:-----|
| as400.output_queue_files | files | files |
| as400.output_queue_writers | writers | writers |
| as400.output_queue_status | released | state |

### Per plancache

These metrics refer to individual plancache instances.

Labels:

| Label | Description |
|:------|:------------|
| metric | Metric identifier |

Metrics:

| Metric | Dimensions | Unit |
|:-------|:-----------|:-----|
| as400.plan_cache_summary | value | value |

### Per queueoverview

These metrics refer to individual queueoverview instances.

Labels:

| Label | Description |
|:------|:------------|
| queue_type | Queue_type identifier |
| item_type | Item_type identifier |

Metrics:

| Metric | Dimensions | Unit |
|:-------|:-----------|:-----|
| as400.queues_count | queues | queues |
| as400.queued_items | items | items |

### Per subsystem

These metrics refer to individual subsystem instances.

Labels:

| Label | Description |
|:------|:------------|
| subsystem | Subsystem identifier |
| library | Library identifier |
| status | Status identifier |

Metrics:

| Metric | Dimensions | Unit |
|:-------|:-----------|:-----|
| as400.subsystem_jobs | active, maximum | jobs |

### Per tempstoragebucket

These metrics refer to individual tempstoragebucket instances.

Labels:

| Label | Description |
|:------|:------------|
| bucket | Bucket identifier |

Metrics:

| Metric | Dimensions | Unit |
|:-------|:-----------|:-----|
| as400.temp_storage_bucket | current, peak | bytes |


## Configuration

### File

The configuration file name for this integration is `ibm.d/as400.conf`.

You can edit the configuration file using the `edit-config` script from the
Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory).

```bash
cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config ibm.d/as400.conf
```

### Options

The following options can be defined globally or per job.

| Name | Description | Default | Required | Min | Max |
|:-----|:------------|:--------|:---------|:----|:----|
| update_every | Data collection frequency | `5` | no | 1 | - |
| Vnode | Vnode allows binding the collector to a virtual node. | `` | no | - | - |
| DSN | DSN provides a full IBM i ODBC connection string if manual override is needed. | `` | no | - | - |
| Timeout | Timeout controls how long to wait for SQL statements and RPCs. | `2000000000` | no | - | - |
| Hostname | Hostname is the remote IBM i host to monitor. | `` | no | - | - |
| Port | Port is the TCP port for the IBM i Access ODBC server. | `8471` | no | 1 | 65535 |
| Username | Username supplies the credentials used for authentication. | `` | no | - | - |
| Password | Password supplies the password used for authentication. | `` | no | - | - |
| Database | Database selects the IBM i database (library) to use when building the DSN. | `*SYSBAS` | no | - | - |
| ConnectionType | ConnectionType selects how the collector connects (currently only "odbc"). | `odbc` | no | - | - |
| ODBCDriver | ODBCDriver specifies the driver name registered on the host. | `IBM i Access ODBC Driver` | no | - | - |
| UseSSL | UseSSL enables TLS for the ODBC connection when supported by the driver. | `false` | no | - | - |
| ResetStatistics | ResetStatistics toggles destructive SQL services that reset system statistics on each query. | `false` | no | - | - |
| CollectDiskMetrics | CollectDiskMetrics toggles collection of disk unit statistics. | `auto` | no | - | - |
| CollectSubsystemMetrics | CollectSubsystemMetrics toggles collection of subsystem activity metrics. | `auto` | no | - | - |
| CollectActiveJobs | CollectActiveJobs toggles collection of detailed per-job metrics. | `auto` | no | - | - |
| CollectHTTPServerMetrics | CollectHTTPServerMetrics toggles collection of IBM HTTP Server statistics. | `auto` | no | - | - |
| CollectPlanCacheMetrics | CollectPlanCacheMetrics toggles collection of plan cache analysis metrics. | `auto` | no | - | - |
| CollectMessageQueueTotals | CollectMessageQueueTotals enables expensive aggregate counting across all message queues. | `auto` | no | - | - |
| CollectJobQueueTotals | CollectJobQueueTotals enables expensive aggregate counting across all job queues. | `auto` | no | - | - |
| CollectOutputQueueTotals | CollectOutputQueueTotals enables expensive aggregate counting across all output queues. | `auto` | no | - | - |
| SlowPath | SlowPath enables the asynchronous slow-path worker for heavy queries. | `true` | no | - | - |
| SlowPathUpdateEvery | SlowPathUpdateEvery controls the beat interval for the slow-path worker. | `10000000000` | no | - | - |
| SlowPathMaxConnections | SlowPathMaxConnections caps the number of concurrent queries the slow-path worker may run. | `1` | no | - | - |
| BatchPath | BatchPath enables the long-period batch worker for expensive queue aggregates. | `false` | no | - | - |
| BatchPathUpdateEvery | BatchPathUpdateEvery controls the beat interval for the batch worker. | `60000000000` | no | - | - |
| BatchPathMaxConnections | BatchPathMaxConnections caps concurrent queries for the batch worker. | `1` | no | - | - |
| MaxDisks | MaxDisks caps how many disk units may be charted. | `100` | no | - | - |
| MaxSubsystems | MaxSubsystems caps how many subsystems may be charted. | `100` | no | - | - |
| DiskSelector | DiskSelector filters disk units by name using glob-style patterns. | `` | no | - | - |
| SubsystemSelector | SubsystemSelector filters subsystems by name using glob-style patterns. | `` | no | - | - |
| ActiveJobs | ActiveJobs lists active jobs to monitor, using fully-qualified job identifiers (JOB_NUMBER/USER/JOB_NAME). When empty, active job collection is disabled. | `nil` | no | - | - |
| MessageQueues | MessageQueues lists message queues to collect, formatted as LIBRARY/QUEUE strings. When empty, message queue collection is disabled. The default configuration monitors QSYS/QSYSOPR, QSYS/QSYSMSG, and QSYS/QHST. | `[QSYS/QSYSOPR QSYS/QSYSMSG QSYS/QHST]` | no | - | - |
| JobQueues | JobQueues lists job queues to collect, formatted as LIBRARY/QUEUE strings. When empty, job queue collection is disabled. | `nil` | no | - | - |
| OutputQueues | OutputQueues lists output queues to collect, formatted as LIBRARY/QUEUE strings. When empty, output queue collection is disabled. | `nil` | no | - | - |

### Examples

#### Basic configuration

IBM i (AS/400) monitoring with default settings.

<details>
<summary>Config</summary>

```yaml
jobs:
  - name: local
    endpoint: dummy://localhost
```

</details>

## Troubleshooting

### Debug Mode

To troubleshoot issues with the `as400` collector, run the `ibm.d.plugin` with the debug option enabled.
The output should give you clues as to why the collector isn't working.

- Navigate to the `plugins.d` directory, usually at `/usr/libexec/netdata/plugins.d/`
- Switch to the `netdata` user
- Run the `ibm.d.plugin` to debug the collector:

```bash
sudo -u netdata ./ibm.d.plugin -d -m as400
```

## Getting Logs

If you're encountering problems with the `as400` collector, follow these steps to retrieve logs and identify potential issues:

- **Run the command** specific to your system (systemd, non-systemd, or Docker container).
- **Examine the output** for any warnings or error messages that might indicate issues. These messages will typically provide clues about the root cause of the problem.

### For systemd systems (most Linux distributions)

```bash
sudo journalctl -u netdata --reverse | grep as400
```

### For non-systemd systems

```bash
sudo grep as400 /var/log/netdata/error.log
sudo grep as400 /var/log/netdata/collector.log
```

### For Docker containers

```bash
sudo docker logs netdata 2>&1 | grep as400
```
