<!--
  SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
  SPDX-License-Identifier: Apache-2.0

  See LICENSE.txt for more license information
-->

# NCCL Device API Pure GIN AlltoAll Example

This example demonstrates NCCL's GPU-Initiated Networking (GIN) capabilities for performing AlltoAll collective operations directly from GPU kernels using only network-based communication.

## Overview

This example showcases **pure GIN communication** where all data exchange happens through the network, without any Load Store Access (LSA) optimizations. This is particularly useful for:

- Multi-node environments where ranks cannot use LSA
- Testing network performance without local optimizations
- Understanding the baseline GIN communication patterns
- Scenarios where all communication must go through the network

## What This Example Does

1. **Creates device communicators** using `ncclDevCommCreate` for GPU kernel access to NCCL operations
2. **Registers symmetric memory windows** with `ncclCommWindowRegister` for direct peer-to-peer access
3. **Launches GPU kernel** that performs AlltoAll operations using pure GIN for all peer communication

## Building and Running

The advanced examples can be built using either pthread or MPI for parallelization. pthread is the default choice. To use MPI the user needs to set `MPI=1` at build time and can optionally provide a valid MPI installation under `MPI_HOME`.

### Build
```bash
make [MPI=1] [MPI_HOME=<path-to-mpi>] [NCCL_HOME=<path-to-nccl>] [CUDA_HOME=<path-to-cuda>]
```

### Run when compiled for pthreads (default)
```bash
[NTHREADS=N] ./alltoall_gin
```

### Run when compiled for MPI
```bash
mpirun -np <num_processes> ./alltoall_gin
```

## Code Walk-through

### Device Communicator Creation (Host-side)
The `ncclDevComm` is the core component enabling GPU kernels to perform network communication directly. For pure GIN communication, we configure the device communicator with GIN-specific resources. The `ncclDevCommRequirements` specifies GIN barriers for network synchronization and signals for completion detection. Unlike LSA-based examples, we don't need LSA barriers since all communication goes through the network.

```cpp
ncclDevComm devComm;
ncclDevCommRequirements reqs = NCCL_DEV_COMM_REQUIREMENTS_INITIALIZER;
// GIN barriers enable cross-node synchronization over the network
reqs.worldGinBarrierCount = NCCL_DEVICE_CTA_COUNT;
// GIN signals provide completion notifications for asynchronous operations
reqs.ginSignalCount = NCCL_DEVICE_CTA_COUNT;
// Enable full GIN connectivity, i.e., connect each rank to all other ranks
reqs.ginConnectionType = NCCL_GIN_CONNECTION_FULL;

// Create device communicator with pure GIN support
NCCLCHECK(ncclDevCommCreate(comm, &reqs, &devComm));
```

### Memory Window Registration (Host-side)
The device API requires symmetric memory windows registered using `NCCL_WIN_COLL_SYMMETRIC`. These windows enable GPU kernels to access remote memory through GIN operations. Unlike LSA which provides direct memory access, GIN windows are accessed through network put/get operations.

```cpp
ncclWindow_t send_win;
ncclWindow_t recv_win;

// Register symmetric windows for GIN network access
NCCLCHECK(ncclCommWindowRegister(comm, d_sendbuff, size_bytes, &send_win, NCCL_WIN_COLL_SYMMETRIC));
NCCLCHECK(ncclCommWindowRegister(comm, d_recvbuff, size_bytes, &recv_win, NCCL_WIN_COLL_SYMMETRIC));
```

### GIN Barriers (Device-side)
GIN barriers enable cross-node synchronization from device code over the network. Each thread block uses `blockIdx.x` to select its dedicated barrier, allowing blocks to progress independently while coordinating with corresponding blocks on other nodes. An **acquire** sync at kernel entry ensures all ranks are ready before the AlltoAll exchange; a **release** sync after `flush` (see below) pairs with that acquire so ranks do not leave the kernel until the collective is complete.

```cpp
  ncclGinBarrierSession<ncclCoopCta> bar { ncclCoopCta(), gin, ncclTeamTagWorld(), blockIdx.x };
  bar.sync(ncclCoopCta(), cuda::memory_order_acquire, ncclGinFenceLevel::Relaxed);
```

### GIN Put Operations (Device-side)
GIN provides one-sided put operations for direct remote memory writes over the network. Each thread handles a subset of destination ranks, writing its rank's data to the appropriate location in each peer's receive buffer. The `ncclGin_SignalInc` parameter increments a signal counter, enabling asynchronous completion detection.

```cpp
  const size_t size = count * sizeof(T);
  for (int r = tid; r < devComm.nRanks; r += nthreads) {
    gin.put(ncclTeamWorld(devComm), r,
        recvwin, recvoffset + devComm.rank * size,
        sendwin, sendoffset + r * size,
        size, ncclGin_SignalInc{signalIndex});
  }
```

### Signal-based Completion (Device-side)
GIN uses signals for asynchronous completion detection of network operations. The kernel waits for the signal value to reach the expected count (initial value + number of ranks), indicating all put operations have completed. The `gin.flush()` ensures all pending operations are committed. A final `bar.sync` with **release** ordering runs after `flush` so every rank completes the GIN work before any rank proceeds past the kernel—pairing with the **acquire** at entry and avoiding races where a successor kernel might observe incomplete data.

```cpp
  // Wait only on the CTA whose blockIdx.x (signalIndex) accumulates all puts to this rank.
  int receivingCta = (devComm.rank % nthreads) / blockDim.x;
  if (blockIdx.x == receivingCta)
    gin.waitSignal(ncclCoopCta(), signalIndex, signalValue + devComm.nRanks);

  gin.flush(ncclCoopCta());
  bar.sync(ncclCoopCta(), cuda::memory_order_release, ncclGinFenceLevel::Relaxed);
```

### Receiving CTA (Device-side)

`signalIndex` is `blockIdx.x`, so each `gin.put` increments the destination rank's signal for that sender CTA index. Only the CTA with `blockIdx.x == receivingCta` calls `waitSignal`; `receivingCta = (devComm.rank % nthreads) / blockDim.x` picks one waiting CTA per destination rank. The wait uses `signalValue + devComm.nRanks` because every peer sends one GIN put to this rank (pure GIN: all peers are remote).

## Expected Output

```
Starting Pure GIN AlltoAll initialization
  Rank 0 using GPU device 0
  Rank 1 using GPU device 1
  Rank 0 initialized NCCL communicator for 2 total ranks
  Rank 1 initialized NCCL communicator for 2 total ranks
  Rank 0 initialized send data
  Rank 1 initialized send data
  Rank 0 created device communicator with GIN support
  Rank 1 created device communicator with GIN support
Starting Pure GIN AlltoAll with 1024 elements per rank (2048 total elements, 0 MB)

=== Executing Pure GIN AlltoAll ===
  Rank 0 completed pure GIN AlltoAll kernel
  Rank 1 completed pure GIN AlltoAll kernel
Pure GIN AlltoAll result: PASSED
```

## When to Use

- **Multi-node environments**: When ranks cannot use LSA
- **Testing network performance**: Without local optimizations

## Performance Considerations

- **Network overhead**: All communication goes through the network stack
- **Signal-based completion**: Enables asynchronous operation patterns
- **Barrier synchronization**: Ensures proper ordering of network operations
- **Multiple GIN contexts**: Can improve parallel communication performance

## Common Issues and Solutions

### Issue: Deadlock at util_broadcast
**Solution:** Ensure you're running with multiple GPUs/processes
```bash
NTHREADS=2 ./alltoall_gin  # For 2 GPUs
```

### Issue: CUDA out of memory
**Solution:** Reduce the data size in the example

### Issue: Network errors
**Solution:** Ensure proper network configuration for multi-node setups

## Performance Notes

- These are educational examples, not optimized for performance
- Real implementations should consider:
  - Optimal GIN context usage for parallel operations
  - Signal pool management for high-throughput scenarios
  - Memory coalescing patterns for network operations
  - Network topology-aware communication strategies

## Error Handling

The example uses comprehensive error checking for CUDA, NCCL, and GIN operations. Device kernels should implement proper error handling for network operations and signal management.

## Next Steps

After understanding this example, explore:
- **Performance optimization**: Fine-tune GIN context usage and signal management
- **Hybrid approaches**: Combine GIN with LSA for topology-aware optimizations
- **Integration with compute**: Fuse network communication with computation kernels
