# all_gather

Gathers a tensor from each distributed rank into a list. The tensor for the local rank is the original one, with the gradients while the others have no gradients.

- If torch.distributed is available and initialized:   1. Creates a list of tensors (each sized like the input `tensor`).   2. Gathers tensors from each rank into that list.   3. Replaces the local tensor in the list with the original tensor that retains gradients.

- If torch.distributed is either unavailable, uninitialized, or   `world_size == 1`, it returns a list containing only the   original tensor and throws a warning to notify the user (helpful when using a single GPU setup).

## Parameters

- **tensor** (*'torch.Tensor'*)
