a h @s^ddlmZddlZddlmZddlmZee Z d dddddd Z ddd d d Z dS)) annotationsN)loggingFz torch.Tensorbool)tensor with_gradreturncs~trltrl|r$tjj}n:t}fddt|D}t|t }||<tj |ddSd}t |S)ap Gathers a tensor from each distributed rank into a list. Always retains gradients for the local rank's tensor, and optionally retains gradients for the gathered tensors if `with_grad` is True. Args: tensor (torch.Tensor): The tensor to gather from each rank. with_grad (bool, optional): If True, the local rank's tensor retains its gradients. Defaults to False. Returns: torch.Tensor: A tensor containing the gathered tensors from all ranks, concatenated along the first dimension. If torch.distributed is not available or not initialized, returns the original tensor. csg|]}tqS)torchZ zeros_like).0_rrb/var/www/html/assistant/venv/lib/python3.9/site-packages/sentence_transformers/util/distributed.py zall_gather..r)dimzTrying to gather while torch.distributed is not available or has not been initialized, returning the original (local) tensor. This is expected if you are only using one GPU; consider not using gathering to remove this warning.) distZ is_availableZis_initializedr distributednn all_gatherZget_world_sizerangeZget_rankcatloggerZ warning_once)rrZgathered_tensorsZ world_sizeZ local_rankwarningrr r r s  r)rrcCs t|ddS)a Gathers a tensor from each distributed rank into a list, retaining gradients for the local rank's tensor. Args: tensor (torch.Tensor): The tensor to gather from each rank. Returns: torch.Tensor: A tensor containing the gathered tensors from all ranks, concatenated along the first dimension. If torch.distributed is not available or not initialized, returns the original tensor. T)r)rr rrr all_gather_with_grad2s r)F) __future__rr Ztorch.distributedrrZtransformers.utilsrZ get_logger__name__rrrrrrr s    '