a h T@sdZddlmZddlmZmZddlZddlmZeGdddZ deej eej e e fej eeed d d Zdeej eej e e fej eeed d d Zdej ejeedddZdej ejeedddZdeej e e fejejeeeeej dddZdS)a IMPORTANT NOTICE: Every class and function in this file is deprecated in favor of using the much more general `masking_utils.py` primitives. New code should not rely on it, it is only kept for backward compatibility for now, and will be removed in the future. ) dataclass)OptionalUnionN)is_torchdynamo_compilingc @s eZdZUdZeed<eed<deeedddZdeeee j e e j d fee j d d d Zd e j ee j eee j d ddZed!e je j e j eeedddZed"e j e j eedddZee jedddZed#ee j e j eeeeedddZdS)$AttentionMaskConvertera9 A utility attention mask class that allows one to: - Create a causal 4d mask - Create a causal 4d mask with slided window - Convert a 2d attention mask (batch_size, query_length) to a 4d attention mask (batch_size, 1, query_length, key_value_length) that can be multiplied with attention scores Examples: ```python >>> import torch >>> from transformers.modeling_attn_mask_utils import AttentionMaskConverter >>> converter = AttentionMaskConverter(True) >>> converter.to_4d(torch.tensor([[0, 0, 0, 1, 1]]), 5, key_value_length=5, dtype=torch.float32) tensor([[[[-3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38], [-3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38], [-3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38], [-3.4028e+38, -3.4028e+38, -3.4028e+38, 0.0000e+00, -3.4028e+38], [-3.4028e+38, -3.4028e+38, -3.4028e+38, 0.0000e+00, 0.0000e+00]]]]) ``` Parameters: is_causal (`bool`): Whether the attention mask should be a uni-directional (causal) or bi-directional mask. sliding_window (`int`, *optional*): Optionally, the sliding window masks can be created if `sliding_window` is defined to a positive integer. is_causalsliding_windowNrr cCs6||_||_|jdur2|jdkr2td|jddS)NrzaMake sure that when passing `sliding_window` that its value is a strictly positive integer, not ``)rr ValueError)selfrr ra/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/modeling_attn_mask_utils.py__init__?s  zAttentionMaskConverter.__init__cpustr) batch_size query_lengthkey_value_lengthdtypedevicereturnc Cs\|jstd|jd||f}||}d}|ddksB|jdurX|j|||||jd}|S)z Creates a causal 4D mask of (bsz, head_dim=1, query_length, key_value_length) shape and adds large negative bias to upper right hand triangular matrix (causal mask). z"Please use `to_causal_4d` only if z has `is_causal` set to True.Nrrpast_key_values_lengthr )rr __class__r _make_causal_mask) r rrrrr input_shapercausal_4d_maskrrr to_causal_4dHs z#AttentionMaskConverter.to_causal_4d)attention_mask_2drrrrc Cs|jd|f}d}|ddks(|jdur`|jr`|dur>td||}|j|||j||jd}n|jdurrtd|j|||dd|j}|dur| | t |j }|} | S) a Converts 2D attention mask to 4D attention mask by expanding mask to (bsz, head_dim=1, query_length, key_value_length) shape and by adding a large negative bias to not-attended positions. If attention_mask is causal, a causal mask will be added. rNrrzpThis attention mask converter is causal. Make sure to pass `key_value_length` to correctly create a causal mask.rz?Sliding window is currently only implemented for causal masking)tgt_len)shaper rr rrNotImplementedError _expand_maskto masked_fillbooltorchfinfomin) r r!rrrrrrZexpanded_attn_maskexpanded_4d_maskrrrto_4dis0  zAttentionMaskConverter.to_4dr)input_ids_shaperrrr c Cs|\}}tj||ft|j|d}tj|d|d}|||d|ddkd||}|dkrtj tj ||||d|gdd}|dur||d} tj tj |tj d| d } tr|}|| t|j|ddddddf|d|||S) zJ Make causal mask used for bi-directional self-attention. )rrrrrr)dimNr)diagonal)r)fullr*r+ZarangesizeZ masked_fill_viewr&catZzerosZtrilZ ones_liker(rcloneexpand) r.rrrr bszr"maskZ mask_condr2Z context_maskrrrrs "   z(AttentionMaskConverter._make_causal_maskr:rr"cCst|\}}|dur|n|}|ddddddf|d|||}tjd|d|}||tjt|jS)zg Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. Nr?r1) r4r8r&r)Ztensorr'r(r*r+)r:rr"r9Zsrc_len expanded_mask inverted_maskrrrr%s  *z#AttentionMaskConverter._expand_maskr= min_dtypecCs0|jtjkrtd|tj||kdddS)a Attend to all tokens in masked rows from the expanded attention mask, for example the relevant first rows when using left padding. This is required by F.scaled_dot_product_attention memory-efficient attention path. Details: https://github.com/pytorch/pytorch/issues/110213 `expanded_mask` is [bsz, num_masks, tgt_seq_len, src_seq_len] or [bsz, tgt_seq_len, src_seq_len]. `attention_mask` is [bsz, src_seq_len]. The dimension num_masks of `expanded_mask` is most often 1, but it can also be the number of heads in the case of alibi attention bias. For example, if `expanded_mask` is (e.g. here left-padding case) ``` [[[[0, 0, 0], [0, 0, 0], [0, 0, 1]]], [[[1, 0, 0], [1, 1, 0], [1, 1, 1]]], [[[0, 0, 0], [0, 1, 0], [0, 1, 1]]]] ``` then the modified `expanded_mask` will be ``` [[[[1, 1, 1], <-- modified [1, 1, 1], <-- modified [0, 0, 1]]], [[[1, 0, 0], [1, 1, 0], [1, 1, 1]]], [[[1, 1, 1], <-- modified [0, 1, 0], [0, 1, 1]]]] ``` z\AttentionMaskConverter._unmask_unattended expects a float `expanded_mask`, got a BoolTensor.rT)r0Zkeepdim)rr)r(r mulallr?rrr_unmask_unattendeds * z)AttentionMaskConverter._unmask_unattendedF)attention_mask inputs_embedsrr is_trainingrc Cs|jd|jd}}||}tjp:t|tjjp:t}d} |durv|sP|s|dks`||kr|dusp||krd} nH|dus||krt|jdkrdS|st |dkr|dks||krd} | S)a9 Detects whether the optional user-specified attention_mask & the automatically created causal mask can be ignored in case PyTorch's SDPA is used, rather relying on SDPA's `is_causal` argument. In case no token is masked in the `attention_mask` argument, if `query_length == 1` or `key_value_length == query_length`, we rather rely on SDPA `is_causal` argument to use causal/non-causal masks, allowing to dispatch to the flash attention kernel (that can otherwise not be used if a custom `attn_mask` is passed). rrFNT) r#r)jit is_tracing isinstancefxProxyrlenrB) rDrErr rF_rrrIignore_causal_maskrrr_ignore_causal_mask_sdpas2 z/AttentionMaskConverter._ignore_causal_mask_sdpa)N)r)N)rN)N)NF)__name__ __module__ __qualname____doc__r(__annotations__intrrr)rrrTensorr r- staticmethodSizerr%Z FloatTensorfloatrCrPrrrrrs`   & /! 0r)rDrrErr c Cstd|d}|d|}|durHt|jdkrH|j||d||jd}n|durt|jdkr|dd |d |f}t|j|krtd t|jd |d qd |}||t j t |jj }n |j |d|d||j|jd}|S)a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)` Args: attention_mask (`torch.Tensor` or `None`): A 2D attention mask of shape `(batch_size, key_value_length)` input_shape (`tuple(int)` or `list(int)` or `torch.Size`): The input shape should be a tuple that defines `(batch_size, query_length)`. inputs_embeds (`torch.Tensor`): The embedded inputs as a torch Tensor. past_key_values_length (`int`): The length of the key value cache. sliding_window (`int`, *optional*): If the model uses windowed attention, a sliding window should be passed. Tr rN)rrrGrrz#Incorrect 4D attention_mask shape: z ; expected: .r<r/)rrMr#r-rtupler r'r&r)r(r*r+r r) rDrrErr attn_mask_converterrZexpected_shaper>rrr!_prepare_4d_causal_attention_mask4s(  r_c Cstd|d}|d|}tjp4t|tjjp4t}tj||||d}|rRd} n||dur||j |d|d||j |j d} nR| dkr|} n|j ||d|j |d } |s| j jd krtj| t|j jd } | S) a Prepares the correct `attn_mask` argument to be used by `torch.nn.functional.scaled_dot_product_attention`. In case no token is masked in the `attention_mask` argument, we simply set it to `None` for the cases `query_length == 1` and `key_value_length == query_length`, and rely instead on SDPA `is_causal` argument to use causal/non-causal masks, allowing to dispatch to the flash attention kernel (that can otherwise not be used if a custom `attn_mask` is passed). Tr r)rDrErr Nrr/rG)rrcuda)r@)rr)rHrIrJrKrLrrPr rrr0r-typerCr*r+) rDrrErr r^rrIrOr,rrr*_prepare_4d_causal_attention_mask_for_sdpais8    rbr;cCstj|||dS) Creates a non-causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)` Args: mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` dtype (`torch.dtype`): The torch dtype the created mask shall have. tgt_len (`int`): The target length or query length the created mask shall have. r;)rr%r;rrr_prepare_4d_attention_masks rdcCsb|j\}}|dur|n|}tjp6t|tjjp6t}|sNt|dkrNdSt j |||dSdS)rcNrr;) r#r)rHrIrJrKrLrrBrr%)r:rr"rNrrIrrr#_prepare_4d_attention_mask_for_sdpas re)rrrrr rcCs8td|d}||d}|j|d|d|||d}|S)a/ Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` Args: input_shape (`tuple(int)` or `list(int)` or `torch.Size`): The input shape should be a tuple that defines `(batch_size, query_length)`. dtype (`torch.dtype`): The torch dtype the created mask shall have. device (`int`): The torch device the created mask shall have. sliding_window (`int`, *optional*): If the model uses windowed attention, a sliding window should be passed. Tr rrr/)rr )rrrrr r^rrDrrr _create_4d_causal_attention_masks   rf)N)N)N)N)rN)rT dataclassesrtypingrrr)Zutils.import_utilsrrrWrYr]listrVr_rbrrdrerrfrrrrsH   : :