a hz@sdZddlZddlmZmZddlZddlZddlmZddlm Z ddl m Z ddl m Z dd lmZmZmZdd lmZmZmZdd lmZdd lmZmZd dlmZmZmZmZm Z d dl!m"Z"d dl#m$Z$ddl%m&Z&erddl'm(Z(m)Z)Gddde Z*eGdddeZ+GdddeZ,GdddeZ-eGddde+Z.eddGd d!d!e+e Z/Gd"d#d#eZ0Gd$d%d%e"Z1Gd&d'd'eZ2gd(Z3dS))zPyTorch PLBART model.N)OptionalUnion)nn)CrossEntropyLoss)Cache)GenerationMixin)AttentionMaskConverter_prepare_4d_attention_mask#_prepare_4d_attention_mask_for_sdpa)BaseModelOutputSeq2SeqLMOutputSeq2SeqModelOutput)PreTrainedModel)auto_docstringis_torch_flex_attn_available)BartClassificationHead BartDecoder BartEncoderBartForCausalLMBartScaledWordEmbedding)'BigBirdPegasusForSequenceClassification)shift_tokens_right) PLBartConfig) BlockMaskmake_flex_block_causal_maskc@s eZdZdS)PLBartScaledWordEmbeddingN__name__ __module__ __qualname__r#r#e/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/plbart/modular_plbart.pyr7src@seZdZUeed<dZdZddgZdZdZ dZ e e j dfe j ddd Zee e j d fe j e j ed d d Zee j eee je j edddZe e j dfe e j dfe je j dddZdS)PLBartPreTrainedModelconfigmodelTZPLBartDecoderLayerZPLBartEncoderLayerN)attention_mask inputs_embedscCsv|durr|jjdkr&d|vr |nd}nL|jjdkr@t||j}n2|jjdkrft|tjrrt|dd}n t||j}|S)Nflash_attention_2rsdpaflex_attentionF) is_causal r&_attn_implementationr dtype isinstancetorchTensorrr )selfr(r)r#r#r$_update_full_maskFs     z'PLBartPreTrainedModel._update_full_maskr)r( input_tensorcache_positionpast_key_valuesc Csf|jjdkrRt|tjr"t|}n,|durNttj|jd|jdf|jd}|S|jjdkrz|durv|dk rv|SdS|dur| nd}|dur|j nd}|jjdkr|st j ||||jd rdS|j}|jd}|r|} n"t|tjr|jd n ||d} |j||| |||jdd } |jjdkrb|durb|jjd vrbt|j} t | | } | S) Nr,rr)sizedevicer*gFr+)r)Zpast_key_values_lengthZ is_training)sequence_length target_lengthr0r7 batch_size)cudaZxpuZnpu)r&r/r1r2r3rZonesshaper:anyZget_seq_lengthZis_compileabler Z_ignore_causal_mask_sdpaZtrainingr0Zget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positiontypefinfominZ_unmask_unattended) r4r(r6r7r8Zpast_seen_tokensZusing_compilable_cacher0r<r= causal_mask min_dtyper#r#r$_update_causal_mask]sd            z)PLBartPreTrainedModel._update_causal_mask)r(r<r=r0r7r>c KsF|dur|dkr|}n&t|j}tj||f|||jd}|dkrVtj|dd}|tj||jd|ddk9}|ddddddf |ddd}|durB| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N)Z fill_valuer0r:r)Zdiagonalr:r;r)dimr2rDrEfullr:ZtriuZarangeZreshapeexpandcloner@toZ masked_fill) r(r<r=r0r7r>kwargsrFrGZ mask_lengthZ padding_maskr#r#r$rBs*  $  6  zKPLBartPreTrainedModel._prepare_4d_causal_attention_mask_with_cache_position)encoder_hidden_statesencoder_attention_mask input_shaper)cCs|dur|dur|jjdkr.d|vr(|nd}nb|jjdkrPt||j|dd}n@|jjdkr|t|tjrt||ddd}nt||j|dd}|S) Nr*rr+r;)Ztgt_lenr,F)Z query_lengthr-r.)r4rQrRrSr)r#r#r$_update_cross_attn_masks(     z-PLBartPreTrainedModel._update_cross_attn_mask)r r!r"r__annotations__base_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_supports_flash_attnZ_supports_sdpaZ_supports_flex_attnrr2r3r5rrrH staticmethodintr0rBSizerTr#r#r#r$r%;s8    L9  r%c@s eZdZdS) PLBartEncoderNrr#r#r#r$rZsrZc@s eZdZdS) PLBartDecoderNrr#r#r#r$r[ sr[cseZdZddgZedfdd ZddZdd Zd d Zd d Z e de e j e e j e e j e e je e je e j e e je ee je ee e je e je ee ee ee ee e j eee jefdddZZS) PLBartModelencoder.embed_tokens.weightdecoder.embed_tokens.weightr&cslt||j|j}}|jr,t|jnd}t||j||d|_ t ||j |_ t ||j |_ |dS)Ng?) embed_scale)super__init__ pad_token_id vocab_sizeZscale_embeddingmathsqrtd_modelrsharedrZencoderr[decoder init_weights)r4r&Z padding_idxrdr` __class__r#r$rbs zPLBartModel.__init__cCs|jSN)rhr4r#r#r$get_input_embeddingssz PLBartModel.get_input_embeddingscCs||_|j|j_|j|j_dSrn)rhri embed_tokensrj)r4valuer#r#r$set_input_embeddings!s z PLBartModel.set_input_embeddingscCs0|jjr,||jj|j||jj|jdSrn)r&Ztie_word_embeddingsZ_tie_or_clone_weightsrirqrhrjror#r#r$ _tie_weights&szPLBartModel._tie_weightscCs|jSrn)riror#r#r$ get_encoder+szPLBartModel.get_encoderN) input_idsr(decoder_input_idsdecoder_attention_mask head_maskdecoder_head_maskcross_attn_head_maskencoder_outputsr8r)decoder_inputs_embeds use_cacheoutput_attentionsoutput_hidden_states return_dictr7returncCs6| dur | n|jj} |dur |n|jj}| dur4| n|jj} |durH|n|jj}|durn| durnt||jj}|dur|j|||| | ||d}nH|rt|t st |dt |dkr|dndt |dkr|dndd}|j |||d|||| | | | |||d }|s||St |j |j|j|j|j|j |j|jdS) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. cross_attn_head_mask (: obj:*torch.Tensor* of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. N)rvr(ryr)rrrrrr)last_hidden_state hidden_states attentions) rvr(rQrRryr{r8r)r~rrrr7)rr8decoder_hidden_statesdecoder_attentionscross_attentionsencoder_last_hidden_staterQencoder_attentions)r&rrr~use_return_dictrrcrir1r lenrjrrr8rrr)r4rvr(rwrxryrzr{r|r8r)r}r~rrrr7Zdecoder_outputsr#r#r$forward.sd1 zPLBartModel.forward)NNNNNNNNNNNNNNNN)r r!r"_tied_weights_keysrrbrprsrtrurrr2 LongTensorr3list FloatTensorrboolrtuplerr __classcell__r#r#rlr$r\sT  r\zv The PLBART Model with a language modeling head. Can be used for code-to-text, text-to-code and code-to-code. )Z custom_introcs(eZdZdZdgZgdZedfdd ZddZd d Z de e e e e jd fdd Ze d dddZede eje eje eje eje eje eje eje eeje ee eje eje eje e e e e e e e e ejeeejefdddZejdddZZS)PLBartForConditionalGenerationr'final_logits_bias)r]r^zlm_head.weightr_csXt|t||_|dtd|jjjft j |j |jjjdd|_ | dS)NrrF)Zbias)rarbr\r'register_bufferr2zerosrhZnum_embeddingsrZLinearrglm_headrk)r4r&rlr#r$rbs   z'PLBartForConditionalGeneration.__init__cCs |jSrn)r'ruror#r#r$rusz*PLBartForConditionalGeneration.get_encodercCs |jSrn)r' get_decoderror#r#r$rsz*PLBartForConditionalGeneration.get_decoderNT)new_num_tokenspad_to_multiple_of mean_resizingrcs&t|||}||jjd|S)Nr)raresize_token_embeddings_resize_final_logits_biasweightr@)r4rrrZnew_embeddingsrlr#r$rsz6PLBartForConditionalGeneration.resize_token_embeddings)rrcCsj|jjd}||kr,|jddd|f}n.tjd||f|jjd}tj|j|gdd}|d|dS)Nr;rrJ)rKr)rr@r2rr:catr)r4rZold_num_tokensZnew_biasZ extra_biasr#r#r$rs  z8PLBartForConditionalGeneration._resize_final_logits_bias)rvr(rwrxryrzr{r|r8r)r}labelsr~rrrr7rcCs|dur |n|jj}| dur:|dur:| dur:t| |jj}|j||||||||| | | | ||||d}||d}||j|j}d}| durt }|| d|jj | d}|s|f|dd}|dur|f|S|St |||j |j|j|j|j|j|jd S)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. cross_attn_head_mask (: obj:*torch.Tensor* of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example Mask-filling: ```python >>> from transformers import AutoTokenizer, PLBartForConditionalGeneration >>> model = PLBartForConditionalGeneration.from_pretrained("uclanlp/plbart-base") >>> tokenizer = AutoTokenizer.from_pretrained("uclanlp/plbart-base") >>> # en_XX is the language symbol id for English >>> TXT = " Is 0 the Fibonacci number ? en_XX" >>> input_ids = tokenizer([TXT], add_special_tokens=False, return_tensors="pt").input_ids >>> logits = model(input_ids).logits >>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item() >>> probs = logits[0, masked_index].softmax(dim=0) >>> values, predictions = probs.topk(5) >>> tokenizer.decode(predictions).split() ['first', 'same', 'highest', 'result', 'number'] ``` N)r(rwr|rxryrzr{r8r)r}r~rrrr7rr;r) ZlossZlogitsr8rrrrrQr)r&rrrcr'rrrOr:rviewrdr r8rrrrrQr)r4rvr(rwrxryrzr{r|r8r)r}rr~rrrr7outputsZ lm_logitsZmasked_lm_lossZloss_fctoutputr#r#r$rsTKz&PLBartForConditionalGeneration.forward)rcCst||jjSrn)rr&rc)r4rr#r#r$%prepare_decoder_input_ids_from_labels@szDPLBartForConditionalGeneration.prepare_decoder_input_ids_from_labels)NT)NNNNNNNNNNNNNNNNN)r r!r"rVZ_keys_to_ignore_on_load_missingrrrbrurrXrrrZ Embeddingrrrr2rr3rrrrrr rrrr#r#rlr$rsf   zrc@s eZdZdS)PLBartClassificationHeadNrr#r#r#r$rDsrcseZdZfddZZS)PLBartForSequenceClassificationc stjfi|dS)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. cross_attn_head_mask (: obj:*torch.Tensor* of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrarZ super_kwargsrlr#r$rIs!z'PLBartForSequenceClassification.forward)r r!r"rrr#r#rlr$rHsrcs eZdZefddZZS)PLBartForCausalLMc stjfi|dS)a cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, PLBartForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("uclanlp/plbart-base") >>> model = PLBartForCausalLM.from_pretrained("uclanlp/plbart-base", add_cross_attention=False) >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```Nrrrlr#r$rnszPLBartForCausalLM.forward)r r!r"rrrr#r#rlr$rmsr)rrrr\r%)4__doc__retypingrrr2Ztorch.utils.checkpointrZtorch.nnrZ cache_utilsrZ generationrZmodeling_attn_mask_utilsr r r Zmodeling_outputsr r rZmodeling_utilsrutilsrrZbart.modeling_bartrrrrrZ(bigbird_pegasus.modeling_bigbird_pegasusrZmbart.modeling_mbartrZconfiguration_plbartrZintegrations.flex_attentionrrrr%rZr[r\rrrr__all__r#r#r#r$sH        K#%!