a hA@s2dZddlZddlmZmZddlZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZmZm Z dd l!m"Z"ddl#m$Z$m%Z%ddl&m'Z'ddl(m)Z)e%*e+Z,ej-e.e.dddZ/Gdddej0Z1Gdddej2Z3GdddeZ4GdddeZ5Gdddej2Z6Gdd d ej2Z7e$Gd!d"d"e"Z8Gd#d$d$e8Z9Gd%d&d&e8Z:e$Gd'd(d(e8Z;e$d)d*Gd+d,d,e8eZGd2d3d3e8Z?Gd4d5d5e8eZ@gd6ZAdS)7zPyTorch MVP model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)_prepare_4d_attention_mask!_prepare_4d_causal_attention_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutput)PreTrainedModel)auto_docstringlogging)deprecate_kwarg) MvpConfig) input_ids pad_token_iddecoder_start_token_idcCsh||j}|ddddf|ddddf<||dddf<|durTtd||dk||S)z1 Shift input ids one token to the right. Nrrz1self.model.config.pad_token_id has to be defined.i)Z new_zerosshapeclone ValueErrorZ masked_fill_)rrr Zshifted_input_idsr%`/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/mvp/modeling_mvp.pyshift_tokens_right4s (r'csDeZdZdZeedfdd Zd ejeejdfdd ZZ S) MvpLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. )num_embeddings embedding_dimcsd|_t||j|dSN)offsetsuper__init__)selfr)r* __class__r%r&r/Jsz&MvpLearnedPositionalEmbedding.__init__rN)rpast_key_values_length position_idscs\|dur@|jdd\}}tj|||tj|jjd|d}n |d}t ||j S)z3`input_ids' shape is expected to be [bsz x seqlen].Nr,)dtypedevicer!r) r"torcharangelongweightr6expandZ unsqueezer.forwardr-)r0rr3r4bszZseq_lenr1r%r&r<Ps z%MvpLearnedPositionalEmbedding.forward)rN) __name__ __module__ __qualname____doc__intr/r7Tensorr< __classcell__r%r%r1r&r(Esr(cseZdZdZdeeeeeeeeeedfdd Ze d d d d de j ee j ee ee j ee j ee j eee j e e j ee j ee e j fd ddZZS) MvpAttentionz=Multi-headed attention from 'Attention Is All You Need' paperFTN) embed_dim num_headsdropout is_decoderbias layer_idxcst||_||_||_|||_|j||jkrNtd|jd|d|jd|_||_||_ t j |||d|_ t j |||d|_ t j |||d|_t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rK)r.r/rGrHrIhead_dimr$scalingrJrLrLineark_projv_projq_projout_proj)r0rGrHrIrJrKrLr1r%r&r/as$    zMvpAttention.__init__past_key_valuepast_key_values4.58new_nameversion) hidden_stateskey_value_statesrVattention_masklayer_head_mask attn_promptoutput_attentionscache_positionreturnc Cs|du} |\} } } |||j} |dur^t|trZ|j|j}| rR|j}q^|j }n|}| rf|n|}| r|dur|r|j |jj }|j |jj }n| |}||}|| d|j|jdd}|| d|j|jdd}|dur&| s|nd}||||jd|i\}}| r&d|j|j<|durtj|d| ddd|gdd}tj|d| ddd|gdd}|durt| d| |dd|j}tj||gdd}| |jd|jf}| | | |j|jdd} | j|} |j|}|j|}|d}t| |dd}|| |j| |fkrZtd | |j| |fd ||dur|| d| |fkrtd | d| |fd ||| |j| ||}|| |j| |}tjj|dd}|durB||jfkr td |jfd ||dddd|| |j| |}|| |j| |}|rp|| |j| |}|| |j| |}nd}tjj ||j |j!d }t||}|| |j| |jfkrtd| |j| |jfd ||| |j| |j}|dd}|| | |j"}|#|}||fS)z#Input shape: Batch x Time x ChannelNr!rr,raTrdimz$Attention weights should be of size z , but is z!Attention mask should be of size z/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )$sizerSrO isinstancer is_updatedgetrLZcross_attention_cacheZself_attention_cachelayerskeysvaluesrQrRviewrHrNZ transposeupdater7catr;zerostor6ZreshapeZbmmr$r functionalZsoftmaxrIrgrGrT)r0r[r\rVr]r^r_r`raZis_cross_attentionr=tgt_len_Z query_statesrjZcurr_past_key_valueZcurrent_statesZ key_statesZ value_statesZ prompt_maskZ proj_shapeZsrc_len attn_weightsZattn_weights_reshapedZ attn_probsZ attn_outputr%r%r&r<~s        "" "       "   zMvpAttention.forward)rFFTN)NNNNNFN)r>r?r@rArBrfloatboolr/rr7rCr tupler<rDr%r%r1r&rE^s@ rEc sXeZdZedfdd Zdejejejejeee ejeejfdddZ Z S) MvpEncoderLayerconfigcst|j|_t|j|j|jd|_t |j|_ |j |_ t |j |_|j|_t|j|j|_t|j|j|_t |j|_dS)N)rGrHrI)r.r/d_modelrGrEencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrIr activation_function activation_fnactivation_dropoutrPZencoder_ffn_dimfc1fc2final_layer_normr0r}r1r%r&r/s  zMvpEncoderLayer.__init__F)r[r]r^self_attn_promptr`rbc Cs|}|j|||||d\}}tjj||j|jd}||}||}|}|||}tjj||j|jd}| |}tjj||j|jd}||}| |}|j t j krt |st |rt |j jd}t j|| |d}||fS)a@ Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. self_attn_prompt (`torch.FloatTensor`): prompt of self attention of shape `(2, encoder_attention_heads, pro_len, head_dim)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r[r]r^r_r`rei)minmax)rrrtrIrgrrrrrrr5r7Zfloat16isinfanyisnanZfinforclamp) r0r[r]r^rr`residualrwZ clamp_valuer%r%r&r<s4       zMvpEncoderLayer.forward)F) r>r?r@rr/r7 FloatTensorrryrzr<rDr%r%r1r&r{sr{cseZdZdedfdd Zedddddejeejeejeejeejeejeejeejee ee ee eeje ej ee ej ej ffd d d Z ZS)MvpDecoderLayerNr|cst|j|_t|j|j|jd|d|_|j|_t |j |_ |j |_ t |j|_t|j|j|jd|d|_t |j|_t |j|j|_t |j|j|_t |j|_dS)NT)rGrHrIrJrL)rIrJrL)r.r/r~rGrEdecoder_attention_headsrrrIr rrrrrr encoder_attnencoder_attn_layer_normrPZdecoder_ffn_dimrrr)r0r}rLr1r%r&r/Bs0  zMvpDecoderLayer.__init__rUrVrWrXFT) r[r]encoder_hidden_statesencoder_attention_maskr^cross_attn_layer_head_maskrcross_attn_promptrVr` use_cacherarbc  Cs|} |j|| |||| | d\}}tjj||j|jd}| |}||}d}|dur|} |j|||||| | d\}}tjj||j|jd}| |}||}|} || |}tjj||j |jd}| |}tjj||j|jd}| |}| |}|f}| r|||f7}|S)aD Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size `(decoder_attention_heads,)`. self_attn_prompt (`torch.FloatTensor`): prompt of self attention of shape `(2, decoder_attention_heads, pro_len, head_dim)`. cross_attn_prompt (`torch.FloatTensor`): prompt of cross attention of shape `(2, decoder_attention_heads, pro_len, head_dim)`. past_key_values (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r[rVr]r^r_r`rareN)r[r\r]r^r_rVr`) rrrtrIrgrrrrrrrr)r0r[r]rrr^rrrrVr`rrarZself_attn_weightsZcross_attn_weightsoutputsr%r%r&r<^sN&       zMvpDecoderLayer.forward)N) NNNNNNNNFTN)r>r?r@rr/rr7rCrr ryrzrr<rDr%r%r1r&rAs8 rcs@eZdZdZeeeedfdd ZejejdddZ Z S)MvpClassificationHeadz-Head for sentence-level classification tasks.) input_dim inner_dim num_classespooler_dropoutcs8tt|||_tj|d|_t|||_dS)Nrf)r.r/rrPdenseDropoutrIrT)r0rrrrr1r%r&r/s zMvpClassificationHead.__init__)r[rbcCs6||}||}t|}||}||}|SN)rIrr7tanhrT)r0r[r%r%r&r<s      zMvpClassificationHead.forward) r>r?r@rArBrxr/r7rCr<rDr%r%r1r&rs rcs8eZdZdZfddZejeejdddZZ S) MvpPromptz)Layer-wise prompt for encoder or decoder.c st|j|_||_||_|j||_tj|j d|_ t |j|j|_ t t |j|jtt |j|d|j|_dS)Nrr,)r.r/ prompt_length num_layersrHr~rNrrrI Embeddingprompt_embeddingZ SequentialrPZprompt_mid_dimZGELU prompt_trans)r0r}rrHr1r%r&r/s  zMvpPrompt.__init__) prompt_idsrbcCsN|||}||j|jd|j|j}||}|gd d}|S)Nr,)rr,rr) rrrorrrHrNrIZpermutesplit)r0rpromptr%r%r&r<s  zMvpPrompt.forward) r>r?r@rAr/r7rCrzr<rDr%r%r1r&rs rc@s2eZdZUeed<dZdZddZeddZ dS) MvpPreTrainedModelr}modelTcCs||jj}t|tjr>|jjjd|d|jdurx|jj n:t|tj rx|jjjd|d|j durx|jj|j  dS)NrF)meanstd) r}Zinit_stdrirrPr:dataZnormal_rKZzero_r padding_idx)r0modulerr%r%r& _init_weightss    z MvpPreTrainedModel._init_weightscCs>|jj}tjgddddd|gg|jd}|||d}|S)N)r r,r r,r6)r]r)r}rr7Ztensorr6ne)r0Z pad_tokenr dummy_inputsr%r%r&rs "zMvpPreTrainedModel.dummy_inputsN) r>r?r@r__annotations__Zbase_model_prefixZsupports_gradient_checkpointingrpropertyrr%r%r%r&rs  rc seZdZdZd eeejeedfdd Z d ee j ee j ee j ee j eeeeeeeeefddd ZZS) MvpEncodera Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`MvpEncoderLayer`]. Args: config: MvpConfig embed_tokens (nn.Embedding): output embedding use_prompt (bool): whether to use prompt NFr} embed_tokens use_promptcstj|_j|_j}j|_j|_ j rBt |nd|_ |durX||_ntj||j|_tj||_tfddtjD|_t||_||_|rʈj|_tjj|_d|_| dS)N?csg|] }tqSr%)r{).0rvr|r%r& &z'MvpEncoder.__init__..F)!r.r/rIZencoder_layerdrop layerdropr~rrmax_position_embeddingsZmax_source_positionsscale_embeddingmathsqrt embed_scalerrr vocab_sizer(embed_positions ModuleListrangeZencoder_layersrlrlayernorm_embeddingrrrrrgradient_checkpointing post_init)r0r}rrrGr1r|r&r/s4   zMvpEncoder.__init__)rr] head_mask inputs_embedsr`output_hidden_states return_dictrbcCs~|dur |n|jj}|dur |n|jj}|dur4|n|jj}|durV|durVtdn\|durz|}|j} |d| d}n8|dur|dd} |dddddf}ntd|dur|||j }| |} || } | | } t j j| |j|jd} |jr"t|j|j} || } |dur8t||j}|rBdnd}|rPdnd}|dur|dt|jkrtdt|jd |dd t|jD]\}}|r|| f}d }|jrtg}||jkrd }|rd }n<|| ||dur||nd|jr| |nd|d}|d} |r||df}q|rR|| f}|sptdd| ||fDSt| ||dS)a~ Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzDYou cannot specify both input_ids and inputs_embeds at the same timer!z5You have to specify either input_ids or inputs_embedsrer%rz&The head_mask should be specified for  layers, but it is for .FT)NN)r^rr`rcss|]}|dur|VqdSrr%rvr%r%r& rz%MvpEncoder.forward..last_hidden_stater[ attentions) r}r`ruse_return_dictr$r"rorhrrrrrrtrIrgrr7r8rrsr6rrr5lenrl enumeraterandrrzr)r0rr]rrr`rrinput input_shapeZ embed_posr[rrZencoder_statesZall_attentionsidxZ encoder_layerZto_dropdropout_probability layer_outputsr%r%r&r<6sz.            zMvpEncoder.forward)NF)NNNNNNN)r>r?r@rArrrrryr/r7 LongTensorrCrrrzrr<rDr%r%r1r&rs, ( rcseZdZdZd eeejeedfdd Z d ee j ee j ee j ee j ee j ee j eee j ee j eeeeeeeeee j eeefddd ZZS) MvpDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`MvpDecoderLayer`] Args: config: MvpConfig embed_tokens (nn.Embedding): output embedding use_prompt (bool): whether to use prompt NFrcstj|_j|_j|_j|_j r>t j nd|_ |durT||_ntjj |j|_tjj |_tfddtjD|_tj |_||_|rވj|_tjj|_tjj|_d|_ |!dS)Nrcsg|]}t|dqS))rL)r)rir|r%r&rrz'MvpDecoder.__init__..F)"r.r/rIZdecoder_layerdroprrrrZmax_target_positionsrrrr~rrrrrr(rrrZdecoder_layersrlrrrrrrrrrr)r0r}rrr1r|r&r/s<  zMvpDecoder.__init__)rr]rrrcross_attn_head_maskrVrrr`rrrarbcCs| dur | n|jj} | dur | n|jj} | dur4| n|jj} | durH| n|jj} |durj|durjtdn\|dur|}|j}|d|d}n8|dur|dd}|dddddf}ntd|dur| ||j }|j r|j r| rt dd} | r<|dur<|dur0tt|jdt|jdn t|jd}| rbt|trbt dt|}|durt|nd }t||||}|dur|durt||j|dd }|||}||}||}tjj||j|j d }|jrt|j !|j"}|#|}|$|}| rd nd}| r*d nd}| rB|durBd nd}t%||gd dgD]V\}}|durX|d t&|j'krXtd|dt&|j'd|d dqXt(|j'D]\}}| r||f7}|j rt)g}||j*krq||||||dur||nd|dur$||nd|jr6||nd|jrH||nd|| | | d }|d }| r||df7}|dur||df7}q| r||f7}| stdd|||||fDSt+|||||dS)a  Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer!zEYou have to specify either decoder_input_ids or decoder_inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fr|zPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.r)rurer%rrzThe `z` should be specified for rr) rr^rrrrVr`rrarr,css|]}|dur|VqdSrr%rr%r%r&rsz%MvpDecoder.forward..)rrVr[rcross_attentions),r}r`rrrr$r"rorhrrrrgloggerZ warning_oncer r rirzZfrom_legacy_cacheZget_seq_lengthrrr5rrrrtrIrr7r8rrsr6rrziprrlrrrr)r0rr]rrrrrVrrr`rrrarrr3Z positionsr[rrrZall_hidden_statesZall_self_attnsZall_cross_attentionsZ attn_maskZ mask_namerZ decoder_layerrrr%r%r&r<sQ                 zMvpDecoder.forward)NF) NNNNNNNNNNNNN)r>r?r@rArrrrryr/r7rrCrlistrrzrr<rDr%r%r1r&rsD *  rcseZdZdgZddgZedfdd ZddZd d Zd d Z d dZ e de e je e je e je e je e je e je e je ee je ee je e je e je ee ee ee ee e jeeefdddZZS)MvpModelfinal_logits_biasencoder.embed_tokens.weightdecoder.embed_tokens.weightr|csdt||j|j}}|j|_t||j||_t ||j|j|_ t ||j|j|_ | dSr)r.r/rrrrrr~sharedrencoderrdecoderr)r0r}rrr1r%r&r/s zMvpModel.__init__cCs|jSr)rr0r%r%r&get_input_embeddingsszMvpModel.get_input_embeddingscCs||_|j|j_|j|j_dSr)rrrrr0valuer%r%r&set_input_embeddingss zMvpModel.set_input_embeddingscCs|jSr)rrr%r%r& get_encoderszMvpModel.get_encodercCsF|jsJd|d|jjd|jjd|jjddS)NzHIf you want to use lightweight tuning, make sure that `use_prompt=True`.FT)rrequires_grad_rrrrrr%r%r&set_lightweight_tunings  zMvpModel.set_lightweight_tuningN)rr]decoder_input_idsdecoder_attention_maskrdecoder_head_maskrencoder_outputsrVrdecoder_inputs_embedsrr`rrrarbcCsL|dur4| dur4|dur tdt||jj|jj}| dur@| n|jj} |durT|n|jj}| durh| n|jj} |dur||n|jj}|dur|j |||| | ||d}nH|rt |t st |dt |dkr|dndt |dkr|dndd}|j |||d|||| | | | |||d }|s$||St|j|j|j|j|j|j|j|jd S) a* decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. NzIf no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.)rr]rrr`rrrrr,r) rr]rrrrrVrrr`rrra)rrVdecoder_hidden_statesdecoder_attentionsrencoder_last_hidden_staterencoder_attentions)r$r'r}rr r`rrrrrirrrrrrVr[rr)r0rr]rrrrrrrVrrrr`rrraZdecoder_outputsr%r%r&r<sp3 zMvpModel.forward)NNNNNNNNNNNNNNNN)r>r?r@Z"_keys_to_ignore_on_load_unexpected_tied_weights_keysrr/rrrrrrr7rrCrrryrrzrr<rDr%r%r1r&rsV    rze The MVP Model with a language modeling head. Can be used for various text generation tasks. )Z custom_introcs&eZdZgdZedfdd ZddZddZdee ee e j d fd d Z ed dddZddZede eje eje eje eje eje eje eje eeje eeje eje eje eje e e e e e e e e ejeeefdddZejdddZZS)MvpForConditionalGeneration)rrlm_head.weightr|csXt|t||_|dtd|jjjft j |j |jjjdd|_ | dS)NrrFrM)r.r/rrregister_bufferr7rrrr)rrPr~lm_headrrr1r%r&r/fs   z$MvpForConditionalGeneration.__init__cCs |jSr)rrrr%r%r&rosz'MvpForConditionalGeneration.get_encodercCs |jSr)r get_decoderrr%r%r&rrsz'MvpForConditionalGeneration.get_decoderNT)new_num_tokenspad_to_multiple_of mean_resizingrbcst|||}|||Sr)r.resize_token_embeddings_resize_final_logits_bias)r0rrrZnew_embeddingsr1r%r&rus z3MvpForConditionalGeneration.resize_token_embeddings)rrbcCsj|jjd}||kr,|jddd|f}n.tjd||f|jjd}tj|j|gdd}|d|dS)Nr!rrrcr)rr"r7rrr6rqr)r0rZold_num_tokensZnew_biasZ extra_biasr%r%r&r|s  z5MvpForConditionalGeneration._resize_final_logits_biascCs|j|jddSNFrrrrrr%r%r&rs z2MvpForConditionalGeneration.set_lightweight_tuning)rr]rrrrrrrVrrlabelsrr`rrrarbcCs|dur |n|jj}| durR| r*tdd} |durR| durRt| |jj|jj}|j||||||||| | | | ||||d}||d|j }d}| durt }|| d|jj | d}|s|f|dd}|dur|f|S|St |||j|j|j|j|j|j|jd S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example of summarization: Fine-tuning a model ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp") >>> inputs = tokenizer( ... "Summarize: You may want to stick it to your boss and leave your job, but don't do it if these are your reasons.", ... return_tensors="pt", ... ) >>> labels = tokenizer("Bad Reasons To Quit Your Job", return_tensors="pt")["input_ids"] >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... generated_ids = model.generate(**inputs) >>> generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True) ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F)r]rrrrrrrVrrrr`rrrarr!r losslogitsrVr r rr rr )r}rrwarningr'rr rrrrrorrrVr r rr rr )r0rr]rrrrrrrVrrrrr`rrrarZ lm_logitsZmasked_lm_lossloss_fctoutputr%r%r&r<s\R z#MvpForConditionalGeneration.forward)rcCst||jj|jjSr)r'r}rr )r0rr%r%r&%prepare_decoder_input_ids_from_labelsszAMvpForConditionalGeneration.prepare_decoder_input_ids_from_labels)NT)NNNNNNNNNNNNNNNNN)r>r?r@r rr/rrrBrryrrrrrrr7rrCrrrrzrr<r!rDr%r%r1r&r^sf      rz Mvp model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cseZdZddgZedfdd ZddZed ee j ee j ee j ee j ee j ee j ee j ee e j ee j ee j ee j eeeeeeeeeeefd d d ZZS) MvpForSequenceClassificationrrr|c sBtj|fi|t||_t|j|j|j|j|_| dSr) r.r/rrrr~ num_labelsZclassifier_dropoutclassification_headr)r0r}kwargsr1r%r&r/s z%MvpForSequenceClassification.__init__cCs|j|jddSr)rrr$rrr%r%r&r)s z3MvpForSequenceClassification.set_lightweight_tuningN)rr]rrrrrrrrrrr`rrrbcCsB|dur |n|jj}| dur d} |durB| durBtd|jj|j||||||||| | | | ||d}|d}||jj|j }t t | ddkrtd||ddf|dd|ddddddf}||}d}| dur|jjdurX|jjdkrd |j_n<|jjdkrP| jt jksF| jt jkrPd |j_nd |j_|jjd krt}|jjdkr||| }n ||| }nP|jjd krt}||d|jj| d}n|jjd krt}||| }|s|f|dd}|dur|f|S|St|||j|j|j|j|j |j!|j"d S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example of single-label classification: Fine-tuning a model on `num_labels` classes ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForSequenceClassification >>> num_labels = 2 # for example, this is a binary classification task >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForSequenceClassification.from_pretrained("RUCAIBox/mvp", num_labels=num_labels) >>> inputs = tokenizer("Classify: Hello, my dog is cute", return_tensors="pt") >>> labels = torch.tensor(1) # the real label for inputs >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax() ``` NFz8Passing input embeddings is currently not supported for r]rrrrrrrrrr`rrrrz7All examples must have the same number of tokens.r!Z regressionZsingle_label_classificationZmulti_label_classificationr)#r}rNotImplementedErrorr2r>reqZ eos_token_idrsr6rr7Zunique_consecutivesumr$rorhr$Z problem_typer#r5r9rBrsqueezerrrrVr r rr rr )r0rr]rrrrrrrrrrr`rrrr[Zeos_maskZsentence_representationrrrr r%r%r&r<-sM $   *   z$MvpForSequenceClassification.forward)NNNNNNNNNNNNNNN)r>r?r@r rr/rrrr7rrCrrryrrzrr<rDr%r%r1r&r"sJ   r"cseZdZddgZfddZddZed eej eej eej eej eej eej eej ee ej eej eej eej eej ee ee ee ee eeefdd d ZZS) MvpForQuestionAnsweringrrcsBt|d|_|j|_t||_t|j|j|_| dSr+) r.r/r#rrrrP hidden_size qa_outputsrrr1r%r&r/s   z MvpForQuestionAnswering.__init__cCs|j|jddSr)rrr-rrr%r%r&rs z.MvpForQuestionAnswering.set_lightweight_tuningN)rr]rrrrrrstart_positions end_positionsrrrr`rrrbcCs|dur |n|jj}| dur(| dur(d} |j||||||||| | | |||d}|d}||}|jddd\}}|d}|d}d}| dur&| dur&t| dkr| d} t| dkr| d} |d}| d|} | d|} t |d}||| }||| }||d }|sX||f|dd}|durT|f|S|St ||||j |j |j|j|j|j|jd S) a` decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. Example: Fine-tuning a model for extrative question answering, and our model also supports generative question answering using `BartForConditionalGeneration` ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForQuestionAnswering >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForQuestionAnswering.from_pretrained("RUCAIBox/mvp") >>> inputs = tokenizer( ... "Answer the following question: Who was Jim Henson? [SEP] Jim Henson was a nice puppet", ... return_tensors="pt", ... ) >>> target_start_index = torch.tensor([18]) >>> target_end_index = torch.tensor([19]) >>> loss = model(**inputs, start_positions=target_start_index, end_positions=target_end_index).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... outputs = model(**inputs) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1] >>> predict_answer = tokenizer.decode(predict_answer_tokens) ``` NFr&rrr!rc)Z ignore_indexr,) r start_logits end_logitsrVr r rr rr )r}rrr-rr* contiguousrrhrrrrVr r rr rr )r0rr]rrrrrrr.r/rrrr`rrrZsequence_outputrr0r1Z total_lossZ ignored_indexrZ start_lossZend_lossr r%r%r&r<spS           zMvpForQuestionAnswering.forward)NNNNNNNNNNNNNNNN)r>r?r@r r/rrrr7rCrrrryrrzrr<rDr%r%r1r&r+sN   r+cs(eZdZdZfddZddZZS)MvpDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. cst|t||_dSr)r.r/rrrr1r%r&r/us zMvpDecoderWrapper.__init__cOs|j|i|Sr)r)r0argsr%r%r%r&r<yszMvpDecoderWrapper.forward)r>r?r@rAr/r<rDr%r%r1r&r3os r3cseZdZdgZfddZddZddZdd Zd d Zd d Z e de e j e e je e je e je e je e je ee je e je e j e ee ee ee ee e jeeefdddZZS)MvpForCausalLMrcsDd|_d|_t|t||_tj|j|j dd|_ | dS)NTFrM) rJZis_encoder_decoderr.r/r3rrrPr,rrrrr1r%r&r/s   zMvpForCausalLM.__init__cCs |jjjSrrrrrr%r%r&rsz#MvpForCausalLM.get_input_embeddingscCs||jj_dSrr6rr%r%r&rsz#MvpForCausalLM.set_input_embeddingscCs ||j_dSrrr)r0rr%r%r& set_decoderszMvpForCausalLM.set_decodercCs|jjSrr7rr%r%r&rszMvpForCausalLM.get_decodercCs|j|jddSrrrr%r%r&rs z%MvpForCausalLM.set_lightweight_tuningN)rr]rrrrrVrrrr`rrrarbcCs| dur | n|jj} | dur | n|jj} | dur4| n|jj} |jj||||||||| | | | d }||d}d}| durt}||d|jj | d}| s|f|dd}|dur|f|S|St |||j |j |j |jdS)a cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, MvpForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForCausalLM.from_pretrained("RUCAIBox/mvp", add_cross_attention=False) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> list(logits.shape) [1, 8, 50267] ```N) rr]rrrrrVrrr`rrrr!r)rrrVr[rr)r}r`rrrrrrrorrrVr[rr)r0rr]rrrrrVrrrr`rrrarrrrr r%r%r&r<sD-zMvpForCausalLM.forward)NNNNNNNNNNNNNN)r>r?r@r r/rrr8rrrrr7rrCrrryrrzrr<rDr%r%r1r&r5}sN   r5)r5rr+r"rr)BrArtypingrrr7Ztorch.utils.checkpointrZtorch.nnrrrZ activationsr Z cache_utilsr r r Z generationr Zmodeling_attn_mask_utilsrrZmodeling_layersrZmodeling_outputsrrrrrrrZmodeling_utilsrutilsrrZutils.deprecationrZconfiguration_mvprZ get_loggerr>rrCrBr'rr(ModulerEr{rrrrrrrrr"r+r3r5__all__r%r%r%r&sf    $    !Cw5 1-*v