a hb@s|dZddlZddlZddlmZmZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZmZmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZmZdd l m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)ddl*m+Z+ddl,m-Z-e&rddl.m/Z/ddl0m1Z1e)2e3Z4Gdddej5Z6Gdddej5Z7Gdddej5Z8Gdddej5Z9Gdddej5Z:Gdddej5Z;Gd d!d!ej5Ze%Gd&d'd'e!Z?Gd(d)d)e?Z@e%Gd*d+d+e?ZAe%d,d-Gd.d/d/e?eZBe%Gd0d1d1e?ZCe%d2d-Gd3d4d4e?ZDe%Gd5d6d6e?ZEe%Gd7d8d8e?ZFgd9ZGdS):zPyTorch UMT5 model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)AttentionMaskConverter)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) DUMMY_INPUTS DUMMY_MASKauto_docstringis_torch_flex_attn_availableis_torch_fx_proxyis_torchdynamo_compilinglogging)deprecate_kwarg) UMT5Config) BlockMask)make_flex_block_causal_maskcs&eZdZdfdd ZddZZS) UMT5LayerNormư>cs&ttt||_||_dS)ze Construct a layernorm module in the UMT5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__b/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/umt5/modeling_umt5.pyr'?s zUMT5LayerNorm.__init__cCs\|tjdjddd}|t||j}|jjtj tj fvrR||jj}|j|S)NT)Zkeepdim) tor)Zfloat32powmeanZrsqrtr,r+dtypefloat16Zbfloat16)r- hidden_statesZvariancer2r2r3forwardGs zUMT5LayerNorm.forward)r%)__name__ __module__ __qualname__r'r< __classcell__r2r2r0r3r$>sr$cs*eZdZedfdd ZddZZS)UMT5DenseActDenseconfigcsTttj|j|jdd|_tj|j|jdd|_t|j |_ t |j |_ dSNFbias)r&r'rLineard_modeld_ffwiwoDropout dropout_ratedropoutr dense_act_fnactr-rCr0r2r3r'Ys  zUMT5DenseActDense.__init__cCsl||}||}||}t|jjtjr^|j|jjjkr^|jjjtj kr^| |jjj}||}|SN) rJrPrN isinstancerKr+r)Tensorr9int8r6r-r;r2r2r3r<`s    zUMT5DenseActDense.forwardr=r>r?r!r'r<r@r2r2r0r3rAXsrAcs*eZdZedfdd ZddZZS)UMT5DenseGatedActDenserBcsjttj|j|jdd|_tj|j|jdd|_tj|j|jdd|_t |j |_ t |j |_dSrD)r&r'rrGrHrIwi_0wi_1rKrLrMrNr rOrPrQr0r2r3r'ps  zUMT5DenseGatedActDense.__init__cCsz|||}||}||}||}t|jjtjrl|j |jjj krl|jjj tj krl| |jjj }||}|SrR) rPrYrZrNrSrKr+r)rTr9rUr6)r-r;Z hidden_geluZ hidden_linearr2r2r3r<xs   zUMT5DenseGatedActDense.forwardrWr2r2r0r3rXosrXcs*eZdZedfdd ZddZZS) UMT5LayerFFrBcsJt|jrt||_n t||_t|j|jd|_ t |j |_ dS)Nr/)r&r'Z is_gated_actrXDenseReluDenserAr$rHlayer_norm_epsilon layer_normrrLrMrNrQr0r2r3r's    zUMT5LayerFF.__init__cCs&||}||}|||}|SrR)r_r]rN)r-r;Zforwarded_statesr2r2r3r<s  zUMT5LayerFF.forwardrWr2r2r0r3r[s r[c seZdZdZdeedfdd Zejejddd Z d d Z dd d Z e dddddejeejee ejeejeejeejdddZZS) UMT5Attentionz7 T5's attention using relative_attention_bias. FN layer_idxcst|j|_||_|j|_|j|_|j|_|j|_|j |_ |j |_ |j |j|_ ||_|dur|jrtd|jjdtj|j|j dd|_tj|j|j dd|_tj|j|j dd|_tj|j |jdd|_|jrt|j|j |_t|_dS)NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.FrE)r&r' is_decoderhas_relative_attention_biasrelative_attention_num_bucketsrelative_attention_max_distancerHd_kvkey_value_proj_dim num_headsn_headsrMrNZ inner_dimrblogger warning_oncer1r=rrGqkvo Embeddingrelative_attention_biasset pruned_heads)r-rCrdrbr0r2r3r's, zUMT5Attention.__init__) projectionreturncCs6|dd|j|jf}||dddd}|S)Nr5rr4r r)sizerjrhviewpermute)r-ruZnew_projection_shapeZnew_projectionr2r2r3_shapeszUMT5Attention._shapec Csd}|j}|j}|jsB|d}||dktj|7}t|}nt|t| }|d}||k}t | |t ||}|||}||tj}t|t ||d}|t |||7}|S)a Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593 Translate relative position to a bucket number for relative attention. The relative position is defined as memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr4r )rerfrcr6r)longabsminZ zeros_likelogfloatmathZ full_likewhere) r-relative_positionZrelative_bucketsZ num_bucketsZ max_distanceZ max_exactZis_smallZ log_ratioZrelative_position_if_larger2r2r3_relative_position_buckets$   z'UMT5Attention._relative_position_bucketc Cs|dur|jjj}|dur:tj|tj|ddddf}n|dddf}tj|tj|ddddf}||}||}||} | gdd} | S)z%Compute binned relative position biasN)r9device)r4rr r) rrr+rr)aranger{rryZ unsqueeze) r-Z query_length key_lengthrcache_positionZcontext_positionZmemory_positionrZrelative_position_bucketvaluesr2r2r3 compute_biass    zUMT5Attention.compute_biaspast_key_valuepast_key_values4.58new_nameversion)r;encoder_hidden_statesrattention_masklayer_head_maskrcCs|jdd\}}|du} ||} | |d|j|jdd} |durtt|trt|j |j } | rl|j } qx|j } n|} | r|n|} | r|dur| r| j |j j}| j |j j}n|| }|| }||d|j|jdd}||d|j|jdd}|durB| s|nd}| |||j d|i\}}| rBd|j|j <t| |dd}|durl||n|}|jd}|jstjd|j||f|j|jd}n6|j|||j|d }|dddd| dddf}|dur|ddddddd|jdf}||}|jrLt|jd}d |t|j<|dd|f}n|}||7}tj j!|"dd #|}tj j$||j$|j%d }|dur||}t||}|dd&}|||d}|'|}||fS) Nr4r5r rTr)rr9)rrrdim)ptraining)(shapermrxrjrhZ transposerSr is_updatedgetrbZcross_attention_cacheself_attention_cacheZlayerskeysrrnroupdater)matmulget_seq_lengthrdZzerosrr9rrtr*listboolrZ functionalZsoftmaxrZtype_asrNr contiguousrp)r-r;rrrrr batch_size seq_lengthZis_cross_attentionZ query_statesrZcurr_past_key_valueZcurrent_statesZ key_statesZ value_statesZscoresZreal_seq_lengthrZ position_bias causal_maskmaskZposition_bias_maskedZ attn_weightsZ attn_outputr2r2r3r<sl         " &   zUMT5Attention.forward)FN)NN)NNNNN)r=r>r?__doc__rintr'r)rTrzrrrtupler<r@r2r2r0r3r`s&/   r`cs@eZdZd eedfdd Zeddddd d d ZZS) UMT5LayerSelfAttentionNracs>tt|d|d|_t|j|jd|_t |j |_ dS)NTrdrbr\) r&r'r` SelfAttentionr$rHr^r_rrLrMrNr-rCrbr0r2r3r'fs zUMT5LayerSelfAttention.__init__rrrrc CsF||}|j|||||d}|||d}|f|dd}|S)Nrrrrrr )r_rrN) r-r;rrrrnormed_hidden_statesattention_outputoutputsr2r2r3r<ls zUMT5LayerSelfAttention.forward)N)NNNN r=r>r?rrr'rr<r@r2r2r0r3res rcs@eZdZd eedfdd Zeddddd d d ZZS) UMT5LayerCrossAttentionNracs>tt|d|d|_t|j|jd|_t |j |_ dS)NFrr\) r&r'r`EncDecAttentionr$rHr^r_rrLrMrNrr0r2r3r's z UMT5LayerCrossAttention.__init__rrrrc CsH||}|j||||||d}|||d} | f|dd} | S)Nrrrrrrr )r_rrN) r-r;rrrrrrrZ layer_outputrr2r2r3r<s zUMT5LayerCrossAttention.forward)N)NNNNNrr2r2r0r3rs rc s@eZdZd eedfdd Zeddddd d d ZZS) UMT5BlockNracs^t|j|_t|_|jt||d|jrJ|jt||d|jt |dS)Nra) r&r'rcr ModuleListlayerappendrrr[rr0r2r3r's  zUMT5Block.__init__rrrrFc CsT|jd||||| d\}} |jtjkrdt|jj} tt|| d| } tj || | d}d}|j ot|du}|r|jd|||||| d\}}|jtjkrt|jj} tt|| d| } tj || | d}|jd|}|jtjkr8t|jj} tt|| d| } tj || | d}|f}| rP|| |f7}|S)Nrri)r}maxr rr5) rr9r)r:finforrisinfanyclamprc)r-r;rrencoder_attention_maskrcross_attn_layer_head_maskr use_cacheoutput_attentionsrZself_attn_weightsZ max_dtypeZ clamp_valueZcross_attn_weightsZdo_cross_attentionrr2r2r3r<sF   zUMT5Block.forward)N) NNNNNNFFNrr2r2r0r3rs rcs:eZdZdZedfdd ZejejdddZZ S)UMT5ClassificationHeadz-Head for sentence-level classification tasks.rBcsBtt|j|j|_tj|jd|_t|j|j |_ dS)N)r) r&r'rrGrHdenserLclassifier_dropoutrN num_labelsout_projrQr0r2r3r's zUMT5ClassificationHead.__init__)r;rvcCs6||}||}t|}||}||}|SrR)rNrr)tanhrrVr2r2r3r<s      zUMT5ClassificationHead.forward) r=r>r?rr!r'r)rTr<r@r2r2r0r3rsrc@sJeZdZUeed<dZdZdZdgZdgZ e ddZ dd Z d d Z d S) UMT5PreTrainedModelrC transformerTrrKcCs$tt}tt}|||d}|S)N)decoder_input_ids input_idsdecoder_attention_mask)r)Ztensorrr)r-rZ input_mask dummy_inputsr2r2r3rs  z UMT5PreTrainedModel.dummy_inputscCs|jj}t|tr(|jj|dnt|ttt t fr|j jjj d|ddt |dr||jjs||jjjj d|ddt |dr|jjjj d||jjdd|jjjn2t|trt |dr|jjjj d|dd|jjjnt|tr|jjjj d||jjddt |jdrL|jjd urL|jjj|jjjj d||jjddt |jdr|jjd ur|jjjnPt|tr>|jjjj d||jjddt |jdr|jjd ur|jjj|jjjj d||jjddt |jdr|jjd ur|jjjnt|tr*|jjjj d||jjddt |jdr|jjd ur|jjj|jjjj d||jjddt |jdr|jjd ur|jjj|jjjj d||jjddt |jdr|jjd ur|jjjnt|t r|jj}|jj!}|jj"}|j#jjj d|||dd|j$jjj d||dd|j%jjj d||dd|j&jjj d|||dd|j'r|j(jjj d||ddd S) zInitialize the weights?)r8Zstdlm_head qa_outputs classifierrFN))rCZinitializer_factorrSr$r+dataZfill_ UMT5ModelUMT5ForConditionalGenerationUMT5EncoderModelUMT5ForQuestionAnsweringsharedZnormal_hasattrtie_word_embeddingsrrrHrFZzero_UMT5ForTokenClassificationrrrrrArJrKrIrXrYrZr`rgrirmrnrorprdrr)r-modulefactorrHrhrjr2r2r3 _init_weightssn                   z!UMT5PreTrainedModel._init_weightscCs|jj}|jj}|dur tdt|rbt|jddd|}tj||dddfgdd}n4| |j}|dddf |dddf<||d<|durtd| |d k||S) Nzself.model.config.decoder_start_token_id has to be defined. In UMT5 it is usually set to the pad_token_id. See UMT5 docs for more information.r5)r .rr ).rz1self.model.config.pad_token_id has to be defined.) rCdecoder_start_token_id pad_token_id ValueErrorrr)fullrcatZ new_zeroscloneZ masked_fill_)r-rrrZshifted_input_idsr2r2r3 _shift_rightUs    z UMT5PreTrainedModel._shift_rightN)r=r>r?r!__annotations__Zbase_model_prefixZsupports_gradient_checkpointingZ_can_compile_fullgraphZ_no_split_modulesZ_keep_in_fp32_modulespropertyrrrr2r2r2r3rs  Brc sxeZdZdfdd ZddZdddZdeejd fejeje e d d d Z e eje e ejeje d ddZZS) UMT5StackNcslt||_j|_tfddtjD|_t j j d|_ t j|_d|_|dS)Ncsg|]}t|dqS)ra)r).0irBr2r3 vz&UMT5Stack.__init__..r\F)r&r' embed_tokensrcrrrange num_layersblockr$rHr^final_layer_normrLrMrNgradient_checkpointing post_init)r-rCrr0rBr3r'rs  zUMT5Stack.__init__cCs ||_dSrR)rr-Znew_embeddingsr2r2r3set_input_embeddings~szUMT5Stack.set_input_embeddingsc# Cs| dur | n|jj} | dur | n|jj} | dur4| n|jj} | durH| n|jj} |dur|dur|jrjdnd}td|d|dn`|dur|}|d|d}n>|dur|dd}n$|jrdnd}td|d|d |j r |j r | r t d d } |dur2|j dur(td | |}|\}}| d ur\|js\td|d|jr| r|dur|jjrtt|jdt|jd}n t|jd}n |jsd}|dur|nd}| durtj||||jd} |durts||}tj|||jd}|jrD|||| t|tr:|jn|| }nL|dur|ddddddf}|j|jd}d|t|jj}nd}|jr|dur|\}}}||f}|durtj||jd}||}nd}|||jj }|||jj }| r dnd}| rdnd}| r0|jr0dnd}|!|}t"|j#D]x\}}||} ||}!| rp||f}|||||| |!|| | | d }"|"d}| rH||"df7}|jrH||"df7}qH|$|}|!|}| r||f}| st%dd|||||fDSt&|||||dS)NZdecoder_zYou cannot specify both zinput_ids and zinputs_embeds at the same timer5zYou have to specify either z input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fzsz$UMT5Stack.forward..)last_hidden_staterr; attentionscross_attentions)'rCrroutput_hidden_statesuse_return_dictrcrrwrxrrrkrlris_encoder_decoderr r rr)rrrr*_update_causal_maskrSrr6r9rr}Zinvert_attention_maskZ get_head_maskrrN enumeraterrrr)#r-rrrrr head_maskcross_attn_head_maskrrrr return_dictrZerr_msg_prefixZ input_shaperrpast_key_values_lengthZmask_seq_lengthrZencoder_batch_sizeZencoder_sequence_length_Zencoder_hidden_shapeZencoder_extended_attention_maskZall_hidden_statesZall_attentionsZall_cross_attentionsr;rZ layer_modulerrZ layer_outputsr2r2r3r<s                  zUMT5Stack.forwardFr")r input_tensorrrrc CsB|jjdkr(|dur$|dkr$|SdS|jjdkrLt|tjrHt|}|S|dur\|nd}|durn|jnd}|jjdkr|s|st j ||||j drdS|j }|j d} |r|} n"t|tjr|j d n || d} |j|| | |||j dd } |jjdkr>|dur>|jjd vr>|s>t|j} t | | } | S) NZflash_attention_2rZflex_attentionrFZsdpa)rrZ is_trainingr r5)sequence_length target_lengthr9rr)cudaZxpuZnpu)rCZ_attn_implementationrrSr)rTr#rZis_compileablerZ_ignore_causal_mask_sdparr9rZget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrtyperr}Z_unmask_unattended) r-rr rrrZpast_seen_tokensZusing_compilable_cacher9r r r min_dtyper2r2r3r-sZ           zUMT5Stack._update_causal_mask)rr r r9rrc KsF|dur|dkr|}n&t|j}tj||f|||jd}|dkrVtj|dd}|tj||jd|ddk9}|ddddddf |ddd}|durB| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N)Z fill_valuer9rr )Zdiagonalrr5r)rr)rr}rrZtriurZreshapeexpandrrr6Z masked_fill) rr r r9rrkwargsrrZ mask_lengthZ padding_maskr2r2r3rqs*  $  6  z?UMT5Stack._prepare_4d_causal_attention_mask_with_cache_position)N) NNNNNNNNNNNNN)F)r=r>r?r'rr<rr)rTr rr staticmethodrr9rr@r2r2r0r3rqsB  3  DrcseZdZUdZdZeed<ddgZfddZdd Z d d Z d d Z ddZ ddZ edeejeejeejeejeejeejeejeeeejeeeejeejeeeeeeeeeejeeejefdddZZS)rao Examples: ```python >>> from transformers import UMT5Model, AutoTokenizer >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> noisy_text = "UN Offizier sagt, dass weiter werden muss in Syrien." >>> label = " verhandelt" >>> inputs = tokenizer(inputs, return_tensors="pt") >>> labels = tokenizer(label=label, return_tensors="pt") >>> outputs = model(input_ids=inputs["input_ids"], decoder_input_ids=labels["input_ids"]) >>> hidden_states = outputs.last_hidden_state ```umt5rCencoder.embed_tokens.weightdecoder.embed_tokens.weightcst|t|j|j|_t|}d|_ d|_ d|_ t ||j|_ t|}d|_ d|_ |j|_t ||j|_|dSNFT)r&r'rrq vocab_sizerHrcopydeepcopyrcrtie_encoder_decoderrencodernum_decoder_layersrdecoderrr-rCencoder_configZdecoder_configr0r2r3r's   zUMT5Model.__init__cCs|jSrRrr-r2r2r3get_input_embeddingsszUMT5Model.get_input_embeddingscCs"||_|j||j|dSrRrrrrrr2r2r3rs zUMT5Model.set_input_embeddingscCs0|jjr,||jj|j||jj|jdSrRrCr_tie_or_clone_weightsrrrrr#r2r2r3 _tie_weightsszUMT5Model._tie_weightscCs|jSrRrr#r2r2r3 get_encoderszUMT5Model.get_encodercCs*|D]\}}|jj|j|qdS) Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrZ attention prune_headsr-Zheads_to_prunerZheadsr2r2r3 _prune_headsszUMT5Model._prune_headsN)rrrrrdecoder_head_maskrencoder_outputsrrdecoder_inputs_embedsrrrrrrvcCs| dur | n|jj} |dur |n|jj}|durJ|j||| || ||d}nH|rt|tst|dt|dkrt|dndt|dkr|dndd}|d}|j||| | ||||| | |||d }|s||St|j |j |j |j |j |j |j |j dS) a+ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. Example: ```python >>> from transformers import AutoTokenizer, UMT5Model >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for UMT5Model. >>> # This is not needed for torch's UMT5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nrrrrrrrrr r4rr;r rrrrrrrrrrrrr)rrdecoder_hidden_statesdecoder_attentionsrencoder_last_hidden_staterencoder_attentions)rCrrrrSrlenrrrrr;rr)r-rrrrrr0rr1rrr2rrrrrr;decoder_outputsr2r2r3r<s\Q zUMT5Model.forward)NNNNNNNNNNNNNNNN)r=r>r?r model_typer!r_tied_weights_keysr'r$rr(r*r/rrr) LongTensor FloatTensor BoolTensorrTrr rrrr<r@r2r2r0r3rs\  rz< UMT5 Model with a `language modeling` head on top. )Z custom_introcseZdZdZdZgdZfddZddZdd Zd d Z d d Z e de e je e je e je e je e je e je e je eee je ee e je e je e je ee ee ee ee e jeee jefdddZe jdddZZS)ra Examples: ```python >>> from transformers import UMT5ForConditionalGeneration, AutoTokenizer >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> summary = "Weiter Verhandlung in Syrien." >>> inputs = tokenizer(article, text_target=summary, return_tensors="pt") >>> outputs = model(**inputs) >>> loss = outputs.loss ```r)rrzlm_head.weightcst||j|_t|j|j|_t |}d|_ d|_ d|_ t ||j|_t |}d|_ d|_ |j|_t ||j|_tj|j|jdd|_|dS)NFTrE)r&r'rH model_dimrrqrrrrrcrrrrrrrrGrrr r0r2r3r's   z%UMT5ForConditionalGeneration.__init__cCs|jSrRr"r#r2r2r3r$sz1UMT5ForConditionalGeneration.get_input_embeddingscCs"||_|j||j|dSrRr%rr2r2r3rs z1UMT5ForConditionalGeneration.set_input_embeddingscCs0|jjr,||jj|j||jj|jdSrRr&r#r2r2r3r(sz)UMT5ForConditionalGeneration._tie_weightscCs|jSrRr)r#r2r2r3r*sz(UMT5ForConditionalGeneration.get_encoderN)rrrrrr0rr1rrr2labelsrrrrrrvcCs| dur | n|jj} |dur |n|jj}|durJ|j||| ||||d}nH|rt|tst|dt|dkrt|dndt|dkr|dndd}|d}| dur|dur| dur|| }|j||| | ||||| ||||d }|d}|jj r||j d}| |}d}| durJt d d }| |j} ||d |d | d }|s~|f|dd|}|durz|f|S|St|||j|j|j|j|j|j|jd S) aK input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, UMT5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer("Studies have shown that good for you", return_tensors="pt").input_ids >>> outputs = model.generate(input_ids) >>> tokenizer.decode(outputs[0], skip_special_tokens=True) ```Nr3rr r4r4r5rrZ ignore_indexr5 losslogitsrr6r7rr8rr9)rCrrrrSrr:rrrrArrr6rrxrwrrr;rrr)r-rrrrrr0rr1rrr2rBrrrrrr;r;sequence_outputZ lm_logitsrEloss_fctoutputr2r2r3r<svU      z$UMT5ForConditionalGeneration.forward)rBcCs ||SrR)r)r-rBr2r2r3%prepare_decoder_input_ids_from_labels`szBUMT5ForConditionalGeneration.prepare_decoder_input_ids_from_labels)NNNNNNNNNNNNNNNNN)r=r>r?rr<r=r'r$rr(r*rrr)r>r?r@rTrr rrrr<rJr@r2r2r0r3rys` #rc seZdZdZdZdgZfddZddZdd Zd d Z d d Z ddZ e de eje eje eje eje ee ee eeeejefdddZZS)ra Examples: ```python >>> from transformers import UMT5EncoderModel, AutoTokenizer >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> input_ids = tokenizer(article, return_tensors="pt").input_ids >>> outputs = model(input_ids) >>> hidden_state = outputs.last_hidden_state ```rrcsNt|t|j|j|_t|}d|_ d|_ t ||j|_ | dSNF)r&r'rrqrrHrrrrrrrr)r-rCr!r0r2r3r'xs  zUMT5EncoderModel.__init__cCs|jSrRr"r#r2r2r3r$sz%UMT5EncoderModel.get_input_embeddingscCs||_|j|dSrR)rrrrr2r2r3rsz%UMT5EncoderModel.set_input_embeddingscCs|jjr||jj|jdSrR)rCrr'rrrr#r2r2r3r(szUMT5EncoderModel._tie_weightscCs|jSrRr)r#r2r2r3r*szUMT5EncoderModel.get_encodercCs0|D]"\}}|jj|jdj|qdS)r+rN)r,rrrrr-r.r2r2r3r/szUMT5EncoderModel._prune_headsN)rrrrrrrrvc Cs0|dur |n|jj}|j|||||||d}|S)aQ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). Example: ```python >>> from transformers import AutoTokenizer, UMT5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nr3)rCrr) r-rrrrrrrr1r2r2r3r<s# zUMT5EncoderModel.forward)NNNNNNN)r=r>r?rr<r=r'r$rr(r*r/rrr)r>r?rrrrr<r@r2r2r0r3rds6 rz UMT5 model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cseZdZdgZddgZedfdd Zed ee j ee j ee j ee j ee j ee j ee j ee e j ee j ee j ee j eeeeeeeeeeefdd d ZZS) UMT5ForSequenceClassificationFdecoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weightrrrBcs2t|t||_t||_|d|_dSrK)r&r'rrrclassification_headrZmodel_parallelrQr0r2r3r's    z&UMT5ForSequenceClassification.__init__N)rrrrrr0rr1rr2rBrrrrrvcCsx|dur |n|jj}| dur d} |durB| durBtd|jj|durl| durl|durbtd||}|j||||||||| | | | ||d}|d}||jj  |j }t t |ddkrtd|j\}}}||ddf|d |ddd ddf}||}d}| dur | |j } |jjdur|jjdkrRd |j_n<|jjdkr| jt jks|| jt jkrd |j_nd |j_|jjd krt}|jjdkr||| }n ||| }nP|jjd krt}||d |jj| d }n|jjd kr t}||| }|sP|f|dd}|durL|f|S|St|||j|j|j|j |j!|j"|j#d S)as input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.) rrrrr0rr1rr2rrrrrr z7All examples must have the same number of tokens.r5Z regressionZsingle_label_classificationZmulti_label_classificationrD)$rCrNotImplementedErrorr1r=rrreqZ eos_token_idr6rr:r)Zunique_consecutivesumrrxrNZ problem_typerr9r{rrsqueezerrrrr6r7rr8rr9)r-rrrrrr0rr1rr2rBrrrrrrGZeos_maskrr r.Zsentence_representationrFrErHrIr2r2r3r<s>   ,    *   z%UMT5ForSequenceClassification.forward)NNNNNNNNNNNNNNN)r=r>r?"_keys_to_ignore_on_load_unexpectedr=r!r'rrr)r>rTrr?rrrrr<r@r2r2r0r3rLsJ   rLc seZdZdgZdgZedfdd Zed ee j ee j ee j ee j ee j ee ee ee e e e j efd dd ZZS) rrMz'transformer.encoder.embed_tokens.weightrBcsJt||j|_t||_t|j|_t |j |j|_ | dSrR) r&r'rrrrrLrrNrGr.rrrQr0r2r3r's   z#UMT5ForTokenClassification.__init__N) rrrrrBrrrrvc  Cs|dur |n|jj}|j|||||||d} | d} || } || } d} |durtt} | | d|j|d} |s| | ddf}| dur| f|S|St| | | j | j dS)aB input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. N)rrrrrrrr5r4)rErFr;r) rCrrrNrrrxrrr;r)r-rrrrrBrrrrr;rFrErHrIr2r2r3r<s4   z"UMT5ForTokenClassification.forward)NNNNNNNN)r=r>r?rTr=r!r'rrr)rTrrrrr<r@r2r2r0r3rzs. rcseZdZddgZfddZddZddZd d Zd d Ze de e j e e j e e j e e je e j e e j e e je eee je e j e e j e e j e e j e ee ee ee eeee j efdddZZS)rrrcst||j|_t|j|j|_t |}d|_ d|_ d|_ t ||j|_t |}d|_ d|_ |j|_t ||j|_|j|_t|j|j|_|dSr)r&r'rHrArrqrrrrrcrrrrrrrrrGrrr r0r2r3r's    z!UMT5ForQuestionAnswering.__init__cCs|jSrRr"r#r2r2r3r$sz-UMT5ForQuestionAnswering.get_input_embeddingscCs"||_|j||j|dSrRr%rr2r2r3rs z-UMT5ForQuestionAnswering.set_input_embeddingscCs0|jjr,||jj|j||jj|jdSrRr&r#r2r2r3r(sz%UMT5ForQuestionAnswering._tie_weightscCs|jSrRr)r#r2r2r3r*sz$UMT5ForQuestionAnswering.get_encoderN)rrrrrr0rr1start_positions end_positionsrr2rrrrrvcCsn|dur |n|jj}| dur | n|jj} | dur<| durr?r=r'r$rr(r*rrr)r>r?r@rTrrrrr<r@r2r2r0r3rsT r)rrrrLrrr)Hrrrtypingrrr)rZtorch.nnrrrZ activationsr Z cache_utilsr r r Z generationr Zmodeling_attn_mask_utilsrZmodeling_layersrZmodeling_outputsrrrrrrrZmodeling_utilsrutilsrrrrrrrZutils.deprecationrZconfiguration_umt5r!Z!torch.nn.attention.flex_attentionr"Zintegrations.flex_attentionr#Z get_loggerr=rkModuler$rArXr[r`rrrrrrrrrrLrr__all__r2r2r2r3sp     $ $     GKr;Ogl$LN