a h3@sdZddlZddlmZmZddlZddlmZddlmZddl m Z ddl m Z m Z mZdd lmZdd lmZmZdd lmZdd lmZmZdd lmZddlmZmZddlmZddl m!Z!e"e#Z$Gdddej%Z&Gdddej%Z'Gdddej(Z)Gdddej(Z*GdddeZ+eGdddeZ,Gddde,Z-ed d!Gd"d#d#e,Z.ed$d!Gd%d&d&e,eZ/d&dgZ0dS)'z/PyTorch TrOCR decoder model (based on RoBERTa).N)OptionalUnion)nn)CrossEntropyLoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)_prepare_4d_attention_mask!_prepare_4d_causal_attention_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModel)auto_docstringlogging)deprecate_kwarg) TrOCRConfigcsDeZdZdZeedfdd Zd ejeejdfdd ZZ S) TrOCRLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. )num_embeddings embedding_dimcsd|_t||j|dS)N)offsetsuper__init__)selfrr __class__d/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.pyr0sz(TrOCRLearnedPositionalEmbedding.__init__rN) input_idspast_key_values_length position_idscs\|dur@|jdd\}}tj|||tj|jjd|d}n |d}t ||j S)z3`input_ids' shape is expected to be [bsz x seqlen].Nr)dtypedevicer) shapetorcharangelongweightr'expand unsqueezerforwardr)rr#r$r%bszseq_lenrr!r"r06s z'TrOCRLearnedPositionalEmbedding.forward)rN) __name__ __module__ __qualname____doc__intrr*Tensorr0 __classcell__r!r!rr"r+srcsFeZdZdZd eeeeedfdd Zej dfdd Z Z S) TrOCRScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. ?)rr padding_idx embed_scalecst|||||_dSN)rrr=)rrrr<r=rr!r"rJsz!TrOCRScaledWordEmbedding.__init__)r#cst||jSr>)rr0r=)rr#rr!r"r0Nsz TrOCRScaledWordEmbedding.forward)r;) r3r4r5r6r7rfloatrr*r8r0r9r!r!rr"r:Esr:cseZdZdZdeeeedfdd ZedeeeedddZe de j ed d d Z de j eeed ddZ ZS)"TrOCRSinusoidalPositionalEmbeddingzDThis module produces sinusoidal positional embeddings of any length.N) num_positionsrr<csBtd|_||_||_|||||_|dt ddS)Nr _float_tensorr) rrrrr< get_embeddingweightsZregister_bufferr* FloatTensor)rrArr<rr!r"rUs  z+TrOCRSinusoidalPositionalEmbedding.__init__)rrr<cCs|d}td|d}ttj|tjd| }tj|tjdd|d}tjt |t |gdd |d}|ddkrtj|t |dgdd}|durd||ddf<| tS) z Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.5 of "Attention Is All You Need". ri'r)r&rdimr(N)mathlogr*expr+Zint64r?r/catsincosviewZzerostoZget_default_dtype)rrr<Zhalf_dimZembr!r!r"rC]s $& z0TrOCRSinusoidalPositionalEmbedding.get_embeddingr)r#r$cCs|\}}|||j||j}|jd|}|jdusL||jdkr`|||j|j|_|j|j|_|j d| d ||d }|S)Nrrr() size"create_position_ids_from_input_idsr<rOr'rDrCrrBZ index_selectrNdetach)rr#r$r1r2r%Zmax_posxr!r!r"r0ps "z*TrOCRSinusoidalPositionalEmbedding.forward)r#r<r$cCs6||}tj|dd|||}||S)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. rrF)ner7r*ZcumsumZtype_asr,)rr#r<r$maskZincremental_indicesr!r!r"rQszETrOCRSinusoidalPositionalEmbedding.create_position_ids_from_input_ids)N)N)r)r)r3r4r5r6r7rr staticmethodrCr*Zno_gradr8r0rQr9r!r!rr"r@Rs r@cseZdZdZdeeeeeeeeeeeeeeeed fdd Ze d d d d de j ee j ee ee j ee j eeee j e e j ee j ee e j fd ddZZS)TrOCRAttentionz>Multi-headed attention from 'Attention Is All You Need' paper.NFT) embed_dim num_headskdimvdimdropout is_decoderbiasis_cross_attention layer_idxc st||_|dur|n||_|dur.|n||_||_||_|||_|j||jksrtd|jd|d|jd|_ ||_ | |_ t j |j||d|_t j |j||d|_t j |||d|_t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩r_)rrrYr[r\rZr]head_dim ValueErrorscalingr^rarLineark_projv_projq_projout_proj) rconfigrYrZr[r\r]r^r_r`rarr!r"rs(    zTrOCRAttention.__init__past_key_valuepast_key_values4.58new_nameversion) hidden_stateskey_value_statesrmattention_masklayer_head_maskoutput_attentionscache_positionreturncCs|du}|\} } } |||j} |dur^t|trZ|j|j} |rR|j}q^|j }n|}|rf|n|}|r|dur| r|j |jj }|j |jj }n| |}||}|| d|j|jdd}|| d|j|jdd}|dur&|s|nd}||||jd|i\}}|r&d|j|j<| |jd|jf}| | | |j|jdd} | j|} |j|}|j|}|d}t| |dd}|| |j| |fkrtd| |j| |fd||dur8|| d| |fkrtd | d| |fd||| |j| ||}|| |j| |}tjj|dd }|dur||jfkrtd |jfd||dddd|| |j| |}|| |j| |}|r|| |j| |}|| |j| |}nd}tjj||j|jd }t||}|| |j| |jfkrJtd | |j| |jfd||| |j| |j}|dd}|| | | }||}||fS)z#Input shape: Batch x Time x ChannelNr(rrrwTz$Attention weights should be of size z , but is z!Attention mask should be of size rFz/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )rPrire isinstancer is_updatedgetraZcross_attention_cacheZself_attention_cachelayerskeysvaluesrgrhrNrZrcZ transposeupdateZreshaper*Zbmmrdr functionalZsoftmaxr]r{rj)rrrrsrmrtrurvrwr`r1tgt_lenrYZ query_statesr}Zcurr_past_key_valueZcurrent_statesZ key_statesZ value_statesZ proj_shapeZsrc_lenZ attn_weightsZattn_weights_reshapedZ attn_probsZ attn_outputr!r!r"r0s              "   zTrOCRAttention.forward)NNrXFTFN)NNNNFN)r3r4r5r6r7rr?boolrrr*r8rtupler0r9r!r!rr"rWsH# rWc seZdZdedfdd Zedddddejeejeejeejeejeejee ee ee eejd d d Z Z S)TrOCRDecoderLayerNrkc st|j|_t||j|j|jd|d|_|j|_t |j |_ |j |_ t |j|_|jrt||j|j|j|j|jdd|d |_t |j|_t |j|j|_t |j|j|_t |j|_dS)NT)rYrZr]r^ra)rYrZr[r\r]r^r`ra)rr hidden_sizerYrWZdecoder_attention_headsZattention_dropout self_attnr]rZactivation_function activation_fnactivation_dropoutr LayerNormself_attn_layer_normr^Zcross_attention_hidden_size encoder_attnencoder_attn_layer_normrfZdecoder_ffn_dimfc1fc2final_layer_norm)rrkrarr!r"r+s<   zTrOCRDecoderLayer.__init__rlrmrnroFT) rrrtencoder_hidden_statesencoder_attention_maskrucross_attn_layer_head_maskrmrv use_cacherwc  Cs|} |j|||||| d\}} tjj||j|jd}| |}||}d} |dur|} |j||||||| d\}} tjj||j|jd}| |}||}|} || |}tjj||j |jd}| |}tjj||j|jd}| |}| |}|f}|r|| | f7}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size *(decoder_attention_heads,)*. past_key_values (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrrmrtrurvrwryN)rrrsrtrurmrvrw) rrrr]r{rrrrrrrr)rrrrtrrrurrmrvrrwZresidualZself_attn_weightsZcross_attn_weightsoutputsr!r!r"r0OsL        zTrOCRDecoderLayer.forward)N) NNNNNNFTN) r3r4r5rrrr*r8rrrr0r9r!r!rr"r*s.$ rc@s,eZdZUeed<dZdZdgZddZdS)TrOCRPreTrainedModelrkmodelTrcCs|jj}t|tjtjfrD|jjjd|d|j dur~|j j n:t|tj r~|jjjd|d|j dur~|jj|j  dS)NrX)meanstd) rkZinit_stdr|rrfZConv1dr-dataZnormal_r_Zzero_ Embeddingr<)rmodulerr!r!r" _init_weightss   z"TrOCRPreTrainedModel._init_weightsN) r3r4r5r__annotations__Zbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesrr!r!r!r"rs rc s0eZdZdZedfdd ZdddZZS) TrOCRDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`TrOCRDecoderLayer`] Args: config: TrOCRConfig rcstj|_j|_j|_jr6t j nd}t j j |j|d|_ jrjtjj |_ntj|jdj |j|_jrtj |_nd|_tfddtjD|_d|_|dS)Nr;)r=rcsg|]}t|dqS))ra)r).0irr!r" z)TrOCRDecoder.__init__..F)rrr]Zdecoder_layerdrop layerdropZ pad_token_idr<Zscale_embeddingrHsqrtrr: vocab_size embed_tokensuse_learned_position_embeddingsrZmax_position_embeddingsembed_positionsr@layernorm_embeddingrrZ ModuleListrangeZdecoder_layersrgradient_checkpointing post_init)rrkr=rrr"rs*  zTrOCRDecoder.__init__Nc Cs| dur | n|jj} | dur | n|jj} | dur4| n|jj} | durH| n|jj} |durj|durjtdnX|dur|}|d|jd}n8|dur|dd}|dddddf}ntd|j r|j r| rt dd} | r |dur |durt t|jdt|jdn t|jd}| rFt|trFt dt |}|durX|nd }|durp||}|jjr|j||d }n|j||d }||}|jdur||}tjj||j|j d }|j}t||||}|dur|durt||j|dd }| rd nd}| r d nd}| r8|dur8d nd}t||gddgD]V\}}|durN|d t|jkrNtd|dt|jd|d dqNt |jD]\}}| r||f7}|j rt!"g}||j#krq||||||dur||nd|dur||nd|| | | d }|d }| r||df7}|dur||df7}q| rr||f7}| stdd|||||fDSt$|||||dS)a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in encoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer(zEYou have to specify either decoder_input_ids or decoder_inputs_embedsz^`use_cache = True` is incompatible with gradient checkpointing. Setting `use_cache = False`...FrzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.r)r$ry)rr! head_maskcross_attn_head_maskzThe `z` should be specified for z layers, but it is for .)rrurrmrvrrwrrcss|]}|dur|VqdSr>r!)rvr!r!r" sz'TrOCRDecoder.forward..)Zlast_hidden_statermrr attentionscross_attentions)%rkrvoutput_hidden_statesruse_return_dictrdrNr)rPrr{loggerZ warning_oncer r r|rZfrom_legacy_cacheZget_seq_lengthrrrrrrr]r r r&ziplenr enumerater*Zrandrr)rr#rtrrrrrm inputs_embedsrrvr return_dictrwinputZ input_shaper$Z embed_posrrZall_hidden_statesZall_self_attnsZall_cross_attentionsZ attn_maskZ mask_nameidxZ decoder_layerZdropout_probabilityZ layer_outputsr!r!r"r0sQ                   zTrOCRDecoder.forward) NNNNNNNNNNNNN)r3r4r5r6rrr0r9r!r!rr"rs !ra The TrOCR Model with a language modeling head. Can be used for summarization. This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. )Z custom_introcs$eZdZfddZddZZS)TrOCRDecoderWrappercst|t||_dSr>)rrrdecoderrrkrr!r"rs zTrOCRDecoderWrapper.__init__cOs|j|i|Sr>)r)rargskwargsr!r!r"r0szTrOCRDecoderWrapper.forward)r3r4r5rr0r9r!r!rr"rs rzy The TrOCR Decoder with a language modeling head. Can be used as the decoder part of [`EncoderDecoderModel`] and cseZdZdgZfddZddZddZdd Zd d Zd d Z ddZ e de e je e je e je e je e je e je eee je e je e je ee ee ee ee e jeeefdddZZS)TrOCRForCausalLMzoutput_projection.weightcsDd|_d|_t|t||_tj|j|j dd|_ | dS)NTFrb) r^Zis_encoder_decoderrrrrrrfrroutput_projectionrrrr!r"rs   zTrOCRForCausalLM.__init__cCs |jjjSr>rrrrr!r!r"get_input_embeddingssz%TrOCRForCausalLM.get_input_embeddingscCs||jj_dSr>r)rvaluer!r!r"set_input_embeddingssz%TrOCRForCausalLM.set_input_embeddingscCs|jSr>rrr!r!r"get_output_embeddingssz&TrOCRForCausalLM.get_output_embeddingscCs ||_dSr>r)rZnew_embeddingsr!r!r"set_output_embeddingssz&TrOCRForCausalLM.set_output_embeddingscCs ||j_dSr>rr)rrr!r!r" set_decoderszTrOCRForCausalLM.set_decodercCs|jjSr>rrr!r!r" get_decoderszTrOCRForCausalLM.get_decoderN)r#rtrrrrrmrlabelsrrvrrrwrxcCs| dur | n|jj} | dur | n|jj} | dur4| n|jj} |jj||||||||| | | | |d }||d}d}| durt}||d|jj | d}| s|f|dd}|dur|f|S|St |||j |j |j |jdS)a cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import ( ... TrOCRConfig, ... TrOCRProcessor, ... TrOCRForCausalLM, ... ViTConfig, ... ViTModel, ... VisionEncoderDecoderModel, ... ) >>> import requests >>> from PIL import Image >>> # TrOCR is a decoder model and should be used within a VisionEncoderDecoderModel >>> # init vision2text model with random weights >>> encoder = ViTModel(ViTConfig()) >>> decoder = TrOCRForCausalLM(TrOCRConfig()) >>> model = VisionEncoderDecoderModel(encoder=encoder, decoder=decoder) >>> # If you want to start from the pretrained model, load the checkpoint with `VisionEncoderDecoderModel` >>> processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten") >>> model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten") >>> # load image from the IAM dataset >>> url = "https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg" >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB") >>> pixel_values = processor(image, return_tensors="pt").pixel_values >>> text = "industry, ' Mr. Brown commented icily. ' Let us have a" >>> # training >>> model.config.decoder_start_token_id = processor.tokenizer.eos_token_id >>> model.config.pad_token_id = processor.tokenizer.pad_token_id >>> model.config.vocab_size = model.config.decoder.vocab_size >>> labels = processor.tokenizer(text, return_tensors="pt").input_ids >>> outputs = model(pixel_values, labels=labels) >>> loss = outputs.loss >>> round(loss.item(), 2) 5.30 >>> # inference >>> generated_ids = model.generate(pixel_values) >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> generated_text 'industry, " Mr. Brown commented icily. " Let us have a' ```N) r#rtrrrrrmrrrvrrrwrr(r)losslogitsrmrrrr)rkrvrrrrrrrNrrrmrrrr)rr#rtrrrrrmrrrrvrrrwrrrZloss_fctoutputr!r!r"r0sFMzTrOCRForCausalLM.forward)NNNNNNNNNNNNNN)r3r4r5Z_tied_weights_keysrrrrrrrrrr*Z LongTensorr8rErrrrr0r9r!r!rr"rsP  r)1r6rHtypingrrr*rZtorch.nnrZ activationsrZ cache_utilsrr r Z generationr Zmodeling_attn_mask_utilsr r Zmodeling_layersrZmodeling_outputsrrZmodeling_utilsrutilsrrZutils.deprecationrZconfiguration_trocrrZ get_loggerr3rrrr:Moduler@rWrrrrr__all__r!r!r!r"sH          >z|