a h9 @sdZddlZddlmZddlmZmZmZddlZddl Zddlm Z ddl m Z ddl mZmZdd lmZdd lmZdd lmZdd lmZmZmZmZdd lmZmZddlmZm Z m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z,e&-e.Z/ee$ddGddde#Z0Gddde j1Z2Gddde j1Z3Gddde j1Z4de3iZ5Gddde j1Z6Gd d!d!e j1Z7Gd"d#d#e j1Z8Gd$d%d%eZ9Gd&d'd'e j1Z:e$Gd(d)d)eZ;Gd*d+d+e j1Zej>ej>eej>e?e?d/d0d1Z@Gd2d3d3e j1ZAGd4d5d5eZBGd6d7d7e j1ZCGd8d9d9e j1ZDe$d:dGd;d<dd>e j1ZFe$d?dGd@dAdAe;ZGe$dBdGdCdDdDe;eZHgdEZIdS)GzPyTorch GIT model.N) dataclass)CallableOptionalUnion)nn)ACT2FN)Cache DynamicCache)GenerationMixin)_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringcan_return_tuplelogging torch_int)deprecate_kwarg) GitConfigGitVisionConfigz} Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. )Z custom_introc@sjeZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dS)GitVisionModelOutputz image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__r!rtorch FloatTensor__annotations__r"r#tupler$r-r-`/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/git/modeling_git.pyr 5s r csLeZdZdZfddZd eejeejeeje ej dddZ Z S) GitEmbeddingsz;Construct the embeddings from word and position embeddings.csttj|j|j|jd|_t|j|j|_ tj |j|j d|_ t |j |_t|dd|_|jdt|jddddS) N) padding_idxepsposition_embedding_typeabsolute position_idsrF persistent)super__init__r Embedding vocab_size hidden_sizeZ pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutgetattrr3register_bufferr)arangeexpandselfconfig __class__r-r.r;Ks zGitEmbeddings.__init__Nr) input_idsr5 inputs_embedspast_key_values_lengthreturnc Cs|dur|}n|dd}|d}|durL|jdd|||f}|dur`||}n|}|jdkr||}||7}||}||}|S)Nr7rr4)sizer5r?r3rArBrF) rLrPr5rQrR input_shape seq_length embeddingsrAr-r-r.forwardZs      zGitEmbeddings.forward)NNNr) r%r&r'r(r;rr)Z LongTensorr*intTensorrX __classcell__r-r-rNr.r/Hs r/c sheZdZd fdd Zeddddd ejeejeejee ee ee e ejd d d Z Z S)GitSelfAttentionNcsXt|j|jdkr>t|ds>td|jd|jd||_|durbtd|j j d|j|_t |j|j|_ |j|j |_ t |jj|jjdd |_|jdur|j|j9_t|j|j |_t|j|j |_t|j|j |_t|j|_|pt|d d |_|jd ks2|jd krT|j|_td|jd |j |_dS)NrZembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()zInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.rr3r4 relative_keyrelative_key_query) r:r;r>num_attention_headshasattr ValueError layer_idxlogger warning_oncerOr%rYattention_head_size all_head_size vision_config image_size patch_sizeimage_patch_tokensnum_image_with_embeddingrLinearquerykeyvaluerDZattention_probs_dropout_probrFrGr3r@r<distance_embeddingrLrMr3rdrNr-r.r;ys8   zGitSelfAttention.__init__past_key_valuepast_key_values4.58new_nameversionFr#attention_mask head_maskruoutput_attentionspixel_values_presentrSc Cs|j\}}} |||d|j|jdd} |r8|jnd} |||d|j|jdd} |||d|j|jdd} |dur2| | dddd| dddf| dddd| dddf|j \}}t j | ddddd| ddf|gdd} t j | ddddd| ddf|gdd} t | | dd}|jdks^|jdkrZ| jd| jd}}|durt j|dt j|jd dd}nt j|t j|jd dd}t j|t j|jd dd}||}|||jd}|j| jd }|jdkr&t d | |}||}n4|jdkrZt d | |}t d | |}|||}|t|j}|dur|||}tjj|dd}||}|dur||}t || }|dddd }| dd|j!f}||}||fS)Nr7rr^rdimr_r`dtypedevicerzbhld,lrd->bhlrzbhrd,lrd->bhlrr)"shaperoviewrarg transposerlrprqupdaterdr)catmatmulr3ZtensorlongrrIrrr@torZeinsummathsqrtr functionalsoftmaxrFpermute contiguousrTrh)rLr#r{r|rur}r~ batch_sizerV_Z query_layercutoffZ key_layerZ value_layerZkey_layer_pastZvalue_layer_pastZattention_scoresZ query_lengthZ key_lengthZposition_ids_lZposition_ids_rZdistanceZpositional_embeddingZrelative_position_scoresZrelative_position_scores_queryZrelative_position_scores_keyZattention_probsZ context_layerZnew_context_layer_shaper-r-r.rXsr        @..          zGitSelfAttention.forward)NN)NNNFF)r%r&r'r;rr)rZrr*r boolr,rXr[r-r-rNr.r\xs " r\cs4eZdZfddZejejejdddZZS) GitSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr1) r:r;rrnr>denserBrCrDrErFrKrNr-r.r;s zGitSelfOutput.__init__r# input_tensorrScCs&||}||}|||}|SNrrFrBrLr#rr-r-r.rXs  zGitSelfOutput.forwardr%r&r'r;r)rZrXr[r-r-rNr.rs reagerc speZdZdfdd ZddZedddd dejeej eej ee ee ee e ejd d d Z ZS) GitAttentionNcs6tt|j|||d|_t||_t|_dS)N)r3rd) r:r;GIT_SELF_ATTENTION_CLASSES_attn_implementationrLroutputset pruned_headsrsrNr-r.r;s   zGitAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)lenrrLrargrrrorprqrrrhunion)rLheadsindexr-r-r. prune_headss zGitAttention.prune_headsrtrurvrwFrzc Cs,|||||||\}}|||} | |fSr)rLr) rLr#r{r|rur}r~ attn_outputZself_attn_weightsattention_outputr-r-r.rX!s  zGitAttention.forward)NN)NNNFF)r%r&r'r;rrr)rZrr*r rr,rXr[r-r-rNr.rs"  rcs0eZdZfddZejejdddZZS)GitIntermediatecsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr) r:r;rrnr>intermediate_sizer isinstance hidden_actstrrintermediate_act_fnrKrNr-r.r;9s   zGitIntermediate.__init__r#rScCs||}||}|Sr)rrrLr#r-r-r.rXAs  zGitIntermediate.forwardrr-r-rNr.r8s rcs4eZdZfddZejejejdddZZS) GitOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r:r;rrnrr>rrBrCrDrErFrKrNr-r.r;Is zGitOutput.__init__rcCs&||}||}|||}|Srrrr-r-r.rXOs  zGitOutput.forwardrr-r-rNr.rHs rc speZdZdfdd Zedddddejeejeejee ee ee e ejd d d Z d d Z ZS)GitLayerNcs>t|j|_d|_t||d|_t||_t||_ dS)Nr)rd) r:r;chunk_size_feed_forward seq_len_dimr attentionr intermediaterr)rLrMrdrNr-r.r;Ws   zGitLayer.__init__rtrurvrwFrzc Cs6|j||||||d\}}t|j|j|j|} | |fS)N)r}rur~)rrfeed_forward_chunkrr) rLr#r{r|rur}r~rZself_attention_weights layer_outputr-r-r.rX_s  zGitLayer.forwardcCs||}|||}|Sr)rr)rLrZintermediate_outputrr-r-r.rxs  zGitLayer.feed_forward_chunk)N)NNNFF)r%r&r'r;rr)rZrr*r rr,rXrr[r-r-rNr.rVs" rc seZdZfddZd ejeejeejeee e e ejfee ee ee ee ee ee eje fd ddZ ZS) GitEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}t|qSr-)r).0irMr-r. z'GitEncoder.__init__..F) r:r;rMr ModuleListrangenum_hidden_layerslayergradient_checkpointingrKrNrr.r;s  zGitEncoder.__init__NFT) r#r{r|ru use_cacher}output_hidden_statesr~ return_dictrSc Cs|jr|jr|rtdd}t|tdtfs8td|rP|durPt|j d}|rXdnd} |rddnd} t |j D]V\} } |r| |f} |dur|| nd}| ||||||}|d}|rr| |df} qr|r| |f} | st dd ||| | fDSt ||| | d S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FzBThe `past_key_values` should be either a `Cache` object or `None`.rr-rrcss|]}|dur|VqdSrr-)rvr-r-r. sz%GitEncoder.forward..r"rur#r$)rtrainingrerfrtyper rcr rM enumeraterr,r)rLr#r{r|rurr}rr~rZall_hidden_statesZall_self_attentionsrZ layer_moduleZlayer_head_mask layer_outputsr-r-r.rXsV         zGitEncoder.forward)NNNNFFFT)r%r&r'r;r)rZrr*rr r,rrrXr[r-r-rNr.r~s* rc@s&eZdZUeed<dZdZddZdS)GitPreTrainedModelrMgitTcCst|trRtjj|jd|jjdtjj|jj |jjdtjj|j j |jjdt|tj r|j j jd|jjd|j dur|j j nft|tjr|j j jd|jjd|jdur|j j |jn&t|tjr|j j |j j ddS)zInitialize the weights)meanstd)rNg?)rGitVisionEmbeddingsrinitZnormal_class_embeddingrMZinitializer_rangepatch_embeddingweightposition_embeddingrndatabiasZzero_r<r0rBZfill_)rLmoduler-r-r. _init_weightss       z GitPreTrainedModel._init_weightsN)r%r&r'rr+Zbase_model_prefixZsupports_gradient_checkpointingrr-r-r-r.rs rcsPeZdZedfdd ZejeeejdddZd ej ejdd d Z Z S) rrcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF)Z in_channelsZ out_channelsZ kernel_sizeZstriderr^rr5r6r8)r:r;rMr> embed_dimrjrkr Parameterr)ZrandnrZConv2dZ num_channelsr num_patches num_positionsr<rrHrIrJrKrNr-r.r;s"  zGitVisionEmbeddings.__init__)rWheightwidthrSc Cs|jdd}|jjd}|jdd}tjsP||krP||krP||jS|ddddf}|ddddf}|jd} ||j} ||j} t |d} | d| | | }| dddd}t j j|| | fdd d }| dddddd| }tj||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr7g?rr^ZbicubicF)rTmodeZ align_cornersr)rrrZ unsqueezer)Zjit is_tracingr5rkrreshaperrrZ interpolaterr) rLrWrrrrrZclass_pos_embedZpatch_pos_embedrZ new_heightZ new_widthZsqrt_num_positionsr-r-r.interpolate_pos_encodings*      z,GitVisionEmbeddings.interpolate_pos_encodingF) pixel_valuesrSc Cs|j\}}}}|sL||jks&||jkrLtd|d|d|jd|jd |jjj}||j|d}|ddd}|j |dd} t j | |gdd } |r| | | ||} n| ||j} | S) NzInput image size (*z) doesn't match model ().rr^rr7r)rrjrcrrrrflattenrrrJr)rrrr5) rLrrrrrrZ target_dtypeZ patch_embedsZ class_embedsrWr-r-r.rX&s  zGitVisionEmbeddings.forward)F) r%r&r'rr;r)rZrYrr*rXr[r-r-rNr.rs)rcs0eZdZfddZejejdddZZS) GitVisionMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSr) r:r;rMrr activation_fnrrnr>rfc1fc2rKrNr-r.r;:s   zGitVisionMLP.__init__rcCs"||}||}||}|Sr)rrrrr-r-r.rXAs   zGitVisionMLP.forwardrr-r-rNr.r9s rr)rrorprqr{scalingrFc Ks|t||dd|}|dur(||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)Nr7r)rr)prrr^) r)rrrrrZfloat32rrrFrr) rrorprqr{rrFkwargs attn_weightsrr-r-r.eager_attention_forwardIs  rc sZeZdZdZfddZd ejeejeejeee ejeejfdddZ Z S) GitVisionAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkrZtd|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: rgF)r:r;rMr>rra num_headshead_dimrcscaleZattention_dropoutrF is_causalrrnk_projv_projq_projout_projrKrNr-r.r;cs$   zGitVisionAttention.__init__NFr#r{causal_attention_maskr}rSc CsP|j\}}}||}||} ||} ||||j|jdd}| |||j|jdd} | |||j|jdd} |jj dkr|dur|dur||}q|dur|}n |du|_ t } |jj dkr|jj dkr|rt dn t|jj } | ||| | ||j |j|jsdn|jd \} } | |||} || } |sHd} | | fS) z#Input shape: Batch x Time x Channelrr^Zflash_attention_2NrZsdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r)rrrF)rrrrrrrrrMrrrrerfrrrrFrrr)rLr#r{rr}rrVrZquerieskeysvaluesZattention_interfacerrr-r-r.rXwsF           zGitVisionAttention.forward)NNF) r%r&r'r(r;r)rZrrr,rXr[r-r-rNr.r`s rcsJeZdZedfdd Zdejejejeee ej dddZ Z S) GitVisionEncoderLayerrcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSr) r:r;r>rr self_attnrrBrC layer_norm1rmlp layer_norm2rKrNr-r.r;s    zGitVisionEncoderLayer.__init__FrcCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r`||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r#r{rr})r rr r )rLr#r{rr}Zresidualroutputsr-r-r.rXs"     zGitVisionEncoderLayer.forward)F) r%r&r'rr;r)rZrrr,r*rXr[r-r-rNr.rs rc sdeZdZdZedfdd Zed eej eej ee ee ee e e e fdddZZS) GitVisionEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`GitVisionEncoderLayer`]. Args: config: GitVisionConfig rcs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr-)rrrrr-r.rrz-GitVisionEncoder.__init__..F) r:r;rMrrrrlayersrrKrNrr.r;s  zGitVisionEncoder.__init__N)r{rr}rrrSc Cs|dur |n|jj}|dur |n|jj}|dur4|n|jj}|rDdnd}|rPdnd}|} t|jD]@\} } |rx|| f}| | |||d} | d} |rb|| df}qb|r|| f}t| ||dS)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr-)r}rrr"r#r$)rMr}ruse_return_dictrrr) rLrQr{rr}rrZencoder_statesZall_attentionsr#idxZ encoder_layerrr-r-r.rXs0'    zGitVisionEncoder.forward)NNNNN)r%r&r'r(rr;rrr)rZrrr,rrXr[r-r-rNr.r s  r c s^eZdZedfdd Zed eejee ee ee ee e e e fdddZ ZS) GitVisionTransformerrcsRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSr) r:r;rMr>rrWrrBrC pre_layrnormr encoderpost_layernorm)rLrMrrNr-r.r;<s   zGitVisionTransformer.__init__NFrr}rrrrSc Cs|dur |n|jj}|dur |n|jj}|dur4|n|jj}|durLtd|j||d}||}|j||||d}|d}||}|s|f|ddSt ||j |j dS)Nz You have to specify pixel_valuesr)rQr}rrrrr) rMr}rrrcrWrrrrr#r$) rLrr}rrrr#encoder_outputsr"r-r-r.rXFs.   zGitVisionTransformer.forward)NNNFN)r%r&r'rr;rrr)r*rrr,rrXr[r-r-rNr.r:s  rzY The vision model from CLIP, used in GIT, without any head or projection on top. c sxeZdZUeed<dZedfdd ZejdddZ e de e j e ee eee eeeefd d d ZZS)GitVisionModelrMrrcs"t|t||_|dSr)r:r;r vision_model post_initrKrNr-r.r;zs  zGitVisionModel.__init__)rScCs |jjjSr)rrWrrLr-r-r.get_input_embeddingssz#GitVisionModel.get_input_embeddingsNFrcCs(|dur |n|jj}|j|||||dS)a{ Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, GitVisionModel >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = GitVisionModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```N)rr}rrr)rMrr)rLrr}rrrr-r-r.rXszGitVisionModel.forward)NNNFN)r%r&r'rr+Zmain_input_namer;rModulerrrr)r*rrr,rrXr[r-r-rNr.rps$  rcs6eZdZedfdd ZejejdddZZS) GitProjectionrcs@t||_tt|jj|jtj|j|jj d|_ dSr) r:r;rMrZ Sequentialrnrir>rBrCvisual_projectionrKrNr-r.r;s  zGitProjection.__init__)rWrScCs ||Sr)r!)rLrWr-r-r.rXszGitProjection.forward) r%r&r'rr;r)rZrXr[r-r-rNr.r sr zy The bare GIT Model transformer consisting of a CLIP image encoder and text decoder outputting raw hidden-states cseZdZfddZddZddZddZeej ej ej d d d Z dd dZ edeej eej eej eej eej eej eeeeejfeeeeeeeeeeeej efd ddZZS)GitModelcsrt|_t|_tj|_t|_ t |_ j durft fddtj D|_|dS)Nc3s&|]}ttddjjVqdS)rN)rrr)zerosrir>rrr-r.rsz$GitModel.__init__..)r:r;rMr/rWrri image_encoderrrr r!rmrZ ParameterListrimg_temperal_embeddingrrKrNrr.r;s       zGitModel.__init__cCs|jjSrrWr?rr-r-r.rszGitModel.get_input_embeddingscCs ||j_dSrr&)rLrqr-r-r.set_input_embeddingsszGitModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)rLZheads_to_prunerrr-r-r. _prune_headsszGitModel._prune_heads)rTrrrScCs4tjtj||||ddd}||dktd}|S)Nrrr)Zdiagonal-inf)r)ZtriuZonesZ masked_fillfloat)rLrTrrmaskr-r-r._generate_future_maskszGitModel._generate_future_maskNcCs|jd}|jd}|j}|j} tj||f|| d} tj|||ftd|j| d} tj||f| |jd} |dkrtj|jd|jd|f| |jd}tj| | fdd} tj| || fdd}tj| |fdddddf}|dur tj|jd|jdfd|d}|jtj kr t d tj ||jd }td||<| |jd|||||f}| }|ddddd|f}|dddddf}|||ddddd|f<|dddddddf}|S) Nrr*r+rrrF)Z fill_valuerz1Memory key padding mask must be a boolean tensor.r)rrrr)r#fullr,rrrrcZ zeros_likerJclone)rLtgtmemorytgt_maskrRZmemory_key_padding_maskZnum_tgtZ num_memoryrrtop_left top_right bottom_leftleftrightZfull_attention_maskZzero_negative_infinityZ origin_leftrr-r-r.create_attention_masksP      zGitModel.create_attention_maskF) rPr{r5rr|rQrurr}rrrrSc  Cs| dur | n|jj} | dur | n|jj} |dur4|n|jj}| durH| n|jj} |durj|durjtdn@|dur||||} n"|dur|dd} ntd| d}d}|durt|t s| n| }| ||jj }d}|dur|j dkr|j|| dj}n~|j d krg}t|jdD]F}|j|dd|ddddf| dj}||j|7}||q2tj|dd }ntd ||}|j||||d }|durtj|jdd|jd f|j|jd}||d|ddd}tj||fdd }|||j|j}|j||||d}|durt||j| dd|j}|dkr|dddd| dddf}n4|dddd| d d| d df|7<|j |||||| | | |dud }|d}| s|f|ddSt!||j"|j#|j$dS)a Examples: ```python >>> from transformers import AutoProcessor, AutoModel >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = AutoModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "this is an image of two cats" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer7z5You have to specify either input_ids or inputs_embedsrrrrz#pixel_values must be of rank 4 or 5)rPr5rQrRr^r)r1r2r3rR)Ztgt_len)r{r|rurr}rrr~r)%rMr}rrrrcZ%warn_if_padding_and_no_attention_maskrTrr get_seq_lengthZ get_head_maskrndimr$r"rrr%appendr)rr!rWr#rrrepeatr.r9r rrrrur#r$)rLrPr{r5rr|rQrurr}rrrrUrVrRZprojected_visual_featuresZvisual_featuresZ frame_idxZvisual_features_frameZembedding_outputr#r3Zcombined_attention_maskZexpanded_attn_maskrsequence_outputr-r-r.rXs%              $4 zGitModel.forward)N) NNNNNNNNNNFN)r%r&r'r;rr'r)rYr)rrrZr.r9rrrr listr*rr,rrXr[r-r-rNr.r"sD  2r"z` GIT Model with a `language modeling` head on top for autoregressive language modeling. cseZdZdgZfddZddZddZedee j ee j ee j ee j ee j ee j ee j ee e e e j feeeeeeeeee ee j efd d d Zdd dZZS)GitForCausalLMz output.weightcs4t|t||_t|j|j|_| dSr) r:r;r"rrrnr>r=rrrKrNr-r.r;s  zGitForCausalLM.__init__cCs|jSrrrr-r-r.get_output_embeddingssz$GitForCausalLM.get_output_embeddingscCs ||_dSrrC)rLZnew_embeddingsr-r-r.set_output_embeddingssz$GitForCausalLM.set_output_embeddingsNF)rPr{r5rr|rQlabelsrurr}rrrrScKs | dur | n|jj} |dur d} |j|||||||| | | | | d }|d}||}d}|dur|jjjdjjj}|dd|dddf }|ddddf }|j | d|jj | dfd|jj i|}| s|f|dd}|dur|f|S|St |||j|j|jdS) a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` Examples: Image captioning example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-coco") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-coco") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> print(generated_caption) two cats sleeping on a pink blanket next to remotes. ``` Visual question answering (VQA) example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> from huggingface_hub import hf_hub_download >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-textvqa") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-textvqa") >>> file_path = hf_hub_download(repo_id="nielsr/textvqa-sample", filename="bus.png", repo_type="dataset") >>> image = Image.open(file_path).convert("RGB") >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> question = "what does the front of the bus say at the top?" >>> input_ids = processor(text=question, add_special_tokens=False).input_ids >>> input_ids = [processor.tokenizer.cls_token_id] + input_ids >>> input_ids = torch.tensor(input_ids).unsqueeze(0) >>> generated_ids = model.generate(pixel_values=pixel_values, input_ids=input_ids, max_length=50) >>> print(processor.batch_decode(generated_ids, skip_special_tokens=True)) ['what does the front of the bus say at the top? special'] ``` Video captioning example: ```python >>> import av >>> import numpy as np >>> from PIL import Image >>> from huggingface_hub import hf_hub_download >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-vatex") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-vatex") >>> # set seed for reproducibility >>> np.random.seed(45) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # load video >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample frames >>> num_frames = model.config.num_image_with_embedding >>> indices = sample_frame_indices( ... clip_len=num_frames, frame_sample_rate=4, seg_len=container.streams.video[0].frames ... ) >>> frames = read_video_pyav(container, indices) >>> pixel_values = processor(images=list(frames), return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> print("Generated caption:", processor.batch_decode(generated_ids, skip_special_tokens=True)) Generated caption: ['a woman is sitting at a table and she is talking about the food she is holding.'] ``` NF) r{r5rr|rQrurr}rrrrr7rr=)losslogitsrur#r$)rMrrrrrrrLrlrZ loss_functionrr=rrur#r$)rLrPr{r5rr|rQrFrurr}rrrrr r@rHrGZnum_image_tokensZshifted_logitsrr-r-r.rXsV zGitForCausalLM.forwardc Kst|durF|}|jd|kr$|}n|jdd}|dd|df}|j}|dur^||}|||d||dS)Nrr)rPr{rrur)r<rZnew_onesget) rLrPrur{rrZ past_lengthZremove_prefix_lengthrUr-r-r.prepare_inputs_for_generations z,GitForCausalLM.prepare_inputs_for_generation) NNNNNNNNNNNFN)NNN)r%r&r'Z_tied_weights_keysr;rDrErrr)rZrr rArr,rrXrJr[r-r-rNr.rBsJ ErB)rBr"rr)r)Jr(r dataclassesrtypingrrrr)Ztorch.utils.checkpointrZ activationsrZ cache_utilsr r Z generationr Zmodeling_attn_mask_utilsr Zmodeling_layersr Zmodeling_outputsrrrrZmodeling_utilsrrZ pytorch_utilsrrrutilsrrrrrZutils.deprecationrZconfiguration_gitrrZ get_loggerr%rer rr/r\rrrrrrrrrrrZr,rrrr rrr r"rB__all__r-r-r-r.s         0z3(LS P3W65 w