a h @sdZddlZddlmZddlmZmZmZmZddl Z ddl m Z ddl Z ddl mZddlmZddlmZmZmZmZdd lmZmZdd lmZmZmZdd lmZmZm Z m!Z!m"Z"d d l#m$Z$m%Z%m&Z&e!'e(Z)e j*e j*dddZ+e j*e j*dddZ,eeGdddeZ-Gddde j.Z/Gddde j.Z0Gddde j.Z1de0iZ2Gddde j.Z3Gdd d e j.Z4Gd!d"d"e j.Z5Gd#d$d$eZ6Gd%d&d&e j.Z7Gd'd(d(e j.Z8dHe j.e j*e j*e j*ee j*e9e9d*d+d,Z:Gd-d.d.e j.Z;Gd/d0d0e j.ZGd5d6d6e j.Z?eGd7d8d8eZ@Gd9d:d:e j.ZAGd;d<dGd?d@d@e@ZCGdAdBdBe@ZDGdCdDdDe@ZEdIdEdFZFgdGZGdS)JzPyTorch AltCLIP model.N) dataclass)AnyCallableOptionalUnion)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling,BaseModelOutputWithPoolingAndCrossAttentions'BaseModelOutputWithPoolingAndProjection)ALL_ATTENTION_FUNCTIONSPreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringcan_return_tuplelogging torch_int) AltCLIPConfigAltCLIPTextConfigAltCLIPVisionConfig)logitsreturncCstj|tjt||jdS)Ndevice)nn functionalZ cross_entropytorcharangelenr)rr%h/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/altclip/modeling_altclip.pycontrastive_loss,sr') similarityrcCs t|}t|}||dS)Ng@)r't)r(Z caption_lossZ image_lossr%r%r& clip_loss0s r*c@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eed<dZeed <eed d d ZdS) AltCLIPOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`AltCLIPTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`AltCLIPVisionModel`]. text_model_output (`BaseModelOutputWithPooling`): The output of the [`AltCLIPTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`AltCLIPVisionModel`]. Nlosslogits_per_imagelogits_per_text text_embeds image_embedstext_model_outputvision_model_outputrcstfddDS)Nc3s,|]$}|dvr|n t|VqdS))r1r2N)getattrto_tuple).0kselfr%r& Vsz)AltCLIPOutput.to_tuple..)tuplekeysr8r%r8r&r5Us zAltCLIPOutput.to_tuple)__name__ __module__ __qualname____doc__r,rr" FloatTensor__annotations__r-r.r/r0r1r r2r;rr5r%r%r%r&r+6s   r+cs2eZdZdZfddZd ddZdd ZZS) AltRobertaEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|jdt|jddd|jd tj|jtjd dd|j|_tj|j|j|jd|_ dS) N) padding_idxepsposition_embedding_typeabsolute position_idsrF persistenttoken_type_idsdtype)super__init__r EmbeddingZ vocab_size hidden_sizeZ pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsZtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutr4rGregister_bufferr"r#expandzerosrIsizelongrDr9config __class__r%r&rRcs"  zAltRobertaEmbeddings.__init__Nrc Cs|dur*|dur t||j|}n ||}|dur<|}n|dd}|d}|durt|dr|jddd|f}||d|} | }ntj|tj |j j d}|dur| |}| |} || } |jdkr||} | | 7} || } || } | S)NrKrrNrrPrrH)"create_position_ids_from_input_idsrD&create_position_ids_from_inputs_embedsrahasattrrNr_r"r`rbrIrrUrXrGrWrYr]) r9 input_idsrNrI inputs_embedspast_key_values_length input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedrX embeddingsrWr%r%r&forward|s0         zAltRobertaEmbeddings.forwardcCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor NrKrrgr)rar"r#rDrbr unsqueezer_)r9rlrnZsequence_lengthrIr%r%r&ris z;AltRobertaEmbeddings.create_position_ids_from_inputs_embeds)NNNNr)r=r>r?r@rRrsri __classcell__r%r%rer&rC]s   (rCcsNeZdZdfdd Zd ejeejeejeee ejdddZ Z S) AltRobertaSelfAttentionNcst|j|jdkr>t|ds>td|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|pt|dd|_|jdks|jd kr|j|_t d |jd |j|_dS) NrZembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()rGrH relative_keyrelative_key_queryr)rQrRrTnum_attention_headsrj ValueErrorintattention_head_size all_head_sizer Linearquerykeyvaluer[Zattention_probs_dropout_probr]r4rGrVrSdistance_embeddingr9rdrGrer%r&rRs(  z AltRobertaSelfAttention.__init__F hidden_statesattention_mask head_maskoutput_attentionsrcCs |jdd}g|d|jR}|||dd}|||dd}|||dd} t||dd} |j dks|j dkrd|jd|jd} } tj | tj |j ddd} tj | tj |j ddd}| |}| ||jd}|j|jd}|j dkr0td ||}| |} n4|j dkrdtd ||}td ||}| ||} | t|j} |dur| |} tjj| dd }||}|dur||}t|| }|d ddd }|dd|jf}||}|r||fn|f}|S)NrKrrzrxryrgrOzbhld,lrd->bhlrzbhrd,lrd->bhlrdimrr)shaper~rview transposerrr"matmulrGr#rbrrrVtorPZeinsummathsqrtr r!softmaxr]permute contiguousrar)r9rrrrrnZ hidden_shapeZ query_layerZ key_layerZ value_layerZattention_scoresZ query_lengthZ key_lengthZposition_ids_lZposition_ids_rZdistanceZpositional_embeddingZrelative_position_scoresZrelative_position_scores_queryZrelative_position_scores_keyZattention_probsZ context_layerZnew_context_layer_shapeoutputsr%r%r&rssB         zAltRobertaSelfAttention.forward)N)NNF) r=r>r?rRr"TensorrrAboolr;rsrur%r%rer&rvsrvcs4eZdZfddZejejejdddZZS)AltRobertaSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNrE) rQrRr rrTdenserYrZr[r\r]rcrer%r&rRs zAltRobertaSelfOutput.__init__r input_tensorrcCs&||}||}|||}|SNrr]rYr9rrr%r%r&rss  zAltRobertaSelfOutput.forwardr=r>r?rRr"rrsrur%r%rer&r s reagercsVeZdZd fdd ZddZd ejeejeejee e ejddd Z Z S) AltRobertaAttentionNcs4tt|j||d|_t||_t|_dS)N)rG) rQrR"ALT_ROBERTA_SELF_ATTENTION_CLASSES_attn_implementationr9routputset pruned_headsrrer%r&rR!s   zAltRobertaAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)r$rr9r{r~rrrrrrrrunion)r9headsindexr%r%r& prune_heads)s zAltRobertaAttention.prune_headsFrcCs8|j||||d}||d|}|f|dd}|SN)rrrrr)r9r)r9rrrrZ self_outputsattention_outputrr%r%r&rs;szAltRobertaAttention.forward)N)NNF) r=r>r?rRrr"rrrArr;rsrur%r%rer&r srcs0eZdZfddZejejdddZZS)AltRobertaIntermediatecsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr) rQrRr rrTintermediate_sizer isinstance hidden_actstrrintermediate_act_fnrcrer%r&rROs   zAltRobertaIntermediate.__init__rrcCs||}||}|Sr)rrr9rr%r%r&rsWs  zAltRobertaIntermediate.forwardrr%r%rer&rNs rcs4eZdZfddZejejejdddZZS)AltRobertaOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) rQrRr rrrTrrYrZr[r\r]rcrer%r&rR_s zAltRobertaOutput.__init__rcCs&||}||}|||}|Srrrr%r%r&rses  zAltRobertaOutput.forwardrr%r%rer&r^s rcsTeZdZfddZd ejeejeejeee ejdddZ dd Z Z S) AltRobertaLayercs:t|j|_d|_t||_t||_t||_ dS)Nr) rQrRchunk_size_feed_forward seq_len_dimr attentionr intermediaterrrcrer%r&rRns    zAltRobertaLayer.__init__NFrc KsP|j|f|||d|}|d}|dd}t|j|j|j|} | f|}|Sr)rrfeed_forward_chunkrr) r9rrrrkwargsZself_attention_outputsrr layer_outputr%r%r&rsvs   zAltRobertaLayer.forwardcCs||}|||}|Sr)rr)r9rZintermediate_outputrr%r%r&rs  z"AltRobertaLayer.feed_forward_chunk)NNF) r=r>r?rRr"rrrArr;rsrrur%r%rer&rms  rc sdeZdZfddZed ejeejeejee ee ee e e eje fdddZ ZS) AltRobertaEncodercs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr%)r)r6irdr%r& z.AltRobertaEncoder.__init__..F) rQrRrdr ModuleListrangenum_hidden_layerslayergradient_checkpointingrcrerr&rRs  zAltRobertaEncoder.__init__NFT)rrrroutput_hidden_states return_dictrcKs|rdnd}|rdnd} t|jD]\\} } |r8||f}|durH|| nd} | f||| |d|} | d}|r"| | df} q"|r||f}t||| dS)Nr%)rrrrrrlast_hidden_stater attentions) enumeraterr )r9rrrrrrrZall_hidden_statesZall_self_attentionsrZ layer_moduleZlayer_head_mask layer_outputsr%r%r&rss0    zAltRobertaEncoder.forward)NNFFT)r=r>r?rRrr"rrrArrr;r rsrur%r%rer&rs  rcs0eZdZfddZejejdddZZS)AltRobertaPoolercs*tt|j|j|_t|_dSr)rQrRr rrTrZTanh activationrcrer%r&rRs zAltRobertaPooler.__init__rcCs(|dddf}||}||}|S)Nr)rr)r9rZfirst_token_tensor pooled_outputr%r%r&rss  zAltRobertaPooler.forwardrr%r%rer&rs r)modulerrrrscalingr]c Ks|t||dd|}|dur(||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)NrKr)rrP)ptrainingrrz) r"rrr r!rZfloat32rrPr]rr) rrrrrrr]r attn_weights attn_outputr%r%r&eager_attention_forwards  rc sZeZdZdZfddZd ejeejeejeee ejeejfdddZ Z S) AltCLIPAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkrZtd|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: ).F)rQrRrdrT embed_dimr{ num_headshead_dimr|scaleZattention_dropoutr] is_causalr rk_projv_projq_projout_projrcrer%r&rRs$   zAltCLIPAttention.__init__NFrrcausal_attention_maskrrc CsP|j\}}}||}||} ||} ||||j|jdd}| |||j|jdd} | |||j|jdd} |jj dkr|dur|dur||}q|dur|}n |du|_ t } |jj dkr|jj dkr|rt dn t|jj } | ||| | ||j |j|jsdn|jd \} } | |||} || } |sHd} | | fS) z#Input shape: Batch x Time x ChannelrrzZflash_attention_2NrZsdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r)rrr])rrrrrrrrrdrrrloggerZ warning_oncerrrr]reshaperr)r9rrrr batch_sizerorZqueriesr<valuesZattention_interfacerrr%r%r&rssF           zAltCLIPAttention.forward)NNF) r=r>r?r@rRr"rrrr;rsrur%r%rer&rs rcs0eZdZfddZejejdddZZS) AltCLIPMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSr) rQrRrdrr activation_fnr rrTrfc1fc2rcrer%r&rR@s   zAltCLIPMLP.__init__rcCs"||}||}||}|Sr)rrrrr%r%r&rsGs   zAltCLIPMLP.forwardrr%r%rer&r?s rcsJeZdZedfdd Zdejejejeee ej dddZ Z S) AltCLIPEncoderLayerrcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSr) rQrRrTrr self_attnr rYrZ layer_norm1rmlp layer_norm2rcrer%r&rROs    zAltCLIPEncoderLayer.__init__FrcCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r`||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrrr)rrrr)r9rrrrZresidualrrr%r%r&rsWs"     zAltCLIPEncoderLayer.forward)F) r=r>r?rrRr"rrrr;rArsrur%r%rer&rNs rc sdeZdZdZedfdd Zed eej eej ee ee ee e e e fdddZZS) AltCLIPEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`AltCLIPEncoderLayer`]. Args: config: AltCLIPConfig rcs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr%)r)r6_rr%r&rrz+AltCLIPEncoder.__init__..F) rQrRrdr rrrlayersrrcrerr&rRs  zAltCLIPEncoder.__init__N)rrrrrrc Cs|dur |n|jj}|dur |n|jj}|dur4|n|jj}|rDdnd}|rPdnd}|} t|jD]@\} } |rx|| f}| | |||d} | d} |rb|| df}qb|r|| f}t| ||dS)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr%)rrrr)rdrruse_return_dictrrr ) r9rlrrrrrZencoder_statesZall_attentionsridxZ encoder_layerrr%r%r&rss0'    zAltCLIPEncoder.forward)NNNNN)r=r>r?r@rrRrrr"rrrr;r rsrur%r%rer&rs  rcsPeZdZedfdd ZejeeejdddZd ej ejdd d Z Z S) AltCLIPVisionEmbeddingsrcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF)Z in_channelsZ out_channelsZ kernel_sizeZstridebiasrzrrIrJrL)rQrRrdrTr image_size patch_sizer Parameterr"Zrandnclass_embeddingZConv2dZ num_channelspatch_embedding num_patches num_positionsrSposition_embeddingr^r#r_rcrer%r&rRs"  z AltCLIPVisionEmbeddings.__init__)rrheightwidthrc Cs|jdd}|jjd}|jdd}tjsP||krP||krP||jS|ddddf}|ddddf}|jd} ||j} ||j} t |d} | d| | | }| dddd}t j j|| | fdd d }| dddddd| }tj||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNrKg?rrzZbicubicF)ramodeZ align_cornersr)rrweightrtr"Zjit is_tracingrIrrrrr r!Z interpolatercat) r9rrrr rrrZclass_pos_embedZpatch_pos_embedrZ new_heightZ new_widthZsqrt_num_positionsr%r%r&interpolate_pos_encodings*      z0AltCLIPVisionEmbeddings.interpolate_pos_encodingF) pixel_valuesrc Cs|j\}}}}|sL||jks&||jkrLtd|d|d|jd|jd |jjj}||j|d}|ddd}|j |dd} t j | |gdd } |r| | | ||} n| ||j} | S) NzInput image size (*z) doesn't match model (rrOrzrrKr)rrr|rr rPrflattenrrr_r"r rrrI) r9rrrrrr Z target_dtypeZ patch_embedsZ class_embedsrrr%r%r&rss  zAltCLIPVisionEmbeddings.forward)F) r=r>r?rrRr"rr}rrArsrur%r%rer&rs)rc@s*eZdZUeed<dZdZgZddZdS)AltCLIPPreTrainedModelrdZaltclipTcCs|jj}t|trt|jj}tjj|jd|jd|dtjj|j j |jj |dtjj|j j |jj |dnt|t r|jj}|jdd|jjd|}|jd|}tjj|jj |dtjj|jj |dtjj|jj |dtjj|jj |dnt|tr||jj}|jjdd|jjd|}d|jjd|}tjj|jj |dtjj|jj |dn t|trtjj|jj |jd|jjdd|j_tjj|jj |jd|jjdd|j_nt|tjr|jj |j j!dnt|tj"rF|j jjd|jjd|jdur|jj nBt|tj#r|j jjd|jjd|j$dur|j j|j$ dS) zInitialize the weightsrr)meanstd)rrzTg?N)%rdZinitializer_factorrrr initZnormal_rrrr Zinitializer_rangerrrrrrrrrTrr AltCLIPModeltext_projectiontext_embed_dimZ_is_hf_initializedvisual_projectionvision_embed_dimrYrdataZzero_Zfill_rrSrD)r9rfactorZ in_proj_stdZ out_proj_stdZfc_stdr%r%r& _init_weights2sT          z$AltCLIPPreTrainedModel._init_weightsN) r=r>r?rrBZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulerr%r%r%r&r+s rc sbeZdZedfdd Zeed eej ee ee ee ee e e e fdddZZS) AltCLIPVisionTransformerrcsRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSr) rQrRrdrTrrrr rYrZ pre_layrnormrencoderpost_layernorm)r9rdrrer%r&rRas   z!AltCLIPVisionTransformer.__init__NF)rrrrrrc Cs|dur |n|jj}|dur |n|jj}|dur4|n|jj}|durLtd|j||d}||}|j|||dd}|d}|dddddf} || } t || |j |j dS)Nz You have to specify pixel_values)rT)rlrrrrr pooler_outputrr) rdrrrr|rrrr r!r rr) r9rrrrrrencoder_outputsrrr%r%r&rsks.   z AltCLIPVisionTransformer.forward)NNNNF)r=r>r?rrRrrrr"rArrr;r rsrur%r%rer&r`s   rc sxeZdZUeed<dZedfdd ZejdddZ e de e j e ee eee eeeefd d d ZZS)AltCLIPVisionModelrdrrcs"t|t||_|dSr)rQrRr vision_model post_initrcrer%r&rRs  zAltCLIPVisionModel.__init__r3cCs |jjjSr)r&rrrr8r%r%r&get_input_embeddingssz'AltCLIPVisionModel.get_input_embeddingsNFrrrrrrcCs(|dur |n|jj}|j|||||dS)a Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AltCLIPVisionModel >>> model = AltCLIPVisionModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```Nrrrrr)rdrr&)r9rrrrrr%r%r&rsszAltCLIPVisionModel.forward)NNNFN)r=r>r?rrBZmain_input_namerRr Moduler(rrr"rArrr;r rsrur%r%rer&r%s$  r%aE The model behaves as an encoder following the architecture described in *Attention is all you need*_ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. .. _*Attention is all you need*: https://huggingface.co/papers/1706.03762 )Z custom_introcseZdZUeed<dfdd ZddZddZd d Ze de e j e e j e e j e e j e e j e e j e e e e e e eee j efd d dZZS)AltRobertaModelrdTcsDt|||_t||_t||_|r2t|nd|_| dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) rQrRrdrCrrrr rpoolerr')r9rdadd_pooling_layerrer%r&rRs    zAltRobertaModel.__init__cCs|jjSrrrrUr8r%r%r&r(sz$AltRobertaModel.get_input_embeddingscCs ||j_dSrr/r9rr%r%r&set_input_embeddingssz$AltRobertaModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr rrr)r9Zheads_to_prunerrr%r%r& _prune_headsszAltRobertaModel._prune_headsN) rkrrNrIrrlrrrrc Cs|dur |n|jj}|dur |n|jj}| dur4| n|jj} |durV|durVtdn@|durt||||} n"|dur|dd} ntd| \} } |dur|jn|j} |durtj | | f| d}|dur t |j dr|j j ddd| f}| | | }|}ntj| tj| d}||| }|||jj}|j ||||d}|j|||||dd }|d }|jdur||nd}t|||j|jd S) NzDYou cannot specify both input_ids and inputs_embeds at the same timerKz5You have to specify either input_ids or inputs_embedsrrNrg)rkrIrNrlT)rrrrrrr")rdrrrr|Z%warn_if_padding_and_no_attention_maskrarr"ZonesrjrrrNr_r`rbZget_extended_attention_maskZ get_head_maskrr r-r rr)r9rkrrNrIrrlrrrrnrrorrprqZextended_attention_maskZembedding_outputr$sequence_outputrr%r%r&rss\      zAltRobertaModel.forward)T) NNNNNNNNN)r=r>r?rrBrRr(r1r3rrr"rrrr;r rsrur%r%rer&r,s6 r,cseZdZUeed<fddZejdddZej ddd d Z de e ej d fd d Z eede eje eje eje eje eje eje ee ee eeeefd ddZZS)AltCLIPTextModelrdcsLt|t|dd|_t|j|j|_tj |j|j d|_ | dS)NF)r.rE) rQrRr,robertar rrT project_dimtransformationrYrZpre_LNr'rcrer%r&rRCs  zAltCLIPTextModel.__init__r3cCs |jjjSrr6rrrUr8r%r%r&r(Jsz%AltCLIPTextModel.get_input_embeddingsN)rrcCs||jj_dSrr:r0r%r%r&r1Msz%AltCLIPTextModel.set_input_embeddings)new_num_tokensrcs t|Sr)rQresize_token_embeddings)r9r;rer%r&r<Psz(AltCLIPTextModel.resize_token_embeddings) rkrrNrIrrlrrrrc  Csp|dur |n|jj}|j|||||||| dd } | d} || } || } | dddf} t| | | j| jdS)a+ Examples: ```python >>> from transformers import AutoProcessor, AltCLIPTextModel >>> model = AltCLIPTextModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> texts = ["it's a cat", "it's a dog"] >>> inputs = processor(text=texts, padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```NT) rkrrNrIrrlrrrrr")rdrr6r9r8r rr)r9rkrrNrIrrlrrrrr4Zprojection_stater#r%r%r&rsSs,    zAltCLIPTextModel.forward)N) NNNNNNNNN)r=r>r?rrBrRr r+r(rSr1rr}r<rrr"rrrr;r rsrur%r%rer&r5@s8   r5cseZdZUeed<edfdd Zedeej eej eej ee ee ee ej dddZ edeej ee ee e ee ej d d d Z edeejeej eej eejeej ee ee ee e ee eeefd ddZZS)rrdrcst|t|jts.tdt|jdt|jtsPtdt|jd|j}|j}|j |_ |j |_ |j |_ |j |_t||_t||_tj|j|j dd|_tj|j |j dd|_tt|jj|_|dS)NzRconfig.vision_config is expected to be of type AltCLIPVisionConfig but is of type .zNconfig.text_config is expected to be of type AltCLIPTextConfig but is of type F)r)rQrRr vision_configr TypeErrortype text_configrrZprojection_dimr7rrTrr5 text_modelrr&r rrrrr"ZtensorrdZlogit_scale_init_value logit_scaler')r9rdrAr>rer%r&rRs2     zAltCLIPModel.__init__N)rkrrIrrrrc Csj|dur |n|jj}|dur |n|jj}|dur4|n|jj}|j|||||||d}|d} || } | S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`AltCLIPTextModel`]. Examples: ```python >>> from transformers import AutoProcessor, AltCLIPModel >>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> text_features = model.get_text_features(**inputs) ```N)rkrrIrNrrrr)rdrrrrBr) r9rkrrIrNrrr text_outputsrZ text_featuresr%r%r&get_text_featuress   zAltCLIPModel.get_text_featuresFr)c Csf|dur |n|jj}|dur |n|jj}|dur4|n|jj}|j|||||d}|d}||}|S)a* Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`AltCLIPVisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AltCLIPModel >>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```Nr*r)rdrrrr&r) r9rrrrrvision_outputsrZimage_featuresr%r%r&get_image_featuress zAltCLIPModel.get_image_features) rkrrrIrN return_lossrrrrrc  Cs,|dur |n|jj}|dur |n|jj}| dur4| n|jj} |j||||||| d} |j|||| | d} | d} || } | d}||}| | jdddd} ||jdddd}|j }t || |}|j}d}|rt|}| s|||| | | f}|dur|f|S|St||||| | | d S) a return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AltCLIPModel >>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP") >>> processor = AutoProcessor.from_pretrained("BAAI/AltCLIP") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```N)rkrrNrIrrrr*rrzrKT)rrZkeepdim)r,r-r.r/r0r1r2)rdrrrrBr&rrZnormrCexpr"rr)Tr*r+)r9rkrrrIrNrHrrrrrDrFr0r/rCr.r-r,rr%r%r&rssX%    zAltCLIPModel.forward)NNNNNNN)NNNFN) NNNNNNNNFN)r=r>r?rrBrRrrr"rrrArErGZ LongTensorrr;r+rsrur%r%rer&rsr !./ rcCs6||}tj|dd|||}||S)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rr)ner}r"ZcumsumZtype_asrb)rkrDrmmaskZincremental_indicesr%r%r&rhvs rh)rr%r5r)r)r)Hr@r dataclassesrtypingrrrrr"Ztorch.nnr Ztorch.utils.checkpointZ activationsrZmodeling_layersr Zmodeling_outputsr r r r Zmodeling_utilsrrZ pytorch_utilsrrrutilsrrrrrZconfiguration_altcliprrrZ get_loggerr=rrr'r*r+r+rCrvrrrrrrrrfloatrrrrrrrrr%r,r5rrh__all__r%r%r%r&st     $YW.)2 P2XS445 nSd