a h@sdZddlZddlZddlmZddlmZmZddl Z ddl Z ddl m Z ddl m Z ddlmZdd lmZdd lmZmZmZmZmZmZdd lmZdd lmZmZmZdd lm Z m!Z!ddl"m#Z#e!$e%Z&ee ddGdddeZ'Gddde j(Z)Gddde j(Z*Gddde j(Z+Gddde j(Z,Gddde j(Z-Gddde j(Z.Gd d!d!e j(Z/Gd"d#d#e j(Z0Gd$d%d%eZ1Gd&d'd'e j(Z2e Gd(d)d)eZ3e Gd*d+d+e3Z4Gd,d-d-e j(Z5e d.dGd/d0d0e3Z6Gd1d2d2e j(Z7Gd3d4d4e j(Z8e d5dGd6d7d7e3Z9e d8dGd9d:d:e3Z:e d;dGdd?d?e3Ztj!t"||dd}|#tj|||||gddqtj|dd}||dddf|dddff| d | }||dddf|dddff| d }||dddf|dddff| d d}||dddf|dddff| d | }|j$| d d }tj||fdd}tj|j dddddfdddddf| d d |fdd}||}|%|}tjt"|jdd||gdd}|||fffS)Nrsizerrdimc sBg|]:\}}tjtjj||fdddd|d|fqS)ZbilinearT)rBmodeZ align_cornersr)r functionalpad interpolate).0hw)height spatial_poswidthr&r' ks z/ViltEmbeddings.visual_embed..Zij)ZindexingdeviceF)as_tuplecs$g|]}dddf|kqSNrr&rIu) valid_idxr&r'rOcs$g|]}dddf|kqSrUr&rV) non_valid_idxr&r'rOrYcSsg|]}|dqSrrArIvr&r&r'rOrYcSsg|]}|dqSr[rAr\r&r&r'rOrYcsg|] }|qSr&r&r\max_image_lengthr&r'rOrYT) replacement)&r3 projectionweightshapefloatrrFrHlongsumr; image_size patch_sizer5 transposeviewr!catzipflattenstackrarangetorSexpand isinstanceintmaxminZnonzerounique enumerateZ multinomialonesappendr1r:)r< pixel_values pixel_maskr__phpwxZx_maskZx_hZx_w batch_size num_channelsZ patch_dimZ pos_embed patch_indexZeffective_resolutionZ unique_rowsZ valid_row_idxZnon_valid_row_idxZ valid_numsZnon_valid_numsZpad_numsselectir]nvpZ valid_choiceZ pad_choiceZ cls_tokensr&)rLr_rZrMrXrNr' visual_embed^sx  $$$0  &      (.,..8 &zViltEmbeddings.visual_embedrc  Cs|j|||d} |dur4|j|||jjd\}} } n |d} |durJd}| |tj|tj| j d} ||tj | |tj| j d}tj | |gdd} tj || gdd} | | fS)N) input_idstoken_type_ids inputs_embedsr^rdtyperSrC) r-rr;r_rmr7r!Z zeros_likererSZ full_likerk)r<rattention_maskrrzr{r image_embedsimage_token_type_idxZ text_embedsZ image_masksr embeddingsZmasksr&r&r'forwards&   zViltEmbeddings.forward)r?)r)rrrr r+rr __classcell__r&r&r=r'r(Fs   ar(cs*eZdZdZfddZdddZZS)r,zGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|jdt|jddd|jd tj|jtjd dddS) N) padding_idxepsposition_embedding_typeabsolute position_ids)rrQF) persistentrr)r*r+rr6 vocab_sizer0Z pad_token_idword_embeddingsZmax_position_embeddingsr5Ztype_vocab_sizer7 LayerNormlayer_norm_epsr8r9r:getattrrZregister_bufferr!rorqr/rrBrer<r;r=r&r'r+s zTextEmbeddings.__init__Nc Cs|dur|}n|dd}|d}|durH|jddd|f}|durt|dr|jddd|f}||d|}|}ntj|tj|jjd}|dur| |}| |} || } |j dkr| |} | | 7} | | } || } | S)NrQrrrrr)rBrhasattrrrqr!r/rerSrr7rr5rr:) r<rrrr input_shape seq_lengthZbuffered_token_type_idsZ buffered_token_type_ids_expandedr7rr5r&r&r'rs,        zTextEmbeddings.forward)NNNNrrrr r+rrr&r&r=r'r,s r,cs(eZdZdZfddZddZZS)r2z# Image to Patch Embedding. cst|j|j}}|j|j}}t|tjj r8|n||f}t|tjj rR|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr)Z kernel_sizeZstride)r*r+rgrhrr0rr collectionsabcIterabler4rConv2dra)r<r;rgrhrr0r4r=r&r'r+s  zViltPatchEmbeddings.__init__cCs@|j\}}}}||jkr td|jjj}||j|d}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r)rcr ValueErrorrarbrrp)r<rzrrrLrNZ target_dtyperr&r&r'r-s  zViltPatchEmbeddings.forwardrr&r&r=r'r2s r2cs&eZdZfddZdddZZS)ViltSelfAttentioncst|j|jdkr>t|ds>td|jd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)NrZembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .bias)r*r+r0num_attention_headsrrrsattention_head_size all_head_sizerLinearZqkv_biasquerykeyvaluer8Zattention_probs_dropout_probr:rr=r&r'r+9s  zViltSelfAttention.__init__NFcCs*|j\}}}|||d|j|jdd}|||d|j|jdd} |||d|j|jdd} t || dd} | t |j} |dur| |} t j dd| } || } |dur| |} t | | } | dddd} | dd|jf}| j|} |r | | fn| f}|S)NrQrr@rPrCrr)rcrrjrrrirrr!matmulmathsqrtrZSoftmaxr:Zpermute contiguousrBr)r<rr head_maskoutput_attentionsrrr|Z query_layerZ key_layerZ value_layerZattention_scoresZattention_probsZ context_layerZnew_context_layer_shapeoutputsr&r&r'rKsB          zViltSelfAttention.forward)NNFrrrr+rrr&r&r=r'r8s rcs>eZdZdZedfdd ZejejejdddZZ S)ViltSelfOutputz The residual connection is defined in ViltLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r;cs.tt|j|j|_t|j|_dSN) r*r+rrr0denser8r9r:rr=r&r'r+s zViltSelfOutput.__init__r input_tensorreturncCs||}||}|Srrr:r<rrr&r&r'rs  zViltSelfOutput.forward) rrrr rr+r!Tensorrrr&r&r=r'r{srcs.eZdZfddZddZd ddZZS) ViltAttentioncs*tt||_t||_t|_dSr)r*r+r attentionroutputset pruned_headsrr=r&r'r+s   zViltAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)NrrrC)lenrrrrrrrrrrrrunion)r<headsindexr&r&r' prune_headss zViltAttention.prune_headsNFcCs6|||||}||d|}|f|dd}|S)Nrr)rr)r<rrrrZ self_outputsattention_outputrr&r&r'rszViltAttention.forward)NNF)rrrr+rrrr&r&r=r'rs rcs6eZdZedfdd ZejejdddZZS)ViltIntermediatercsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr) r*r+rrr0intermediate_sizerrr hidden_actstrrintermediate_act_fnrr=r&r'r+s   zViltIntermediate.__init__)rrcCs||}||}|Sr)rrr<rr&r&r'rs  zViltIntermediate.forward rrrrr+r!rrrr&r&r=r'rsrcs:eZdZedfdd ZejejejdddZZS) ViltOutputrcs.tt|j|j|_t|j|_ dSr) r*r+rrrr0rr8r9r:rr=r&r'r+s zViltOutput.__init__rcCs ||}||}||}|Srrrr&r&r'rs  zViltOutput.forwardrr&r&r=r'rsrcs*eZdZdZfddZdddZZS) ViltLayerz?This corresponds to the Block class in the timm implementation.csbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)Nrr)r*r+Zchunk_size_feed_forwardZ seq_len_dimrrr intermediaterrrrr0rlayernorm_beforelayernorm_afterrr=r&r'r+s    zViltLayer.__init__NFc Csj|j|||||d}|d}|dd}|||j}||}||}|||}|f|}|S)N)rrr)rrrprSrrr) r<rrrrZself_attention_outputsrrZ layer_outputr&r&r'rs     zViltLayer.forward)NNFrr&r&r=r'rs rcs&eZdZfddZdddZZS) ViltEncodercs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr&)r)rIr|rr&r'rOrYz(ViltEncoder.__init__..F) r*r+r;rZ ModuleListrangenum_hidden_layerslayerZgradient_checkpointingrr=rr'r+s  zViltEncoder.__init__NFTc Cs|rdnd}|rdnd}t|jD]R\} } |r8||f}|durH|| nd} | ||| |} | d}|r"|| df}q"|r||f}|stdd|||fDSt|||dS)Nr&rrcss|]}|dur|VqdSrr&r\r&r&r' rYz&ViltEncoder.forward..)last_hidden_staterr)rwrr%r ) r<rrrroutput_hidden_states return_dictZall_hidden_statesZall_self_attentionsrZ layer_moduleZlayer_head_maskZ layer_outputsr&r&r'rs&    zViltEncoder.forward)NNFFTrr&r&r=r'rs rc@s.eZdZUeed<dZdZddgZddZdS) ViltPreTrainedModelr;viltTr(rcCst|tjtjfr@|jjjd|jjd|j dur|j j nft|tj r|jjjd|jjd|j dur|jj|j  n&t|tj r|j j |jjddS)zInitialize the weightsg)meanZstdNg?)rrrrrrbdataZnormal_r;Zinitializer_rangerZzero_r6rrZfill_)r<moduler&r&r' _init_weights%s     z!ViltPreTrainedModel._init_weightsN) rrrrr#Zbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesrr&r&r&r'rs rcseZdZdfdd ZddZddZdd Zedee j ee j ee j ee j ee j ee j ee j ee j ee ee ee ee eeee j fd d d ZZS) ViltModelTcsXt|||_t||_t||_tj|j |j d|_ |rFt |nd|_ |dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)r*r+r;r(rrencoderrrr0r layernorm ViltPoolerpooler post_init)r<r;add_pooling_layerr=r&r'r+8s   zViltModel.__init__cCs |jjjSrrr-rr<r&r&r'get_input_embeddingsIszViltModel.get_input_embeddingscCs||jj_dSrr)r<rr&r&r'set_input_embeddingsLszViltModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r<Zheads_to_prunerrr&r&r' _prune_headsOszViltModel._prune_headsN) rrrrzr{rrrrrrrrc  Cs| dur | n|jj} | dur | n|jj} | dur4| n|jj} |durV|durVtdn@|durt||||} n"|dur|dd} ntd| \}}|dur|jn|j}|durtj ||f|d}|dur|durtdn|dur|durtd|dur|j dn|j d}||kr.td |durTtj ||jj |jj f|d}| ||jj }|j|||||||| d \}}||| }|j|||| | | d }|d}||}|jdur||nd}| s||f|d dSt|||j|jd S)ak image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. image_token_type_idx (`int`, *optional*): - The token type ids for images. Examples: ```python >>> from transformers import ViltProcessor, ViltModel >>> from PIL import Image >>> import requests >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "hello world" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltModel.from_pretrained("dandelin/vilt-b32-mlm") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerQz5You have to specify either input_ids or inputs_embedsrRzFYou cannot specify both pixel_values and image_embeds at the same timez7You have to specify either pixel_values or image_embedsrzAThe text inputs and image inputs need to have the same batch size)r)rrrrrr)r pooler_outputrr)r;rruse_return_dictrZ%warn_if_padding_and_no_attention_maskrBrSr!rxrcrgZ get_head_maskrrZget_extended_attention_maskrrrr rr)r<rrrrzr{rrrrrrrrZtext_batch_sizerrSZimage_batch_sizeZembedding_outputZextended_attention_maskZencoder_outputssequence_output pooled_outputr&r&r'rWsp*        zViltModel.forward)T) NNNNNNNNNNNN)rrrr+rrrrrr! LongTensorr"rsboolrr r%rrr&r&r=r'r6s@rcs$eZdZfddZddZZS)rcs*tt|j|j|_t|_dSr)r*r+rrr0rZTanh activationrr=r&r'r+s zViltPooler.__init__cCs(|dddf}||}||}|SrU)rr)r<rZfirst_token_tensorrr&r&r'rs  zViltPooler.forwardrr&r&r=r'rs rzU ViLT Model with a language modeling head on top as done during pretraining. cseZdZddgZfddZddZddZed ee j ee j ee j ee j ee j ee j ee j ee j ee j ee ee ee e eee j fd d d ZZS)ViltForMaskedLMzmlm_score.decoder.weightzmlm_score.decoder.biascs,t|t||_t||_|dSr)r*r+rr ViltMLMHead mlm_scorerrr=r&r'r+s   zViltForMaskedLM.__init__cCs|jjSr)rdecoderrr&r&r'get_output_embeddingssz%ViltForMaskedLM.get_output_embeddingscCs||j_|j|j_dSr)rrr)r<Znew_embeddingsr&r&r'set_output_embeddingssz%ViltForMaskedLM.set_output_embeddingsN rrrrzr{rrrlabelsrrrrc  Cs| dur | n|jj} |j||||||||| | | d } | dd\}}|durV|jdn|jd}|ddd|f|dd|df}}||}d}| durt}| |j} ||d|jj | d}| s|f| dd}|dur|f|S|St ||| j | j dS)a/ image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Labels for computing the masked language modeling loss. Indices should be in *[-100, 0, ..., config.vocab_size]* (see *input_ids* docstring) Tokens with indices set to *-100* are ignored (masked), the loss is only computed for the tokens with labels in *[0, ..., config.vocab_size]* Examples: ```python >>> from transformers import ViltProcessor, ViltForMaskedLM >>> import requests >>> from PIL import Image >>> import re >>> import torch >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "a bunch of [MASK] laying on a [MASK]." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltForMaskedLM.from_pretrained("dandelin/vilt-b32-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> tl = len(re.findall("\[MASK\]", text)) >>> inferred_token = [text] >>> # gradually fill in the MASK tokens, one by one >>> with torch.no_grad(): ... for i in range(tl): ... encoded = processor.tokenizer(inferred_token) ... input_ids = torch.tensor(encoded.input_ids) ... encoded = encoded["input_ids"][0][1:-1] ... outputs = model(input_ids=input_ids, pixel_values=encoding.pixel_values) ... mlm_logits = outputs.logits[0] # shape (seq_len, vocab_size) ... # only take into account text features (minus CLS and SEP token) ... mlm_logits = mlm_logits[1 : input_ids.shape[1] - 1, :] ... mlm_values, mlm_ids = mlm_logits.softmax(dim=-1).max(dim=-1) ... # only take into account text ... mlm_values[torch.tensor(encoded) != 103] = 0 ... select = mlm_values.argmax().item() ... encoded[select] = mlm_ids[select].item() ... inferred_token = [processor.decode(encoded)] >>> selected_token = "" >>> encoded = processor.tokenizer(inferred_token) >>> output = processor.decode(encoded.input_ids[0], skip_special_tokens=True) >>> print(output) a bunch of cats laying on a couch. ```N rrrzr{rrrrrrr@rrQrrrr) r;rrrcrrrprSrjrr rr)r<rrrrzr{rrrrrrrrrrZ text_seq_lenZ text_featuresr|Z mlm_logitsZmasked_lm_lossloss_fctrr&r&r'rs@I*  zViltForMaskedLM.forward) NNNNNNNNNNNN)rrrZ_tied_weights_keysr+rrrrr!rr"rrr r%rrr&r&r=r'rs@ rcs$eZdZfddZddZZS)ViltPredictionHeadTransformcsVtt|j|j|_t|jtr6t |j|_ n|j|_ tj |j|j d|_ dS)Nr) r*r+rrr0rrrrrrtransform_act_fnrrrr=r&r'r+js   z$ViltPredictionHeadTransform.__init__cCs"||}||}||}|Sr)rr rrr&r&r'rss   z#ViltPredictionHeadTransform.forwardrr&r&r=r'r is r cs.eZdZdfdd ZddZddZZS) rNcsbt||_t||_tj|j|jdd|_ t t |j|_ |durT||j _|j |j _ dS)NFr)r*r+r;r  transformrrr0rrr.r!r/rrb)r<r;rbr=r&r'r+{s  zViltMLMHead.__init__cCs|j|j_dSr)rrrr&r&r' _tie_weightsszViltMLMHead._tie_weightscCs||}||}|Sr)r r)r<rr&r&r'rs  zViltMLMHead.forward)N)rrrr+rrrr&r&r=r'rzs rz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for visual question answering, e.g. for VQAv2. cseZdZfddZedeejeejeejeejeejeejeejeejeejee ee ee e e e ejfd ddZ ZS)ViltForQuestionAnsweringc slt||j|_t||_tt|j|jdt |jdt t|jd|j|_ | dS)Nr@) r*r+ num_labelsrrr Sequentialrr0rGELU classifierrrr=r&r'r+s  z!ViltForQuestionAnswering.__init__Nrc  Cs| dur | n|jj} |j||||||||| | | d } | r>| jn| d}||}d}| dur| |j} tj || | j d}| s|f| dd}|dur|f|S|St ||| j | j dS)aX image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.FloatTensor` of shape `(batch_size, num_labels)`, *optional*): Labels for computing the visual question answering loss. This tensor must be either a one-hot encoding of all answers that are applicable for a given example in the batch, or a soft encoding indicating which answers are applicable, where 1.0 is the highest score. Examples: ```python >>> from transformers import ViltProcessor, ViltForQuestionAnswering >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "How many cats are there?" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: 2 ```Nrrr@r )r;rrrrrprSrrFZ binary_cross_entropy_with_logitsrcrrr)r<rrrrzr{rrrrrrrrrrrrr&r&r'rs:1  z ViltForQuestionAnswering.forward) NNNNNNNNNNNNrrrr+rrr!rr"rrrr%rrr&r&r=r'rs: rz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text or text-to-image retrieval, e.g. MSCOCO and F30K. cseZdZfddZedeejeejeejeejeejeejeejeejeejee ee ee e e e ejfd ddZ ZS)ViltForImageAndTextRetrievalcs2t|t||_t|jd|_|dSr)) r*r+rrrrr0 rank_outputrrr=r&r'r+s  z%ViltForImageAndTextRetrieval.__init__Nrc  Cs| dur | n|jj} d} | dur(td|j||||||||| | | d }| rR|jn|d}||}| s|f|dd}| dur| f|S|St| ||j|jdS)ad image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels are currently not supported. Examples: ```python >>> from transformers import ViltProcessor, ViltForImageAndTextRetrieval >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> model = ViltForImageAndTextRetrieval.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> # forward pass >>> scores = dict() >>> for text in texts: ... # prepare inputs ... encoding = processor(image, text, return_tensors="pt") ... outputs = model(**encoding) ... scores[text] = outputs.logits[0, :].item() ```NzTraining is not yet supported.rrr@r ) r;rNotImplementedErrorrrrrrr)r<rrrrzr{rrrrrrrrrrrrr&r&r'rs8- z$ViltForImageAndTextRetrieval.forward) NNNNNNNNNNNNrr&r&r=r'rs: rzq Vilt Model transformer with a classifier head on top for natural language visual reasoning, e.g. NLVR2. cseZdZfddZedeejeejeejeejeejeejeejeejeejee ee ee e e e ejfd ddZ ZS)"ViltForImagesAndTextClassificationc svt||j|_t||_|j}tt|j ||j |t |j |t t|j ||j|_ | dSr)r*r+rrr num_imagesrrrr0rrrr)r<r;rr=r&r'r+is  z+ViltForImagesAndTextClassification.__init__Nrc CsN| dur | n|jj} | dur | n|jj} | dur4| n|jj} |durX|jdkrX|d}|durt|jdkrt|d}|dur|jdnd} | dur|dur|jdnd} | |jjkrtdg}| rgnd}| rgnd}t | D]}|j ||||dur|dd|ddddddfnd|durB|dd|ddddfnd|||durn|dd|ddddfnd|d| | | d }| r|j n|d}| || r| |j | r| |jqtj|dd}||}d}| durt}| |j} ||d|j| d}| s>|||f}|dur:|f|S|St||||d S) a3 image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Binary classification labels. Examples: ```python >>> from transformers import ViltProcessor, ViltForImagesAndTextClassification >>> import requests >>> from PIL import Image >>> image1 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_0.jpg", stream=True).raw) >>> image2 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_1.jpg", stream=True).raw) >>> text = "The left image contains twice the number of dogs as the right image." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> model = ViltForImagesAndTextClassification.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> # prepare inputs >>> encoding = processor([image1, image2], text, return_tensors="pt") >>> # forward pass >>> outputs = model(input_ids=encoding.input_ids, pixel_values=encoding.pixel_values.unsqueeze(0)) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: True ```Nrrz\Make sure to match the number of images in the model with the number of images in the input.) rrrzr{rrrrrrrrQrCr )r;rrrndimZ unsqueezercrrrrrryrrr!rkrrrprSrjrr)r<rrrrzr{rrrrrrrrZpooler_outputsrrrrrrrrr rr&r&r'r{sn/      .((      z*ViltForImagesAndTextClassification.forward) NNNNNNNNNNNN)rrrr+rrr!rr"rrrr%rrr&r&r=r'rcs: rcseZdZfddZedeejeejeejeejeejeejeejeejeejee ee ee e e e ejfd ddZ ZS)ViltForTokenClassificationcsNt||j|_t|dd|_t|j|_t |j |j|_ | dS)NF)r) r*r+rrrrr8r9r:rr0rrrr=r&r'r+s  z#ViltForTokenClassification.__init__Nrc  Cs| dur | n|jj} |j||||||||| | | d } | d}|durN|jdn|jd}||}||ddd|f}d}| durt}| |j} || d|j | d}| s|f| dd}|dur|f|S|St ||| j | j dS)a/ image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size, text_sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. NrrrrQr@r )r;rrrcr:rrrprSrjrrrr)r<rrrrzr{rrrrrrrrrZtext_input_sizerrr rr&r&r'rs@  z"ViltForTokenClassification.forward) NNNNNNNNNNNN)rrrr+rrr!rr"rrrr%rrr&r&r=r'rs: r)rrrrrrrr)>r collections.abcrr dataclassesrtypingrrr!Ztorch.utils.checkpointrZtorch.nnrZ activationsrZmodeling_layersr Zmodeling_outputsr r r r rrZmodeling_utilsrZ pytorch_utilsrrrutilsrrZconfiguration_viltrZ get_loggerrloggerrModuler(r,r2rrrrrrrrrrrr rrrrr__all__r&r&r&r'sz         9C# &,j]O