a h+1 @sdZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZmZmZmZdd lmZmZdd lmZddlmZm Z m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+m,Z,ddl-m.Z.m/Z/m0Z0e'1e2Z3ee%ddGddde#Z4Gddde j5Z6dCe j5e j7e j7e j7ee j7e8e8dddZ9Gddde j5Z:Gd d!d!e j5Z;Gd"d#d#eZGd(d)d)e=Z?Gd*d+d+e j5Z@Gd,d-d-e j5ZAGd.d/d/e j5ZBGd0d1d1e j5ZCGd2d3d3e j5ZDGd4d5d5eZEGd6d7d7e j5ZFGd8d9d9e j5ZGGd:d;d;e=ZHe%dd>e=ZIe%d?dGd@dAdAe=eZJgdBZKdS)DzPyTorch InstructBLIP model.N) dataclass)AnyCallableOptionalUnion)nn)ACT2FN)GenerationMixin)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsBaseModelOutputWithPooling,BaseModelOutputWithPoolingAndCrossAttentions)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging torch_int) AutoModelAutoModelForCausalLMAutoModelForSeq2SeqLM)InstructBlipConfigInstructBlipQFormerConfigInstructBlipVisionConfigzQ Class defining the outputs of [`InstructBlipForConditionalGeneration`]. )Z custom_introc@seZdZUdZdZeeeje d<dZ eeeje d<dZ eeje d<dZ eeeje d<dZ eeeje d<eedd d ZdS) /InstructBlipForConditionalGenerationModelOutputa loss (`torch.FloatTensor`, *optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Language modeling loss from the language model. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head of the language model. vision_outputs (`BaseModelOutputWithPooling`): Outputs of the vision encoder. qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`): Outputs of the Q-Former (Querying Transformer). language_model_outputs (`CausalLMOutputWithPast` or `Seq2SeqLMOutput`): Outputs of the language model. Nlosslogitsvision_outputsqformer_outputslanguage_model_outputsreturncstfddDS)Nc3s,|]$}|dvr|n t|VqdS)r(r)r*N)getattrto_tuple).0kselfr/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/instructblip/modeling_instructblip.py Js zKInstructBlipForConditionalGenerationModelOutput.to_tuple..)tuplekeysr2r4r2r5r/Is z8InstructBlipForConditionalGenerationModelOutput.to_tuple)__name__ __module__ __qualname____doc__r&rr7torch FloatTensor__annotations__r'r(r)r*rr/r4r4r4r5r%.s  r%csReZdZedfdd ZejeeejdddZd ej e ejdd d Z Z S) InstructBlipVisionEmbeddingsconfigcst||_|j|_|j|_|j|_tt dd|j|_ tj d|j|j|jd|_ |j|jd|_|jd|_tt d|j|j|_dS)Nr!r)Z in_channelsZ out_channelsZ kernel_sizeZstrider)super__init__rB hidden_size embed_dimZ image_size patch_sizer Parameterr=Zrandnclass_embeddingConv2dpatch_embedding num_patches num_positionsposition_embeddingr3rB __class__r4r5rDTs  z%InstructBlipVisionEmbeddings.__init__) embeddingsheightwidthr,c Cs|jdd}|jjdd}tjs>||kr>||kr>|jS|jddddf}|jddddf}|jd}||j} ||j} t|d} |d| | |}|dddd}t j j || | fdd d }|dddd dd|}tj ||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 r!Ng?rrrZbicubicF)sizemodeZ align_cornersdim)shaperNr=Zjit is_tracingrGrreshapepermuter functionalZ interpolateviewcat) r3rRrSrTrLrMZclass_pos_embedZpatch_pos_embedrYZ new_heightZ new_widthZsqrt_num_positionsr4r4r5interpolate_pos_encodingfs(     z5InstructBlipVisionEmbeddings.interpolate_pos_encodingF) pixel_valuesrar,c Cs|j\}}}}|jjj}||j|d}|ddd}|j|dd|} t j | |gdd} |rx| | ||} n|j } | | ddd| dddf|} | S)Ndtyperr!rUrX)rZrKweightrdtoflatten transposerIexpandr=r`rarNrV) r3rbra batch_size_rSrTZ target_dtypeZ patch_embedsZ class_embedsrRrNr4r4r5forwards *z$InstructBlipVisionEmbeddings.forward)F) r9r:r;r$rDr=Tensorintrar>boolrl __classcell__r4r4rPr5r@Ss(r@)modulequerykeyvalueattention_maskscalingdropoutc Kspt||dd|}|dur(||}tjj|dd}tjj|||jd}t||} | dd} | |fS)NrUrX)ptrainingr!r) r=matmulrhrr^Zsoftmaxrxr{ contiguous) rrrsrtrurvrwrxkwargs attn_weights attn_outputr4r4r5eager_attention_forwards  rc sreZdZdZfddZejeedddZd eje eje e e eje eje e ejfd d d Z Z S) InstructBlipAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkrZtd|jd|jd|jd|_ d|_ |j |_ t j |jd|jdd|_|jrt t|j}t t|j}nd}d}|durt|tj|dd|f}t ||j_t |j|j|_dS) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFr)bias)Z requires_grad)rCrDrBrErFnum_attention_heads num_headshead_dim ValueErrorscaleZ is_causalattention_dropoutrLinearqkvqkv_biasrHr=zerosr`Z zeros_liker projection)r3rBZq_biasZv_biasrrPr4r5rDs0   zInstructBlipAttention.__init__)tensorseq_lenbszcCs ||||j|jddS)Nr!r)r_rrrhr})r3rrrr4r4r5_shapeszInstructBlipAttention._shapeNF) hidden_states head_maskoutput_attentionsr,cKs|\}}}||}|||d|j||jddddd}|d|d|d} } } t} |jjdkr|jjdkr|rt dn t |jj} | || | | fd |j sd n|j |j d |\} }| ||d } || } |r| |fn| d f}|S) z#Input shape: Batch x Time x Channelrrrr!eagerZsdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.Nrq)rvrxrwrU)rVrr\rr]rrBZ_attn_implementationloggerZ warning_oncerr{rrr}r)r3rrrr~rZtgt_lenrFZ mixed_qkvZ query_statesZ key_statesZ value_statesZattention_interfacerroutputsr4r4r5rls:       zInstructBlipAttention.forward)NF)r9r:r;r<rDr=rmrnrrror7rlrpr4r4rPr5rs rcs0eZdZfddZejejdddZZS)InstructBlipMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) rCrDrBr hidden_act activation_fnrrrEintermediate_sizefc1fc2rOrPr4r5rD s   zInstructBlipMLP.__init__rr,cCs"||}||}||}|Sr)rrrr3rr4r4r5rls   zInstructBlipMLP.forwardr9r:r;rDr=rmrlrpr4r4rPr5r s rcsFeZdZedfdd Zdejejeee ej dddZ Z S) InstructBlipEncoderLayerrAcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSNeps) rCrDrErFr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rOrPr4r5rDs    z!InstructBlipEncoderLayer.__init__F)rrvrr,cCsb|}||}|j|||d\}}||}|}||}||}||}|f}|r^||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrr)rrrr)r3rrvrZresidualrrr4r4r5rl$s      z InstructBlipEncoderLayer.forward)F) r9r:r;r"rDr=rmrror7r>rlrpr4r4rPr5rs rc@sBeZdZUeed<dZdZdZdZdZ dZ dZ gdZ ddZ dS)InstructBlipPreTrainedModelrBZblipT)InstructBlipQFormerEmbeddingsr%InstructBlipQFormerMultiHeadAttentionInstructBlipQFormerSelfOutputcCs|jj}t|tjtjfrD|jjjd|d|j dur|j j nt|tj rd|jjjd|dnvt|tj r|j j |jj dnNt|trtjj|jd|dtjj|jd|dnt|ttfr|jj dS)zInitialize the weightsrq)meanZstdN?)rBZinitializer_range isinstancerrrJredataZnormal_rZzero_ EmbeddingrZfill_r@initZ trunc_normal_rNrI$InstructBlipForConditionalGenerationInstructBlipModel query_tokens)r3rrfactorr4r4r5 _init_weights^s     z)InstructBlipPreTrainedModel._init_weightsN)r9r:r;r"r?Zbase_model_prefixZsupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_no_split_modulesrr4r4r4r5rKs rcsXeZdZdZedfdd Zd eejee ee ee e e e fdddZ ZS) InstructBlipEncodera Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`InstructBlipEncoderLayer`]. Args: config (`InstructBlipConfig`): The corresponding vision configuration for the `InstructBlipEncoder`. rAcs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr4)r)r0rkrAr4r5 z0InstructBlipEncoder.__init__..F) rCrDrBr ModuleListrangenum_hidden_layerslayersgradient_checkpointingrOrPrAr5rD}s  zInstructBlipEncoder.__init__N)rvroutput_hidden_states return_dictr,c Cs|dur |n|jj}|dur |n|jj}|dur4|n|jj}|rDdnd}|rPdnd}|}t|jD]>\} } |rx||f}| |||d} | d}|rb|| df}qb|r||f}|stdd|||fDSt|||dS) a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Embedded representation of the inputs. Should be float, not int tokens. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr4)rvrrr!css|]}|dur|VqdSrr4r0vr4r4r5r6rz.InstructBlipEncoder.forward..)last_hidden_stater attentions)rBrruse_return_dict enumeraterr7r ) r3 inputs_embedsrvrrrZencoder_statesZall_attentionsridxZ encoder_layer layer_outputsr4r4r5rls2    zInstructBlipEncoder.forward)NNNN)r9r:r;r<r"rDrr=rmrorr7r rlrpr4r4rPr5rss   rc speZdZUdZeed<edfdd Zed ee j ee ee ee e e e efdd d Zd d ZZS)InstructBlipVisionModelrbrBrAcsJt|||_|j}t||_t||_tj ||j d|_ | dSr) rCrDrBrEr@rRrencoderrrrpost_layernorm post_init)r3rBrFrPr4r5rDs   z InstructBlipVisionModel.__init__NF)rbrrrrar,c Cs|dur |n|jj}|dur |n|jj}|dur4|n|jj}|durLtd|j||d}|j||||d}|d}||}|dddddf} || } |s|| f|ddSt|| |j |j dS)Nz You have to specify pixel_values)ra)rrrrrr!)r pooler_outputrr) rBrrrrrRrrrrr) r3rbrrrrarencoder_outputsr pooled_outputr4r4r5rls2   zInstructBlipVisionModel.forwardcCs|jSr)rRr2r4r4r5get_input_embeddingssz,InstructBlipVisionModel.get_input_embeddings)NNNNF)r9r:r;main_input_namer$r?rDrrr=r>rorr7rrlrrpr4r4rPr5rs$   *rcsPeZdZdfdd ZddZddZdd Zd d Zd d ZdddZ Z S)rFcs$t||_|j|jdkr>t|ds>td|j|jf|j|_t|j|j|_|j|j|_ t |j|j |_ |rt |j |j |_t |j |j |_n$t |j|j |_t |j|j |_t |j|_t|dd|_|jdks|jdkr|j|_t d|jd |j|_d |_dS) NrZembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)position_embedding_typeabsolute relative_keyrelative_key_queryrr!F)rCrDrBrErhasattrrrnattention_head_size all_head_sizerrrsZencoder_hidden_sizertruDropoutZattention_probs_dropout_probrxr.rmax_position_embeddingsrdistance_embeddingsave_attentionr3rBis_cross_attentionrPr4r5rDs.  z.InstructBlipQFormerMultiHeadAttention.__init__cCs ||_dSrattn_gradients)r3rr4r4r5save_attn_gradientssz9InstructBlipQFormerMultiHeadAttention.save_attn_gradientscCs|jSrrr2r4r4r5get_attn_gradients!sz8InstructBlipQFormerMultiHeadAttention.get_attn_gradientscCs ||_dSr attention_map)r3rr4r4r5save_attention_map$sz8InstructBlipQFormerMultiHeadAttention.save_attention_mapcCs|jSrrr2r4r4r5get_attention_map'sz7InstructBlipQFormerMultiHeadAttention.get_attention_mapcCs6|dd|j|jf}|j|}|ddddS)NrUrrr!r)rVrrr_r])r3xZ new_x_shaper4r4r5transpose_for_scores*s z:InstructBlipQFormerMultiHeadAttention.transpose_for_scoresNcCs.|du}|r2|||}|||} |}n |||}|||} ||} || } t| |dd} |jdks|jdkrV|d} tj | tj |j d dd}tj | tj |j d dd}||}| ||jd}|j| jd}|jdkr"td| |}| |} n4|jdkrVtd| |}td ||}| ||} | t|j} | j}|dur~| |} tjdd | |}|r|jr||||j||}|dur||}t|| }|d d dd }|dd|jf}|j |}|r$||fn|f}|S)NrUryrrr!rddevicerczbhld,lrd->bhlrzbhrd,lrd->bhlrrXrrr)rrtrursr=r|rhrrVarangelongrr_rrrfrdZeinsummathsqrtrrZSoftmaxrr register_hookrrxr]r}r)r3rrvrencoder_hidden_statesencoder_attention_maskrrZ key_layerZ value_layerZmixed_query_layerZ query_layerZattention_scores seq_lengthZposition_ids_lZposition_ids_rZdistanceZpositional_embeddingZrelative_position_scoresZrelative_position_scores_queryZrelative_position_scores_keyZattention_scores_dtypeZattention_probsZattention_probs_droppedZ context_layerZnew_context_layer_shaperr4r4r5rl/sR               z-InstructBlipQFormerMultiHeadAttention.forward)F)NNNNF) r9r:r;rDrrrrrrlrpr4r4rPr5rsrcs4eZdZfddZejejejdddZZS)rcsBtt|j|j|_tj|j|jd|_t|j |_ dSr) rCrDrrrEdenserrrhidden_dropout_probrxrOrPr4r5rDs z&InstructBlipQFormerSelfOutput.__init__r input_tensorr,cCs&||}||}|||}|Srrrxrr3rrr4r4r5rls  z%InstructBlipQFormerSelfOutput.forwardrr4r4rPr5r~s rc sfeZdZd fdd ZddZd ejeejeejeejeejee e ejddd Z Z S) InstructBlipQFormerAttentionFcs,tt|||_t||_t|_dSr)rCrDr attentionroutputset pruned_headsrrPr4r5rDs   z%InstructBlipQFormerAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrr!rX)lenrrrrr rrsrtrurrrunion)r3headsindexr4r4r5 prune_headss z(InstructBlipQFormerAttention.prune_headsN)rrvrrrrr,c Cs<|j||||||d}||d|}|f|dd} | S)N)rrvrrrrrr!)rr) r3rrvrrrrZ self_outputsattention_outputrr4r4r5rls z$InstructBlipQFormerAttention.forward)F)NNNNF) r9r:r;rDrr=rmrr>ror7rlrpr4r4rPr5rs rcs0eZdZfddZejejdddZZS)InstructBlipQFormerIntermediatecsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr) rCrDrrrErrrrstrr intermediate_act_fnrOrPr4r5rDs   z(InstructBlipQFormerIntermediate.__init__rcCs||}||}|Sr)rrrr4r4r5rls  z'InstructBlipQFormerIntermediate.forwardrr4r4rPr5rs rcs4eZdZfddZejejejdddZZS)InstructBlipQFormerOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) rCrDrrrrErrrrrrxrOrPr4r5rDs z"InstructBlipQFormerOutput.__init__rcCs&||}||}|||}|Srrrr4r4r5rls  z!InstructBlipQFormerOutput.forwardrr4r4rPr5rs rcs6eZdZfddZd ddZdd Zd d ZZS) InstructBlipQFormerLayercs~t|j|_d|_t||_||_||jdkrLt|dd|_d|_ nd|_ t ||_ t ||_ t ||_t ||_dS)Nr!rT)rF)rCrDchunk_size_feed_forward seq_len_dimrr layer_idxZcross_attention_frequencycrossattentionhas_cross_attentionr intermediaterrintermediate_query output_query)r3rBrrPr4r5rDs     z!InstructBlipQFormerLayer.__init__NFrc Cs|j||||d}|d} |dd} |dkr| ddd|ddf} |jr|dur^td|j| |||||d} | d} | | dd} t|j|j|j| } | jd|krt|j |j|j| dd|dddf | j }t j | |gdd} nt|j |j|j| } | f| } | S)N)rvrrrr!z>encoder_hidden_states must be given for cross-attention layers)rvrrrrrX)rrrrrfeed_forward_chunk_queryrrrZfeed_forward_chunkrfrr=r`)r3rrvrrrr query_lengthZself_attention_outputsrrZquery_attention_outputZcross_attention_outputs layer_outputZlayer_output_textr4r4r5rls^   z InstructBlipQFormerLayer.forwardcCs||}|||}|Sr)rrr3rZintermediate_outputr!r4r4r5r/s  z+InstructBlipQFormerLayer.feed_forward_chunkcCs||}|||}|Sr)rrr"r4r4r5r4s  z1InstructBlipQFormerLayer.feed_forward_chunk_query)NNNNFr)r9r:r;rDrlrrrpr4r4rPr5rs  ?rcs&eZdZfddZd ddZZS) InstructBlipQFormerEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}t|qSr4)r)r0rrAr4r5r@rz7InstructBlipQFormerEncoder.__init__..F) rCrDrBrrrrlayerrrOrPrAr5rD<s  z#InstructBlipQFormerEncoder.__init__NFTrc  Cs|rdnd} |rdnd} |r dnd} t|jjD]|} |j| }|rL| |f} |dur\|| nd}|||||||| d}|d}|r0| |df} | dkr0|jr0| |df} q0|r| |f} |stdd|| | | fDSt|| | | dS) Nr4)rrr rr!rcss|]}|dur|VqdSrr4rr4r4r5r6osz5InstructBlipQFormerEncoder.forward..)rrrcross_attentions)rrBrr$rr7r)r3rrvrrrrrrr Zall_hidden_statesZall_self_attentionsZall_cross_attentionsiZ layer_moduleZlayer_head_maskrr4r4r5rlDsL        z"InstructBlipQFormerEncoder.forward)NNNNFFTr)r9r:r;rDrlrpr4r4rPr5r#;s r#cs*eZdZdZfddZdddZZS) rz;Construct the embeddings from word and position embeddings.csttj|j|j|jd|_t|j|j|_ tj |j|j d|_ t |j|_|jdt|jdddt|dd|_||_dS) N)Z padding_idxr position_ids)r!rUF) persistentrr)rCrDrr vocab_sizerEZ pad_token_idword_embeddingsrposition_embeddingsrr layernormrrrxZregister_bufferr=rrir.rrBrOrPr4r5rDs z&InstructBlipQFormerEmbeddings.__init__NrcCs|dur|d}nd}|dur@|jdd|||f}|dur||}|jdkrv|||j}||}|durtj ||fdd}n|}||j j j }| |}| |}|S)Nr!rrrX)rVr'cloner*rr+rfrr=r`r,rerdrx)r3 input_idsr' query_embedsZpast_key_values_lengthrrRr+r4r4r5rls"    z%InstructBlipQFormerEmbeddings.forward)NNNr)r9r:r;r<rDrlrpr4r4rPr5rs rcseZdZdZdZdZdZdZedfdd Z ddZ dd Z d d Z de jeee jee jd d dZde jee jee jee jee jee jee jeeeeeeeee jefd ddZZS)InstructBlipQFormerModelz Querying Transformer (Q-Former), used in InstructBLIP. Slightly modified from BLIP-2 as it also takes the instruction as input. FrAcs2t|||_t||_t||_|dSr)rCrDrBrrRr#rrrOrPr4r5rDs    z!InstructBlipQFormerModel.__init__cCs|jjSrrRr*r2r4r4r5rsz-InstructBlipQFormerModel.get_input_embeddingscCs ||j_dSrr1r3rur4r4r5set_input_embeddingssz-InstructBlipQFormerModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrr$rr)r3Zheads_to_pruner$r r4r4r5 _prune_headssz%InstructBlipQFormerModel._prune_heads)rv input_shaper has_queryr,cCs|dkr*|dddddddf}n>|dkrP|ddddddf}ntd|d|jd|j|jd}d|d }|S) a> Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`tuple[int]`): The shape of the input to the model. device: (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. rNrz!Wrong shape for input_ids (shape z) or attention_mask (shape )rcrg)rYrrZrfrd)r3rvr6rr7extended_attention_maskr4r4r5get_extended_attention_masks    z4InstructBlipQFormerModel.get_extended_attention_maskN) r.rvr'r/rrrrrrr,c  s|dur |njj}| dur | njj} | dur4| njj} |durT|durTtd|durf|jdnd} j|||d} | dd} | \}}| j}|durt j ||f|d} || |}|durNt |t r|d\}}}n|\}}}||f}t |t rfdd |D}n.|durBt j ||d}|}n |}nd}|jj}j| |||||| | | d }|d}|dddddf}| s||f|ddSt|||j|j|j|jd S) a encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`Cache` of length `config.n_layers` with each tuple having 4 tensors of: shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Nz7You have to specify query_embeds when input_ids is Noner!r)r.r'r/rU)rcsg|]}|qSr4)invert_attention_mask)r0maskr2r4r5rHrz4InstructBlipQFormerModel.forward..)rvrrrrrrr )rrpast_key_valuesrrr%)rBrrrrrZrRrVrr=onesr:rlistr;Z get_head_maskrrrr=rrr%)r3r.rvr'r/rrrrrrr Zembedding_outputr6rjrrr9Zencoder_batch_sizeZencoder_sequence_lengthrkZencoder_hidden_shapeZencoder_extended_attention_maskrZsequence_outputrr4r2r5rlsl        z InstructBlipQFormerModel.forward)F) NNNNNNNNN)r9r:r;r<rrrrr#rDrr3r5r=rmr7rnrror: LongTensorrr>rrrlrpr4r4rPr5r0sN   .r0z[ InstructBLIP base Model consisting of language model, qformer and vision encoder. cseZdZdZdgZedfdd ZddZdd Zd d Z d d Z e j e j dddZeede j e j ee j ee j ee j ee j ee j ee jeeeeeeeeeeeeeefdddZZS)rrbrrAcst|t|j|_ttd|j |j j |_ t |j |_t|j j |jj |_t|j|_|jjdur|j|jj|jjdur|j|jj|dSNr!)rCrDr vision_config vision_modelrrHr=rnum_query_tokensqformer_configrErr0qformerr text_configlanguage_projectionr from_configlanguage_modelrextend_keep_in_fp32_modulesrrOrPr4r5rD|s     zInstructBlipModel.__init__cCs |jSrrJrr2r4r4r5rsz&InstructBlipModel.get_input_embeddingscCs|j|dSrrJr3r2r4r4r5r3sz&InstructBlipModel.set_input_embeddingscCs(|jjs$|jj|jj_|jj|jj_dSrrBuse_decoder_only_language_modelrJZsharedrZ embed_tokensdecoderr2r4r4r5 _tie_weightsszInstructBlipModel._tie_weightscCsL|j}t|dkr2d|vr2tjdkr2tdt|jdrHd|jj _ dSz Some pre-processing hacks to make the model `accelerate` compatible. Check https://github.com/huggingface/transformers/pull/21707 for more details. r!rJaThe `language_model` is not in the `hf_device_map` dictionary and you are running your script in a multi-GPU environment. this may lead to unexpected behavior when using `accelerate`. Please pass a `device_map` that contains `language_model` to remove this warning. Please refer to https://github.com/huggingface/blog/blob/main/accelerate-large-models.md for more details on creating a `device_map` for large models._hf_hookTN hf_device_mapr r=cudaZ device_countrwarningrrJrTZio_same_devicer3rVr4r4r5_preprocess_accelerates" z(InstructBlipModel._preprocess_accelerater.rcCs`|dur8||tj|jjtj|jdk}|d}n ||jjk}|d | |j}|SzZ Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`. NrrU rr=rrBZimage_token_idrrallZ unsqueezeZ expand_asrfr3r.rspecial_image_maskr4r4r5get_placeholder_masks  z&InstructBlipModel.get_placeholder_maskNF)rbqformer_input_idsqformer_attention_maskr.rvdecoder_input_idsdecoder_attention_maskrrrrra use_cacher~r,c Ks| dur | n|jj} |j|| | | | d}|d}tj|ddtj|jd}|j |j ddd}tj|ddtj|jd}|durt |}tj ||gdd}|j |||||| | | d}|dddd|dddf}|dur|j|}|durt |}||}||j|j}|j||d }|||}|jjrr|jf||| | | | d |}n"|jf||||| | | | d |}t|||d S) a qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the language model. Input tokens can optionally be provided to serve as text prompt, which the language model can continue. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. N)rbrrrrarrUrr!rX)r.rvr/rrrrrrrrvrrrrf)rrvrdrerrrrfr-)rBrrCr=r>rVrrrrirZ ones_liker`rFrJrrHrfrdramasked_scatterrPr%)r3rbrbrcr.rvrdrerrrrrarfr~r( image_embedsimage_attention_maskrquery_attention_mask query_outputs query_outputlanguage_model_inputsr`rr4r4r5rls|1    $         zInstructBlipModel.forward) NNNNNNNNNFN)r9r:r;rrLr"rDrr3rRrZr=r@r>rarrrrmrorr rr7r%rlrpr4r4rPr5rssL ra InstructBLIP Model for generating text given an image and an optional text prompt. The model consists of a vision encoder, Querying Transformer (Q-Former) and a language model. One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the language model continue the prompt. Otherwise, the language model starts generating text from the [BOS] (beginning-of-sequence) token. cseZdZUeed<dZdZdgZedfdd Zdd Z d d Z d d Z e j dddZddZddZddZddZd'ejejeejeeeedddZejejddd Zeed(ejejeejeejeejeejeejeejeeeeeejeeeeeeeeee fd!d"d#Z!e"d)ejeejeejeejeejeejeejd$d%d&Z#Z$S)*rrBrbTrrAcst|t|j|_tt d|j |j j |_ t|j |_t|j j |jj |_|jrnt|j}n t|j}|jdur|j|j|jdur|j|j||_|dSrA)rCrDr _from_configrBrCrrHr=rrDrErErr0rFrrGrHrPrrIr rrKrLrJr)r3rBrJrPr4r5rDIs    z-InstructBlipForConditionalGeneration.__init__cCs |jSrrMr2r4r4r5rcsz9InstructBlipForConditionalGeneration.get_input_embeddingscCs|j|dSrrNr2r4r4r5r3fsz9InstructBlipForConditionalGeneration.set_input_embeddingscCs|j|dSr)rJset_output_embeddings)r3Znew_embeddingsr4r4r5rrisz:InstructBlipForConditionalGeneration.set_output_embeddingsr+cCs |jSr)rJget_output_embeddingsr2r4r4r5rslsz:InstructBlipForConditionalGeneration.get_output_embeddingscCs |jSr)rJ get_encoderr2r4r4r5rtosz0InstructBlipForConditionalGeneration.get_encodercCs |jSr)rJ get_decoderr2r4r4r5rursz0InstructBlipForConditionalGeneration.get_decodercCs(|jjs$|jj|jj_|jj|jj_dSrrOr2r4r4r5rRvsz1InstructBlipForConditionalGeneration._tie_weightscCsL|j}t|dkr2d|vr2tjdkr2tdt|jdrHd|jj _ dSrSrUrYr4r4r5rZ|s" z;InstructBlipForConditionalGeneration._preprocess_accelerateNF)rbrbrcrarcCs|j||dd}|d}tj|ddtj|jd}|j|jddd} tj| ddtj|jd} |durt |}tj | |gdd}|j ||| ||dd } | dddd| dddf} | | } |r| || fS| S) a$ Encodes images into continuous embeddings that can be forwarded to the language model. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. T)rbrarrNrUrr!rX)r.rvr/rrr) rCr=r>rVrrrrirZrir`rFrH)r3rbrbrcrarr(rkrlrrmrnrorpr4r4r5get_image_featuress2   $  z7InstructBlipForConditionalGeneration.get_image_featuresr[cCs`|dur8||tj|jjtj|jdk}|d}n ||jjk}|d | |j}|Sr\r]r_r4r4r5ras  z9InstructBlipForConditionalGeneration.get_placeholder_mask)rbrbrcr.rvrdrerrrlabelsrrarfr~r,c Ksd| dur | n|jj} |j|||| dd\}}}| s:|n|}| sJ|n|}|durb||}|durtt|}||j|j }|j ||d}| ||}|jj r|j f||| | | |d|}| r|jn|d}d}| durR|jf|| |jjjd|}nL|j f||||| | | | |d |}| r6|jn|d}| rJ|jn|d }t|||||d S) a+ qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the language model. Input tokens can optionally be provided to serve as text prompt, which the language model can continue. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration >>> import torch >>> from PIL import Image >>> import requests >>> model = InstructBlipForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> processor = InstructBlipProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> model.to(device) # doctest: +IGNORE_RESULT >>> url = "https://raw.githubusercontent.com/salesforce/LAVIS/main/docs/_static/Confusing-Pictures.jpg" >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB") >>> prompt = "What is unusual about this image?" >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... min_length=1, ... top_p=0.9, ... repetition_penalty=1.5, ... length_penalty=1.0, ... temperature=1, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) The unusual aspect of this image is that a man is ironing clothes on the back of a yellow SUV, which is parked in the middle of a busy city street. This is an unconventional approach to ironing clothes, as it requires the man to balance himself and his ironing equipment on top of the vehicle while navigating through traffic. Additionally, the presence of taxis and other vehicles in the scene further emphasizes the unusual nature of this situation. ```NTrbrcrarrgrhr)r'rwr)) rrvrdrerrrrwrfr!)r&r'r(r)r*)rBrrvr/rr=rirfrrdrarjrPrJr'Z loss_functionrGr)r&r%)r3rbrbrcr.rvrdrerrrrwrrarfr~rpr(rnr`rr'r&r4r4r5rlstX          z,InstructBlipForConditionalGeneration.forward)rbrbrcr.rvrrar,cKst|dr||jd} |j||||dd\} } } |dur|dur|jjg|jj} | |jjjg}t j |gt j |j d}| | d}||}|durt |}| |j |j} |j||d}||| }||d }|jjjs||d <|jjfi||}|S) a Overrides `generate` function to be able to use the model as a conditional generator. Args: pixel_values (`torch.FloatTensor` of shape (batch_size, num_channels, height, width)): Input images to be processed. qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt for the generation. attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Embedded representation of the inputs. Should be float, not int tokens. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the positional encoding of the image embeddings. Returns: captions (list): A list of strings of length batch_size * num_captions. rVrTrxNrr!rg)rrvr.)rrZrZrvrBZimage_token_indexrDrGZ bos_token_idr=rrrrepeatrrirfrdrarjrJZis_encoder_decodergenerate)r3rbrbrcr.rvrraZgenerate_kwargsrjrpr(rnZ image_tokensZ start_tokensr`inputsrr4r4r5rzgs6"         z-InstructBlipForConditionalGeneration.generate)NFF) NNNNNNNNNNFN)NNNNNF)%r9r:r;r"r?rrrLrDrr3rrrModulersrtrurRrZr=r>r@rrorvrarrrrrr7r%rlZno_gradrzrpr4r4rPr5r9s  1 r)r0rrrr)rq)Lr<r dataclassesrtypingrrrrr=Ztorch.utils.checkpointrZ activationsr Z generationr Zmodeling_flash_attention_utilsr Zmodeling_layersr Zmodeling_outputsr rrrZmodeling_utilsrrZprocessing_utilsrZ pytorch_utilsrrrutilsrrrrrrautorrr Zconfiguration_instructblipr"r#r$Z get_loggerr9rr%r|r@rmfloatrrrrrrrrrrrrrr#rr0rr__all__r4r4r4r5s~         R U0'P>}0`F3@B n