a h @sdZddlmZddlmZmZmZmZddlZddl m m Z ddl Zddlm Z ddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZmZm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0m1Z1e'r>ddl2m3Z3ddl4m5Z5e(6e7Z8ee%ddGdddeZ9ee%ddGdddeZ:dCd d!Z;gfd"d#ZGd&d'd'e j?Z@Gd(d)d)e jAZBGd*d+d+ej jAZCd,d-ZDdDd.d/ZEGd0d1d1e jAZFdEe jAejGejGejGeejGeHeHd3d4d5ZIGd6d7d7e jAZJGd8d9d9eZKGd:d;d;eZLe%Gdd?d?eMZNGd@dAdAeMeZOgdBZPdS)FzPyTorch Idefics model.) dataclass)AnyCallableOptionalUnionN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs)GradientCheckpointingLayer) ModelOutput)ALL_ATTENTION_FUNCTIONSPretrainedConfigPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availablelogging)deprecate_kwarg) IdeficsConfig)IdeficsPerceiverResampler)IdeficsVisionEmbeddingsIdeficsVisionTransformer) BlockMask)make_flex_block_causal_maskz{ Base class for Idefics model's outputs that may also contain a past key/values (to speed up sequential decoding). )Z custom_introc@seZdZUdZdZeejed<dZ ee e ejed<dZ ee ejed<dZ ee ejed<dZ ee ejed<dS)IdeficsBaseModelOutputWithPastal last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and optionally if `config.is_encoder_decoder=True` 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if `config.is_encoder_decoder=True` in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder, and optionally by the perceiver Nlast_hidden_statepast_key_values hidden_states attentionsimage_hidden_states)__name__ __module__ __qualname____doc__r#rtorch FloatTensor__annotations__r$tupler%r&r'r0r0h/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/idefics/modeling_idefics.pyr"7s r"zS Base class for Idefics causal language model (or autoregressive) outputs. c@seZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZee ejed<dS) IdeficsCausalLMOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder, and optionally by the perceiver Nlosslogitsr$r%r&r')r(r)r*r+r3rr,r-r.r4r$listr%r/r&r'r0r0r0r1r2[s r2FcKs:t|jdddd|d|j}|d|}|d|d<|d|d<|d|d<|d|d<d|vr|d}|d||d<|dur|d||d <|ddur|dd||d<|ddur|dd||d<nF|ddur|dd||d<n"|ddur2|dd||d<||fS) Nrr pixel_valuesimage_encoder_embeddingsperceiver_embeddingsimage_attention_masktoken_type_idsattention_mask) r,arangeshapeviewrepeattodeviceZ index_selectget) input_idsZ expand_sizeis_encoder_decoderr<Zencoder_outputs model_kwargsZexpanded_return_idxr;r0r0r1expand_inputs_for_generation|s6 ,    rGcsftjtjtjdfdd|D}|D]4|rVtfdd|DrVdq,dq,|S)N) LayerNormLinear Embeddingcsg|] }|qSr0r0).0m)mappingr0r1 z freeze_model..c3s|]}t|VqdSN) isinstance)rKt)moduler0r1 rOzfreeze_model..TF)rrHrIrJmodulesanyrequires_grad_)modelmodule_exceptionsZmodule_exceptions_mappedr0)rMrSr1 freeze_models   rZcsDeZdZdZd eeddfdd ZddZed d d Z Z S) IdeficsDecoupledEmbeddinga Implements a decoupling of parameters to allow freezing (or not) a subset of the embeddings. In practise, the regular `weight` can be trained or frozen (i.e. `partially_freeze=True`), and if `num_additional_embeddings` > 0, then it will create `num_additional_embeddings` additional parameters that are always trained. If `num_additional_embeddings=0`, then the module defaults back to the regular behavior of `nn.Embedding`. FN)partially_freezereturnc  s|dur$||kr$td|d|tjf|||||d|||_||_||_||_|rj|jd|jdkrt j |j|||d|_ dS)a) Args: num_embeddings (`int`): Size of the dictionary of embeddings num_additional_embeddings (`int`): Number of additional embeddings. Only useful when you `partially_freeze=True`. embedding_dim (`int`): The size of each embedding vector partially_freeze: (`bool`, *optional*, defaults to `False`): If `True`, the regular `weight` will be frozen. `additional_weight` is never frozen. padding_idx (`int`, *optional*): The padding index (needs to be less than num_embeddings) Note: there are a lot of other parameters to initialize a standard `nn.Embedding` such as `padding_idx`, `max_norm` or `norm_type`. We are not supporting these. Nz/padding_idx must be within num_embeddings. Got z and )num_embeddings embedding_dimrBdtype padding_idxFr)r^r_rBr`) ValueErrorsuper__init__r^ranum_additional_embeddingsr\weightrWrrJadditional_embedding) selfr^rer_r\rBr`rakwargs __class__r0r1rds0  z"IdeficsDecoupledEmbedding.__init__cCsj|jdkrt||jS|}t||jk}||}|||j}d||<t||j}|||<|S)a we have 2 embeddings, with different indices - one pretrained self.weight and another self.additional_embedding.weight that is being trained. in order to make a lookup of the input ids, we: 1. find out the indices of the entries belonging to the 2nd embedding 2. extract those values while subtracting the size of the first embedding (num_embeddings), since the 2nd embedding starts from 0 and not num_embeddings 3. perform the 2nd embedding lookup 4. now we handle the 1st embedding, we overwrite indices belonging to the 2nd embedding with a padding index 5. perform the 1st embedding lookup 6. now we overwrite the values in the 1st embedding lookup with the values of the 2nd embedding lookup note: for the 1st embedding lookup we could have looked up only the low indices and not do the padding, but then we have to create a new tensor and populate it with 2 tensors that are spread out across various indices - i.e. not a simple concat - I haven't benchmarked the complex case if it's any faster, given that seqlens are usually relatively short it's probably not faster or if faster not by much - but might be a good idea to measure. r) reFZ embeddingrfcloner,wherer^rg)rhrDZadditional_vocab_indicesZinput_ids_additional_vocabZadditional_embeddingsZ full_vectorr0r0r1forwards z!IdeficsDecoupledEmbedding.forwardr]cCs$d|jd|jd|jd|jS)Nznum_embeddings=z, num_additional_embeddings=z, embedding_dim=, partially_freeze=)r^rer_r\rhr0r0r1 extra_reprsz$IdeficsDecoupledEmbedding.extra_repr)FNNN) r(r)r*r+rboolrdrostrrs __classcell__r0r0rjr1r[s 5'r[csTeZdZdZdeeeeeddfdd Zejejdd d Z e d d d Z Z S)IdeficsDecoupledLineara Implements a decoupling of parameters to allow freezing (or not) a subset of the parameters. In practise, the regular `weight` can be trained or frozen (i.e. `partially_freeze=True`), and if `out_additional_features` > 0, then it will create `out_additional_features * in_features` additional parameters that are always trained. If `out_additional_features=0`, then the module defaults back to the regular behavior of `nn.Linear`. rTN) in_features out_featuresout_additional_featuresbiasr\r]csnt|||||||_||_||_||_|rL|jd|rL|jd|dkrjt j |||||d|_ dS)aG out_additional_features: int. Number of additional trainable dimensions. Only makes sense when `partially_freeze=True`. partially_freeze: bool. If True, the regular `weight` will be frozen and extra parameters (if any) will be trainable. If False, default to the regular behavior of nn.Linear. Fr)rxryr{rBr`N) rcrdrzr\rxryrfrWr{rrI additional_fc)rhrxryrzr{r\rBr`rjr0r1rd*s"  zIdeficsDecoupledLinear.__init__)inputr]cCs:t||j|j}|jdkr6||}t||fd}|S)Nrr6)rlZlinearrfr{rzr|r,cat)rhr}outputZadditional_featuresr0r0r1roNs   zIdeficsDecoupledLinear.forwardrpc Cs0d|jd|jd|jd|jdud|j S)z=Overwriting `nn.Linear.extra_repr` to include new parameters.z in_features=z, out_features=z, out_additional_features=z, bias=Nrqrxryrzr{r\rrr0r0r1rsWsz!IdeficsDecoupledLinear.extra_repr)rTTNN) r(r)r*r+intrtrdr,Tensorrorursrvr0r0rjr1rw!s  $ rwcs.eZdZdfdd ZddZddZZS) IdeficsRMSNormư>cs&ttt||_||_dS)z= IdeficsRMSNorm is equivalent to T5LayerNorm N)rcrdr Parameterr,onesrfvariance_epsilon)rh hidden_sizeepsrjr0r1rd^s zIdeficsRMSNorm.__init__cCs\|tjdjddd}|t||j}|jjtj tj fvrR||jj}|j|S)Nr6T)Zkeepdim) rAr,float32powmeanZrsqrtrrfr`Zfloat16Zbfloat16)rhr%Zvariancer0r0r1rofs zIdeficsRMSNorm.forwardcCst|jjd|jS)Nz, eps=)r/rfr>rrrr0r0r1rspszIdeficsRMSNorm.extra_repr)r)r(r)r*rdrorsrvr0r0rjr1r]s rcs0eZdZd fdd ZddZd dd ZZS) IdeficsEmbedding'Ncszt||_||_||_d|jtjd|jdtjdj|tj d|j}|j d|dd|j ||j j td dS) N?rrr`rBr`inv_freqF persistentseq_lenrBr`)rcrddimmax_position_embeddingsbaser,r=int64rAfloatregister_buffer_set_cos_sin_cacherrBZget_default_dtype)rhrrrrBrrjr0r1rdvs &zIdeficsEmbedding.__init__cCs|||_tj|j|tjd|j}td||j}tj||fdd}|jd| |dd|jd| |dddS) Nrzi,j->ijr6r cos_cachedFr sin_cached) max_seq_len_cachedr,r=rZtype_asrZeinsumr~rcosrAsin)rhrrBr`rRZfreqsZembr0r0r1rs z#IdeficsEmbedding._set_cos_sin_cachecCsN||jkr|j||j|jd|jd|j|jd|jd|j|jdfS)Nrr)rrrBr`rrAr)rhxrr0r0r1ros  zIdeficsEmbedding.forward)rrN)N)r(r)r*rdrrorvr0r0rjr1rus rcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr6rr)r>r,r~)rx1Zx2r0r0r1 rotate_halfsrcCsL|||}|||}||t||}||t||}||fS)anApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`): The position indices of the tokens corresponding to the query and key tensors. For example, this can be used to pass offsetted position ids when working with a KV-cache. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkrr position_idsZ unsqueeze_dimZq_embedZk_embedr0r0r1apply_rotary_pos_embs rcs.eZdZeeedfdd ZddZZS) IdeficsMLPrintermediate_size hidden_actcsNttj||dd|_tj||dd|_tj||dd|_t||_dS)NFr{) rcrdrrI gate_proj down_projup_projr act_fn)rhrrrrjr0r1rds  zIdeficsMLP.__init__cCs ||||||SrP)rrrr)rhrr0r0r1roszIdeficsMLP.forward)r(r)r*rrurdrorvr0r0rjr1rs  r)rSquerykeyvaluer<scalingdropoutc Ks|t||dd|}|dur(||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)Nr6)rr`ptrainingrr) r,matmul transposer functionalZsoftmaxrrAr`rr contiguous) rSrrrr<rrri attn_weights attn_outputr0r0r1eager_attention_forwards  rcseZdZdZdeeeeeeeedfdd Z e j eedd d Z e d d d dde j ee j ee j ee jeee j eeee jee j ee j eee j fd ddZZS)IdeficsAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrFN)r num_headsris_cross_attentionconfigqk_layer_norms layer_idxc st||_||_||_|||_||_d|_|jd|_||_ |durbt d|j j d|j||jkrtd|jd|d||_ttjdstd |jrt|jd s|jn|jj}tj|j||jd d |_tj|||jd d |_tj|||jd d |_nNtj|j||jd d |_tj|j||jd d |_tj|j||jd d |_tj||j|d d |_t|j|_||_|jrt|j|jd |_t|j|jd |_ dS)NTgzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.z?hidden_size must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).Zscaled_dot_product_attentionz)this model requires pytorch 2.0 or higher embed_dimFrr)!rcrdrrrhead_dimrZ is_causalrrlogger warning_oncerkr(rbrhasattrrr vision_configrrIq_projk_projv_projo_projr rotary_embrr rms_norm_eps q_layer_norm k_layer_norm) rhrrrrrrrZ kv_input_dimrjr0r1rdsz       zIdeficsAttention.__init__)tensorrbszcCs ||||j|jddS)Nrr)r?rrrr)rhrrrr0r0r1_shape>szIdeficsAttention._shapepast_key_valuer$4.58new_nameversion) r%key_value_statesr<rr$output_attentions use_cachecache_positionr]c Ks |jp |du} |\} } } ||| | |j|jdd}| s||| | |j|jdd}||| | |j|jdd}nR|\} }} ||| ||j|jdd}||| ||j|jdd}|j d}|dur||d7}| s*|j |t || d\}}t |||||\}}|durRd|i}| |||j|\}}|jrn||}||}t}|jjdkr|jjdkr|rtd n t|jj}||||||f|jsd n|j|jd | \}}|| | d }||}|rd}||fS) Nrrrr)rreagersdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r)rrr6)rsizerr?rrrrrr>rmaxrupdaterrrrrr_attn_implementationrrrrrrreshaperr)rhr%rr<rr$rrrrirrZq_len_Z query_statesZ key_statesZ value_statesZkv_lenZ kv_seq_lenrrZ cache_kwargsZattention_interfacerrr0r0r1roAs\""$"         zIdeficsAttention.forward)rFNFN)NNNNFFN)r(r)r*r+rrrtrrrdr,rrr LongTensorr/rorvr0r0rjr1rsFQ  rcseZdZd eeedfdd Zedddddej eej eej ee ej ee ee eej e ej ee ej ej ffd d d ZZS)IdeficsDecoderLayerNrrcsrt|j|_t|j|j|j||d|_t|j|j|j d|_ t |j|j d|_ t |j|j d|_|j|_dS)N)rrrrrrr)rcrdrrnum_attention_headsr self_attnrrrmlprrinput_layernormpost_attention_layernormrhrrrjr0r1rds" zIdeficsDecoderLayer.__init__rr$rrF)r%r<rr$rrrr]c Ks|} ||}|jf|||||||d|\}} tjj||j|jd}| |}|} ||}||}tjj||j|jd}| |}|f} |r| | f7} | S)a^ Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states )r%r<rr$rrrr)rrrrrrrr) rhr%r<rr$rrrriresidualself_attn_weightsoutputsr0r0r1ros0     zIdeficsDecoderLayer.forward)N)NNNFFN)r(r)r*rrrrdrr,rrr/rtr-rorvr0r0rjr1rs$  rcseZdZd eeedfdd Zedddddej eej eej eej eej ee ee ee ej e ej ee ej ej ffd d d Z ZS)IdeficsGatedCrossAttentionLayerNrc st|j|_t|j|jd|j||j|d|_t|j|j |j d|_ t |j|j d|_t |j|j d|_|j|_t|_t|_|jdkr|jdkrttdd|j|_ttdd|j|_nD|jdkrttd|_ttd|_ntd |jd nV|jd kr|jdkrbttdd|j|_ttdd|j|_nD|jdkrttd|_ttd|_ntd |jd n|jd vr\|jdkrttjd |jdd|jfd|_ttjd |jdd|jfd|_nT|jdkrHttjd |jdd|_ttjd |jdd|_ntd |jd nt d|jdt!|drt!|dstddS)NT)rrrrrrrrrzerosZvectorrrz Unknown value for `alpha_type` ()r>gaussianrandomnormalr)rstdrzAlpha initialization scheme z not yet implemented!alpha_cross_attn alpha_densez+Alpha parameters not initialized correctly!)"rcrdrrrrr cross_attnrrrrrrrrrrZTanhact_cross_attn act_densealpha_initializerZ alpha_typerr,rrrrbrralphas_initializer_rangeNotImplementedErrorrrrjr0r1rdsl             z(IdeficsGatedCrossAttentionLayer.__init__rr$rrF) r%r<r'r:cross_attention_gaterrr$r]c Ks|durtd|dur td|dur0td|} ||}|jf||||d| \}} tjj||j|jd}| |dkdddddfd}| | |j |}|} | |}| |}tjj||j|jd}| ||j|}|f} |r| | f7} | S) a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. image_attention_mask (`torch.FloatTensor`, *optional*): image attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. cross_attention_gate (`torch.FloatTensor`, *optional*): gate of size `(batch, seq_len)` used to zero-out cross-attention output for tokens attending no images. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states Nzt`image_hidden_states` is required for Idefics cross attention module which are visual features to be conditioned on.z`cross_attention_gate` is required for Idefics cross attention module to zero-out the cross-attention hidden_states attending to no images.zMPast key value states are not implemented for Idefics cross attention module.)r%rr<rrrr)rbr rrrrrrr masked_fillrrrrrr) rhr%r<r'r:r rrr$rirrrr0r0r1ro"s@  "   z'IdeficsGatedCrossAttentionLayer.forward)N)NNNNFFN)r(r)r*rrrrdrr,rrtr/r-rorvr0r0rjr1rs(B  rc@s>eZdZUeed<dZdZddgZdZdZ dZ dZ ddZ d S) IdeficsPreTrainedModelrrXTrrFcCs|jj}t|tjtjfrF|jjjd|d|j durB|j j nLt|tj r|jjjd|d|j dur|jj|j  nt|tj r|jjd|j j nt|tr|jjdnt|tr|jjnt|trz|jjdkr|jj |jj nf|jjdkr>|jjd|jjdn:|jjdvr|jjjd|jjd|jjjd|jjdnt|tr|jjdS)Nr)rrrrr>rrr)rZinitializer_rangerQrrIZConv2drfdataZnormal_r{Zzero_rJrarHZfill_rrZclass_embeddingrr rrr rZlatents)rhrSrr0r0r1 _init_weightszs8         z$IdeficsPreTrainedModel._init_weightsN) r(r)r*rr.Zbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_supports_sdpaZ_supports_flash_attnZ_can_compile_fullgraphZ_supports_attention_backendrr0r0r0r1rns rcs"eZdZdZedfdd ZdddZgfdd Zgfd d Ze e de e j e e je e j e ee e je e je e je e je e je ee ee ee ee ee e j eeeeefd ddZdee jdfe je jeedddZee jeee je jedddZZS) IdeficsModelz Transformer decoder consisting of `config.num_hidden_layers` layers. Each layer is a [`IdeficsDecoderLayer`] Args: config: IdeficsConfig rcst|_j|_j|_tjjjj |jd|_ j j |_ j |_ j |j _ tj |_jrj}tj j|j|j|j|j|_tfddtjD|_j|_j|j}tfddt|D|_d|_ t!jj"d|_#|$|%dS)N)r^rer_r\racsg|]}t|dqS)r)rrKirr0r1rNrOz)IdeficsModel.__init__..csg|]}t|dqSr)rrrr0r1rNrOFr)&rcrdrZ pad_token_idra vocab_sizer[additional_vocab_sizerfreeze_text_layers embed_tokensrZ image_sizerr vision_model use_resamplerperceiver_configrrZresampler_depthZresampler_n_headsZresampler_head_dimZresampler_n_latentsperceiver_resamplerrZ ModuleListrangeZnum_hidden_layerslayerscross_layer_intervalgated_cross_attn_layersgradient_checkpointingrrnorm post_initfreeze_relevant_params)rhrrZnum_cross_layersrjrr1rdsJ      zIdeficsModel.__init__NcCs:|dur|j}|jr ||j|jr6t|j|jddSN)rY)rrZfreeze_text_module_exceptionsfreeze_vision_layersrZrZfreeze_vision_module_exceptions)rhrr0r0r1r%s  z#IdeficsModel.freeze_relevant_paramscCs"|j|jfD]}t||dq dSr&)rr#rZ)rhrYrSr0r0r1rszIdeficsModel.freeze_text_layerscCst|j|ddSr&)rZr)rhrYr0r0r1r'sz!IdeficsModel.freeze_vision_layersF)rDr<rr$ inputs_embedsr7r8r9r:rroutput_hidden_statesinterpolate_pos_encoding return_dictrrir]c( Ks|dur|jn|j}| dur | n|jj} | dur4| n|jj} | durH| n|jj} |dur\|n|jj}|du|duAr|td|jr|jr| rt dd} |dur| |}t |t dtfstd| r|durt|jd}|j\}}}|dur|nd}||}|dur,tj|||jd|jd }|durz|durz|d d}||dkd|dd| df}n|dur|d}td d |||fDd krtdn|dur|j|j|d}|jdd \}}|j||g|jd dR}|j|| dj}n<|durT|\}}}}|j|j|d}|||||}|jj r|dur|!|}|d|d }}n|\}}}}|}n*|dur|d|d }}ntd|||||}| d}| d } | "ddd|} | ||||} |durV|\}}}||f}| durJtj#||d } |$| } nd} | dkj%d dj|jdj&dd|}|durtj#||ftj'|jd}|(||||| }|} | rdnd}!| rdnd}"t)|j*D]\}#}$| r|!| f7}!|#|j+dkr@|j,|#|j+}%|%| ||f| || | dd|}&|&d} |$| f|||| | |d|}'|'d} | r|"|'df7}"q|-| } | r|!| f7}!|||||}t.| ||!|"|dS)ab image_encoder_embeddings (`torch.FloatTensor`, *optional*): The output of the image encoder. perceiver_embeddings (`torch.FloatTensor`, *optional*): The output of the perceiver resampler. image_attention_mask (`torch.LongTensor`, *optional*): The attention mask for the image encoder. Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FzBThe `past_key_values` should be either a `Cache` object or `None`.rrrrBr6cSsg|] }|duqSrPr0)rKrr0r0r1rN3rOz(IdeficsModel.forward..rz_Exactly 1 of pixel_values, image_encoder_embeddings or perceiver_embeddings has to be not-None.)r`rB)r7r*zBIf `perceiver_embeddings` are passed, use_resampler should be Truerrrr0)r:r rrr$)r<rr$rrr)r#r$r%r&r')/rBrrr)ruse_return_dictrbr"rrrrrQtyper r r>get_seq_lengthr,r=longZcumsumZ masked_fill_rsumrAr`rr?rr#rrrr@rZinvert_attention_maskrVZsqueezert_update_causal_mask enumeraterr r!r#r")(rhrDr<rr$r(r7r8r9r:rrr)r*r+rrirB batch_size seq_lengthrpast_key_values_lengthZseq_length_with_pastZ num_imagesr'Z image_seq_lenZimage_hidden_sizeZ text_seq_lenZimage_batch_sizeZimage_sequence_lengthZimage_hidden_shaper r%Zall_hidden_statesZall_self_attnsidxZ decoder_layerZcross_attn_blockrZ layer_outputsr0r0r1ros        $           "        zIdeficsModel.forwardr )r< input_tensorrr$rc CsB|jjdkr(|dur$|dkr$|SdS|jjdkrLt|tjrHt|}|S|dur\|nd}|durn|jnd}|jjdkr|s|st j ||||j drdS|j }|j d} |r|} n"t|tjr|j d n || d} |j|| | |||j dd } |jjdkr>|dur>|jjd vr>|s>t|j} t | | } | S) NZflash_attention_2rZflex_attentionrFr)r(r6Z is_trainingrr6)sequence_length target_lengthr`rr4)cudaZxpuZnpu)rrrVrQr,rr!r/Zis_compileabler Z_ignore_causal_mask_sdparr`r>Zget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrBr.finfominZ_unmask_unattended) rhr<r8rr$rZpast_seen_tokensZusing_compilable_cacher`r9r: causal_mask min_dtyper0r0r1r2sZ           z IdeficsModel._update_causal_mask)r<r9r:r`rr4c KsF|dur|dkr|}n&t|j}tj||f|||jd}|dkrVtj|dd}|tj||jd|ddk9}|ddddddf |ddd}|durB| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N)Z fill_valuer`rBr)Zdiagonalr,r6r)rr,r=r>fullrBZtriur=rexpandrmr>rAr ) r<r9r:r`rr4rir?r@Z mask_lengthZ padding_maskr0r0r1r<s*  $  6  zBIdeficsModel._prepare_4d_causal_attention_mask_with_cache_position)N)NNNNNNNNNNNNFNN)F)r(r)r*r+rrdr%rr'rrrr,rrr r-rtrrrr/r"ror2 staticmethodrr`r<rvr0r0rjr1rst2   O  DrcseZdZddgZdfdd ZddZeedee j ee j ee j ee ee j ee j ee j ee j ee j ee j eeeeeeeeeeee j eeeeefd d d Zdfd d Zdeeeefeeeefdfdd ZZS)IdeficsForVisionText2Textzmodel.embed_tokens.weightzlm_head.weightNcs>t|t||_t|j|j|jd|jd|_ | dS)NFr) rcrdrrXrwrrrZfreeze_lm_headlm_headr$)rhrrrjr0r1rd2s   z"IdeficsForVisionText2Text.__init__cCs|}|}t|jddrL|j|_|jdkrL|j|jks@J|jj|j_t |drt |dr|j |_ t |drt |dr|j|_dS) z Overwrite `transformers.modeling_utils.PreTrainedModel.tie_weights` to handle the case of IdeficsDecoupledLinear and IdeficsDecoupledEmbedding. Ztie_word_embeddingsTrryr^rzreN) Zget_output_embeddingsZget_input_embeddingsgetattrrrfrerzrgr|rr^ry)rhZoutput_embeddingsZinput_embeddingsr0r0r1 tie_weightsAs   z%IdeficsForVisionText2Text.tie_weightsF)rDr<rr$r(r7r8r9r:labelsrrr)r*r+rrir]cKs| dur | n|jj} | dur | n|jj} |dur4|n|jj}|jf||||||||| | | | |d|d|}|d}||}d}| dur|jf|| |jjd|}t|||j |j |j |j dS)aC image_encoder_embeddings (`torch.FloatTensor`, *optional*): The output of the image encoder. perceiver_embeddings (`torch.FloatTensor`, *optional*): The output of the perceiver resampler. image_attention_mask (`torch.LongTensor`, *optional*): The attention mask for the image encoder. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoProcessor, IdeficsForVisionText2Text >>> model = IdeficsForVisionText2Text.from_pretrained("HuggingFaceM4/idefics-9b") >>> processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics-9b") >>> dogs_image_url_1 = "https://huggingface.co/datasets/hf-internal-testing/fixtures_nlvr2/raw/main/image1.jpeg" >>> dogs_image_url_2 = "https://huggingface.co/datasets/hf-internal-testing/fixtures_nlvr2/raw/main/image2.jpeg" >>> prompts = [ ... [ ... "User:", ... dogs_image_url_1, ... "Describe this image.\nAssistant: An image of two dogs.\n", ... "User:", ... dogs_image_url_2, ... "Describe this image.\nAssistant:", ... ] ... ] >>> inputs = processor(prompts, return_tensors="pt") >>> generate_ids = model.generate(**inputs, max_new_tokens=6) >>> processor.batch_decode(generate_ids, skip_special_tokens=True) ```NT)rDr<rr$r(r7r8r9r:rrr)r*r+rr)r4rIr)r3r4r$r%r&r') rrr)r-rXrFZ loss_functionrr2r$r%r&r')rhrDr<rr$r(r7r8r9r:rIrrr)r*r+rrirr%r4r3r0r0r1roVsH< z!IdeficsForVisionText2Text.forwardc  si} |dur(|jjr|| d<q0|| d<n|| d<| dd| d<tj|f|||||| | d| | } | dur|dur| djd}| dd| df| d <| S) Nr9r8r7r*F)r$r<r(rrrr:rDrr:)rrpoprcprepare_inputs_for_generationr>)rhrDr<rr(r$rr7r'r:rriZ images_kwargsZ model_inputsr5rjr0r1rKs4     z7IdeficsForVisionText2Text.prepare_inputs_for_generation)rrFrEr]c s~tj|||fi|}d|vrp|d}|dddddfd}|ddrZ||d<ntj||gdd|d<|j|d<|S)Nr:r6rrTrr')rc#_update_model_kwargs_for_generationrrCr,r~r')rhrrFrErir:Z last_maskrjr0r1rLs   z=IdeficsForVisionText2Text._update_model_kwargs_for_generation)N)NNNNNNNNNNNNNFNN) NNNNNNNNN)F)r(r)r*Z_tied_weights_keysrdrHrrrr,rrr r-rtrrrr/r2rorKrdictrurrLrvr0r0rjr1rE/st g1  rE)rErr)rFNN)r)r)Qr+ dataclassesrtypingrrrrr,Ztorch.nn.functionalrrrlZtorch.utils.checkpointZ activationsr Z cache_utilsr r Z generationr Zmodeling_attn_mask_utilsr Zmodeling_flash_attention_utilsrZmodeling_layersrZmodeling_outputsrZmodeling_utilsrrrZprocessing_utilsrutilsrrrrrZutils.deprecationrZconfiguration_ideficsrZ perceiverrZvisionrrZ!torch.nn.attention.flex_attentionr Zintegrations.flex_attentionr!Z get_loggerr(rr"r2rGrZrJr[rIrwModulerrrrrrrrrrrrrrE__all__r0r0r0r1s                - i<'  'O-V