a h4 @srdZddlZddlmZmZmZddlZddlmm Z ddl ZddlmZddl m Z ddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZmZmZdd lmZmZddl m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,ddl-m.Z.m/Z/ddl0m1Z1m2Z2m3Z3e)rFddl4m5Z5ddl6m7Z7e*8e9Z:ej;eej;ej;fdddZ?ej;ed?d?eZSGd@dAdAejBZTe'GdBdCdCe"ZUe'dDdEGdFdGdGeUZVe'dHdEGdIdJdJeUZWe'dKdEGdLdMdMeUeZXe'dNdEGdOdPdPeUZYe'dQdEGdRdSdSeUeZZgdTZ[dS)WzPyTorch Mllama model.N)CallableOptionalUnion)nn)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availablelogging)deprecate_kwarg)OutputRecordercheck_model_inputs) MllamaConfigMllamaTextConfigMllamaVisionConfig) BlockMask)make_flex_block_causal_mask)cross_attention_masknum_vision_tokensdtypereturnc Cs|j^}}}|j|dd}|||d}|d}d||}||tjt|j }t|j }||kj dd |d}||9}||fS)Nrdimr?).N) shapeZrepeat_interleaveview unsqueezeto masked_filltorchboolfinfominanyZtype_as) r$r%r& batch_sizeZtext_total_length_Zinverted_cross_attn_maskZnegative_inf_valuefull_text_row_masked_out_maskr9f/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/mllama/modeling_mllama.py_prepare_cross_attention_mask1s   r;)aspect_ratio_mask num_patches target_lengthr&r'cCs|j\}}|||dd|}|dd|d}||}d|dddd| df<d|}||||d}||ddt|j}| d}|S)Nrrr*) r,r-r/repeatreshape transposer1r3r4r.)r<r=r>r&r6 max_num_tilesattention_maskZ pad_patchesr9r9r:$_prepare_aspect_ratio_attention_maskMs  rEcs>eZdZdeedfdd ZejejejdddZZ S) %MllamaPrecomputedAspectRatioEmbeddingTconfigis_gatedcs^t|j|_|j|_|j|_||_t|jd|j|j|_|rZt t d|_ dS)Nr) super__init__rC hidden_sizemax_aspect_ratio_idrIr Embedding embedding Parameterr1zerosgateselfrHrI __class__r9r:rKis z.MllamaPrecomputedAspectRatioEmbedding.__init__ hidden_stateaspect_ratio_idsr'cCs>||}|d|jd|j}|jr2||j}||}|S)Nr*r)rOrArCrLrIrRtanh)rTrXrYZ embeddingsr9r9r:forwardts  z-MllamaPrecomputedAspectRatioEmbedding.forward)T) __name__ __module__ __qualname__r!r2rKr1Tensorr[ __classcell__r9r9rUr:rFhs rFcs:eZdZedfdd ZejejejdddZZS)"MllamaPrecomputedPositionEmbeddingrHcst|j|_|j|_|j|jdd|_|j|_|jd|_t t d|_ t |j|j}t |j||_t |jd|j|j|j|_dS)Nr)rJrKrCrM image_size patch_sizer=rLscalerrPr1rQrRrandnrOrNtile_embedding)rTrHZposition_embeddingrUr9r:rKs  z+MllamaPrecomputedPositionEmbedding.__init__rWcCspd|j|j}||dd|j|j}||}|jd}|||j |j|j}|j|}||}|S)Nrr) rRrZrOr-r=rLrir,rArC)rTrXrYZgated_position_embeddingZtile_position_embeddingr6Zgated_tile_position_embeddingr9r9r:r[s  z*MllamaPrecomputedPositionEmbedding.forward) r\r]r^r!rKr1r_r[r`r9r9rUr:rasracs0eZdZfddZejejdddZZS)MllamaVisionMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) rJrKrHr hidden_act activation_fnrLinearrLintermediate_sizefc1fc2rTrHrUr9r:rKs   zMllamaVisionMLP.__init__) hidden_statesr'cCs"||}||}||}|Srk)rprmrq)rTrsr9r9r:r[s   zMllamaVisionMLP.forward)r\r]r^rKr1r_r[r`r9r9rUr:rjs rj)rsn_repr'cCs^|j\}}}}|dkr|S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r,expandrA)rsrtbatchnum_key_value_headsslenhead_dimr9r9r: repeat_kvs 0rz)modulequerykeyvaluerDscalingdropoutkwargsc Kst||j}t||j} t||dd|} |durf|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)Nrcrr?r*)r)r&)ptrainingr)rznum_key_value_groupsr1matmulrBr,r functionalZsoftmaxfloat32r/r&rr contiguous) r|r}r~rrDrrr key_states value_states attn_weights causal_mask attn_outputr9r9r:eager_attention_forwards  & rcsNeZdZedfdd ZdejeejeejeejfdddZ Z S) MllamaVisionAttentionrbcst||_|j|_|j|_|j|j|_|jd|_d|_ t j |j|j|jdd|_ t j |j|j|jdd|_ t j |j|j|jdd|_t j |j|j|jdd|_dS)NrdrFbias)rJrKrHrLZ embed_dimattention_heads num_headsryrrrrnq_projk_projv_projo_projrrrUr9r:rKs  zMllamaVisionAttention.__init__N)rXrDr'cKs||}||}||}|j\}}} |j\} } } ||||j|jdd}||| |j|jdd}||| |j|jdd}t} |j j dkrt |j j } | |||||fd|j d|\} } | ||d} || } | | fS)Nrrceagerr{rrr*)rrrr,r-rryrBrrH_attn_implementationrrrArr)rTrXrDrr}r~rr6Z q_seq_lenr7Z kv_seq_lenattention_interfacerrr9r9r:r[s4         zMllamaVisionAttention.forward)N) r\r]r^r!rKr1r_rtupler[r`r9r9rUr:rsrcs@eZdZd eedfdd Zd ejeejdddZ Z S) MllamaVisionEncoderLayerFrGcst|j|_|j|_||_|j|_t||_t ||_ t j |j|j d|_t j |j|j d|_|rt tdtjd|_t tdtjd|_dS)Nepsr)rJrKrLrnum_attention_headsrIror self_attnrjmlpr LayerNormZnorm_epsinput_layernormpost_attention_layernormrPr1onesmathpi gate_attngate_ffnrSrUr9r:rKs   z!MllamaVisionEncoderLayer.__init__NrXrDcCst|}||}|j||d\}}|jr4|j|}||}|}||}||}|jrh|j|}||}|S)NrD)rrrIrrZrrr)rTrXrDresidualrr9r9r:r[&s   z MllamaVisionEncoderLayer.forward)F)N) r\r]r^r!r2rKr1r_rr[r`r9r9rUr:rs rcsDeZdZdZd edfdd Zd ejeeje dd d Z Z S) MllamaVisionEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`MllamaEncoderLayer`]. Args: config: MllamaConfig Frbcs@t|_tfddt|D|_d|_|_dS)Ncsg|]}tqSr9)r).0r7rGr9r: Jz0MllamaVisionEncoder.__init__..F)rJrKrHr ModuleListrangelayersgradient_checkpointing)rTrHZ num_layersrIrUrGr:rKGs   zMllamaVisionEncoder.__init__N)rsrDr'cCs2d}|jD]}|||d}||f}q t||dS)a8 Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) r9r)last_hidden_staters)rr)rTrsrDZencoder_statesZ encoder_layerr9r9r:r[Ns  zMllamaVisionEncoder.forward)rF)N) r\r]r^__doc__r!rKr1r_rrr[r`r9r9rUr:r>s rcs.eZdZdfdd ZddZddZZS) MllamaTextRMSNormư>cs&ttt||_||_dS)z@ MllamaTextRMSNorm is equivalent to T5LayerNorm N)rJrKrrPr1rweightvariance_epsilon)rTrLrrUr9r:rKos zMllamaTextRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)Nrcr*T)Zkeepdim) r&r/r1rpowmeanZrsqrtrr)rTrsZ input_dtypeZvariancer9r9r:r[ws  zMllamaTextRMSNorm.forwardcCst|jjd|jS)Nz, eps=)rrr,rrTr9r9r: extra_repr~szMllamaTextRMSNorm.extra_repr)r)r\r]r^rKr[rr`r9r9rUr:rnsrcseZdZdZd eeeedfdd Zedddd de j ee j ee ee j ee ee j ee j ee j eee j fd d d ZZS)MllamaTextCrossAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNrH layer_idxcst||_|jj|_|jj|_|j|_|j|_|j|j|_||_ |j|j|_ |jd|_ t j |j|j|jdd|_t j |j|j|jdd|_t j |j|j|jdd|_t j |j|j|jdd|_t|j|jd|_t|j|jd|_dS)NrdFrr)rJrKrHrrrwrrLryrrrrrnrrrrr rms_norm_epsq_normk_normrTrHrrUr9r:rKs     z!MllamaTextCrossAttention.__init__past_key_valuepast_key_values4.58new_nameversion)rscross_attention_statesrrD use_cachecache_positionr'cKsh|\}} } ||} | || |j|jdd} || } |dur||} ||} | |d|j |jdd} | |d|j |jdd} | | } |dur| | | |j d|i\} } n4|ddkr|j |j j|j |j j} } ntdt}|jjdkrt|jj}||| | | |f|js,d n|j|jd |\}}||| d}||}||fS) z#Input shape: Batch x Time x ChannelrrcNr*rrz^Cross attention layer can't find neither `cross_attn_states` nor cached values for key/values!rr{r)sizerr-rryrBrrrrwrupdaterrkeysvalues ValueErrorrrHrrrrrrArr)rTrsrrrDrrrbszq_lenr7 query_statesrrrrrr9r9r:r[sN             z MllamaTextCrossAttention.forward)NN)NNNNN)r\r]r^rrr intrKrr1r_rr2 LongTensorrr[r`r9r9rUr:rs, rcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr*rcr()r,r1cat)xx1Zx2r9r9r: rotate_halfsrcCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. )r.r)qkcossin position_idsZ unsqueeze_dimZq_embedZk_embedr9r9r:apply_rotary_pos_embs   rcsNeZdZeedfdd Zeddddd ejejeje d d d Z Z S)MllamaTextSelfAttentionrcst||_|j|_|j|_|j|_|j|_|j|j|_|j|j|_ |jd|_ |j |_ ||_ t j|j|j|jdd|_t j|j|j|jdd|_t j|j|j|jdd|_t j|j|j|jdd|_dS)NrdFr)rJrKrHrrrrLrwryrrZ rope_thetarrrnrrrrrrUr9r:rKs  z MllamaTextSelfAttention.__init__rrrrFN)rsrDposition_embeddingsrcKs4|\}} } ||} ||} ||} | || |j|jdd} | || |j|jdd} | || |j|jdd} |\}}t | | ||\} } |dur|||d}| | | |j |\} } t }|j jdkrt|j j}||| | | |f|jsdn|j|jd|\}}||| d}||}||fS)Nrrc)rrrrr{rr*)rrrrr-rryrBrwrrrrrHrrrrrrArr)rTrsrDrrrrrrrr7rrrrrZ cache_kwargsrrrr9r9r:r[s<         zMllamaTextSelfAttention.forward)FNN) r\r]r^r rrKrr1r_r2r[r`r9r9rUr:rs rcs$eZdZfddZddZZS) MllamaTextMLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFr) rJrKrHrLrorrn gate_projup_proj down_projrrlact_fnrrrUr9r:rKGs zMllamaTextMLP.__init__cCs$||||||}|Srk)rrrr)rTrrr9r9r:r[Rs zMllamaTextMLP.forward)r\r]r^rKr[r`r9r9rUr:rFs rcseZdZeedfdd Zeddddd eje eje eje eje e ejejfe ej e e e e e ej e e ejejfeee eje e ejejffd d d ZZS)MllamaSelfAttentionDecoderLayerrcsXt|j|_t||d|_t||_t|j|jd|_ t|j|jd|_ ||_ dS)Nrr) rJrKrLrrrrrrrrrrrUr9r:rKYs  z(MllamaSelfAttentionDecoderLayer.__init__rrrrNF rsrr$rDr8rrrrrrr'c  Ks^|} ||}|jf|||||| | d| \}} | |}|} ||}||}| |}|S)aZ Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )rsrDrrrrr)rrrr)rTrsrr$rDr8rrrrrrrZself_attn_weightsr9r9r:r[es&#    z'MllamaSelfAttentionDecoderLayer.forward) NNNNNNFNN)r\r]r^r rrKrr1r_rrrrr2rr FloatTensorr[r`r9r9rUr:rXs2 rcseZdZdZeeddfdd Zedddd dej ej ej ej e ej ej fe ej e e e ee ej e ej eee ej d d d ZZS) MllamaCrossAttentionDecoderLayerzLCross-attention transformer block with tanh-gated attention and feedforward.N)rHrr'csxt||_t||d|_t|j|jd|_t j t d|_ t||_t|j|jd|_t j t d|_dS)N)rrr)rJrKrr cross_attnrrLrrr1rrPrQcross_attn_attn_gaterrrcross_attn_mlp_gaterrUr9r:rKs  z)MllamaCrossAttentionDecoderLayer.__init__rrrrFrc Ks|} ||}|jf||||| d| \}} | |j|}|} ||}||}|durt|dddf|}| |j|}|S)N)rsrDrrrr)rrrrZrrr)rTrsrr$rDr8rrrrrrrrr9r9r:r[s&    z(MllamaCrossAttentionDecoderLayer.forward)NNFNN)r\r]r^rr rrKrr1r_rrrrr2rr r[r`r9r9rUr:rs, rcsDeZdZUejed<dedfdd Zee ddZ Z S) MllamaRotaryEmbeddinginv_freqNrbcsht|jd|_|j|_|j|_||_t|j|_ | |j|\}|_ |j d|dd|j |_ dS)N rope_typerF) persistent)rJrKZ rope_scalingrZmax_position_embeddingsZmax_seq_len_cachedZoriginal_max_seq_lenrHrZ rope_init_fnattention_scalingZregister_bufferrZoriginal_inv_freq)rTrHdevicerrUr9r:rKs   zMllamaRotaryEmbedding.__init__c Cs|jddddf|jddd}|dddddf}t|jjtrd|jjdkrd|jjnd}tj |ddV|| dd}tj ||fdd }| |j }||j } Wdn1s0Y|j|jd | j|jd fS) Nrr*rZmpscpuF) device_typeZenabledrcr()r&)rfloatrur, isinstancertypestrr1ZautocastrBrrrrr/r&) rTrrZinv_freq_expandedZposition_ids_expandedrZfreqsZembrrr9r9r:r[s(&,zMllamaRotaryEmbedding.forward)N) r\r]r^r1r___annotations__r rKZno_gradrr[r`r9r9rUr:rs   rc@seZdZUeed<dZdZgdZdZdZ dZ dZ dZ e egeedddeedd deedd dgd Zd d Zdeejd fejejeedddZeejeeejejedddZdS)MllamaPreTrainedModelrHT)rrrFrr)indexZ layer_namer)rs attentionscCst|jd|jj}t|tjtjfrT|jj j d|d|j durP|j j nVt|tj r|jj j d|d|jdur|jj |j nt|tjr|jj d|j j nt|tr|jj dnt|trtjj |jj |dnt|tr&tjj |jj |dtj|jj nt|trd|jrdtjj |jj |dtjj |jj |dnFt|tr|jj |jj n t|t r|jr|jj dS)Ninitializer_ranger{)rstdr+)r)!getattrrHget_text_configrrrrnConv2drdataZnormal_rZzero_rN padding_idxrZfill_rMllamaVisionModelinitclass_embeddingrarOZzeros_rRrrIrrrrrrF)rTr|rr9r9r: _init_weightss8          z#MllamaPreTrainedModel._init_weightsr")rD input_tensorrroutput_attentionsc CsB|jjdkr(|dur$|dkr$|SdS|jjdkrLt|tjrHt|}|S|dur\|nd}|durn|jnd}|jjdkr|s|st j ||||j drdS|j }|j d} |r|} n"t|tjr|j d n || d} |j|| | |||j dd } |jjdkr>|dur>|jjd vr>|s>t|j} t | | } | S) NZflash_attention_2r{Zflex_attentionrFZsdpa) inputs_embedsZpast_key_values_lengthZ is_trainingrr*)sequence_lengthr>r&rr6)cudaZxpuZnpu)rHrr5rr1r_r#get_seq_lengthZis_compileabler Z_ignore_causal_mask_sdparr&r,Zget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrrr3r4Z_unmask_unattended) rTrDrrrrpast_seen_tokensZusing_compilable_cacher&rr>r min_dtyper9r9r:_update_causal_mask.sZ           z)MllamaPreTrainedModel._update_causal_mask)rDrr>r&rr6c KsF|dur|dkr|}n&t|j}tj||f|||jd}|dkrVtj|dd}|tj||jd|ddk9}|ddddddf |ddd}|durB| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nr)Z fill_valuer&rr)Zdiagonalrr*r)r)r1r3r4fullrZtriuarangerArucloner,r/r0) rDrr>r&rr6rrrZ mask_lengthZ padding_maskr9r9r:rrs*  $  6  zKMllamaPreTrainedModel._prepare_4d_causal_attention_mask_with_cache_positionN)F)r\r]r^rrbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modules_can_compile_fullgraphZ_supports_sdpaZ_supports_flash_attnZ_supports_flex_attnZ_supports_attention_backendrrrrrZ_can_record_outputsrrr1r_rr2r staticmethodrr&rr9r9r9r:rs@     &  DrzH The Mllama Vision Model which consists of two vision encoders. )Z custom_introcsneZdZUeed<dZedfdd ZddZej ej dd d Z e e ej ej ej e d d d ZZS)rrH vision_modelrbcs t||j|_|j|_|j|_|j|_|j|_|j|_|j|jdd|_|jd|_ t j |j|j|j|jddd|_ t |j t|j|_t||_t|dd|_t|dd|_t |j|_t |j|_t||jdd|_t||jdd|_|dS) NrcrrdZvalidF)Z in_channelsZ out_channelsZ kernel_sizeZstridepaddingrT)rI)rJrKrerfrCrL num_channelsintermediate_layers_indicesr=rgrr patch_embeddingrPr1rhrragated_positional_embeddingrFpre_tile_positional_embeddingpost_tile_positional_embeddingr layernorm_prelayernorm_postrnum_hidden_layers transformerZnum_global_layersglobal_transformer post_initrrrUr9r:rKs4    zMllamaVisionModel.__init__cCs|jS)zg This function is used to fetch the first embedding layer to activate grads on inputs. )r'rr9r9r:get_input_embeddingssz&MllamaVisionModel.get_input_embeddings)rXr'cCs2|j\}}}|j|d|}tj||gdd}|S)Nrr()r,rrur1r)rTrXr6r7rLrr9r9r:apply_class_embeddings z'MllamaVisionModel.apply_class_embedding) pixel_valuesrYr<r'c s|j\}}}}} } |||||| | }|||d}|jjj} |jjj} ||| | } | ddd}|j\}}}||||d|}| ||}||||||}| |}|d7}||||||}| ||}| |}d|jddd}ddd|f}t j||ddd}|dkr4| nd }|||d}t||j|jd|jd }|||d|}|j||d j}||}|||||||}|||}|||||||}|j||d }|j}|||||||}|d d d d d |f}||||||}fd d |jD}tj|dd}||||||d}|d d d d d |f}|||||d}tj||gdd}t|dS)a5 aspect_ratio_ids (`torch.Tensor` of shape `(batch_size, max_num_images)`, *optional*): Aspect ratio ids used to select the appropriate precomputed tile embeddings based on the aspect ratio of each input image. These ids correspond to indices in the model's list of supported aspect ratios, offset by 1. For example, if the model supports aspect ratios [[1, 1], [1, 2], [2, 1]]: - An image with aspect ratio [1, 1] would have ID 1 - An image with aspect ratio [1, 2] would have ID 2 - An image with aspect ratio [2, 1] would have ID 3 The id 0 is reserved for padding (i.e., no image). If an image has aspect ratio [1, 2], that means it was split into 2 tiles horizontally, and its `aspect_ratio_id` would be 2. aspect_ratio_mask (`torch.Tensor` of shape `(batch_size, max_num_images, max_num_tiles)`, *optional*): Mask to avoid performing attention on padding tiles. Mask values selected in `[0, 1]`: - 1 for tiles that are **not masked**, - 0 for tiles that are **masked**. Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, MllamaVisionModel >>> checkpoint = "meta-llama/Llama-3.2-11B-Vision" >>> model = MllamaVisionModel.from_pretrained(checkpoint) >>> processor = AutoProcessor.from_pretrained(checkpoint) >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> output = model(**inputs) >>> print(output.last_hidden_state.shape) torch.Size([1, 1, 4, 1025, 7680]) ``` r*rcrr?rZconstant)moderN)r<r=r>r&rcsg|]}j|qSr9)rs)rioutputr9r:r^rz-MllamaVisionModel.forward..r()r)r,rAr'rr&rr/flattenrBr)r2r(r+FpadrEr=r-r.rr,r*r/r&r1stackrr)rTr3rYr<rr6Znum_concurrent_mediaZ num_tilesr%heightwidthZ target_dtypeZ target_deviceZ patch_embedsrXr7r=r)Znum_padding_patchesr$Z slice_indexrDZ global_outputZall_intermediate_hidden_statesZintermediate_hidden_statesr9r7r:r[s|-           zMllamaVisionModel.forward)r\r]r^r!rr rKr1r1r_r2rrrr[r`r9r9rUr:rs % rzc The Mllama Text Model which consists of transformer with self and cross attention layers. cseZdZUeed<dZedfdd Zeee d e e j e e j e e j e e je e j e ee j e j fe eeee jfe e je ee e j eeed dd ZZS) MllamaTextModelrHlanguage_model.modelrbcst||j|_|j|_t|jd|j|j|_|j |_ g}t |j D]0}||j vrl| t ||qL| t||qLt||_t|j|jd|_t|d|_d|_|dS)Nr4rrbF)rJrK pad_token_idr  vocab_sizerrNrL embed_tokenscross_attention_layersrr-appendrrrrrrnormr rotary_embrr0)rTrHrrrUr9r:rKys    zMllamaTextModel.__init__N) input_idsrDrrr$r8rrrrrr'c  KsN| dur | n|jj} |du|duAr,td|dur>||}|} | rZ|durZt|jd}| dur|durr|nd} tj| | |jd|j d} |dur| d}| ||| |}| | |}t |jD]l\}}||jv}|dup|duo||dk}|r|dur|rq|| f||||||| | |d | } q|| } t| |dS) aQ cross_attention_states (`torch.FloatTensor`, *optional*): Output of the vision model, used for cross-attention. This tensor contains the processed image features that the language model will attend to. cross_attention_mask (`torch.Tensor` of shape `(batch_size, seq_length, max_num_images, max_num_tiles)`, *optional*): Cross-attention mask to control the interaction between text tokens and image tiles. This 4D tensor defines which image tiles each text token should attend to. For each text token (in seq_length): - 1 indicates the token **should attend** to the corresponding image tile - 0 indicates the token **should not attend** to the corresponding image tile full_text_row_masked_out_mask (`tuple[torch.Tensor, torch.Tensor]`, *optional*): A tuple containing two tensors that mask out rows in the cross-attention mechanism: - The first tensor has shape `(batch_size, 1, seq_length, 1)` and contains values of 0 or 1. A value of 0 indicates that the corresponding text token's entire row in the cross-attention matrix should be masked out (all image tokens ignored). - The second tensor has the same shape and is used internally to apply the masking during the forward pass of cross-attention layers. This mask is derived from the cross_attention_mask and is used to handle cases where a text token should not attend to any image token. Example: ```python >>> from transformers import AutoProcessor, MllamaTextModel >>> checkpoint = "meta-llama/Llama-3.2-11B-Vision" >>> model = MllamaTextModel.from_pretrained(checkpoint) >>> processor = AutoProcessor.from_pretrained(checkpoint) >>> text = "<|image|>If I had to write a haiku for this one" >>> inputs = processor(text=text, return_tensors="pt") >>> output = model(**inputs) >>> print(output.last_hidden_state.shape) torch.Size([1, 13, 4096]) ``` N:You must specify exactly one of input_ids or inputs_embedsrbrrr) rr$rDr8rrrrr)rr)rHrrrCr rr1rr,rr.rrG enumeraterrDrFr)rTrHrDrrr$r8rrrrrrsrrridxZ decoder_layerZis_cross_attention_layerZis_cross_attention_cache_emptyr9r9r:r[sX8        zMllamaTextModel.forward) NNNNNNNNNN)r\r]r^r rr rKrrrrr1rr_rrrrlistr2rr rr[r`r9r9rUr:r?ps< r?zE The Mllama Text Model with a language modeling head on top. cseZdZUeed<dZdZdgZfddZe e d e e j e e je e j e e j e e j e ee je jfe eeee jfe e je e j e ee e j eee jfeeeeefd d d ZZS) MllamaForCausalLMrHTlanguage_modellm_head.weightcsVt|||_|jj|_t|j|_tj |jj |jdd|_ | dSr) rJrKr  text_configrBr? _from_configmodelrrnrLlm_headr0rrrUr9r:rK s   zMllamaForCausalLM.__init__Nr)rHrDrrr$r8rrlabelsrrlogits_to_keeprr'c  Ks|jf||||||||| | d | }|j}t| trBt| dn| }||dd|ddf}d}| dur|j|| |jfi| }t |||j |j |j dS)a cross_attention_states (`torch.FloatTensor`, *optional*): Output of the vision model, used for cross-attention. This tensor contains the processed image features that the language model will attend to. cross_attention_mask (`torch.Tensor` of shape `(batch_size, seq_length, max_num_images, max_num_tiles)`, *optional*): Cross-attention mask to control the interaction between text tokens and image tiles. This 4D tensor defines which image tiles each text token should attend to. For each text token (in seq_length): - 1 indicates the token **should attend** to the corresponding image tile - 0 indicates the token **should not attend** to the corresponding image tile full_text_row_masked_out_mask (`tuple[torch.Tensor, torch.Tensor]`, *optional*): A tuple containing two tensors that mask out rows in the cross-attention mechanism: - The first tensor has shape `(batch_size, 1, seq_length, 1)` and contains values of 0 or 1. A value of 0 indicates that the corresponding text token's entire row in the cross-attention matrix should be masked out (all image tokens ignored). - The second tensor has the same shape and is used internally to apply the masking during the forward pass of cross-attention layers. This mask is derived from the cross_attention_mask and is used to handle cases where a text token should not attend to any image token. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, MllamaForCausalLM >>> model = MllamaForCausalLM.from_pretrained("Llama-3.2-11B-Vision") >>> tokenizer = AutoTokenizer.from_pretrained("Llama-3.2-11B-Vision") >>> prompt = "If I had to write a haiku, it would be:" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=40, do_sample=True, temperature=0.6) >>> result = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] >>> print(result) If I had to write a haiku, it would be: "Snowflakes gently fall" - simple, yet peaceful. I love the idea of snowflakes gently falling, each one ``` ) rHrrDrr$r8rrrrNlosslogitsrrsr) rRrrrslicerSr loss_functionrBrrrsr)rTrHrDrrr$r8rrrTrrrUroutputsrs slice_indicesrXrWr9r9r:r[s6?  zMllamaForCausalLM.forward) NNNNNNNNNNNr)r\r]r^r rr!r _tied_weights_keysrKrrrr1rr_rrrrLrr2rrrrr[r`r9r9rUr:rMsF    rMzr The Mllama model which consists of a vision encoder and a language model without language modeling head. cseZdZddiZedfdd ZddZdd Zd d Zd d Z e e e de eje eje eje eje eje eje eje eje ee eje ee ejeeedddZZS) MllamaModelr@rNrbcst||jj|_|jj|_|jj|_|jj|_|jj durH|jj nd|_ t |j|_ t |j|_tj|jj|jjdd|_|dS)Nr*Tr)rJrKrPrBrLZ vision_configrCZvision_output_dimrHrArrQr#r?rNrrnmulti_modal_projectorr0rrrUr9r:rK|s     zMllamaModel.__init__cCs |jSrk)rNr1rr9r9r:r1sz MllamaModel.get_input_embeddingscCs|j|dSrk)rNset_input_embeddingsrTrr9r9r:r`sz MllamaModel.set_input_embeddingscCs ||_dSrkrNrTdecoderr9r9r: set_decoderszMllamaModel.set_decodercCs|jSrkrbrr9r9r: get_decoderszMllamaModel.get_decoderN)rHr3r<rYrDr$rrrrrrrr'c  Ks|du| duArtd|dur0|dur0td|durz|durHtd|j|||d}|j}||d|jd|j}|durt||jj|j d\}}nd}|dur| dur|dddd| f}|dddd| f}|j f||||||| | | | d | }t |j|j |j |jd S) ar aspect_ratio_mask (`torch.Tensor` of shape `(batch_size, max_num_images, max_num_tiles)`, *optional*): Mask to avoid performing attention on padding tiles. Mask values selected in `[0, 1]`: - 1 for tiles that are **not masked**, - 0 for tiles that are **masked**. aspect_ratio_ids (`torch.Tensor` of shape `(batch_size, max_num_images)`, *optional*): Aspect ratio ids used to select the appropriate precomputed tile embeddings based on the aspect ratio of each input image. These ids correspond to indices in the model's list of supported aspect ratios, offset by 1. For example, if the model supports aspect ratios [[1, 1], [1, 2], [2, 1]]: - An image with aspect ratio [1, 1] would have ID 1 - An image with aspect ratio [1, 2] would have ID 2 - An image with aspect ratio [2, 1] would have ID 3 The id 0 is reserved for padding (i.e., no image). If an image has aspect ratio [1, 2], that means it was split into 2 tiles horizontally, and its `aspect_ratio_id` would be 2. cross_attention_mask (`torch.Tensor` of shape `(batch_size, seq_length, max_num_images, max_num_tiles)`, *optional*): Cross-attention mask to control the interaction between text tokens and image tiles. This 4D tensor defines which image tiles each text token should attend to. For each text token (in seq_length): - 1 indicates the token **should attend** to the corresponding image tile - 0 indicates the token **should not attend** to the corresponding image tile cross_attention_states (`torch.FloatTensor`, *optional*): Output of the vision model, used for cross-attention. This tensor contains the processed image features that the language model will attend to. NrIzM`pixel_values` and `cross_attention_states` cannot be provided simultaneouslyzA`aspect_ratio_ids` must be provided if `pixel_values` is provided)r3rYr<r*r?)r%r&) rHrDrrr$r8rrrr)rrrsr)rr#rr_rAr,rLr;r=r&rNrrrsr)rTrHr3r<rYrDr$rrrrrrrZvision_outputsr8r[r9r9r:r[s\0   zMllamaModel.forward) NNNNNNNNNNNN)r\r]r^_checkpoint_conversion_mappingrrKr1r`rerfrrrrr1rrr_rr2rr rr[r`r9r9rUr:r^tsJr^zS The Mllama model which consists of a vision encoder and a language model. cs,eZdZdddddZdgZedfdd Zd d Zd d ZddZ ddZ e ddZ e ddZ eed eejeejeejeejeejeejeejeejeeeeejfeejeejeeeejeeejfeeeeefdddZd!fdd ZfddZ Z!S)"MllamaForConditionalGenerationzmodel.language_modelzmodel.vision_modelzmodel.multi_modal_projectorrS)z^language_model.modelz ^vision_modelz^multi_modal_projectorz^language_model.lm_headrOrbcs<t|t||_tj|jj|jjdd|_ | dSr) rJrKr^rRrrnrPrLrBrSr0rrrUr9r:rKs  z'MllamaForConditionalGeneration.__init__cCs |jSrk)rRr1rr9r9r:r1sz3MllamaForConditionalGeneration.get_input_embeddingscCs|j|dSrk)rRr`rar9r9r:r`sz3MllamaForConditionalGeneration.set_input_embeddingscCs|j|dSrk)rRrercr9r9r:resz*MllamaForConditionalGeneration.set_decodercCs |jSrk)rRrfrr9r9r:rfsz*MllamaForConditionalGeneration.get_decodercCs|jjSrk)rRrNrr9r9r:rN!sz-MllamaForConditionalGeneration.language_modelcCs|jjSrk)rRr#rr9r9r:r#%sz+MllamaForConditionalGeneration.vision_modelNr)rHr3r<rYrDr$rrrrrTrrrUrr'cKs|jf||||||||| | | | d |}|j}t|trFt| dn|}||dd|ddf}d}| dur|j|| |jjj fi|}t |||j |j |j dS)af aspect_ratio_mask (`torch.Tensor` of shape `(batch_size, max_num_images, max_num_tiles)`, *optional*): Mask to avoid performing attention on padding tiles. Mask values selected in `[0, 1]`: - 1 for tiles that are **not masked**, - 0 for tiles that are **masked**. aspect_ratio_ids (`torch.Tensor` of shape `(batch_size, max_num_images)`, *optional*): Aspect ratio ids used to select the appropriate precomputed tile embeddings based on the aspect ratio of each input image. These ids correspond to indices in the model's list of supported aspect ratios, offset by 1. For example, if the model supports aspect ratios [[1, 1], [1, 2], [2, 1]]: - An image with aspect ratio [1, 1] would have ID 1 - An image with aspect ratio [1, 2] would have ID 2 - An image with aspect ratio [2, 1] would have ID 3 The id 0 is reserved for padding (i.e., no image). If an image has aspect ratio [1, 2], that means it was split into 2 tiles horizontally, and its `aspect_ratio_id` would be 2. cross_attention_mask (`torch.Tensor` of shape `(batch_size, seq_length, max_num_images, max_num_tiles)`, *optional*): Cross-attention mask to control the interaction between text tokens and image tiles. This 4D tensor defines which image tiles each text token should attend to. For each text token (in seq_length): - 1 indicates the token **should attend** to the corresponding image tile - 0 indicates the token **should not attend** to the corresponding image tile cross_attention_states (`torch.FloatTensor`, *optional*): Output of the vision model, used for cross-attention. This tensor contains the processed image features that the language model will attend to. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, MllamaForConditionalGeneration >>> checkpoint = "meta-llama/Llama-3.2-11B-Vision" >>> model = MllamaForConditionalGeneration.from_pretrained(checkpoint) >>> processor = AutoProcessor.from_pretrained(checkpoint) >>> prompt = "<|image|>If I had to write a haiku for this one" >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(text=prompt, images=image, return_tensors="pt") >>> # Generate >>> output = model.generate(**inputs, max_new_tokens=15) >>> prompt_len = inputs.input_ids.shape[-1] >>> generated_ids = output[:, prompt_len:] >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False) >>> print(generated_text) [', it would be:.\\nA stop sign in Chinatown.\\n'] ``` ) rHr3r<rYr$rrDrrrrrNrV)rRrrrrYrSrZrHrPrBrrrsr)rTrHr3r<rYrDr$rrrrrTrrrUrr[rsr\rXrWr9r9r:r[)s:P z&MllamaForConditionalGeneration.forwardFc  sTtj|f| | |||||||| | d | }| ddkrPd|d<d|d<d|d<|S)N) rrrrrDr3rYr<r$rrUrr3rYr<)rJprepare_inputs_for_generation)rTrHrrDrr3rYr<r$rrrrUrZ model_inputsrUr9r:ris,  zs             & 7+0Z GJ55AmR