a h* @sDddlZddlmZddlmZmZmZmZddlZddl m Z ddl m m Z ddl mZddlmZddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZm Z ddl!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,m-Z-m.Z.ddl/m0Z0ddl1m2Z2m3Z3m4Z4e.5e6Z7edGddde j8Z9Gddde j8Z:Gddde j8Z;Gddde j8Zd"d#Z?ej@ej@ej@ej@eAej@ej@fd$d%d&ZBej@eCej@d'd(d)ZDdPe j8ej@ej@ej@eej@eEeEe(e*d+d,d-ZFGd.d/d/e j8ZGGd0d1d1eZHGd2d3d3e j8ZId4d5ZJdQd6d7ZKGd8d9d9e j8ZLGd:d;d;e j8ZMGdd?Gd@dAdAe ZOe+GdBdCdCe&ZPGdDdEdEePZQe+GdFdGdGePZRe+GdHdIdIePZSee+dJd?GdKdLdLe ZTGdMdNdNePeZUgdOZVdS)RN) dataclass)AnyCallableOptionalUnion) LayerNorm)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPast ModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torchdynamo_compilinglogging)check_model_inputs) Glm4vConfigGlm4vTextConfigGlm4vVisionConfigZRMSNormcs.eZdZdfdd ZddZddZZS) Glm4vRMSNormư>cs&ttt||_||_dS)z; Glm4vRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__d/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/glm4v/modeling_glm4v.pyr%3s zGlm4vRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)NTZkeepdim) dtypetor(float32powmeanZrsqrtr+r*)r, hidden_statesZ input_dtypeZvariancer1r1r2forward;s  zGlm4vRMSNorm.forwardcCst|jjd|jS)Nz, eps=)tupler*shaper+r,r1r1r2 extra_reprBszGlm4vRMSNorm.extra_repr)r#)__name__ __module__ __qualname__r%r<r@ __classcell__r1r1r/r2r"1sr"cs,eZdZdedfdd ZddZZS) Glm4VisionMlpFbiascslt|j|_|j|_tj|j|j|d|_tj|j|j|d|_tj|j|j|d|_ t |j |_ dSNrF) r$r%r-out_hidden_sizeintermediate_sizer&Linear gate_projup_proj down_projr hidden_actact_fn)r,configrGr/r1r2r%Gs zGlm4VisionMlp.__init__cCs ||||||SN)rNrPrLrMr, hidden_stater1r1r2r<PszGlm4VisionMlp.forward)F)rArBrCboolr%r<rDr1r1r/r2rEFs rEcs8eZdZeddfdd ZejejdddZZS)Glm4vVisionPatchEmbedNrQreturncsVt|j|_|j|_|j|_|j|_|j|j|jg}tj|j|j||d|_ dS)N) kernel_sizestride) r$r% patch_sizetemporal_patch_size in_channelsr- embed_dimr&ZConv3dproj)r,rQrYr/r1r2r%Us zGlm4vVisionPatchEmbed.__init__r;rXcCsD|jjj}|d|j|j|j|j}||j|dd|j}|S)Nr4r6) r_r*r6viewr]r\r[r7r^)r,r;Z target_dtyper1r1r2r<_s  zGlm4vVisionPatchEmbed.forward rArBrCr!r%r(Tensorr<rDr1r1r/r2rVTs rVcsFeZdZUejed<d eeddfdd Zeejddd Z Z S) Glm4vVisionRotaryEmbeddinginv_freq@N)dimthetarXcs>td|tjd|dtjd|}|jd|dddS)N?rr3rarfF persistent)r$r%r(arangefloatregister_buffer)r,rhrirfr/r1r2r%ks  z#Glm4vVisionRotaryEmbedding.__init__)seqlenrXcCs*tj||jj|jjd}t||j}|S)Ndevicer6)r(rmrfrrr6outer)r,rpseqfreqsr1r1r2r<psz"Glm4vVisionRotaryEmbedding.forward)rg) rArBrCr(rd__annotations__intrnr%r<rDr1r1r/r2rehs  recs@eZdZd eeeeddfdd ZejejdddZ Z S) Glm4vVisionPatchMergerFN)rh context_dimrOrGrXcstttj|||d|_t||_tj|||d|_tj|||d|_tj|||d|_ t |_ t ||_ dSrH)r$r%r&rKr_rpost_projection_normrLrMrNZGELUact1r rP)r,rhryrOrGr/r1r2r%ws   zGlm4vVisionPatchMerger.__init__)rTrXcCs:||}|||}||||||SrR)r_r{rzrNrPrLrMrSr1r1r2r<s zGlm4vVisionPatchMerger.forward)F) rArBrCrwstrrUr%r(rdr<rDr1r1r/r2rxvs rxcs2eZdZedfdd ZejdddZZS)Glm4vVisionEmbeddingsrQcsvt||_|j|_|j|_|j|_|j|jd|_|j|_t |j|j|_ |j dt |jddddS)Nr3 position_ids)rr4Frk)r$r%rQr-r^Z image_sizer[Z num_patchesZ num_positionsr& Embeddingposition_embeddingror(rmexpandr,rQr/r1r2r%s zGlm4vVisionEmbeddings.__init__rXcs|jj}|jd}|jd}|j} || || }}|dkrXtjd|| |jd} n|tt rttj | tj dttj stj | tj d|jd} t | d} || | |ddddj| tjd} tfddttDj| tjd}tfddttDj| tjd}|j| tjd}|j| tjd}|d|dd}|d|dd}tj||fd d dd}tj| |d d d d}|dd dd}||j|j} || }|S)a Forward pass with integrated position encoding adaptation using 2D interpolation. Args: embeddings: Input embeddings tensor lengths (torch.Tensor): Sequence lengths for each image in the batch. image_shapes (torch.Tensor): Tensor of shape [batch_size, 3] representing the image shapes (t, h, w). h_coords (torch.Tensor): Tensor of shape [total_seq] representing the h coordinate for each patch. w_coords (torch.Tensor): Tensor of shape [total_seq] representing the w coordinate for each patch. Returns: torch.Tensor: Embeddings with adapted position encoding added. rrrqg?r3cs"g|]}|df|qS)rrepeat.0i image_shapeslengthsr1r2 z1Glm4vVisionEmbeddings.forward..cs"g|]}|df|qS)r3rrrr1r2rrr4rhZbicubicFborder)modeZ align_cornersZ padding_mode)rr*r>rrr7r(emptyr6 isinstancelisttensorlongrdrwrbpermute unsqueezer8catrangelenstackFZ grid_sampleZsqueeze)r, embeddingsrrZh_coordsZw_coordsZpos_embed_weightr-Z total_seqrrZadapted_pos_embedZ orig_size_sqZ orig_sizeZ pos_embed_2dZtarget_hZtarget_wZnorm_wZnorm_hgridZinterpolated_embed_fp32Zadapted_pos_embed_fp32r1rr2r<sN      "" zGlm4vVisionEmbeddings.forwardrcr1r1r/r2r}s r}cCsH|dd|jddf}|d|jdddf}tj| |fddS)*Rotates half the hidden dims of the input..Nr4r3r)r>r(rxx1Zx2r1r1r2 rotate_halfsr)qkcossinrXcCs|j}|j}||}}|d|d}}||t||}||t||}||}||}||fS)N)r6rnrrr7)rrrrZ orig_q_dtypeZ orig_k_dtypeq_embedk_embedr1r1r2apply_rotary_pos_emb_visions  r)r;n_reprXcCs^|j\}}}}|dkr|S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r>rreshape)r;rbatchnum_key_value_headsslenhead_dimr1r1r2 repeat_kvs 0r)modulequerykeyvalueattention_maskscalingdropoutkwargsc Kst||j}t||j} t||dd|} |durf|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)Nr3rrr4rhr6)ptrainingr)rnum_key_value_groupsr(matmul transposer>r& functionalZsoftmaxr8r7r6rr contiguous) rrrrrrrr key_states value_states attn_weights causal_mask attn_outputr1r1r2eager_attention_forwards  & rcsXeZdZeddfdd ZdejejeejeeejejfejdddZ Z S) Glm4vVisionAttentionNrWcst|j|_|j|_|j|j|_d|_tj|j|jd|j d|_ tj|j|jdd|_ |jd|_ ||_ |j|_d|_dS)NrrrFF)r$r%r-rh num_headsrrr&rKZattention_biasqkvr_rrQattention_dropout is_causalrr/r1r2r%s  zGlm4vVisionAttention.__init__r; cu_seqlensrotary_pos_embposition_embeddingsrXc s|jd}||djdddddd\}}} |durptdtj ||fdd} | } | } n|\} } t ||| | \}}| ddd}| ddd}| ddd} tjjdkrtjjjjd krD|dd|dd} ||| fdjjs$d nj||| | d d \}}nX|dd|ddfd d||| fD}fddt|D}tj |dd}||d}|}|S)Nrrr4rr3aSThe attention layers in this model are transitioning from computing the RoPE embeddings internally through `rotary_pos_emb` (2D tensor of RoPE theta values), to using externally computed `position_embeddings` (Tuple of tensors, containing cos and sin). In v4.54 `rotary_pos_emb` will be removed and `position_embeddings` will be mandatory.reagerZflash_attention_2rF)rrrZ cu_seq_lens_qZ cu_seq_lens_kZ max_length_qZ max_length_krcs g|]}tj|ddqS)r3r)r(splittolist)rr)rr1r2r^sz0Glm4vVisionAttention.forward..c sDg|]<\}}}|||fdjjs*dnjdddqS)NrF)rrrrr)rrr)rrrv)attention_interfacerr,r1r2rbs    )r>rrrrZunbindloggerZ warning_oncer(rrrrrrrrQ_attn_implementationrmaxrrrziprr_)r,r;rrrr seq_length query_statesrrembrrZ max_seqlenr_ZsplitsZ attn_outputsr1)rrrr,r2r<'s` (        zGlm4vVisionAttention.forward)NN) rArBrCr!r%r(rdrr=r<rDr1r1r/r2rsrcsVeZdZddfdd ZdejejeejeeejejfejdddZZ S) Glm4vVisionBlockNrcsJtt|j|jd|_t|j|jd|_t||_t |dd|_ dS)Nr.FrF) r$r%r"r- rms_norm_epsnorm1norm2rattnrEmlprr/r1r2r%xs   zGlm4vVisionBlock.__init__rcKs<||j||f|||d|}||||}|S)N)rrr)rrrr)r,r;rrrrr1r1r2r<szGlm4vVisionBlock.forward)NN) rArBrCr%r(rdrr=r<rDr1r1r/r2rws rcsDeZdZUejed<dedfdd Zee ddZ Z S) Glm4vTextRotaryEmbeddingrfNr~cstt|dr8|jdur8|jd|jd|_nd|_|j|_|j|_||_ t |j|_ | |j |\}|_ |j d|dd|j|_dS)N rope_scaling rope_typetypedefaultrfFrk)r$r%hasattrrgetrZmax_position_embeddingsZmax_seq_len_cachedZoriginal_max_seq_lenrQrZ rope_init_fnattention_scalingrorfZoriginal_inv_freq)r,rQrrrfr/r1r2r%s  z!Glm4vTextRotaryEmbedding.__init__c Cs|jdddddfd|jddd}|dddddddf}t|jjtrn|jjdkrn|jjnd}tj |ddV|| dd}tj ||fdd }| |j }||j } Wdn1s0Y|j|jd | j|jd fS) Nrrr4ZmpscpuF) device_typeZenabledr3rra)rfrnrr>rrrrr|r(Zautocastrrrrrr7r6) r,rrZinv_freq_expandedZposition_ids_expandedrrurrrr1r1r2r<s, &,z Glm4vTextRotaryEmbedding.forward)N) rArBrCr(rdrvr r%Zno_gradrr<rDr1r1r/r2rs  rcCs>|ddddf}|ddddf}tj| |fdddS) r.rNr3rr4rr)r(rflattenrr1r1r2rotate_half_llmsrc CsJ|d}tjddt|j|ddDdd|}tjddt|j|ddDdd|}|dd|jddfjddd}|dd|jddfjddd}|jd}|dd|f|d|df}}|dd|f|d|df} } ||t||} | |t| |} tj| |gdd} tj| | gdd} | | fS) aXApplies Rotary Position Embedding with Multimodal Sections to the query and key tensors (https://qwenlm.github.io/blog/qwen2-vl/). Explanation: Multimodal 3D rotary position embedding is an extension to 1D rotary position embedding. The input embedding sequence contains vision (images / videos) embedding and text embedding or just contains text embedding. For vision embedding part, we apply rotary position embedding on temporal, height and width dimension separately. Here we split the channel dimension to 3 chunks for the temporal, height and width rotary position embedding. For text embedding part, we just apply 1D rotary position embedding. The three rotary position index (temporal, height and width) of text embedding is always the same, so the text embedding rotary position embedding has no difference with modern LLMs. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. mrope_section(`List(int)`): Multimodal rope section is for channel dimension of temporal, height and width in rope calculation. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. r3cSsg|]\}}||dqSrr1rrmr1r1r2rrz3apply_multimodal_rotary_pos_emb..r4rcSsg|]\}}||dqSrr1rr1r1r2rr.N)r(r enumeraterrr>repeat_interleaver) rrrr mrope_sectionZ unsqueeze_dimZ rotary_dimZq_rotZq_passZk_rotZk_passrrr1r1r2apply_multimodal_rotary_pos_embs"&&$$ ""rcseZdZdZd eeedfdd Zd ej e ej ej feej eej ee eej e ee ej eej ee ej fdddZZS) Glm4vTextAttentionz Multi-headed attention from 'Attention Is All You Need' paper. and "Generating Long Sequences with Sparse Transformers". NrQ layer_idxcst||_||_|j|_|j|_|j|j|_|j|_|j|j|_ d|_ |j |_ |j |_ |jd|_ tj|j|j|jdd|_tj|j|j|jdd|_tj|j|j|jdd|_tj|j|j|jdd|_dS)NTrrFF)r$r%rQrr-Znum_attention_headsrrrrrrrrr&rKq_projk_projv_projo_projr,rQrr/r1r2r%s   zGlm4vTextAttention.__init__)r;rrrpast_key_valuescache_positionrrXcKs6|\}} } ||} ||} ||} | || d|jdd} | || d|jdd} | || d|jdd} |\}}t| | |||jd\} } |dur|||d}| | | |j |\} } t }|j j dkrt|j j }||| | | |f|jsdn|j|jd|\}}||| d}||}||fS) Nr4rr3r)rrrrr)rr)sizerrrrbrrrrupdaterrrQrrrrrrrr)r,r;rrrrrrZbszZq_lenrrrrrrZ cache_kwargsrrrr1r1r2r<s@         zGlm4vTextAttention.forward)N)NNNN)rArBrC__doc__r rrwr%r(rdr= LongTensorr rrr<rDr1r1r/r2rs rcs0eZdZfddZejejdddZZS) Glm4vTextMLPcsPt||_tj|jd|jdd|_tj|j|jdd|_t |j |_ dS)Nr3FrF) r$r%rQr&rKr-rJ gate_up_projrNr rO activation_fnrr/r1r2r%Ds  zGlm4vTextMLP.__init__r`cCs4||}|jddd\}}|||}||S)Nr3r4r)r chunkr rN)r,r;Z up_statesZgater1r1r2r<Ls zGlm4vTextMLP.forward)rArBrCr%r( FloatTensorr<rDr1r1r/r2r Cs r cseZdZeedfdd Zd ejeejejfe eje ej e eeje e e e e ej eej e eej ej ffd ddZ ZS) Glm4vTextDecoderLayerrcstt|j|_t|||_t||_t|j|jd|_ t|j|jd|_ t|j|jd|_ t|j|jd|_ dS)Nr) r$r%r-r self_attnr rr"rinput_layernormpost_attention_layernormpost_self_attn_layernormpost_mlp_layernormrr/r1r2r%Vs   zGlm4vTextDecoderLayer.__init__NF) r;rrrroutput_attentions use_cacherrXc Kst|} ||}|jf||||||||d| \}} ||}| |}|} ||}||}||}| |}|S)N)r;rrrrrrr)rrrrrr) r,r;rrrrrrrrZresidualrr1r1r2r<`s,       zGlm4vTextDecoderLayer.forward)NNNFFN)rArBrCr rwr%r(rdr=rr rUrr<rDr1r1r/r2rUs$ rzJ Base class for Llava outputs, with hidden states and attentions. )Z custom_introc@steZdZUdZdZejed<dZe e ejed<dZ e e ejed<dZ e e ejed<dZe ejed<dS)Glm4vModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. Nlast_hidden_staterr; attentions rope_deltas)rArBrCrrr(rrvrrrr;r=rrr r1r1r1r2rs  rc@sDeZdZUeed<dZdZddgZdZdZ dZ dZ dZ e edZdS) Glm4vPreTrainedModelrQmodelTrrr)r;rN)rArBrCrrvbase_model_prefixZsupports_gradient_checkpointing_no_split_modulesZ_skip_keys_device_placementZ_supports_flash_attnZ_supports_sdpaZ_can_compile_fullgraphZ_supports_attention_backendrrZ_can_record_outputsr1r1r1r2rs rcsReZdZUeed<dgZddfdd ZddZej ej ej d d d Z Z S) Glm4vVisionModelrQrNrcstj|_j|_t|_t|_jj }t |d|_ t fddtjD|_tjjjd|_tjjd|_t jjjjjd|_tjjd|_d|_|dS)Nr3csg|] }tqSr1)r)rrr~r1r2rrz-Glm4vVisionModel.__init__..)rhryrOr)r]Z out_channelsrYrZF)r$r%spatial_merge_sizer[r}rrV patch_embedr-rrerr& ModuleListrdepthblocksrxrIrJrOmergerr"rpost_conv_layernormZConv2d downsamplepost_layernormgradient_checkpointing post_init)r,rQrr/r~r2r%s*      zGlm4vVisionModel.__init__c Csg}|D]\}}}t|dd|}|||j|j||j|j}|dddd}|}t|d|d}|||j|j||j|j}|dddd}|}|tj ||gdd |dqtj |dd}|ddddf }| |} | |d} | |fS)Nrr4rr3rr)r(rmrrrr rrappendrrrrr) r,grid_thwZpos_idsthwZhpos_idsZwpos_idsZ max_grid_sizeZrotary_pos_emb_fullrr1r1r2 rot_pos_embs4" zGlm4vVisionModel.rot_pos_emb)r;r,rXc Cs\||}||}||\}}tj||fdd}||f}t|dddf|dddf|dddfjdtj r|j ntj d}t j|ddd }|dd|dd}|||||dddf|dddf}|jD]} | |||d }q||}|d|j|j|jd}|dd dd}||d|jj}||}|S) az Args: hidden_states (`torch.Tensor` of shape `(seq_len, hidden_size)`): The final hidden states of the model. grid_thw (`torch.Tensor` of shape `(num_images_or_videos, 3)`): The temporal, height and width of feature shape of each image in LLM. Returns: `torch.Tensor`: hidden_states. r4rNrr3rr)rr)r)rrr)r!r&r0r(rrrrcumsumjit is_tracingr6Zint32rpadrrr$r(rbr r>rr'rQrIr%) r,r;r,rZimage_type_idsrrrZseqlensZblkr1r1r2r<s4  4*   zGlm4vVisionModel.forward) rArBrCr!rvrr%r0r(rdr<rDr1r1r/r2rs rcseZdZUeed<edfdd Zeed ee j ee j ee j ee e j ee j eeee j eeeeefd ddZZS) Glm4vTextModelrQr~cstj|_j|_tjj|j|_t fddt j D|_ t jjd|_td|_d|_|dS)Ncsg|]}t|qSr1)r)rrr~r1r2r.rz+Glm4vTextModel.__init__..rr~F)r$r%Z pad_token_idZ padding_idx vocab_sizer&rr- embed_tokensr"rZnum_hidden_layerslayersr"rnormr rotary_embr)r*rr/r~r2r%'s  zGlm4vTextModel.__init__N) input_idsrrr inputs_embedsrrrrXc Ks8|du|duArtd|r:|dur:tjs:t|jd}|durL||}|dur|durd|nd} tj| | |j d|j d}|dur| ddd d|j dd}n$| dkr|d  d|j dd}t|j|||||d } |} || |} |jD]$} | | f| | |||d |}|} q|| } t| |d S) N:You must specify exactly one of input_ids or inputs_embedsr~rrrrr4rr3)N.)rQZ input_embedsrrrr)rrrrr)rr) ValueErrorr(r2r3r rQr7get_seq_lengthrmr>rrrbrrhrr:r8r9r)r,r;rrrr<rrrZpast_seen_tokensrr;rZ decoder_layerZ layer_outputsr1r1r2r<7sV          zGlm4vTextModel.forward)NNNNNNN)rArBrCr rvr%rrrr(r rdrrrUrrrr=rr<rDr1r1r/r2r5#s,   r5csfeZdZUdZiZdZeed<ddgZfddZ dd Z d d Z d d Z ddZ d eejeejeejeejeejejfdddZd!ejeejdddZd"ejeejdddZd#ejejejejdddZeed$ejeejeejeeejeejeejeejeejeejeejeejeeeeefd ddZZ S)% Glm4vModelFrQrrcs:t|t|j|_t|j|_d|_ | dSrR) r$r%r _from_config vision_configvisualr5 text_configlanguage_modelrr*rr/r1r2r%s  zGlm4vModel.__init__cCs |jSrR)rGget_input_embeddingsr?r1r1r2rHszGlm4vModel.get_input_embeddingscCs|j|dSrR)rGset_input_embeddingsr,rr1r1r2rIszGlm4vModel.set_input_embeddingscCs ||_dSrRrGr,decoderr1r1r2 set_decoderszGlm4vModel.set_decodercCs|jSrRrKr?r1r1r2 get_decoderszGlm4vModel.get_decoderN)r;image_grid_thwvideo_grid_thwrrXc, Cs*|jjj}|jj}|jj}|jj}g} |durT|dusB|durT|} |durXt| }tjd|j d|j d|j |j d} d\} } d}| | j }t | D]\}}|||dk}|}g}d}|D]`}||krd}n ||krd}||kr|s|d q||kr |r |d q|d qg}tt |d d D]>\}}t|}|dd}|ddd}||||fqDg}d}|D]N\}}}t|dkr|ddnd}|d kr|| d|| d|| d}} }!|| ||!|}"}#}$t|"ddd|#|$}%t|#ddd|"d|$}&t|$ddd|"|#d}'|t|%|&|'g|| d7} d}q|d kr||| d|| d}} }!|| ||!|}"}#}$t|"D]}(t|(ddd|#|$}%t|#ddddd|$}&t|$dddd|#d}'|t|%|&|'g|q|d7}||| dkr| d7} d}|d7}n0||})|t|)dddd|d}qtj|dddd}*|* | j | d|||dkf<| |*dt| |qtj| |j dd} | | fS|dur| !dd} | "|dkd| dddd |j } | jddddjdddd}+|+d|j d} nNtj|j d|j ddddd|j dd} tj#|j ddg|j |j d} | | fSdS)aU Calculate the 3D rope index based on image and video's temporal, height and width in LLM. Explanation: Each embedding sequence contains vision embedding and text embedding or just contains text embedding. For pure text embedding sequence, the rotary position embedding has no difference with modern LLMs. Examples: input_ids: [T T T T T], here T is for text. temporal position_ids: [0, 1, 2, 3, 4] height position_ids: [0, 1, 2, 3, 4] width position_ids: [0, 1, 2, 3, 4] For vision and text embedding sequence, we calculate 3D rotary position embedding for vision part and 1D rotary position embedding for text part. Examples: Temporal (Time): 3 patches, representing different segments of the video in time. Height: 2 patches, dividing each frame vertically. Width: 2 patches, dividing each frame horizontally. We also have some important parameters: fps (Frames Per Second): The video's frame rate, set to 1. This means one frame is processed each second. tokens_per_second: This is a crucial parameter. It dictates how many "time-steps" or "temporal tokens" are conceptually packed into a one-second interval of the video. In this case, we have 25 tokens per second. So each second of the video will be represented with 25 separate time points. It essentially defines the temporal granularity. temporal_patch_size: The number of frames that compose one temporal patch. Here, it's 2 frames. interval: The step size for the temporal position IDs, calculated as tokens_per_second * temporal_patch_size / fps. In this case, 25 * 2 / 1 = 50. This means that each temporal patch will be have a difference of 50 in the temporal position IDs. input_ids: [V V V V V V V V V V V V T T T T T], here V is for vision. vision temporal position_ids: [0, 0, 0, 0, 50, 50, 50, 50, 100, 100, 100, 100] vision height position_ids: [0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1] vision width position_ids: [0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1] text temporal position_ids: [101, 102, 103, 104, 105] text height position_ids: [101, 102, 103, 104, 105] text width position_ids: [101, 102, 103, 104, 105] Here we calculate the text start position_ids as the max vision position_ids plus 1. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. Returns: position_ids (`torch.LongTensor` of shape `(3, batch_size, sequence_length)`) mrope_position_deltas (`torch.Tensor` of shape `(batch_size)`) Nrrrr6rr)rrFTimageZvideotextcSs|dS)Nrr1)rr1r1r2rz+Glm4vModel.get_rope_index..r4r3r.r>r5rq)$rQrDr image_token_idvideo_start_token_idvideo_end_token_idr(Z ones_liker)r>r6rrr7rrr+ itertoolsgroupbyrrritemrmrbrrrrrrrrrr1Z masked_fill_Zzeros),r,r;rPrQrr rVrWrXZmrope_position_deltasZtotal_input_idsrZ image_indexZ video_indexZvideo_group_indexrZ input_tokensZinput_token_typeZvideo_check_flgtokenZinput_type_grouprgroup start_indexZ end_indexZllm_pos_ids_listZvideo_frame_numZ modality_typeZ start_idxZend_idxZst_idxr-r.r/Z llm_grid_tZ llm_grid_hZ llm_grid_wZt_indexZh_indexZw_indexZt_idxZtext_lenZ llm_positionsZmax_position_idsr1r1r2get_rope_indexs;       "      """      """ $     zGlm4vModel.get_rope_indexpixel_values_videosrQc Cs||jj}g}|D]:\}}}td||gd|d}||qtj |dd}|j||d} | d|jj d } t | | } | S)a Encodes videos into continuous embeddings that can be forwarded to the language model. Args: pixel_values_videos (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input videos. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. rrrr,r4r3)rrEr6r(rr[rrr+rprodr rr) r,rarQZtemp_frames_hwr-r.r/Z repeated_rowZflattened_video_grid_thw video_embeds split_sizesr1r1r2get_video_featuresXs &  zGlm4vModel.get_video_features pixel_valuesrPcCsF||jj}|j||d}|d|jjd}t||}|S)a Encodes images into continuous embeddings that can be forwarded to the language model. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. rbr4r3)rrEr6rcr rr(r)r,rhrP image_embedsrer1r1r2get_image_featuresps  zGlm4vModel.get_image_features)r;r<image_featuresvideo_featuresc Cs6|durf||tj|jjtj|jdk}|d}||tj|jjtj|jdk}|d}n||jjk}||jjk}| }| d | |j}|dur|| | krtd|d|jd| }| d | |j}|dur.|| | kr.td|d|jd||fS)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. NrRr4z6Image features and image tokens do not match: tokens: z , features rz7Videos features and video tokens do not match: tokens: )rHr(rrQrVrrrallZvideo_token_idsumrZ expand_asr7Znumelr?r>) r,r;r<rkrlZspecial_image_maskZspecial_video_maskZn_image_tokensZn_video_tokensr1r1r2get_placeholder_masks0      zGlm4vModel.get_placeholder_mask) r;rrrr<rhrarPrQrrrrXc  Ks|du|duArtd|dur,||}|durz|||} tj| dd|j|j} |j||| d\}}| || }|dur| || }tj|dd|j|j}|j|||d\}}| ||}|dur~t |t s|n|d}|durB|j dkrBtj|dddfd d d }|jjrB|t|jj}d |}toz|durd|jd d kpz|duoz|jd d k}t o| dur| ddkp|dup|dk}|s|s|jdur|j||| |d \}} | |_n|j\}}}| dur| d|j|jnd}tj||jd}|d d|d}| dur`|j||jddd}||}|dddd}|jfd||||| d| }t|j |j!|j"|j#|jdS)a image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. Nr=rr)rk)rlZfull_attentionrr3)Zdim1Zdim2rj)rr>r4r)r;rrrr<r)rrr;rr)$r?rHrjr(rr7rrr6roZmasked_scatterrfrdictndimZdiagonalZis_floating_pointZfinfominrwrr>r@rr_rmrbrraddrrGrrrr;r)r,r;rrrr<rhrarPrQrrrriZ image_maskrrdZ video_maskZattention_mask_tensorZprefill_compiled_stageZprefill_noncompiled_stageZ batch_sizerdeltaoutputsr1r1r2r<s              zGlm4vModel.forward)NNNN)N)N)NN) NNNNNNNNNNN)!rArBrCr_checkpoint_conversion_mappingaccepts_loss_kwargsrrvrr%rHrIrNrOrr(r rdr=r_rrfrjrorrrrrrrr<rDr1r1r/r2rA|s~   @  )  rAzQ Base class for Glm4v causal language model (or autoregressive) outputs. c@seZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZeejed<dS) Glm4vCausalLMOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. Nlosslogitsrr;rr)rArBrCrrzrr(rrvr{rrr;r=rrr r1r1r1r2rys rycseZdZiZdgZdZfddZddZddZd d Z d d Z d&e j e e jdddZd'e j e e jdddZeddZeddZeed(e je e je e je ee j e e j e e je e je e j e e je e je e je e jeee jfeeeeefdddZd)fdd Zd*e e je e jee je jfdd d!Z d+ee!e e jee je"e#e$ffd#d$d%Z%Z&S),Glm4vForConditionalGenerationzlm_head.weightFcs<t|t||_tj|jj|jjdd|_ | dS)NFrF) r$r%rArr&rKrFr-r6lm_headr*rr/r1r2r%9s  z&Glm4vForConditionalGeneration.__init__cCs |jSrR)rrHr?r1r1r2rH@sz2Glm4vForConditionalGeneration.get_input_embeddingscCs|j|dSrR)rrIrJr1r1r2rICsz2Glm4vForConditionalGeneration.set_input_embeddingscCs|j|dSrR)rrNrLr1r1r2rNFsz)Glm4vForConditionalGeneration.set_decodercCs |jSrR)rrOr?r1r1r2rOIsz)Glm4vForConditionalGeneration.get_decoderNr`cCs|j||SrR)rrf)r,rarQr1r1r2rfLsz0Glm4vForConditionalGeneration.get_video_featuresrgcCs|j||SrR)rrj)r,rhrPr1r1r2rjQsz0Glm4vForConditionalGeneration.get_image_featurescCs|jjSrR)rrGr?r1r1r2rGUsz,Glm4vForConditionalGeneration.language_modelcCs|jjSrR)rrEr?r1r1r2rEYsz$Glm4vForConditionalGeneration.visualr)r;rrrr<labelsrhrarPrQrrlogits_to_keeprrXc Ks|jf|||| | ||||| d |}|d}t| trDt| dn| }||dd|ddf}d}|dur|j|||jjjd}t |||j |j |j |j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Glm4vForConditionalGeneration >>> model = Glm4vForConditionalGeneration.from_pretrained("THUDM/GLM-4.1V-9B-Thinking") >>> processor = AutoProcessor.from_pretrained("THUDM/GLM-4.1V-9B-Thinking") >>> messages = [ { "role": "user", "content": [ {"type": "image"}, {"type": "text", "text": "What is shown in this image?"}, ], }, ] >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) >>> inputs = processor(text=[text], images=[image], vision_infos=[vision_infos]) >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "The image shows a street scene with a red stop sign in the foreground. In the background, there is a large red gate with Chinese characters ..." ```) r;rhrarPrQrrrr<rrN)r{r~r6)rzr{rr;rr)rrrwslicer}Z loss_functionrQrFr6ryrr;rr)r,r;rrrr<r~rhrarPrQrrrrrvr;Z slice_indicesr{rzr1r1r2r<]s8= z%Glm4vForConditionalGeneration.forwardTc  sRtj|f||||||| | | |d | } d| d<|ddkrNd| d<d| d<| S)N) rrr<rrrhrarPrQrrrrhra)r$prepare_inputs_for_generation)r,r;rrr<rrrrhrarPrQrZ model_inputsr/r1r2rs*  z;Glm4vForConditionalGeneration.prepare_inputs_for_generation)r;r<rXc Cs|dur||tj|jjtj|jdkd}||tj|jjtj|jdkd}||tj|jjtj|jdkd}n$||jjk}||jjk}||jjk}tj | | dd}|dk}||@}|j dd} |j dd} | | fS)aa Get the number of images and videos for each sample to calculate the separation length of the sample tensor. These parameters are not passed through the processor to avoid unpredictable impacts from interface modifications. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Returns: image_nums (`torch.LongTensor` of shape `(batch_size, num_images_sample)`) video_nums (`torch.LongTensor` of shape `(batch_size, num_videos_sample)`) NrR).rrrr) rHr(rrQZimage_start_token_idrrrrWrXr1rwrn) r,r;r<Zis_imageZis_video_startZ is_video_endZ video_levelZ inside_videoZstandalone_imagesZ image_countsZ video_countsr1r1r2_get_image_nums_and_video_numss>      z|dkrt|}||||d ||<q>|d krt|t|}d d |D}||||d ||<q>|dkrt|}||||d ||<q>|d kr>|||t|d ||<q>|S)NrPrQr<)r<csDt||}|gdg|dtjfdd|Ddd}|S)Nrcsg|]}|jqSr1rrsampleZ repeat_argsr1r2r7rzGlm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visual.._repeat_interleave_samples..rr)r(rrhr)rr repeat_timessamplesresultr1rr2_repeat_interleave_samples4s zGlm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visual.._repeat_interleave_samplesrhcSsg|]}tj|ddqSrrr(rcrnrr1r1r2r?rz{Glm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visual..)rrracSsg|]}tj|ddqSrrrr1r1r2rKrr)rrr(rr) dict_to_expandrPrQZ image_numsZ video_numsrrrr)rr; model_kwargsr,r1r2"_expand_dict_for_generation_visual-sD         zgGlm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visualcsP|D]F}|dkr||durt||tjr|vr||jdd||<q|S)Nrrr)rr(rdr)rr)r visual_keysr1r2_expand_dict_for_generationZs z`Glm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generationrrZencoder_outputszMIf `is_encoder_decoder` is True, make sure that `encoder_outputs` is defined.)rrr?)r,rrr;rrrr1)rr;rr,rr2_expand_inputs_for_generations - z;Glm4vForConditionalGeneration._expand_inputs_for_generation)N)N) NNNNNNNNNNNNr) NNNNNTNNNN)N)rFN)'rArBrCrwZ_tied_weights_keysrxr%rHrIrNrOr(rrr rfrjpropertyrGrErrrdrrrwrrr=ryr<rrrUrqr|rrrDr1r1r/r2r|3s       _, :r|)r|rArr5)r)r)WrY dataclassesrtypingrrrrr(Ztorch.nnr&Ztorch.nn.functionalrrrZ activationsr Z cache_utilsr r Z generationr Z integrationsr Z masking_utilsrZmodeling_flash_attention_utilsrZmodeling_layersrZmodeling_outputsrrZmodeling_rope_utilsrrZmodeling_utilsrrZprocessing_utilsrutilsrrrrrZ utils.genericrZconfiguration_glm4vrr r!Z get_loggerrArModuler"rErVrerxr}rrdr=rrwrrnrrrrrrrr rrrrr5rAryr|__all__r1r1r1r2s            W  ^& 9K5nXC