a h% @sddlmZmZmZmZmZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZmZdd lmZmZdd lmZmZddlmZddlm Z m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(ddl)m*Z*e(rddl+m,Z,ddl-m.Z.m/Z/ndZ,e'r8ddl0m1Z1m2Z2nd\Z2Z1e#3e4Z5GdddeddZ6GdddZ7Gdddej8Z9d d!Z:ej;ee>ee d&d'd(Z?dId)d*Z@Gd+d,d,ej8ZAGd-d.d.ejj8ZBej;ed>ej8ZKGd?d@d@eZLe!GdAdBdBeZMe!GdCdDdDeMZNe!GdEdFdFeMe ZOgdGZPdS)J)AnyCallableOptional TypedDictUnionN)nn)ACT2FN)Cache)GenerationMixin)use_kernel_forward_from_hub)AttentionMaskConverter)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)deprecate_kwarg)is_causal_conv1d_availableis_mamba_2_ssm_available) BambaConfig)selective_state_update)mamba_chunk_scan_combined mamba_split_conv1d_scan_combined)causal_conv1d_fncausal_conv1d_update)NNc@s@eZdZUdZejed<ejed<eed<eed<ejed<dS)BambaFlashAttentionKwargsa Keyword arguments for advanced Flash Attention, causal-conv1d, and mamba_ssm kernel usage. Use cases include padding-free training and fewer `torch.compile` graph breaks. Attributes: cu_seq_lens_q (`torch.LongTensor`) Gets cumulative sequence length for query state. cu_seq_lens_k (`torch.LongTensor`) Gets cumulative sequence length for key state. max_length_q (`int`): Maximum sequence length for query state. max_length_k (`int`): Maximum sequence length for key state. seq_idx (`torch.IntTensor): Index of each packed sequence. Z cu_seq_lens_qZ cu_seq_lens_kZ max_length_qZ max_length_kseq_idxN) __name__ __module__ __qualname____doc__torch LongTensor__annotations__int IntTensorr/r/d/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/bamba/modeling_bamba.pyr$@s   r$F)totalc@seZdZdZdZejdfedddZdej ej e e e e efeej ej fddd Zejd d d Zde e e dddZdS) HybridMambaAttentionDynamicCachea A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. FNconfigc s0|j|_d|_|j}|j}g|_g|_g|_t|jD]}|j|dkr|jt j |j |j d|j |||dg7_|jt j |j|j||dg7_q6|jt jggdg7_|jt jggdg7_|j|q6fddt|jD|_fddt|jD|_dS) NFmambadevicedtyper8cs g|]}tjggdqSr:r*tensor.0_ batch_sizer8r/r0 z=HybridMambaAttentionDynamicCache.__init__..cs g|]}tjggdqSr;r<r>rAr/r0rCrD)layers_block_typehas_previous_state mamba_d_conv mamba_d_state conv_states ssm_statestransformer_layersrangenum_hidden_layersr*Zzeros mamba_expand hidden_sizemamba_n_groups mamba_n_heads mamba_d_headr=append key_cache value_cache)selfr4rBr9r8conv_kernel_sizessm_state_sizeir/rAr0__init__isB   z)HybridMambaAttentionDynamicCache.__init__) key_states value_states layer_idx cache_kwargsreturncCsz|j|jddkr*||j|<||j|<n