a h@sddlmZmZddlZddlmZddlmZddlmZddl m Z ddl m Z dd l mZd d lmZmZmZmZd d lmZe eZGdddeddZGdddejZGdddeZGdddeZGdddeZGdddeZgdZ dS))Optional TypedDictN)nn)ACT2FN)Cache)Unpack)logging)deprecate_kwarg)GraniteMoeDecoderLayerGraniteMoeForCausalLMGraniteMoeModelGraniteMoePreTrainedModel)GraniteMoeSharedConfigc@s@eZdZUdZejed<ejed<eed<eed<ejed<dS)GraniteFlashAttentionKwargsa Keyword arguments for advanced Flash Attention, causal-conv1d, and mamba_ssm kernel usage. Use cases include padding-free training and fewer `torch.compile` graph breaks. Attributes: cu_seq_lens_q (`torch.LongTensor`) Gets cumulative sequence length for query state. cu_seq_lens_k (`torch.LongTensor`) Gets cumulative sequence length for key state. max_length_q (`int`): Maximum sequence length for query state. max_length_k (`int`): Maximum sequence length for key state. seq_idx (`torch.IntTensor): Index of each packed sequence. Z cu_seq_lens_qZ cu_seq_lens_kZ max_length_qZ max_length_kZseq_idxN) __name__ __module__ __qualname____doc__torch LongTensor__annotations__intZ IntTensorrry/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/granitemoeshared/modular_granitemoeshared.pyr&s   rF)totalcs:eZdZdZedfdd ZejejdddZZ S)GraniteMoeSharedMLPz~ MLP layer for shared experts Args: config: Configuration object with model hyperparameters. configcsZt|j|_|j|_t|j|_tj |j|jddd|_ tj |j|jdd|_ dS)Nr F)Zbias) super__init__Z hidden_sizeZ input_sizeshared_intermediate_sizerZ hidden_act activationrZLinear input_linear output_linearselfr  __class__rrr"Hs   zGraniteMoeSharedMLP.__init__) hidden_statesreturncCs<||}|jddd}||d|d}||}|S)Nr )dimrr)r%chunkr$r&)r(r+Zchunked_hidden_statesrrrforwardQs   zGraniteMoeSharedMLP.forward) rrrrrr"rTensorr0 __classcell__rrr)rr?s rcseZdZeedfdd Zeddddd eje eje ej e e e e e e e ej e e e e ejejfeee eje e ejejffd d d ZZS)GraniteMoeSharedDecoderLayer)r layer_idxcs*t|||jdkrdnt||_dS)Nr)r!r"r#r shared_mlp)r(r r4r)rrr"Zsz%GraniteMoeSharedDecoderLayer.__init__Zpast_key_valuepast_key_valuesz4.58)new_nameversionNF) r+attention_mask position_idsr6output_attentions use_cachecache_positionoutput_router_logitsposition_embeddingskwargsr,c  Ks|} ||}|jf|||||||| d| \}} | ||j}|} ||}||\} }|jdurn| }n| ||}~ | ||j}|f}|r|| f7}|r||f7}|S)aD Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs. Can be used to provide `GraniteFlashAttentionKwargs` for padding-free training and/or improve torch.compile performance. )r+r9r:r6r;r<r=r?N)Zinput_layernormZ self_attnZresidual_multiplierZpost_attention_layernormZblock_sparse_moer5)r(r+r9r:r6r;r<r=r>r?r@ZresidualZself_attn_weightsZmoe_hidden_statesZ router_logitsoutputsrrrr0^s:'       z$GraniteMoeSharedDecoderLayer.forward)NNNFFNFN)rrrrrr"r rr1rrrbooltuplerrZ FloatTensorr0r2rrr)rr3Ys. r3c@seZdZUeed<dgZdS)GraniteMoeSharedPreTrainedModelr r3N)rrrrrZ_no_split_modulesrrrrrDs rDcs"eZdZedfdd ZZS)GraniteMoeSharedModelrcs0ttfddtjD|_dS)Ncsg|]}t|qSr)r3).0r4rrr z2GraniteMoeSharedModel.__init__..)r!r"rZ ModuleListrangeZnum_hidden_layersZlayersr'r)rrr"s zGraniteMoeSharedModel.__init__)rrrrr"r2rrr)rrEsrEcs(eZdZdgZedfdd ZZS)GraniteMoeSharedForCausalLMzlm_head.weightrcs"t|t||_|dS)N)r!r"rEmodelZ post_initr'r)rrr"s  z$GraniteMoeSharedForCausalLM.__init__)rrrZ_tied_weights_keysrr"r2rrr)rrJsrJ)rJrErD)!typingrrrrZ activationsrZ cache_utilsrZprocessing_utilsrutilsr Zutils.deprecationr Zgranitemoe.modeling_granitemoer r rrZconfiguration_granitemoesharedrZ get_loggerrloggerrModulerr3rDrErJ__all__rrrrs"        X