a ½ ÀhÇPã@sœddlZddlmZddlZddlmZddlmZmZddlm Z m Z ddl m Z ddl mZdd lmZd d lmZd d lmZmZmZmZmZmZmZmZd d lmZddlmZe  e!¡Z"dZ#dZ$Gdd„deƒZ%dd„Z&Gdd„dej'ƒZ(Gdd„de(ƒZ)Gdd„de(ƒZ*e(e)e*dœZ+Gdd„deƒZ,Gdd „d eƒZ-Gd!d"„d"eƒZ.Gd#d$„d$eƒZ/Gd%d&„d&eƒZ0Gd'd(„d(eƒZ1Gd)d*„d*eƒZ2gd+¢Z3dS),éN)ÚOptional)Únné)ÚCacheÚ StaticCache)Ú_flash_attention_forwardÚ!flash_attn_supports_top_left_mask)ÚPreTrainedModel)Úlogging)Údeprecate_kwargé)ÚGemmaForCausalLM)ÚLlamaDecoderLayerÚLlamaForQuestionAnsweringÚLlamaForSequenceClassificationÚLlamaForTokenClassificationÚ LlamaModelÚLlamaPreTrainedModelÚapply_rotary_pos_embÚ repeat_kv)Ú MistralMLPé)ÚDiffLlamaConfigzkajuma/DiffLlama-0.3B-handcutrc@s eZdZdS)Ú DiffLlamaMLPN©Ú__name__Ú __module__Ú __qualname__©rrúk/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/diffllama/modular_diffllama.pyr2srcCsddt d|¡S)Ngš™™™™™é?g333333ã?g333333Ó¿)ÚmathÚexp)Ú layer_idxrrrÚlambda_init_fn6sr#csšeZdZdZdeeedœ‡fdd„ Zedddd de j e e j e j fee j ee j ee eee j e e j ee j ee e j fd œd d „ƒZ‡ZS)ÚDiffLlamaAttentionz=Multi-headed attention from 'Attention Is All You Need' paperN©Úconfigr"cs¦tƒ ¡||_||_|dur4t d|jj›d¡|j|_|j |_ |j |_ t |d|j |j ƒ|_ |j|_|j |j|_|j|_|j|_d|_tj|j |j |j |jd|_tj|j |j|j |jd|_tj|j |j|j |jd|_tj|j |j |j |jd|_t|ƒ|_t tjd|j|j fd¡|_ t tjd|j|j fd¡|_!t tjd|j|j fd¡|_"t tjd|j|j fd¡|_#tj$d|j |j%d d |_&dS) NzInstantiating z¹ without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Úhead_dimT)Zbiasr)Úsizer F)ÚepsZelementwise_affine)'ÚsuperÚ__init__r&r"ÚloggerÚ warning_onceÚ __class__rÚattention_dropoutZ hidden_sizeZnum_attention_headsÚ num_headsÚgetattrr'Únum_key_value_headsÚnum_key_value_groupsZmax_position_embeddingsZ rope_thetaÚ is_causalrZLinearZattention_biasÚq_projÚk_projÚv_projÚo_projr#Ú lambda_initÚ ParameterÚtorchÚnormalÚlambda_std_devÚ lambda_q1Ú lambda_k1Ú lambda_q2Ú lambda_k2ZRMSNormZ rms_norm_epsÚ groupnorm©Úselfr&r"©r.rrr+=s4 ÿ zDiffLlamaAttention.__init__Úpast_key_valueÚpast_key_valuesú4.58©Únew_nameÚversionF©Ú hidden_statesÚposition_embeddingsÚattention_maskÚ position_idsrGÚ use_cacheÚcache_positionÚreturncKsb| ¡\} } } | } | |¡} | |¡}| |¡}|  | | |j|j¡ dd¡} | | | |j|j¡ dd¡}| | | |j|j¡ dd¡}|\}}t | |||ƒ\} }|durÈ|||dœ}|  |||j |¡\}}t ||j ƒ}t ||j ƒ}tjtj|ddddd}| dddd¡}t | | dd¡¡t |j¡}|durb|dd…dd…dd…d|jd…f}||}tjj|dtjd | j¡}tjj||j|jd }t tj |j!|j"dtjd¡ | j¡}t tj |j#|j$dtjd¡ | j¡}|||j%}t ||¡}tj|ddd\}}|||}d|j%| &|¡}| dd¡ '¡}| (| | d¡}| )|¡}||fS) Nrr ©ÚsinÚcosrR©Údiméÿÿÿÿréþÿÿÿ©rXÚdtype)ÚpÚtraining)*r(r5r6r7Úviewr0r'Ú transposer2rÚupdater"rr3r;ÚcatÚchunkÚrepeatÚmatmulr ÚsqrtÚshaperÚ functionalZsoftmaxÚfloat32Útor\Údropoutr/r^r!Úsumr>r?r@rAr9rBÚ contiguousÚreshaper8)rDrMrNrOrPrGrQrRÚkwargsÚbszZ target_lenÚ_Úq_lenÚ query_statesÚ key_statesÚ value_statesrVrUÚ cache_kwargsZ attn_weightsÚ causal_maskÚlambda_1Úlambda_2Ú lambda_fullÚ attn_outputÚ attn_output1Ú attn_output2rrrÚforward_sL         & ÿ ÿ   zDiffLlamaAttention.forward)N)NNNFN)rrrÚ__doc__rrÚintr+r r;ÚTensorÚtupleÚ LongTensorrÚboolr~Ú __classcell__rrrErr$:s$" øör$c szeZdZdZ‡fdd„Zeddddd ejeejejfe ej e ej e e e e ej eejdfd œd d „ƒZ ‡ZS)ÚDiffLlamaFlashAttention2aN DiffLlama flash attention module. This module inherits from `DiffLlamaAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cstƒj|i|¤Žtƒ|_dS)N)r*r+rÚ_flash_attn_uses_top_left_mask)rDÚargsrorErrr+¦sz!DiffLlamaFlashAttention2.__init__rFrGrHrINFrLc Cs$t|tƒrtdƒ‚| ¡\}} } | |¡} | |¡} | |¡} |  || |j|j ¡  dd¡} |  || |j |j ¡  dd¡} |  || |j |j ¡  dd¡} |dur¶t   d¡| | |¡\}}n|\}}t| | ||ƒ\} } |durú|||dœ}| | | |j|¡\} } |   dd¡} |   dd¡} |   dd¡} |jr,|jnd}| j}| jjdkrL| jjnd}|tjkrØt ¡r†ttd ƒr|t |¡nt ¡}n"t|jd ƒrž|jj}n |jjj}t   d |›d ¡|   |¡} |   |¡} |   |¡} tj!| ddd \}}| "dddd¡}| "dddd¡}t#| | ||| ||t$|ddƒ|j%|j&d }t#| | ||| ||t$|ddƒ|j%|j&d }tj'||gdd }tj!|ddd \}}t (tj)|j*|j+dtjd¡  | j¡}t (tj)|j,|j-dtjd¡  | j¡}|||j.}|||}d|j.| /|¡}| 0|| d¡ 1¡}| 2|¡}|dfS)NzÈ`static` cache implementation is not compatible with `attn_implementation==flash_attention_2` make sure to use `sdpa` in the mean time, and open an issue at https://github.com/huggingface/transformersrr aYThe attention layers in this model are transitioning from computing the RoPE embeddings internally through `position_ids` (2D tensor with the indexes of the tokens), to using externally computed `position_embeddings` (Tuple of tensors, containing cos and sin). In v4.46 `position_ids` will be removed and `position_embeddings` will be mandatory.rTçZmpsÚcpuÚget_autocast_dtypeÚ_pre_quantization_dtypez¾The input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in Ú.rWÚsliding_window)rPrkrŽZuse_top_left_maskr4rYr[)3Ú isinstancerÚ ValueErrorr(r5r6r7r_r0r'r`r2r,r-Z rotary_embrrar"r^r/r\ÚdeviceÚtyper;riZis_autocast_enabledÚhasattrr‹Zget_autocast_gpu_dtyper&rŒÚweightrjrcrdrr1r‡r4rbr!rlr>r?r@rAr9rBrnrmr8)rDrMrNrOrPrGrQrRrprrrqrsrtrurVrUrvZ dropout_rateZ input_dtypeZ device_typeZ target_dtypeZ value_states1Z value_states2r|r}r{rxryrzrrrr~®sª ÿ   ÿ      ÿý  þÿ    ö  ö  ÿ ÿ  z DiffLlamaFlashAttention2.forward)NNNFN)rrrrr+r r;rr‚rrƒrr„r~r…rrrErr†Ÿs$  ø ÷r†c@s|eZdZdZeddddd ejeejejfeejeej ee e eej eejeejeeejfdœd d „ƒZ dS) ÚDiffLlamaSdpaAttentiona DiffLlama attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from `DiffLlamaAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to SDPA API. rFrGrHrINFrLcKsf| ¡\} } } | |¡} | |¡} | |¡}|  | | |j|j¡ dd¡} |  | | |j|j¡ dd¡} | | | |j|j¡ dd¡}|\}}t | | ||ƒ\} } |durÄ|||dœ}|  | ||j |¡\} }t | |j ƒ} t ||j ƒ}tjtj|ddddd}| dddd¡}|}|dur:|dd…dd…dd…d| jd…f}| jjdkrj|durj|  ¡} |  ¡} | ¡}|duoz| dk}tjjj| | |||jrš|jnd|d }tj|ddd\}}t tj|j|jdtjd ¡  | j!¡}t tj|j"|j#dtjd ¡  | j!¡}|||j$}|||}d|j$| %|¡}| dd¡ ¡}| | | d¡}| &|¡}|dfS) Nrr rTrWrYrZÚcudar‰)Z attn_maskZ dropout_pr4r[)'r(r5r6r7r_r0r'r`r2rrar"rr3r;rbrcrdrgr‘r’rmrrhZscaled_dot_product_attentionr^r/r!rlr>r?rirjr\r@rAr9rBr8)rDrMrNrOrPrGrQrRrorprrrqrsrtrurVrUrvrwr4r{r|r}rxryrzrrrr~<s\        &ú  ÿ ÿ  zDiffLlamaSdpaAttention.forward)NNNFN) rrrrr r;rr‚rrƒrr„r~rrrrr•4s" øör•)ÚeagerZflash_attention_2Zsdpacs$eZdZeedœ‡fdd„ Z‡ZS)ÚDiffLlamaDecoderLayerr%cs&tƒ ||¡t|j||d|_dS)Nr%)r*r+ÚDIFFLLAMA_ATTENTION_CLASSESZ_attn_implementationZ self_attnrCrErrr+‘szDiffLlamaDecoderLayer.__init__)rrrrr€r+r…rrrErr˜sr˜c@seZdZdZdZdd„ZdS)ÚDiffLlamaPreTrainedModelFcCsjt ||¡t|tƒrf|jj d|jj¡|j j d|jj¡|j j d|jj¡|j j d|jj¡dS)Nr) r Ú _init_weightsrr$r>ÚdataZnormal_r&r=r?r@rA)rDÚmodulerrrr››s   z&DiffLlamaPreTrainedModel._init_weightsN)rrrZ_supports_flex_attnZ_supports_attention_backendr›rrrrrš—sršc@s eZdZdS)ÚDiffLlamaModelNrrrrrrž¤sržc@s eZdZdS)ÚDiffLlamaForCausalLMNrrrrrrŸ¨srŸc@s eZdZdS)Ú"DiffLlamaForSequenceClassificationNrrrrrr ¬sr c@s eZdZdS)ÚDiffLlamaForQuestionAnsweringNrrrrrr¡°sr¡c@s eZdZdS)ÚDiffLlamaForTokenClassificationNrrrrrr¢´sr¢)ršržrŸr r¡r¢)4r Útypingrr;rZ cache_utilsrrZmodeling_flash_attention_utilsrrZmodeling_utilsr Úutilsr Zutils.deprecationr Zgemma.modeling_gemmar Zllama.modeling_llamarrrrrrrrZmistral.modeling_mistralrZconfiguration_diffllamarZ get_loggerrr,Z_CHECKPOINT_FOR_DOCZ_CONFIG_FOR_DOCrr#ÚModuler$r†r•r™r˜ršržrŸr r¡r¢Ú__all__rrrrÚsB      (   eVý