a h-@sddlZddlmZddlZddlmZmZddlmZddlm Z e e Z Gdddej ZGd d d ej ZGd d d ej ZGd ddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddeZGdd d ej Zeed!d"d#feeed$d%ifeeeejeeeejeejejejejej ej!ed&Z"ee"Z#d'd(Z$e$d)Z%e$d*Z&e$d+Z'e$d,Z(e$d-Z)e$d.Z*e$d/Z+e$d0Z,dS)1N) OrderedDict)Tensornn)logging)is_torchdynamo_compilingc@s eZdZdZeedddZdS)PytorchGELUTanha& A fast C implementation of the tanh approximation of the GeLU activation function. See https://huggingface.co/papers/1606.08415. This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical match due to rounding errors. inputreturncCstjj|ddS)Ntanh)Z approximate)r functionalgeluselfr rT/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/activations.pyforward%szPytorchGELUTanh.forwardN__name__ __module__ __qualname____doc__rrrrrrrsrc@s eZdZdZeedddZdS)NewGELUActivationz Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 r c Cs6d|dttdtj|dt|dS)N??@Hm?g@)torchr mathsqrtpipowrrrrr/szNewGELUActivation.forwardNrrrrrr)srcsHeZdZdZd edfdd ZeedddZeedd d ZZ S) GELUActivationa Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 F)use_gelu_pythoncs&t|r|j|_n tjj|_dSN)super__init__ _gelu_pythonactrr r)rr$ __class__rrr';s  zGELUActivation.__init__r cCs |ddt|tdS)Nrrr)rerfrr rrrrr(BszGELUActivation._gelu_pythoncCs ||Sr%r)rrrrrEszGELUActivation.forward)F) rrrrboolr'rr(r __classcell__rrr*rr#3sr#c@s eZdZdZeedddZdS)FastGELUActivationz} Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs r cCs*d|dt|ddd||S)Nrrg3E?r)rr rrrrrNszFastGELUActivation.forwardNrrrrrr0Isr0c@s eZdZdZeedddZdS)QuickGELUActivationzr Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs r cCs|td|S)NgZd;?)rsigmoidrrrrrWszQuickGELUActivation.forwardNrrrrrr1Rsr1cs8eZdZdZeedfdd ZeedddZZS)ClippedGELUActivationa Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to https://huggingface.co/papers/2004.09602. Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://huggingface.co/papers/1606.08415 minmaxcs8||krtd|d|dt||_||_dS)Nzmin should be < max (got min: z, max: )) ValueErrorr&r'r5r6)rr5r6r*rrr'hs  zClippedGELUActivation.__init__xr cCstt||j|jSr%)rZcliprr5r6)rr:rrrrpszClippedGELUActivation.forward) rrrrfloatr'rrr/rrr*rr3[s r3cs0eZdZdZfddZeedddZZS)AccurateGELUActivationz Applies GELU approximation that is faster than default and more accurate than QuickGELU. See: https://github.com/hendrycks/GELUs Implemented along with MEGA (Moving Average Equipped Gated Attention) cs ttdtj|_dS)N)r&r'rr r!precomputed_constantrr*rrr'|s zAccurateGELUActivation.__init__r c Cs,d|dt|j|dt|dS)Nrrr)rr r>r"rrrrrszAccurateGELUActivation.forward)rrrrr'rrr/rrr*rr<ts r<cs@eZdZdZfddZeedddZeedddZZS) MishActivationz See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://huggingface.co/papers/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish csttjj|_dSr%)r&r'rr mishr)r?r*rrr's zMishActivation.__init__r cCs|ttj|Sr%)rr rr softplusrrrr _mish_pythonszMishActivation._mish_pythoncCs ||Sr%r-rrrrrszMishActivation.forward) rrrrr'rrDrr/rrr*rrAs rAc@s eZdZdZeedddZdS)LinearActivationz[ Applies the linear activation function, i.e. forwarding input directly to output. r cCs|Sr%rrrrrrszLinearActivation.forwardNrrrrrrEsrEc@seZdZdZdddZdS)LaplaceActivationz Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See https://huggingface.co/papers/2209.10655 Inspired by squared relu, but with bounded range and gradient for better stability 绹۞? ^/ ?cCs*|||td}ddt|S)Nrrr)divrr rr,)rr musigmarrrrszLaplaceActivation.forwardN)rGrHrrrrrrrrrrFsrFc@seZdZdZddZdS)ReLUSquaredActivationz` Applies the relu^2 activation introduced in https://huggingface.co/papers/2109.08668v2 cCstj|}t|}|Sr%)rr relurZsquare)rr Z relu_appliedZsquaredrrrrs  zReLUSquaredActivation.forwardNrLrrrrrMsrMcseZdZfddZZS)ClassInstantiercs4t|}t|tr|n|if\}}|fi|Sr%)r& __getitem__ isinstancetuple)rkeycontentclskwargsr*rrrPs zClassInstantier.__getitem__)rrrrPr/rrr*rrOsrOcs`eZdZdZddddejdffdd Zeedd d Zeedd d Z eed ddZ Z S)XIELUActivationz Applies the xIELU activation function introduced in https://arxiv.org/abs/2411.13010 If the user has installed the nickjbrowning/XIELU wheel, we import xIELU CUDA Otherwise, we emit a single warning and use xIELU Python g?rgưFc stttttj||ddd|_ ttttj|||ddd|_ | dtj||d| dtj||d||_ t |j |_t |j |_d|_zddl}tjj|_d}z$ddlm} | |j|_|d7}Wn>tyT} z$|d | d 7}|j|_WYd} ~ n d} ~ 00t |Wn6ty} zt d t!| WYd} ~ n d} ~ 00dS) N)dtyperrbetaepszUsing experimental xIELU CUDA.)allow_in_graphz& Enabled torch._dynamo for xIELU CUDA.z+ Could not enable torch._dynamo for xIELU (z*) - this may result in slower performance.uCUDA-fused xIELU not available (%s) – falling back to a Python version. For CUDA xIELU (experimental), `pip install git+https://github.com/nickjbrowning/XIELU`)"r&r'r ParameterrlogexpZtensor unsqueezealpha_palpha_nZregister_bufferwith_vector_loadsr;rYdetachcpuitem _beta_scalarrZ _eps_scalar_xielu_cuda_objZ xielu.opsclassesxieluZXIELUZ torch._dynamor[ _xielu_cuda_xielu_cuda_fn Exceptionlogger warning_oncestr) rZ alpha_p_initZ alpha_n_initrYrZrXrbrjmsgr[errr*rrr's8 ,&   zXIELUActivation.__init__r9c Cshtj|j}|jtj|j}t|dk||||j|tt ||j |||j|S)Nr) rr rCr`rYrarwhereexpm1r5rZ)rr:r`rarrr _xielu_pythons$zXIELUActivation._xielu_pythoncCs|j}|dkr|d}q|dkr>|dd|d}||jkrXtd||j|j||j |j |j |j |j }||S)zDFirewall function to prevent torch.compile from seeing .item() callsr@rrz_Warning: xIELU input tensor expects 3 dimensions but got (shape: %s). Reshaping to (shape: %s).)shapedimr_viewsizernrorhrr`rarfrgrb)rr:Zoriginal_shaperesultrrrrks(     zXIELUActivation._xielu_cudar cCs4|jdur*|jr*ts ||Std||S)Nz:torch._dynamo is compiling, using Python version of xIELU.)rhZis_cudarrlrnrorurrrrrs   zXIELUActivation.forward) rrrrrZbfloat16r'rrurkrr/rrr*rrWs + rWi r4r$T)rZgelu_10 gelu_fastgelu_new gelu_pythonZgelu_pytorch_tanhZ gelu_accurateZlaplaceZ leaky_relulinearrB quick_gelurNZrelu2Zrelu6r2siluZswishr ZprelurjcCs0|tvrt|Std|dttdS)Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)Zactivation_stringrrrget_activation5srrr~rr}rrrBr)-r collectionsrrrrutilsrZutils.import_utilsrZ get_loggerrrnModulerrr#r0r1r3r<rArErFrMrOrWZ LeakyReLUZReLUZReLU6ZSigmoidZSiLUZTanhZPReLUZACT2CLSrrrr~rr}rrrBZ linear_actrrrrsf           _