a hB6@sUddlZddlZddlmZmZddlZddlmZejdedZiZ e ej j efe d<ehdZeeeegefdd d Zed d d'd dddejejejeejeeeejdddZeeeedddZejeeejdddZejejeeeeeejdddZeeddddZejejeeeeejddd Zed!d d(d ddddd"dd#ejejejeejeejeejeeeeeeeeeeejejejejfd$d%d&ZdS))N)CallableOptional)_dtype_mappings_T)boundONNX_ATEN_DECOMP_TABLE> )op_type opset_versionreturncsttdfdd }|S)zDDecorator to register an ONNX operator with a custom implementation.)funcrcsPd}tjjdd|dd|}|ttttjj|<|||S)NZopsetzonnx::.)Z mutates_args)torchlibraryZ custom_oprgetattropsZonnxZ register_fake)roverloadZtorch_opr r rP/var/www/html/assistant/venv/lib/python3.9/site-packages/torch/onnx/ops/_impl.py decorators  z_onnx_op..decorator)r)r r rrrr_onnx_ops rZRotaryEmbeddingF) interleaved num_headsrotary_embedding_dim)x cos_cache sin_cache position_idsrrrrcsPjd}jd}tjdkrdjd} t|dkfdd| |} |||| g} t| ttjdkddjd} |dkr| }d d d d d d d |f} d d d d d d |d f} |d}|d ur||}||}n|}|}|d d d d d |f}|d d d d d |f}t|d}t|d}|r| d d d d d d dd df}| d d d d d d dd df}ntj| dd d \}}||||}||||}|r t|d }t|d }tj||fd d }t|| j} ntj||fd d } tj| | fd d }tjdkrLt|j}|S) z_RotaryEmbedding-23 https://onnx.ai/onnx/operators/onnx__RotaryEmbedding.html#rotaryembedding-23rr cs djS)NzKnum_heads must be provided for 3D inputs. Received input tensor with shape )shaperrrr>z%rotary_embedding_23..cSsdS)Nzx should be a 4D tensor by nowrrrrrr'Cr(Ndim)r%lenr_checkZreshapeZ unsqueezechunkcat)rr r!r"rrr batch_sizesequence_length hidden_size head_sizeZ new_shapeZx_rotateZ x_not_rotateZrotary_embedding_dim_halfcossinx1Zx2realimagZx_rotate_concatoutputrr&rrotary_embedding_23+sn         "$  r;)scaler4rcCs|dur |Sdt|S)z/Get the scale factor for attention computation.Ng?)mathsqrt)r<r4rrr_get_scale_factorsr?)tensorr1rrcCs:|jd|jd}}||}|||||ddS)z1Reshape 3D tensor to 4D for multi-head attention.r r$)r%view transpose contiguous)r@r1rr2r3r4rrr_reshape_3d_to_4ds rD)QKcurrent_q_num_headscurrent_kv_num_headsr<qk_matmul_output_moderc Cs6|dkrt|||||Stt||ddSdS)z1Get QK output tensor based on the specified mode.rr*N)_compute_qk_output_for_mode_0rZ zeros_likematmulrB)rErFrGrHr<rIrrr_get_qk_output_for_aten_spdas  rM)rGrHrcs"tdkfdddS)z-Validate Group Query Attention configuration.rcsdddS)Nz q_num_heads (z%) must be divisible by kv_num_heads (z ) for GQArrrHrGrrr'r(z-_validate_gqa_configuration..N)rr.)rGrHrrNr_validate_gqa_configurations  rO)rErFrGrHr<rc Cs`|}||kr"||}|j|dd}t||jd}t|}||} ||} t| | ddS)zDHelper function to compute QK output for qk_matmul_output_mode == 0.r r+r#rJr*)repeat_interleaver?r%r=r>rrLrB) rErFrGrHr<ZK_for_qk repeat_factor scale_factor sqrt_scaleQ_scaledK_scaledrrrrKs  rKZ Attention) is_causal kv_num_heads q_num_headsrIr<softcapsoftmax_precision)rErFV attn_maskpast_key past_valuerWrXrYrIr<rZr[rc( Csd\} }}t|j}|jd}t|jdkrvt|dko>|dkdd|jd}t|||}t|||}t|||}tt|jdkot|jdkot|jdkdd|j|}t| |} |d urtj||g|d n|}|d urtj||g|d n|}||}}|j| }|j| }|j|}|j|}| d koh| dkoh| d uoh|d uph|jtj k}t |||rd }|d ur|jtj kr|n|}tj j j ||||d || t ||kd }t||||| | }n||kr ||}|j|| d }|j|| d }tj|||j|jd }|rht|d uddttj||tj |jd }||td}|d ur|jtj kr||td}n||}t| |jd} t| }!||!}"||!}#t|"|#dd}$|$}|$|}%| dkr|%}| d kr| t|%| }%| dkr|%}| d urr| tvrb|%j}&|%tj| }%tj|%dd }'|'|&}'ntj|%dd }'ntj|%dd }'| dkr|'}t|'|}|dkr|dd !||d}||||fS)zMAttention-23 https://onnx.ai/onnx/operators/onnx__Attention.html#attention-23)r r$r#rr#cSsdS)Nz;q_num_heads and kv_num_heads must be provided for 3D inputsrrrrrr'r(zattention_23..r r)cSsdS)Nz'Q, K, and V should be 4D tensors by nowrrrrrr'r(Nr+rV)r]Z dropout_prWr<Z enable_gqa)dtypedevicecSsdS)Nz'Cannot use both is_causal and attn_maskrrrrrr'@r(z-infrJr*r$)"r-r%rr.rDr?r0cloner`boolrOnnZ functionalZscaled_dot_product_attentionrMrPZzerosraZtrilZonesZ masked_fillfloatr=r>rLrBtanh-_ATTENTION_23_ALLOWED_INTERMEDIATE_PRECISIONStorZONNX_DTYPE_TO_TORCH_DTYPEZsoftmaxrCrA)(rErFr\r]r^r_rWrXrYrIr<rZr[Z num_head_dimZ sequence_dimZhead_dimZinput_shape_lenr1Zq_sequence_lengthZ q_head_sizeZ present_keyZ present_valuerGrHZkv_sequence_lengthZ can_use_sdpaZsdpa_attn_maskr:Z qk_outputrQZ attn_biasZ causal_maskrRrSrTrUZqk_matmul_outputZ qk_with_biasZoriginal_dtypeZ qk_softmaxrrr attention_23s       (                         ri)N)NNN)r=typingrrrZtorch.onnx.opsrTypeVarrrdictZ_opsZ OpOverload__annotations__ frozensetrgstrintrZTensorrcr;rer?rDrMrOrKtuplerirrrrs   U