a h@sddlZddlZddlmZddlmZmZddlZddlm m Z ddlm Z ddl m Z mZmZddlmZddlmZdd lmZdd lmZmZmZmZmZmZdd lmZmZdd l m!Z!dd l"m#Z#m$Z$m%Z%ddl&m'Z'ddl(m)Z)e$r ddl*m+Z+ddl,m-Z-ddl.m/Z/ne0Z-e%1e2Z3Gdddej4j5Z6dPeej7ee8dddZ9Gddde-Z:Gddde j;Zd!d"Z?dQd#d$Z@dRd&ej7ej7ej7eejAeBe8e8fe8e8eeCeeBej7ej7feBej7fd' d(d)ZDejEfd&ej7e:ej7e8eBe8e8fe8e8ejFeBej7d* d+d,ZGd&ej7ej7ej7eejAeBe8e8fe8e8eBej7d- d.d/ZHeGeDeHd0ZIGd1d&d&e j;ZJGd2d3d3eZKe#Gd4d5d5e!ZLdSej7ej7eej7eej7eBej7ej7ej7e8eej7eej7fd6d7d8ZMej7ej7e8e8ej7d9d:d;ZNe#Gdd?d?e j;ZPe#d@dAGdBdCdCeLZQe#dDdAGdEdFdFeLZRe#dGdAGdHdIdIeLZSe#GdJdKdKeLZTe#dLdAGdMdNdNeLZUgdOZVdS)TN) nullcontext)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)PreTrainedModel)auto_docstringis_flash_attn_2_availablelogging)is_triton_available)ModernBertConfig) flash_attn_varlen_qkvpacked_func)RotaryEmbedding) apply_rotaryc@s8eZdZedeejeedddZeddZ dS)ApplyRotaryEmbUnpadN cu_seqlens max_seqlenc Csd|}|j\}}}} |ddddf|d| } t| ||d||ddd||||||_|S)NrFT)seqlen_offsetsr!r" interleavedinplace) contiguousshapeviewrZsave_for_backwardr") ctxqkvcossinr!r" total_nnz_three_nheadsheaddimZqkr3n/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/modernbert/modeling_modernbert.pyforward>s  zApplyRotaryEmbUnpad.forwardc Csn|j\}}}|}|j\}}}}|ddddf|d|} t| ||d||jdddd |ddddddfS)Nr#r$rFT)r%r!r"r&r' conjugate)Z saved_tensorsr(r)r*rr") r+dor-r.r!r/r0r1r2Zdqkr3r3r4backward]s   zApplyRotaryEmbUnpad.backward)NN) __name__ __module__ __qualname__ staticmethodrtorchTensorintr5r8r3r3r3r4r=srr cCst|||||S)a Arguments: qkv: (total_nnz, 3, nheads, headdim) - input tensor for packed QKV. cos, sin: (seqlen_rotary, rotary_dim / 2) interleaved: if True, rotate pairs of even and odd dimensions (GPT-J style) instead of 1st half and 2nd half (GPT-NeoX style). inplace: if True, apply rotary embedding in-place. seqlen_offsets: (batch_size,) or int. Each sequence in x is shifted by this amount. Most commonly used in inference when we have KV cache. cu_seqlens: (batch + 1,) or None max_seqlen: int Return: out: (total_nnz, dim) rotary_dim must be <= headdim Apply rotary embedding to the first rotary_dim of x. )rapply)r,r-r.r!r"r3r3r4apply_rotary_unpaddedtsrAc seZdZdZd eeeeeejeej dfdd Z dej ej eee ej e ej ej ffddd Zed d d ZZS)!ModernBertUnpaddedRotaryEmbeddingzP The rotary position embeddings applied directly to unpadded sequences. @N)dimbaser"devicedtypecsFtj|||dd||_|durB|durB|durB|j|||ddS)a max_seqlen: if max_seqlen, device, and dtype are provided, we precompute the cos_sin_cache up to max_seqlen. If the max_seqlen, device, or dtype during training/inference differ, the cos_sin_cache will be recomputed during the forward pass. F)rDrErFr&NrFrG)super__init__r"_update_cos_sin_cache)selfrDrEr"rFrG __class__r3r4rJs z*ModernBertUnpaddedRotaryEmbedding.__init__)r,r!r"returncCs6|dur|j||j|jdt||j|j||d}|S)z Apply rotary embedding *inplace* to qkv. qkv: (total_nnz, 3, nheads, headdim) cu_seqlens: (batch + 1,) cumulative sequence lengths max_seqlen: int max seq length in the batch NrHr )rKrFrGrAZ _cos_cachedZ _sin_cached)rLr,r!r"r3r3r4r5s z)ModernBertUnpaddedRotaryEmbedding.forward)rOcCsd|jd|jd|jS)Nzdim=z, base=z , scale_base=)rDrEZ scale_baserLr3r3r4 extra_reprsz,ModernBertUnpaddedRotaryEmbedding.extra_repr)rCNNN)N)r9r:r;__doc__r?floatrr=rFrGrJr>rtupler5strrQ __classcell__r3r3rMr4rBs( rBcsheZdZdZedfdd Zejddejej ddd Z de eje ej ej d d d Z Z S)ModernBertEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. configcsRt||_tj|j|j|jd|_tj |j|j |j d|_ t |j|_dS)N)Z padding_idxepsbias)rIrJrYrZ Embedding vocab_size hidden_sizeZ pad_token_idtok_embeddings LayerNormnorm_eps norm_biasnormDropoutZembedding_dropoutdroprLrYrMr3r4rJs  zModernBertEmbeddings.__init__TZdynamic) input_idsrOcCs||||SN)rercr_)rLrhr3r3r4compiled_embeddingssz(ModernBertEmbeddings.compiled_embeddingsN)rh inputs_embedsrOcCsF|dur|||}n(|jjr,||n||||}|Sri)rercrYreference_compilerjr_)rLrhrk hidden_statesr3r3r4r5s zModernBertEmbeddings.forward)NN)r9r:r;rRrrJr=compile LongTensorr>rjrr5rVr3r3rMr4rWs rWcs:eZdZdZedfdd ZejejdddZZ S) ModernBertMLPa6Applies the GLU at the end of each ModernBERT layer. Compared to the default BERT architecture, this block replaces :class:`~transformers.model.bert.modeling_bert.BertIntermediate` and :class:`~transformers.model.bert.modeling_bert.SelfOutput` with a single module that has similar functionality. rXcsft||_tj|jt|jd|jd|_ t |j |_ t |j|_tj|j|j|jd|_dS)Nr#r\)rIrJrYrLinearr^r?Zintermediate_sizeZmlp_biasWir Zhidden_activationactrdZ mlp_dropoutreWorfrMr3r4rJs    zModernBertMLP.__init__rmrOcCs2||jddd\}}|||||S)Nr#r$rD)rschunkrurert)rLrminputZgater3r3r4r5szModernBertMLP.forward) r9r:r;rRrrJr=r>r5rVr3r3rMr4rpsrpcsDeZdZUejed<dedfdd Zee ddZ Z S) ModernBertRotaryEmbeddinginv_freqNrXcstt|dr:t|jtr:|jd|jd|_nd|_|j|_ |j|_ ||_ t |j|_ | |j |\}|_|jd|dd|j|_dS)N rope_scaling rope_typetypedefaultr{F) persistent)rIrJhasattr isinstancer|dictgetr}max_position_embeddingsZmax_seq_len_cachedZoriginal_max_seq_lenrYrZ rope_init_fnattention_scalingZregister_bufferr{Zoriginal_inv_freq)rLrYrFr{rMr3r4rJs  z"ModernBertRotaryEmbedding.__init__c Cs|jddddf|jddd|j}|dddddf}t|jjtrl|jjdkrl|jjnd}t j |ddV|| dd}t j ||fdd }| |j}||j} Wdn1s0Y|j|jd | j|jd fS) Nrr$rmpscpuF) device_typeZenabledr#rw)rG)r{rSexpandr)torFrr~rUr=Zautocast transposecatr-rr.rG) rLx position_idsZinv_freq_expandedZposition_ids_expandedrZfreqsZembr-r.r3r3r4r5 s0&,z!ModernBertRotaryEmbedding.forward)N) r9r:r;r=r>__annotations__rrJno_gradrr5rVr3r3rMr4rzs  rzcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr$r#rw)r)r=r)rx1Zx2r3r3r4 rotate_halfsrcCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkr-r.rZ unsqueeze_dimZq_embedZk_embedr3r3r4apply_rotary_pos_emb s   rFModernBertAttention) moduler,attention_masksliding_window_maskrlocal_attentionbsrDoutput_attentionsrOc Ks|j||d\} } |ddjdd\} } }t| | | | \} } |jd}t| | dd|}|dkrl|}||}tjj |dtj d  | j }tjj ||j|jd }t||}|dd}||d|}|r||fS|fS) Nrr rr#rw࿩r$r$r$rDrG)ptraining) rotary_embrunbindrhead_dimr=matmulr functionalZsoftmaxZfloat32rrGZdropoutattention_dropoutrr(r*)rr,rrrrrrDr_kwargsr-r.querykeyvaluescaleZ attn_weights attn_outputr3r3r4eager_attention_forward;s   r) rr,rr!r"rrrD target_dtyperOc Ks||||d}|jtjtjfv} | rb|j} ||}t||||jrH|jnd|j|d} | | } n"t||||jrv|jnd|j|d} | ||fS)Nr )r!r" dropout_pZ deterministicZ window_size) rGr=Zfloat16bfloat16rrrrdeterministic_flash_attnr*) rr,rr!r"rrrDrrZ convert_dtypeZ orig_dtypeattnr3r3r4flash_attention_forward`s.   r) rr,rrrrrrDrOcKs|j||d\} } |ddjdd\} } } t| | | | \} } |dkrJ|}tj| | | |jr`|jnd|ddd}| |d |}|fS) Nrr rr#rwrr)rZ attn_maskr$) rrrrFZscaled_dot_product_attentionrrr(r*)rr,rrrrrrDrr-r.rrrrr3r3r4sdpa_attention_forwards"  r)flash_attention_2eagersdpacsJeZdZdZd eeedfdd Zd ej ee ej ddd Z Z S) raPerforms multi-headed self attention on a batch of unpadded sequences. If Flash Attention 2 is installed, this module uses Flash Attention to improve throughput. If Flash Attention 2 is not installed, the implementation will use PyTorch's SDPA kernel, which requires padding and unpadding inputs, adding some overhead. See `forward` method for additional details. NrYlayer_idcsft||_||_|j|jdkr@td|jd|jd|j|_|j|_|j|_ |j|j|_ |j |j |_ t j |jd|j |jd|_||jdkr|jd|jdf|_|jdur|jn|j}|j}nd|_|j}|j}|jd krt|j ||d |_nt|}||_t|d |_t j |j|j|jd|_|jd krPt |jnt |_t |_!dS) NrzThe hidden size (z6) is not a multiple of the number of attention heads ()r rqr#rr)rDr"rErXr)"rIrJrYrr^Znum_attention_heads ValueErrorrr num_headsr all_head_sizerrrZattention_biasWqkvZglobal_attn_every_n_layersrZlocal_rope_thetaZglobal_rope_thetar_attn_implementationrBrcopydeepcopy rope_thetarzrurdIdentityout_dropsetZ pruned_heads)rLrYrrr config_copyrMr3r4rJs<     "zModernBertAttention.__init__F)rmrrOc Ks||}|jd}|jjdkr6|dd|j|j}n||dd|j|j}t|jj|f||j|j ||j |d|}|d}| | |}|f|ddS)Nrrr$r )r,rrrrDrr) rr)rYrr*rrMODERNBERT_ATTENTION_FUNCTIONrrrrru)rLrmrkwargsr,r attn_outputsr3r3r4r5s(     zModernBertAttention.forward)N)F) r9r:r;rRrrr?rJr=r>boolr5rVr3r3rMr4rs *c seZdZdeeedfdd Zejddej ej ddd Z dej eej eej eej eej eeee ej d d d Z ZS)ModernBertEncoderLayerNrcspt||_|dkr$t|_ntj|j|j|j d|_t ||d|_ tj|j|j|j d|_ t ||_dS)NrrZr)rIrJrYrr attn_normr`r^rarbrrmlp_normrpmlp)rLrYrrMr3r4rJs  zModernBertEncoderLayer.__init__TrgrvcCs|||Sri)rrrLrmr3r3r4 compiled_mlpsz#ModernBertEncoderLayer.compiled_mlpF)rmrrrr!r"rrOc Csf|j||||||||d}||d}|jjr<||n|||} || }|f|ddS)Nrrrr!r"rrr)rrrYrlrrr) rLrmrrrr!r"rrZ mlp_outputr3r3r4r5s   zModernBertEncoderLayer.forward)N)NNNNNF)r9r:r;rrr?rJr=rnr>rrorr5rVr3r3rMr4rs& rcsveZdZUeed<dZdZddgZdZdZ dZ e j ddd Z deeeed fd d Zd dZfddZZS)ModernBertPreTrainedModelrYmodelTrWrF)rcsj|jjdurdtjtdfdd }|jj|jjtd|jj|jj|jj dd}t |t rz||j |dnt |t r||j|d ||j|d nt |tr||j|d ||j|d nt |tr||j|d nxt |tr ||j|d nZt |ttttfr2||j|d n4t |tjrf|jjd |jdurf|jj dS) Nr rstdcsJtjj|jd| ||dt|tjrF|jdurFtj|jdS)Nr)meanrab)rinitZ trunc_normal_weightrrrr\Zzeros_rZ cutoff_factorr3r4 init_weightAs  z.init_weightg@r)inout embedding final_outrrrrg?)!rYZinitializer_cutoff_factorrModulerSZinitializer_rangemathsqrtnum_hidden_layersr^rrWr_rprsrurrModernBertPredictionHeaddenseModernBertForMaskedLMdecoder#ModernBertForSequenceClassificationModernBertForMultipleChoice ModernBertForTokenClassificationModernBertForQuestionAnswering classifierr`rdataZfill_r\Zzero_)rLrrZstdsr3rr4 _init_weights<sD        z'ModernBertPreTrainedModel._init_weights)attn_implementation is_init_checkrOc sDz|dur|rdn|}Wnttfy2Yn0tj||dS)zR Checks and dispatches to hhe requested attention implementation. Nr)rr)Z_flash_attn_2_can_dispatchr ImportErrorrI%_check_and_adjust_attn_implementation)rLrrrMr3r4rps z?ModernBertPreTrainedModel._check_and_adjust_attn_implementationcCs|jjdurdSt|drBt|jdkrB|jjr:tdd|j_|jjdkrh|jjr`tdd|j_|jjdkr|jjrtdd|j_|jjdurt |j_dS) NF hf_device_maprzqIf `accelerate` split the model across devices, `torch.compile` will not work. Falling back to non-compiled mode.rz|Compiling the model with `torch.compile` and using a `torch.mps` device is not supported. Falling back to non-compiled mode.rz|Compiling the model with `torch.compile` and using a `torch.cpu` device is not supported. Falling back to non-compiled mode.) rYrlrlenrlogger warning_oncerFr~rrPr3r3r4_maybe_set_compiles,    z,ModernBertPreTrainedModel._maybe_set_compilecs<tj|i|}|jjdvr8|jjr0tdd|j_|S)N>TNzcResizing token embeddings with `torch.compile` is not supported. Falling back to non-compiled mode.F)rIresize_token_embeddingsrYrlrr)rLargsrZ model_embedsrMr3r4rs z1ModernBertPreTrainedModel.resize_token_embeddings)F)r9r:r;rrZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_supports_flash_attnZ_supports_sdpaZ_supports_flex_attnrrrrrUrrrrrVr3r3rMr4r2s 5r)inputsrrlabelsrOcCs|jdtjd}tj|dd}t|}tjj tj |dtjdd}| dkrn||}n*|j ^} } } | | } |j| g| R|}|dur||nd} |dur||nd}||||| |fS) a Remove padding from input sequences. Args: inputs: (batch, seqlen, ...) or (batch, seqlen) attention_mask: (batch, seqlen), bool / int, 1 means valid and 0 means not valid. position_ids: (batch, seqlen), int, position ids labels: (batch, seqlen), int, labels Returns: unpadded_inputs: (total_nnz, ...), where total_nnz = number of tokens selected in attention_mask. indices: (total_nnz) cu_seqlens: (batch + 1), the cumulative sequence lengths max_seqlen_in_batch: int unpadded_position_ids: (total_nnz) or None unpadded_labels: (total_nnz) or None r$rF)as_tupler)rrr#N)sumr=Zint32Znonzeroflattenr?maxitemrrpadZcumsumrDr)r*)rrrrZseqlens_in_batchindicesZmax_seqlen_in_batchr!Zunpadded_inputsbatchseqlenrestr)Zunpadded_position_idsZunpadded_labelsr3r3r4_unpad_modernbert_inputs  r )rr r r rOcCs|dkr:tj|||j|jd}|||<|||}nH|j^}}tj||g|R|j|jd}|||<|j||g|R}|S)aQ Add padding to sequences. Args: inputs: (total_nnz, ...) or (total_nnz,), where total_nnz = number of tokens selected in attention_mask. indices: (total_nnz) batch: int, batch size seqlen: int, max sequence length Returns: padded_inputs: (batch, seqlen, ...) or (batch, seqlen) r)rGrF)rDr=ZzerosrGrFr*r))rr r r outputZ padded_inputs_r r3r3r4_pad_modernbert_outputs  "rcseZdZedfdd ZddZddZedee j ee j ee j ee j ee j ee j ee j ee ee ee ee ee ee eee j d fefd d d Ze j e e j d ddZZS)ModernBertModelrXcsft|_t|_tfddtjD|_ tj j j j d|_d|_|dS)Ncsg|]}t|qSr3)r).0rrXr3r4 z,ModernBertModel.__init__..rZF)rIrJrYrW embeddingsrZ ModuleListrangerlayersr`r^rarb final_normZgradient_checkpointing post_initrfrMrXr4rJs  zModernBertModel.__init__cCs|jjSrirr_rPr3r3r4get_input_embeddingssz$ModernBertModel.get_input_embeddingscCs ||j_dSrir)rLrr3r3r4set_input_embeddings sz$ModernBertModel.set_input_embeddingsN.)rhrrrrkr r!r" batch_sizeseq_lenroutput_hidden_states return_dictrOc s| dur | n|jj} | dur | n|jj} | dur4| n|jj} |du|duArTtd| r\dnd}| rhdnd}||dur|||durƈdur|dur|jdd\n|jdd\|dur|jn|j}|durt j f|t j d}d}|jj dkrdur|dur|durd}|durxt &t||d ^}}}}Wdn1sl0Ynt||d ^}}}}n0|durt j|d d }|j|| d \}}|j||d }|jD]V}| r||f}|||||||| d}|d }| rt|dkr||df}q| r<||f}||}|rt|d}|durtfdd|D}| stdd|||fDSt|||dS) sliding_window_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding or far-away tokens. In ModernBert, only every few layers perform global attention, while the rest perform local attention. This mask is used to avoid attending to far-away tokens in the local attention layers when not using Flash Attention. indices (`torch.Tensor` of shape `(total_unpadded_tokens,)`, *optional*): Indices of the non-padding tokens in the input sequence. Used for unpadding the output. cu_seqlens (`torch.Tensor` of shape `(batch + 1,)`, *optional*): Cumulative sequence lengths of the input sequences. Used to index the unpadded tensors. max_seqlen (`int`, *optional*): Maximum sequence length in the batch excluding padding tokens. Used to unpad input_ids and pad output tensors. batch_size (`int`, *optional*): Batch size of the input sequences. Used to pad the output tensors. seq_len (`int`, *optional*): Sequence length of the input sequences including padding tokens. Used to pad the output tensors. Nz:You must specify exactly one of input_ids or inputs_embedsr3r#rHFrT)rr)rFr)r)rhrkrrrr r r c3s|]}t|dVqdS)r"N)r)rhsrr rr3r4 |sz*ModernBertModel.forward..css|]}|dur|VqdSrir3)rvr3r3r4r%r)last_hidden_staterm attentions)rYrruse_return_dictrr%warn_if_padding_and_no_attention_maskr)rFr=onesrrrr aranger_update_attention_maskrrrrrrTr )rLrhrrrrkr r!r"rrrrr Zall_hidden_statesZall_self_attentionsrFZrepadrrmZ encoder_layerZ layer_outputsr3r$r4r5s!     2        zModernBertModel.forward)rrrOcCs|rF|jjdkr$tdd|j_n"|jjdkrFtd|jjdt||j}t|jd d}t ||j }||jj dk d d |j}||t|jj}||fS)NrzOutputting attentions is only supported with the 'eager' attention implementation, not with "sdpa". Falling back to `attn_implementation="eager"`.rzZOutputting attentions is only supported with the eager attention implementation, not with zT. Consider setting `attn_implementation="eager"`. Setting `output_attentions=False`.r#r)rYrrrr rGr=r,r)rabsTrrrFZ masked_fillZ logical_notZfinfomin)rLrrZglobal_attention_maskrowsZdistanceZ window_maskrr3r3r4r-s&    "z&ModernBertModel._update_attention_mask) NNNNNNNNNNNNN)r9r:r;rrJrrrrr=ror>r?rrrTr r5r-rVr3r3rMr4rsD zrcs6eZdZedfdd ZejejdddZZS)rrXcsNt||_t|j|j|j|_t|j |_ tj |j|j |j d|_dS)NrZ)rIrJrYrrrr^Zclassifier_biasrr Zclassifier_activationrtr`rarbrcrfrMr3r4rJs   z!ModernBertPredictionHead.__init__rvcCs||||Sri)rcrtrrr3r3r4r5sz ModernBertPredictionHead.forward) r9r:r;rrJr=r>r5rVr3r3rMr4rsrzd The ModernBert Model with a decoder head on top that is used for masked language modeling. )Z custom_introcseZdZdgZedfdd ZddZejddd Z e j d d e j e j d d dZ edee jee j ee j ee j ee j ee j ee j ee j eeeeeeeeeeeeeee j efdddZZS)rzdecoder.weightrXcs^t|||_t||_t||_tj|j |j |j d|_ |jj |_ |jj|_|dS)Nrq)rIrJrYrrrheadrrrr^r]Z decoder_biasrsparse_predictionsparse_pred_ignore_indexrrfrMr3r4rJs     zModernBertForMaskedLM.__init__cCs|jSrirrPr3r3r4get_output_embeddingssz+ModernBertForMaskedLM.get_output_embeddings)new_embeddingscCs ||_dSrir5)rLr7r3r3r4set_output_embeddingssz+ModernBertForMaskedLM.set_output_embeddingsTrg)rrOcCs|||Sri)rr2)rLrr3r3r4 compiled_headsz#ModernBertForMaskedLM.compiled_headNrhrrrrkrr r!r"rrrrr rOcKs|dur |n|jj}||jjdkr$|dur$|dur$| dur$| dur| dur|durt|jdd\} } n|jdd\} } |dur|jn|j}|durtj| | f|tjd}|durt ,t ||||d\}}}} }}Wdn1s0Ynt ||||d\}}}} }}|j |||||||| | | | | |d }|d}|j r|dur| d}| |jdd}||jk}||}||}|jjr||n|||}d}|dur|j||fd |jji|}|jjdkrH|jjs|durtnt t||| | d }Wdn1s>0Y|sl|f}|durh|f|S|St|||j|jd S) r!Nrr#rH)rrrr rhrrrrkr r!r"rrrrr rr$r]r"losslogitsrmr()rYr)rrr)rFr=r+rrr rr3r*r4rlr9rr2 loss_functionr]Zrepad_logits_with_gradrrrrmr()rLrhrrrrkrr r!r"rrrrr rrFoutputsr'Z mask_tokensr>r=rr3r3r4r5sx#  2   $0zModernBertForMaskedLM.forward)NNNNNNNNNNNNNN)r9r:r;Z_tied_weights_keysrrJr6rrrr8r=rnr>r9rrror?rrrTrr5rVr3r3rMr4rsL  rz` The ModernBert Model with a sequence classification head on top that performs pooling. cseZdZedfdd Zedeejeej eej eej eej eej eej eej ee ee ee ee ee ee e e ej efdddZZS) rrXcs\t||j|_||_t||_t||_tj |j |_ t |j|j|_|dSri)rIrJ num_labelsrYrrrr2r=rrdclassifier_dropoutrerrr^rrrfrMr3r4rJKs   z,ModernBertForSequenceClassification.__init__Nr:cKsl|dur |n|jj}||dur0|||| durn| durn|dur\|jdd\} } n|jdd\} } |dur||jn|j}|durtj| | f|tjd}|j |||||||| | | | | |d }|d}|jj dkr|dddf}n2|jj dkr|| dj d d |j d d d }| |}||}||}d}|dur4|jjdur|jd krld |j_n:|jd kr|jtjks|jtjkrd|j_nd|j_|jjd krt}|jd kr|||}n |||}nN|jjdkrt}||d|j|d}n|jjdkr4t}|||}|sX|f}|durT|f|S|St|||j|jdS)aB sliding_window_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding or far-away tokens. In ModernBert, only every few layers perform global attention, while the rest perform local attention. This mask is used to avoid attending to far-away tokens in the local attention layers when not using Flash Attention. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). indices (`torch.Tensor` of shape `(total_unpadded_tokens,)`, *optional*): Indices of the non-padding tokens in the input sequence. Used for unpadding the output. cu_seqlens (`torch.Tensor` of shape `(batch + 1,)`, *optional*): Cumulative sequence lengths of the input sequences. Used to index the unpadded tensors. max_seqlen (`int`, *optional*): Maximum sequence length in the batch excluding padding tokens. Used to unpad input_ids and pad output tensors. batch_size (`int`, *optional*): Batch size of the input sequences. Used to pad the output tensors. seq_len (`int`, *optional*): Sequence length of the input sequences including padding tokens. Used to pad the output tensors. Nr#rHr;rclsrr$rrwTrDZkeepdimZ regressionZsingle_label_classificationZmulti_label_classificationr<)rYr)rr*r)rFr=r+rrclassifier_poolingrrr2rerZ problem_typerArGlongr?rsqueezerr*rrrmr()rLrhrrrrkrr r!r"rrrrr rrFr@r' pooled_outputr>r=loss_fctrr3r3r4r5Xs'        (    z+ModernBertForSequenceClassification.forward)NNNNNNNNNNNNNN)r9r:r;rrJrrr=ror>r?rrrTrr5rVr3r3rMr4rEsB rzv The ModernBert Model with a token classification head on top, e.g. for Named Entity Recognition (NER) tasks. cseZdZedfdd Zedeejeej eej eej eej eej eej eej ee ee ee ee ee ee e e ej efdddZZS) rrXcsVt||j|_t||_t||_tj |j |_ t |j |j|_|dSrirIrJrArrrr2r=rrdrBrerrr^rrrfrMr3r4rJs   z)ModernBertForTokenClassification.__init__Nr:cCs|dur |n|jj}||j|||||||| | | | | |d }|d}||}||}||}d}|durt}||d|j |d}|s|f|dd}|dur|f|S|St |||j |j dS)a sliding_window_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding or far-away tokens. In ModernBert, only every few layers perform global attention, while the rest perform local attention. This mask is used to avoid attending to far-away tokens in the local attention layers when not using Flash Attention. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. indices (`torch.Tensor` of shape `(total_unpadded_tokens,)`, *optional*): Indices of the non-padding tokens in the input sequence. Used for unpadding the output. cu_seqlens (`torch.Tensor` of shape `(batch + 1,)`, *optional*): Cumulative sequence lengths of the input sequences. Used to index the unpadded tensors. max_seqlen (`int`, *optional*): Maximum sequence length in the batch excluding padding tokens. Used to unpad input_ids and pad output tensors. batch_size (`int`, *optional*): Batch size of the input sequences. Used to pad the output tensors. seq_len (`int`, *optional*): Sequence length of the input sequences including padding tokens. Used to pad the output tensors. Nr;rr$rr<) rYr)rrr2rerrr*rArrmr()rLrhrrrrkrr r!r"rrrrr r@r'r>r=rIrr3r3r4r5sD$   z(ModernBertForTokenClassification.forward)NNNNNNNNNNNNNN)r9r:r;rrJrrr=ror>r?rrrTrr5rVr3r3rMr4rsB rcseZdZedfdd Zedeejeejeejeejeejeejeejeejee ee ee ee ee ee e e eje fdddZZS) rrXcsVt||j|_t||_t||_tj |j |_ t |j |j|_|dSrirJrfrMr3r4rJ/s   z'ModernBertForQuestionAnswering.__init__N)rhrrrstart_positions end_positionsr r!r"rrrrr rOcKs|dur |n|jj}||j||||||| | | | | |d }|d}||}||}||}|jddd\}}|d }|d }d}|dur|dur|j ||||fi|}|s||f|dd}|dur|f|S|St ||||j |j dS)r!N) rrrr r!r"rrrrr rrr$rw)r= start_logits end_logitsrmr()rYr)rrr2rersplitrGr(r?rrmr()rLrhrrrrKrLr r!r"rrrrr rr@r'r>rMrNr=rr3r3r4r5:sH#   z&ModernBertForQuestionAnswering.forward) NNNNNNNNNNNNN)r9r:r;rrJrrr=r>r?rrrTrr5rVr3r3rMr4r-s@ rz The ModernBert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. cseZdZedfdd Zedeejeej eej eej eej eej eej eej ee ee ee ee ee ee e e ej efdddZZS) rrXcsRt|||_t||_t||_tj |j |_ t |j d|_|dS)Nr)rIrJrYrrrr2r=rrdrBrerrr^rrrfrMr3r4rJs   z$ModernBertForMultipleChoice.__init__Nr:cKs|dur |n|jj}|dur&|jdn|jd}|durJ|d|dnd}|durh|d|dnd}|dur|d|dnd}|dur|d|d|dnd}||j|||||||| | | | | |d }|d}|jjdkr|dddf}n2|jjdkr6||dj dd |j dd d }| |}| |}| |}|d|}d}|durt }|||}|s|f|dd}|dur|f|S|St|||j|jd S) a sliding_window_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding or far-away tokens. In ModernBert, only every few layers perform global attention, while the rest perform local attention. This mask is used to avoid attending to far-away tokens in the local attention layers when not using Flash Attention. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. indices (`torch.Tensor` of shape `(total_unpadded_tokens,)`, *optional*): Indices of the non-padding tokens in the input sequence. Used for unpadding the output. cu_seqlens (`torch.Tensor` of shape `(batch + 1,)`, *optional*): Cumulative sequence lengths of the input sequences. Used to index the unpadded tensors. max_seqlen (`int`, *optional*): Maximum sequence length in the batch excluding padding tokens. Used to unpad input_ids and pad output tensors. batch_size (`int`, *optional*): Batch size of the input sequences. Used to pad the output tensors. seq_len (`int`, *optional*): Sequence length of the input sequences including padding tokens. Used to pad the output tensors. Nrr$r;rrCrrwTrDr<)rYr)r)r*sizerrrErrr2rerrrrrmr()rLrhrrrrkrr r!r"rrrrr rZ num_choicesr@r'rHr>Zreshaped_logitsr=rIrr3r3r4r5sb&      z#ModernBertForMultipleChoice.forward)NNNNNNNNNNNNNN)r9r:r;rrJrrr=ror>r?rrrTrr5rVr3r3rMr4rsB r)rrrrrrr)NN)Nr)F)NN)Wrr contextlibrtypingrrr=Ztorch.nn.functionalrrrZtorch.nnrrrZ activationsr Zmodeling_attn_mask_utilsr Zmodeling_layersr Zmodeling_outputsr rrrrrZmodeling_rope_utilsrrZmodeling_utilsrutilsrrrZutils.import_utilsrZconfiguration_modernbertrZflash_attn.flash_attn_interfacerZflash_attn.layers.rotaryrZflash_attn.ops.triton.rotaryrobjectZ get_loggerr9rZautogradFunctionrr>r?rArBrrWrprzrrrorTrrrrGrrrrrrr rrrrrrrr__all__r3r3r3r4s            ; 5$ $  .  ,  $O.$ * .  Z[p