a h_ @sddlZddlmZmZmZddlZddlZddlm Z ddlm Z ddl m Z ddl mZddlmZddlmZmZdd lmZdd lmZdd lmZmZmZdd lmZmZdd lm Z ddl!m"Z"m#Z#m$Z$ddl%m&Z&e#rddl'm(Z(e$)e*Z+Gddde j,Z-Gddde j,Z.GdddeZ/GdddeZ0GdddeZ1Gddde j,Z2Gddde j,Z3dAe j,ej4ej4ej4eej4ee5e5eej4d!d"d#Z6Gd$d%d%e j,Z7Gd&d'd'e j,Z8Gd(d)d)eZ9Gd*d+d+e j,Z:Gd,d-d-e j,Z;Gd.d/d/eZdBe?e@e@fe5e@eejAe@ejBd4d5d6ZCe"Gd7d8d8e>ZDdZEe"d9d:Gd;d<dZFe"d=d:Gd>d?d?e>ZGgd@ZHdS)CN)CallableOptionalUnion)CrossEntropyLoss)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)_prepare_4d_attention_mask#_prepare_4d_attention_mask_for_sdpa)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputCausalLMOutputSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringis_torch_flex_attn_availablelogging) HubertConfig)make_flex_block_causal_maskcs$eZdZfddZddZZS)HubertPositionalConvEmbeddingcsBttj|j|j|j|jd|jd|_d|_|j rJt |j|_ntj j }t tj jdrjtj jj }trddl}|jj|jjdd"||jddd|_Wdn1s0Yt |jdr|jjjj}|jjjj}n|jj}|jj}|j|||j||n||jddd|_t|j|_t|j|_dS) N) kernel_sizepaddinggroups weight_normrZ modifier_rankweight)namedimparametrizations)super__init__nnConv1d hidden_sizenum_conv_pos_embeddingsZnum_conv_pos_embedding_groupsconv batch_normZconv_pos_batch_norm BatchNorm1dutilsrhasattrr$r deepspeedzeroGatheredParametersr!Z original0Z original1weight_gweight_vZregister_external_parameterHubertSamePadLayerrrfeat_extract_activation activation)selfconfigrr0r3r4 __class__f/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/hubert/modeling_hubert.pyr&3s8  0   z&HubertPositionalConvEmbedding.__init__cCsN|dd}|jdur ||}||}||}||}|dd}|S)Nrr) transposer,r+rr7r8 hidden_statesr<r<r=forwardXs       z%HubertPositionalConvEmbedding.forward__name__ __module__ __qualname__r&rA __classcell__r<r<r:r=r2s %rcs$eZdZfddZddZZS)r5cs$t|ddkrdnd|_dS)Nrrr)r%r&num_pad_remove)r8r*r:r<r=r&es zHubertSamePadLayer.__init__cCs,|jdkr(|ddddd|j f}|S)Nr)rGr?r<r<r=rAis zHubertSamePadLayer.forwardrBr<r<r:r=r5ds r5cs&eZdZdfdd ZddZZS)HubertNoLayerNormConvLayerrcsjt|dkr |j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ dS)Nrrrstridebias)r%r&conv_dim in_conv_dim out_conv_dimr'r( conv_kernel conv_stride conv_biasr+rr6r7r8r9layer_idr:r<r=r&ps  z#HubertNoLayerNormConvLayer.__init__cCs||}||}|SN)r+r7r?r<r<r=rA~s  z"HubertNoLayerNormConvLayer.forward)rrBr<r<r:r=rHosrHcs&eZdZdfdd ZddZZS)HubertLayerNormConvLayerrcs|t|dkr |j|dnd|_|j||_tj|j|j|j||j||j d|_ tj |jdd|_ t |j|_dS)NrrrIT)Zelementwise_affine)r%r&rLrMrNr'r(rOrPrQr+ LayerNorm layer_normrr6r7rRr:r<r=r&s  z!HubertLayerNormConvLayer.__init__cCs:||}|dd}||}|dd}||}|S)N)r+r>rWr7r?r<r<r=rAs      z HubertLayerNormConvLayer.forward)rrBr<r<r:r=rUsrUcs&eZdZdfdd ZddZZS)HubertGroupNormConvLayerrcst|dkr |j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ tj|j|jdd|_dS)NrrrIT)Z num_groupsZ num_channelsZaffine)r%r&rLrMrNr'r(rOrPrQr+rr6r7 GroupNormrWrRr:r<r=r&s   z!HubertGroupNormConvLayer.__init__cCs"||}||}||}|SrT)r+rWr7r?r<r<r=rAs   z HubertGroupNormConvLayer.forward)rrBr<r<r:r=rZsrZcs0eZdZdZfddZddZddZZS)HubertFeatureEncoderz.Construct the features from raw audio waveformcstjdkr@tddgfddtjdD}n6jdkrdfddtjD}ntd jd t||_ d |_ d |_ dS) NgrouprrScsg|]}t|ddqS)rr^)rH.0ir9r<r= sz1HubertFeatureEncoder.__init__..rlayercsg|]}t|dqS)r^)rUr_rbr<r=rcz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT) r%r&Zfeat_extract_normrZrangeZnum_feat_extract_layers ValueErrorr' ModuleList conv_layersgradient_checkpointing_requires_grad)r8r9rir:rbr=r&s      zHubertFeatureEncoder.__init__cCs|D] }d|_qd|_dS)NF) parameters requires_gradrkr8paramr<r<r=_freeze_parameterss z'HubertFeatureEncoder._freeze_parameterscCs:|dddf}|jr"|jr"d|_|jD] }||}q(|S)NT)rktrainingrmri)r8 input_valuesr@Z conv_layerr<r<r=rAs    zHubertFeatureEncoder.forward)rCrDrE__doc__r&rprArFr<r<r:r=r\s r\cs$eZdZfddZddZZS)HubertFeatureProjectioncsXt|j|_|jr0tj|jd|jd|_t|jd|j |_ t |j |_ dS)NrYeps)r%r&feat_proj_layer_normr'rVrLlayer_norm_epsrWLinearr) projectionDropoutZfeat_proj_dropoutdropoutr8r9r:r<r=r&s  z HubertFeatureProjection.__init__cCs(|jr||}||}||}|SrT)rwrWrzr|r?r<r<r=rAs    zHubertFeatureProjection.forwardrBr<r<r:r=rts rt)modulequerykeyvalueattention_maskscalingr| head_maskc Ks|dur|dd}t||dd|} |dur>| |} tjj| dd} |durj| |dddd} tjj| ||j d} t| |} | dd } | | fS)NrYrrr#r)prq) sizetorchmatmulr>r' functionalsoftmaxviewr|rq contiguous) rrrrrrr|rkwargs attn_weights attn_outputr<r<r=eager_attention_forwards  rc seZdZdZd eeeeeeeedfdd Z d e j ee j ee j ee j eee e ee j ee j eee j fd d d ZZS)HubertAttentionz=Multi-headed attention from 'Attention Is All You Need' paperr~FTN) embed_dim num_headsr| is_decoderrK is_causalr9cst||_||_||_|||_||_|j||jkrTtd|jd|d|jd|_||_ ||_ t j |||d|_ t j |||d|_t j |||d|_t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r)rK)r%r&rrr|head_dimr9rgrrrr'ryk_projv_projq_projout_proj)r8rrr|rrKrr9r:r<r=r&s&    zHubertAttention.__init__)r@key_value_statesrlayer_head_maskoutput_attentionsrreturncKs|du}|jdd\}} |r(|jdn| } || d|jf} || d|jf} ||j| dd} |rh|n|}||j| dd}||j| dd}t}|jj dkrt |jj }||| |||f|j sdn|j |j ||d|\}}||| d}||}||dfS)z#Input shape: Batch x Time x ChannelNrYrreagerr~)r|rrr)shaperrrr>rrrr9_attn_implementationrrqr|rreshaperr)r8r@rrrrrZis_cross_attentionZbszZtgt_lenZsrc_lenZ q_input_shapeZkv_input_shapeZ query_statesZcurrent_statesZ key_statesZ value_statesZattention_interfacerrr<r<r=rA/s:      zHubertAttention.forward)r~FTFN)NNNF)rCrDrErsintfloatboolrrr&rTensorrr tuplerArFr<r<r:r=r s8"rcs$eZdZfddZddZZS)HubertFeedForwardcsptt|j|_t|j|j|_ t |j t rDt |j |_n|j |_t|j|j|_t|j|_dSrT)r%r&r'r{Zactivation_dropoutintermediate_dropoutryr)Zintermediate_sizeintermediate_dense isinstanceZ hidden_actstrrintermediate_act_fn output_densehidden_dropoutoutput_dropoutr}r:r<r=r&fs  zHubertFeedForward.__init__cCs6||}||}||}||}||}|SrT)rrrrrr?r<r<r=rAss      zHubertFeedForward.forwardrBr<r<r:r=res rcs&eZdZfddZdddZZS)HubertEncoderLayercshtt|j|j|jd|d|_t|j |_ tj |j|j d|_ t||_tj |j|j d|_dS)NFrrr|rr9ru)r%r&rr)num_attention_headsattention_dropout attentionr'r{rr|rVrxrWr feed_forwardfinal_layer_normr}r:r<r=r&~s  zHubertEncoderLayer.__init__NFcCsf|}|j|||d\}}}||}||}||}|||}||}|f}|rb||f7}|SNrr)rr|rWrrr8r@rrZ attn_residualr_outputsr<r<r=rAs     zHubertEncoderLayer.forward)NFrBr<r<r:r=r}s rcsXeZdZfddZd ejeejeeedddZ e ejdfejd d d Z Z S) HubertEncodercsft|_t|_tjjjd|_ t j |_ t fddtjD|_d|_dS)Nrucsg|] }tqSr<)rr`rrbr<r=rcrez*HubertEncoder.__init__..Fr%r&r9rpos_conv_embedr'rVr)rxrWr{rr|rhrfnum_hidden_layerslayersrjr}r:rbr=r&s   zHubertEncoder.__init__NFT)r@rroutput_hidden_states return_dictcCs.|rdnd}|rdnd}|durD|ddd|jd}d||<|||}||} || }||}||}tpt|} |j D]f} |r||f}t g} |j o| |j jk} | r| r| |||d}|d}| rd}|r||df}q|r||f}|s tdd |||fDSt|||d S) Nr<rYrrrrNNcss|]}|dur|VqdSrTr<r`vr<r<r= rez(HubertEncoder.forward..Zlast_hidden_stater@ attentions) unsqueezerepeatr_update_full_maskrrWr|rr rrrandrqr9 layerdroprrr8r@rrrrZall_hidden_statesZall_self_attentionsZexpand_attention_maskZposition_embeddingsZ synced_gpusrdZdropout_probabilityZskip_the_layerZ layer_outputsr<r<r=rAsJ          zHubertEncoder.forwardr inputs_embedscCsv|durr|jjdkr&d|vr |nd}nL|jjdkr@t||j}n2|jjdkrft|tjrrt|dd}n t||j}|SNZflash_attention_2rZsdpaZflex_attentionF)r r9rr dtyperrrrr r8rrr<r<r=rs     zHubertEncoder._update_full_mask)NFFT) rCrDrEr&rtensorrrrrArrrFr<r<r:r=rs  > rcs,eZdZfddZejdddZZS)HubertAttnAdapterLayercsZt|j|_|j|_t|j|_t |j|j|_ t |_ t |j|j|_ dS)z Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed up training throughput. N)r%r&adapter_attn_dimZ input_dimr)Z hidden_dimr'rVnormrylinear_1ZReLUact_fnlinear_2r}r:r<r=r&s  zHubertAttnAdapterLayer.__init__)r@cCs,||}||}||}||}|SrT)rrrrr?r<r<r=rA s     zHubertAttnAdapterLayer.forward)rCrDrEr&r FloatTensorrArFr<r<r:r=rs rcs8eZdZfddZdejeejedddZZ S) !HubertEncoderLayerStableLayerNormcstt|j|j|jd|d|_t|j |_ tj |j|j d|_ t||_tj |j|j d|_t|dddurt||_nd|_dS)NFrrur)r%r&rr)rrrr'r{rr|rVrxrWrrrgetattrr adapter_layerr}r:r<r=r&s   z*HubertEncoderLayerStableLayerNorm.__init__NF)r@rrcCsz|}||}|j|||d\}}}||}||}||||}|jdurb|||}|f}|rv||f7}|Sr)rWrr|rrrrr<r<r=rA+s     z)HubertEncoderLayerStableLayerNorm.forward)NF) rCrDrEr&rrrrrArFr<r<r:r=rs rcsBeZdZfddZd ddZeejdfejdd d ZZ S) HubertEncoderStableLayerNormcsft|_t|_tjjjd|_ t j |_ t fddtjD|_d|_dS)Nrucsg|] }tqSr<)rrrbr<r=rcMrez9HubertEncoderStableLayerNorm.__init__..Frr}r:rbr=r&Fs  z%HubertEncoderStableLayerNorm.__init__NFTcCs.|rdnd}|rdnd}|durD|ddd|jd}d||<|||}||} || }||}tpxt|} |jD]f} |r||f}t g} |j o| |j j k} | r| r| |||d}|d}| rd}|r||df}q||}|r||f}|s tdd |||fDSt|||d S) Nr<rYrrrrrcss|]}|dur|VqdSrTr<rr<r<r=rrez7HubertEncoderStableLayerNorm.forward..r)rrrrrr|rr rrrrqr9rrWrrrr<r<r=rAQsJ          z$HubertEncoderStableLayerNorm.forwardrcCsv|durr|jjdkr&d|vr |nd}nL|jjdkr@t||j}n2|jjdkrft|tjrrt|dd}n t||j}|Srrrr<r<r=rs     z.HubertEncoderStableLayerNorm._update_full_mask)NFFT) rCrDrEr&rArrrrrFr<r<r:r=rEs  @ rc@s`eZdZUeed<dZdZdZdZdZ dZ ddZ e e jefddd Zee jd d d Zd S)HubertPreTrainedModelr9hubertrrTcCst|tjr<|jjjd|jjd|jdur8|jj nxt|tj tj tj frp|jj |jj dnDt|tjr\tr2ddl}t|drt|dr|jj|j|jgdd tj|jjWdn1s0YnD|jj|jdd tj|jjWdn1s&0Yntj|jj|jdur|jj nXt|trt|d r|jjn2t|trt|d r|jj d|jjd dS) zInitialize the weightsr~)meanZstdNg?rr4r3r masked_spec_embed layer_weightsr)rr'ryr!dataZnormal_r9Zinitializer_rangerKZzero_rVr[r-Zfill_r(rr0r/r1r2r4r3initZkaiming_normal_ HubertModelruniform_HubertForSequenceClassificationrr)r8rr0r<r<r= _init_weightss0   02     z#HubertPreTrainedModel._init_weights) input_lengthscCs4dd}t|jj|jjD]\}}||||}q|S)zH Computes the output length of the convolutional layers cSstj|||dddS)Nfloor)Z rounding_moder)rdiv) input_lengthrrJr<r<r=_conv_out_lengthszPHubertPreTrainedModel._get_feat_extract_output_lengths.._conv_out_length)zipr9rOrP)r8rrrrJr<r<r= _get_feat_extract_output_lengthssz6HubertPreTrainedModel._get_feat_extract_output_lengths)feature_vector_lengthrcCs~||dtj}|jd}tj||f|j|jd}d|tj |jd|jd|df<| dg d dg }|S)NrYr)rdevicer)r) rsumtorlongrzerosrrarangeflipZcumsumr)r8rrZoutput_lengths batch_sizer<r<r="_get_feature_vector_attention_masks "z8HubertPreTrainedModel._get_feature_vector_attention_maskN)rCrDrEr__annotations__Zbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingZ_supports_flash_attnZ_supports_sdpaZ_supports_flex_attnrrr LongTensorrrrr<r<r<r=rs !r)r mask_prob mask_lengthr min_masksrcs|\}dkrtdkr6tdddtjdfdd}|durt|d nfd d t|D}tj |ft d }g} |} | d kr|S|D]v} || } tjj t | d| dd} t | d krd}n| d }t| tj| | tjd |g} | | qt| } t| dddddf|| f} | || } t ddddf}t||| f|| }| |} | dkr҈d| | dk<t|| dd |S)an Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `csXt|}t|}|kr2}|d|krTt|dd}|S)z;Given input length, compute how many spans should be maskedrr)rmax)rnum_masked_spanepsilonrrrsequence_lengthr<r=compute_num_masked_spans  z6_compute_mask_indices..compute_num_masked_spanNrYcsg|]}qSr<r<r)r r<r=rc&rez)_compute_mask_indices..rrF)replace)rgnprandomritemdetachrtolistrfrrchoicerlenZ concatenateonesZint32appendarrayZ broadcast_torrZput_along_axis)rrrrrrr rZ spec_aug_maskZspec_aug_mask_idxsZmax_num_masked_spanrr Zspec_aug_mask_idxZdummy_mask_idxoffsetsr<r r=_compute_mask_indicess\    rc seZdZedfdd Zd ejeejeejdddZ e d eej eej eejee ee ee e eefdd d ZZS) rrbcszt|||_t||_t||_|jdks:|jdkrRt t |j |_|jrdt||_n t||_|dS)Nr~)r%r&r9r\feature_extractorrtfeature_projectionmask_time_probmask_feature_probr' Parameterrrr)rrZdo_stable_layer_normrencoderr post_initr}r:r<r=r&fs     zHubertModel.__init__N)r@mask_time_indicesrcCst|jdds|S|\}}}|dur<|j|j||<nZ|jjdkr|jrt||f|jj|jj ||jj d}t j ||j t jd}|j|j||<|jjdkr|jrt||f|jj|jj|jjd}t j ||j t jd}|dddfd|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://huggingface.co/papers/1904.08779). Zapply_spec_augmentTNr)rrrr)rr)rrrrY)rr9rrrrrrqrZmask_time_lengthZmask_time_min_masksrrrrrZmask_feature_lengthZmask_feature_min_masksexpand)r8r@r#rrr r)Zmask_feature_indicesr<r<r=_mask_hidden_statesxs4 zHubertModel._mask_hidden_states)rrrr#rrrrc Cs|dur |n|jj}|dur |n|jj}|dur4|n|jj}||}|dd}|durl||jd|}||}|j ||d}|j |||||d} | d}|s|f| ddSt || j | j dS)a1 mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. Example: ```python >>> from transformers import AutoProcessor, HubertModel >>> from datasets import load_dataset >>> processor = AutoProcessor.from_pretrained("facebook/hubert-large-ls960-ft") >>> model = HubertModel.from_pretrained("facebook/hubert-large-ls960-ft") >>> def map_to_array(example): ... example["speech"] = example["audio"]["array"] ... return example >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.map(map_to_array) >>> input_values = processor(ds["speech"][0], return_tensors="pt").input_values # Batch size 1 >>> hidden_states = model(input_values).last_hidden_state ```Nrr)r#rrrrrr)r9rruse_return_dictrr>rrrr%r!rr@r) r8rrrr#rrrZextract_featuresr@Zencoder_outputsr<r<r=rAs2$   zHubertModel.forward)NN)NNNNN)rCrDrErr&rrrrr%rrrrrrrArFr<r<r:r=rds. . rzn Hubert Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). )Z custom_introc seZdZdeedfdd ZddZddZd d Zd d Z e dee j ee j ee ee ee ee j eeefd ddZZS) HubertForCTCN) target_langcs~t|t||_t|j|_||_|j durFt d|j dt |dr\|j r\|jn|j}t||j |_|dS)a0 target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`HubertForCTC`] with adapters. Uses 'eng' by default. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `HubertForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration. add_adapter)r%r&rrr'r{Z final_dropoutr|r) vocab_sizergr;r/r*output_hidden_sizer)rylm_headr")r8r9r)r,r:r<r=r&s    zHubertForCTC.__init__cCsr|j}|dur2t|jdddur2td|dn<|durXt|jdddurXtdn|durn|j|dddS)a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. NrzCannot pass `target_lang`: z- if `config.adapter_attn_dim` is not defined.z)By default `target_lang` is set to 'eng'.T)Z force_load)r)rr9rgloggerinfoZ load_adapter)r8r)r<r<r= tie_weightss  zHubertForCTC.tie_weightscCstdt|dS) Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. Please use the equivalent `freeze_feature_encoder` method instead.Nwarningswarn FutureWarningfreeze_feature_encoderr8r<r<r=freeze_feature_extractor)s z%HubertForCTC.freeze_feature_extractorcCs|jjdSr1Nrrrpr8r<r<r=r75sz#HubertForCTC.freeze_feature_encodercCs|jD] }d|_q dSz Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated. FNrrlrmrnr<r<r=freeze_base_model<szHubertForCTC.freeze_base_modelrrrrrrlabelsrc Cs|dur |n|jj}|dur>||jjkr>td|jj|j|||||d}|d}||}||} d} |dur@|dur|ntj |tj d}| | d tj } |dk} | d} || }tjj| dtjddd}tjjjd d 6tjj||| | |jj|jj|jjd } Wdn1s60Y|sp| f|td}| durl| f|S|St| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nz$Label values must be <= vocab_size: r&rrrY)r#rrF)Zenabled)blankZ reductionZ zero_infinitylosslogitsr@r)r9r'rr+rgrr|r-rZ ones_likerrrrZ masked_selectr'rZ log_softmaxZfloat32r>backendsZcudnnflagsZctc_lossZ pad_token_idZctc_loss_reductionZctc_zero_infinity_HIDDEN_STATES_START_POSITIONrr@r)r8rrrrrrr@rr@rDrCrZ labels_maskZtarget_lengthsZflattened_targetsZ log_probsoutputr<r<r=rADsL     &  zHubertForCTC.forward)N)NNNNN)rCrDrErrr&r0r9r7r>rrrrrrrrArFr<r<r:r=r(s(  r(z Hubert Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. c szeZdZfddZddZddZddZed ee j ee j ee ee ee ee j e e efd d d ZZS)rcst|t|dr$|jr$tdt||_|jd}|jrTt t |||_ t |j|j|_t |j|j|_|dS)Nr*z]Sequence classification does not support the use of Hubert adapters (config.add_adapter=True)r)r%r&r/r*rgrrruse_weighted_layer_sumr'r rrrryr)Zclassifier_proj_size projector num_labels classifierr")r8r9Z num_layersr:r<r=r&s   z(HubertForSequenceClassification.__init__cCstdt|dS)z Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training. r2Nr3r8r<r<r=r9s z8HubertForSequenceClassification.freeze_feature_extractorcCs|jjdSr:r;r8r<r<r=r7sz6HubertForSequenceClassification.freeze_feature_encodercCs|jD] }d|_q dSr<r=rnr<r<r=r>sz1HubertForSequenceClassification.freeze_base_modelNr?cCs|dur |n|jj}|jjr dn|}|j|||||d}|jjr|t}tj|dd}tjj |j dd} || dddj dd}n|d}| |}|dur|jdd} nV||jd|} | ddd|jd} d || <|j dd| j dd dd} || } d}|dur 1` a classification loss is computed (Cross-Entropy). NTr&rrrYrrr~rB)r9r'rIrrGrstackr'rrrrrrJrrrrrrLrrKrr@r)r8rrrrrrr@rr@Z norm_weightsZ pooled_outputZ padding_maskZexpand_padding_maskrDrCZloss_fctrHr<r<r=rAsH     z'HubertForSequenceClassification.forward)NNNNN)rCrDrEr&r9r7r>rrrrrrrrrArFr<r<r:r=rs&   r)r(rrr)Nr~N)Nr)Ir4typingrrrnumpyrrZtorch.nnr'rZ activationsrZintegrations.deepspeedrZintegrations.fsdpr Zmodeling_attn_mask_utilsr r Zmodeling_flash_attention_utilsr Zmodeling_layersr Zmodeling_outputsrrrZmodeling_utilsrrZprocessing_utilsrr.rrrZconfiguration_hubertrZintegrations.flex_attentionrZ get_loggerrCr.Modulerr5rHrUrZr\rtrrrrrrrrrrrrrrZndarrayrrrGr(r__all__r<r<r<r=s           2 & X$].aJ  w s