a hu @s&ddlZddlZddlmZddlmZmZmZddlZ ddl Z ddl m Z ddl m Z ddlmZddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZmZmZmZm Z dd l!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)ddl*m+Z+e(rddl,m-Z-e).e/Z0ee'ddGdddeZ1Gddde j2Z3Gddde j2Z4GdddeZ5GdddeZ6Gdd d eZ7Gd!d"d"e j2Z8Gd#d$d$e j2Z9dKe j2e j:e j:e j:ee j:ee;e;ee j:d&d'd(ZGd-d.d.eZ?Gd/d0d0e j2Z@Gd1d2d2e j2ZAGd3d4d4eZBGd5d6d6e j2ZCGd7d8d8e j2ZDe'Gd9d:d:e#ZEdLeFeGeGfe;eGee jHeGe jId;dd?d?eEZLe'd@dGdAdBdBeEZMdCZNe'dDdGdEdFdFeEZOe'dGdGdHdIdIeEZPgdJZQdS)MN) dataclass)CallableOptionalUnion)CrossEntropyLoss)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)_prepare_4d_attention_mask#_prepare_4d_attention_mask_for_sdpa)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputCausalLMOutput ModelOutputSequenceClassifierOutputWav2Vec2BaseModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringis_torch_flex_attn_availablelogging)UniSpeechConfig)make_flex_block_causal_maskzh Output type of [`UniSpeechForPreTrainingOutput`], with potential hidden states and attentions. )Z custom_introc@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ ee ejed<dZee ejed<dS) UniSpeechForPreTrainingOutputa loss (*optional*, returned when model is in train mode, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the [official paper](https://huggingface.co/papers/2006.11477). projected_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Hidden-states of the model projected to *config.proj_codevector_dim* that can be used to predict the masked projected quantized states. projected_quantized_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Quantized extracted feature vectors projected to *config.proj_codevector_dim* representing the positive target vectors for contrastive loss. codevector_perplexity (`torch.FloatTensor` of shape `(1,)`): The perplexity of the codevector distribution, used to measure the diversity of the codebook. Nlossprojected_statesprojected_quantized_statescodevector_perplexity hidden_states attentions)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rr r!r"tupler#r,r,l/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/unispeech/modeling_unispeech.pyr:s rcs$eZdZfddZddZZS)UniSpeechSamePadLayercs$t|ddkrdnd|_dS)Nrr)super__init__num_pad_remove)selfnum_conv_pos_embeddings __class__r,r-r1Xs zUniSpeechSamePadLayer.__init__cCs,|jdkr(|ddddd|j f}|S)Nr)r2r3r"r,r,r-forward\s zUniSpeechSamePadLayer.forwardr$r%r&r1r8 __classcell__r,r,r5r-r.Ws r.cs$eZdZfddZddZZS) UniSpeechPositionalConvEmbeddingcs$ttj|j|j|j|jd|jd|_tjj }t tjj drNtjj j }t rddl }|jj|jjdd"||jddd|_Wdn1s0Yt |jdr|jj jj}|jj jj}n|jj}|jj}|j|||j||n||jddd|_t|j|_t|j|_dS) Nr/) kernel_sizepaddinggroups weight_normr)Z modifier_rankweight)namedimparametrizations)r0r1nnConv1d hidden_sizer4Znum_conv_pos_embedding_groupsconvutilsr?hasattrrCr deepspeedzeroZGatheredParametersr@Z original0Z original1weight_gweight_vZregister_external_parameterr.r=rfeat_extract_activation activation)r3configr?rJrLrMr5r,r-r1cs2  0   z)UniSpeechPositionalConvEmbedding.__init__cCs:|dd}||}||}||}|dd}|S)Nrr/) transposerGr=rOr7r,r,r-r8s      z(UniSpeechPositionalConvEmbedding.forwardr9r,r,r5r-r;bs !r;cs&eZdZdfdd ZddZZS)UniSpeechNoLayerNormConvLayerrcsjt|dkr |j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ dS)Nrrr<stridebias)r0r1conv_dim in_conv_dim out_conv_dimrDrE conv_kernel conv_stride conv_biasrGrrNrOr3rPlayer_idr5r,r-r1s  z&UniSpeechNoLayerNormConvLayer.__init__cCs||}||}|SN)rGrOr7r,r,r-r8s  z%UniSpeechNoLayerNormConvLayer.forward)rr9r,r,r5r-rRsrRcs&eZdZdfdd ZddZZS)UniSpeechLayerNormConvLayerrcs|t|dkr |j|dnd|_|j||_tj|j|j|j||j||j d|_ tj |jdd|_ t |j|_dS)NrrrST)Zelementwise_affine)r0r1rVrWrXrDrErYrZr[rG LayerNorm layer_normrrNrOr\r5r,r-r1s  z$UniSpeechLayerNormConvLayer.__init__cCs:||}|dd}||}|dd}||}|S)N)rGrQrarOr7r,r,r-r8s      z#UniSpeechLayerNormConvLayer.forward)rr9r,r,r5r-r_sr_cs&eZdZdfdd ZddZZS)UniSpeechGroupNormConvLayerrcst|dkr |j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ tj|j|jdd|_dS)NrrrST) num_groupsZ num_channelsZaffine)r0r1rVrWrXrDrErYrZr[rGrrNrO GroupNormrar\r5r,r-r1s   z$UniSpeechGroupNormConvLayer.__init__cCs"||}||}||}|Sr^)rGrarOr7r,r,r-r8s   z#UniSpeechGroupNormConvLayer.forward)rr9r,r,r5r-rdsrdcs0eZdZdZfddZddZddZZS)UniSpeechFeatureEncoderz.Construct the features from raw audio waveformcstjdkr@tddgfddtjdD}n6jdkrdfddtjD}ntd jd t||_ d |_ d |_ dS) Ngrouprr]csg|]}t|ddqS)rri)rR.0irPr,r- sz4UniSpeechFeatureEncoder.__init__..rlayercsg|]}t|dqS)ri)r_rjrmr,r-rnsz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT) r0r1Zfeat_extract_normrdrangeZnum_feat_extract_layers ValueErrorrD ModuleList conv_layersgradient_checkpointing_requires_grad)r3rPrsr5rmr-r1s       z UniSpeechFeatureEncoder.__init__cCs|D] }d|_qd|_dS)NF) parameters requires_gradrur3paramr,r,r-_freeze_parameterss z*UniSpeechFeatureEncoder._freeze_parameterscCs:|dddf}|jr"|jr"d|_|jD] }||}q(|S)NT)rutrainingrwrs)r3 input_valuesr"Z conv_layerr,r,r-r8s    zUniSpeechFeatureEncoder.forward)r$r%r&r'r1rzr8r:r,r,r5r-rgs rgcs$eZdZfddZddZZS)UniSpeechFeatureProjectioncsJttj|jd|jd|_t|jd|j|_ t |j |_ dS)Nrceps) r0r1rDr`rVlayer_norm_epsraLinearrF projectionDropoutZfeat_proj_dropoutdropoutr3rPr5r,r-r1s z#UniSpeechFeatureProjection.__init__cCs&||}||}||}||fSr^)rarr)r3r"Znorm_hidden_statesr,r,r-r8s   z"UniSpeechFeatureProjection.forwardr9r,r,r5r-r}s r})modulequerykeyvalueattention_maskscalingr head_maskc Ks|dur|dd}t||dd|} |dur>| |} tjj| dd} |durj| |dddd} tjj| ||j d} t| |} | dd } | | fS)Nrcr/rrBr)pr{) sizer(matmulrQrD functionalsoftmaxviewrr{ contiguous) rrrrrrrrkwargs attn_weights attn_outputr,r,r-eager_attention_forwards  rc seZdZdZd eeeeeeeedfdd Z d e j ee j ee j ee j eee e ee j ee j eee j fd d d ZZS)UniSpeechAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrFTN) embed_dim num_headsr is_decoderrU is_causalrPcst||_||_||_|||_||_|j||jkrTtd|jd|d|jd|_||_ ||_ t j |||d|_ t j |||d|_t j |||d|_t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r)rU)r0r1rrrhead_dimrPrqrrrrDrk_projv_projq_projout_proj)r3rrrrrUrrPr5r,r-r10s&    zUniSpeechAttention.__init__)r"key_value_statesrlayer_head_maskoutput_attentionsrreturncKs|du}|jdd\}} |r(|jdn| } || d|jf} || d|jf} ||j| dd} |rh|n|}||j| dd}||j| dd}t}|jj dkrt |jj }||| |||f|j sdn|j |j ||d|\}}||| d}||}||dfS)z#Input shape: Batch x Time x ChannelNrcrr/eagerr)rrrr)shaperrrrQrrrrP_attn_implementationrr{rrreshaperr)r3r"rrrrrZis_cross_attentionZbszZtgt_lenZsrc_lenZ q_input_shapeZkv_input_shapeZ query_statesZcurrent_statesZ key_statesZ value_statesZattention_interfacerrr,r,r-r8Os:      zUniSpeechAttention.forward)rFTFN)NNNF)r$r%r&r'intfloatboolrrr1r(Tensorrr r+r8r:r,r,r5r-r-s8"rcs$eZdZfddZddZZS)UniSpeechFeedForwardcsptt|j|_t|j|j|_ t |j t rDt |j |_n|j |_t|j|j|_t|j|_dSr^)r0r1rDrZactivation_dropoutintermediate_dropoutrrFZintermediate_sizeintermediate_dense isinstanceZ hidden_actstrrintermediate_act_fn output_densehidden_dropoutoutput_dropoutrr5r,r-r1s  zUniSpeechFeedForward.__init__cCs6||}||}||}||}||}|Sr^)rrrrrr7r,r,r-r8s      zUniSpeechFeedForward.forwardr9r,r,r5r-rs rcs&eZdZfddZdddZZS)UniSpeechEncoderLayercshtt|j|j|jd|d|_t|j |_ tj |j|j d|_ t||_tj |j|j d|_dS)NFrrrrrPr~)r0r1rrFnum_attention_headsattention_dropout attentionrDrrrr`rrar feed_forwardfinal_layer_normrr5r,r-r1s  zUniSpeechEncoderLayer.__init__NFcCsf|}|j|||d\}}}||}||}||}|||}||}|f}|rb||f7}|SNrr)rrrarrr3r"rrZ attn_residualr_outputsr,r,r-r8s     zUniSpeechEncoderLayer.forward)NFr9r,r,r5r-rs rcsXeZdZfddZd ejeejeeedddZ e ejdfejd d d Z Z S) UniSpeechEncodercsft|_t|_tjjjd|_ t j |_ t fddtjD|_d|_dS)Nr~csg|] }tqSr,)rrkrrmr,r-rnz-UniSpeechEncoder.__init__..Fr0r1rPr;pos_conv_embedrDr`rFrrarrrrrrpnum_hidden_layerslayersrtrr5rmr-r1s   zUniSpeechEncoder.__init__NFT)r"rroutput_hidden_states return_dictcCs.|rdnd}|rdnd}|durD|ddd|jd}d||<|||}||} || }||}||}tpt|} |j D]f} |r||f}t g} |j o| |j jk} | r| r| |||d}|d}| rd}|r||df}q|r||f}|s tdd |||fDSt|||d S) Nr,rcrr/rrNNcss|]}|dur|VqdSr^r,rkvr,r,r- rz+UniSpeechEncoder.forward..last_hidden_stater"r#) unsqueezerepeatr_update_full_maskrrarr r rr(randr{rP layerdropr+rr3r"rrrrZall_hidden_statesZall_self_attentionsZexpand_attention_maskZposition_embeddingsZ synced_gpusroZdropout_probabilityZskip_the_layerZ layer_outputsr,r,r-r8sJ          zUniSpeechEncoder.forwardr inputs_embedscCsv|durr|jjdkr&d|vr |nd}nL|jjdkr@t||j}n2|jjdkrft|tjrrt|dd}n t||j}|SNZflash_attention_2rZsdpaZflex_attentionF)r rPrr dtyperr(rrr r3rrr,r,r-rs     z"UniSpeechEncoder._update_full_mask)NFFT) r$r%r&r1r(tensorrrrr8rrr:r,r,r5r-rs  > rcs,eZdZfddZejdddZZS)UniSpeechAttnAdapterLayercsZt|j|_|j|_t|j|_t |j|j|_ t |_ t |j|j|_ dS)z Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed up training throughput. N)r0r1adapter_attn_dimZ input_dimrFZ hidden_dimrDr`normrlinear_1ZReLUact_fnlinear_2rr5r,r-r1s  z"UniSpeechAttnAdapterLayer.__init__)r"cCs,||}||}||}||}|Sr^)rrrrr7r,r,r-r8-s     z!UniSpeechAttnAdapterLayer.forward)r$r%r&r1r(r)r8r:r,r,r5r-rs rcs8eZdZfddZdejeejedddZZ S) $UniSpeechEncoderLayerStableLayerNormcstt|j|j|jd|d|_t|j |_ tj |j|j d|_ t||_tj |j|j d|_t|dddurt||_nd|_dS)NFrr~r)r0r1rrFrrrrDrrrr`rrarrrgetattrr adapter_layerrr5r,r-r18s   z-UniSpeechEncoderLayerStableLayerNorm.__init__NF)r"rrcCsz|}||}|j|||d\}}}||}||}||||}|jdurb|||}|f}|rv||f7}|Sr)rarrrrrrr,r,r-r8Ks     z,UniSpeechEncoderLayerStableLayerNorm.forward)NF) r$r%r&r1r(rrrr8r:r,r,r5r-r7s rcsBeZdZfddZd ddZeejdfejdd d ZZ S) UniSpeechEncoderStableLayerNormcsft|_t|_tjjjd|_ t j |_ t fddtjD|_d|_dS)Nr~csg|] }tqSr,)rrrmr,r-rnmrz.Frrr5rmr-r1fs  z(UniSpeechEncoderStableLayerNorm.__init__NFTcCs.|rdnd}|rdnd}|durD|ddd|jd}d||<|||}||} || }||}tpxt|} |jD]f} |r||f}t g} |j o| |j j k} | r| r| |||d}|d}| rd}|r||df}q||}|r||f}|s tdd |||fDSt|||d S) Nr,rcrr/rrrcss|]}|dur|VqdSr^r,rr,r,r-rrz:UniSpeechEncoderStableLayerNorm.forward..r)rrrrrrr r rr(rr{rPrrar+rrr,r,r-r8qsJ          z'UniSpeechEncoderStableLayerNorm.forwardrcCsv|durr|jjdkr&d|vr |nd}nL|jjdkr@t||j}n2|jjdkrft|tjrrt|dd}n t||j}|Srrrr,r,r-rs     z1UniSpeechEncoderStableLayerNorm._update_full_mask)NFFT) r$r%r&r1r8rr(rrr:r,r,r5r-res  @ rcs4eZdZdZfddZeddZddZZS)UniSpeechGumbelVectorQuantizerz Vector quantization using gumbel softmax. See `[CATEGORICAL REPARAMETERIZATION WITH GUMBEL-SOFTMAX](https://huggingface.co/papers/1611.01144) for more information. cst|j|_|j|_|j|jdkrDtd|jd|jdt t d|j|j|j|j|_ t |jd|j|j|_d|_dS)Nrz`config.codevector_dim z5 must be divisible by `config.num_codevector_groups` z for concatenationrrcr/)r0r1Znum_codevector_groupsreZnum_codevectors_per_groupnum_varscodevector_dimrqrD Parameterr(r) codevectorsrrV weight_proj temperaturerr5r,r-r1s  z'UniSpeechGumbelVectorQuantizer.__init__cCs8|jdd}ttj|t|ddd }|S)NrrgHz>rc)meanr(expsumlog)ZprobsZmarginal_probs perplexityr,r,r-_compute_perplexitys (z2UniSpeechGumbelVectorQuantizer._compute_perplexityc Cs|j\}}}||}||||jd}|jr~tjj||j dd |}t j ||||jddd}| |}nH|jdd}|j|jd|ddd}||||jd}| |}|||d}|d|j} | |||j|jd} | d||d} | |fS)NrcT)tauhardrr?rb)rrrrer{rDrZgumbel_softmaxrrtype_asr(rrZargmaxZ new_zerosZscatter_rrrr) r3r" batch_sizesequence_lengthrFZcodevector_probsZcodevector_soft_distrZcodevector_idxZcodevectors_per_grouprr,r,r-r8s0       z&UniSpeechGumbelVectorQuantizer.forward) r$r%r&r'r1 staticmethodrr8r:r,r,r5r-rs   rc@s`eZdZUeed<dZdZdZdZdZ dZ ddZ e e jefddd Zee jd d d Zd S)UniSpeechPreTrainedModelrP unispeechr|Tc Cst|tr>|jjjjddd|jjjtj |j njt|t rtj j|j jddtd|j jd|j jdtj |j jdnt|trtd|jj}tj j |jj| |dtj j |jj| |dnt|tjr|jjjd|jjd|jdur|jjnt|tjtjfrN|jj|jjdnZt|tjrtj |j|jdurt|j|j|jd}tj j |j| |ddS) zInitialize the weightsrr)rZstdrr/)abNr)rrrr@dataZnormal_rUZzero_rDinituniform_rr;rGmathsqrtr<Z in_channelsZ constant_r}rZ in_featuresrrPZinitializer_ranger`rffill_rEZkaiming_normal_r>)r3rkr,r,r- _init_weightss6       z&UniSpeechPreTrainedModel._init_weights) input_lengthscCs4dd}t|jj|jjD]\}}||||}q|S)zH Computes the output length of the convolutional layers cSstj|||dddS)Nfloor)Z rounding_moder)r(div) input_lengthr<rTr,r,r-_conv_out_length<szSUniSpeechPreTrainedModel._get_feat_extract_output_lengths.._conv_out_length)ziprPrYrZ)r3r rr<rTr,r,r- _get_feat_extract_output_lengths7sz9UniSpeechPreTrainedModel._get_feat_extract_output_lengths)feature_vector_lengthrcCs|jdddddf}||tj}|jd}tj||f|j|jd}d|tj |jd|jd|df<| dgd dg }|S)Nrcrr)rdevicer)r) Zcumsumrtor(longrzerosrrarangeflipr)r3rrZnon_padded_lengthsZoutput_lengthsrr,r,r-"_get_feature_vector_attention_maskFs "z;UniSpeechPreTrainedModel._get_feature_vector_attention_maskN)r$r%r&rr*Zbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingZ_supports_flash_attnZ_supports_sdpaZ_supports_flex_attnr rr( LongTensorrrrr,r,r,r-r s !r)r mask_prob mask_lengthr min_masksrcs|\}dkrtdkr6tdddtjdfdd}|durt|d nfd d t|D}tj |ft d }g} |} | d kr|S|D]v} || } tjj t | d| dd} t | d krd}n| d }t| tj| | tjd |g} | | qt| } t| dddddf|| f} | || } t ddddf}t||| f|| }| |} | dkr҈d| | dk<t|| dd |S)an Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `csXt|}t|}|kr2}|d|krTt|dd}|S)z;Given input length, compute how many spans should be maskedrr)rmax)rnum_masked_spanepsilonrrrrr,r-compute_num_masked_span|s  z6_compute_mask_indices..compute_num_masked_spanNrccsg|]}qSr,r,r)rr,r-rnrz)_compute_mask_indices..rrF)replace)rqnprandomritemdetachrtolistrprrchoicerlenZ concatenateonesZint32appendarrayZ broadcast_torr Zput_along_axis)rrrrrrr$r Z spec_aug_maskZspec_aug_mask_idxsZmax_num_masked_spanrr!Zspec_aug_mask_idxZdummy_mask_idxoffsetsr,r"r-_compute_mask_indicesVs\    r2c seZdZedfdd Zd ejeejeejdddZ e d eej eej eejee ee ee e eefdd d ZZS) UniSpeechModelrmcszt|||_t||_t||_|jdks:|jdkrRt t |j |_|jrdt||_n t||_|dS)Nr)r0r1rPrgfeature_extractorr}feature_projectionmask_time_probmask_feature_probrDrr(rrFrmasked_spec_embedZdo_stable_layer_normrencoderr post_initrr5r,r-r1s     zUniSpeechModel.__init__N)r"mask_time_indicesrcCst|jdds|S|\}}}|dur<|j|j||<nZ|jjdkr|jrt||f|jj|jj ||jj d}t j ||j t jd}|j|j||<|jjdkr|jrt||f|jj|jj|jjd}t j ||j t jd}|dddfd|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://huggingface.co/papers/1904.08779). Zapply_spec_augmentTNr)rrrr)rr)rrrrc)rrPrr8rrr6r{r2Zmask_time_lengthZmask_time_min_masksr(rrrr7Zmask_feature_lengthZmask_feature_min_masksexpand)r3r"r;rrrrFZmask_feature_indicesr,r,r-_mask_hidden_statess4 z"UniSpeechModel._mask_hidden_states)r|rr;rrrrc Cs|dur |n|jj}|dur |n|jj}|dur4|n|jj}||}|dd}|durl||jd|}||\}}|j |||d}|j |||||d} | d}|s||f| ddSt ||| j | j dS)a/ mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. Nrr/)r;rrrrrr)rextract_featuresr"r#)rPrruse_return_dictr4rQrrr5r=r9UniSpeechBaseModelOutputr"r#) r3r|rr;rrrr?r"Zencoder_outputsr,r,r-r8s8  zUniSpeechModel.forward)NN)NNNNN)r$r%r&rr1r(r)rrr=rrrrr+rAr8r:r,r,r5r-r3s. . r3zZ UniSpeech Model with a vector-quantization module and ctc loss for pre-training. c seZdZedfdd ZedddZddZd d Ze de j e j e j ed d dZ e dee jee jeeeeeeeeefdddZZS)UniSpeechForPreTrainingrmcs~t|t||_t|j|_t||_ t |j |j |_ t |j |j|_t |j|j|_t|j|_|dSr^)r0r1r3rrDrZfeat_quantizer_dropoutdropout_featuresr quantizerrrZproj_codevector_dim project_qrF project_hidZnum_ctc_classesctc_proj final_dropoutrr:rr5r,r-r1Ms   z UniSpeechForPreTraining.__init__)rcCs ||j_dS)zb Set the Gumbel softmax temperature to a given value. Only necessary for training N)rDr)r3rr,r,r-set_gumbel_temperature\sz.UniSpeechForPreTraining.set_gumbel_temperaturecCstdt|dSz Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training. The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. Please use the equivalent `freeze_feature_encoder` method instead.Nwarningswarn FutureWarningfreeze_feature_encoderr3r,r,r-freeze_feature_extractorbs z0UniSpeechForPreTraining.freeze_feature_extractorcCs|jjdS Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. Nrr4rzrQr,r,r-rPnsz.UniSpeechForPreTraining.freeze_feature_encoderr)target_featuresnegative_featurespredicted_featuresrcCs@tj||gdd}tj||dd}||}||}|S)z Compute logits for contrastive loss based using cosine similarity as the distance measure between `[positive_feature, negative_features]` and `[predicted_features]`. Additionally, temperature can be applied. rrrc)r(catZcosine_similarityrr)rVrWrXrlogitsr,r,r-compute_contrastive_logitsus  z2UniSpeechForPreTraining.compute_contrastive_logitsN)r|rrrrrcCsN|dur |n|jj}|j|||||d}|d}||d}||\} } || |jjj} | | } t | d| d |jj} | dd} t | |j} | dd} | d} || d| | d} || } || } d}|s6|dur ||| | f|ddS|| | f|ddSt||| | |j|jdS) a Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, UniSpeechForPreTraining >>> feature_extractor = AutoFeatureExtractor.from_pretrained("microsoft/unispeech-large-1500h-cv") >>> model = UniSpeechForPreTraining.from_pretrained("microsoft/unispeech-large-1500h-cv") >>> # TODO: Add full pretraining example ```Nr>rrrcrr/)rrr r!r"r#)rPr@rrCrDrErr@rrFr(emptyrr Z replace_probrQZ bernoullirrrZ masked_fillrrGrr"r#)r3r|rrrrrZtransformer_featuresr?Zquantized_featuresr!Zprob_replace_matrixZsampled_replace_matrixrZrr,r,r-r8sL         zUniSpeechForPreTraining.forward)r)NNNN)r$r%r&rr1rrIrRrPrr(r)r[rrrrrr+rr8r:r,r,r5r-rBGs2  rBr/zq UniSpeech Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). c seZdZdeedfdd ZddZddZd d Zd d Z e dee j ee j ee ee ee ee j eeefd ddZZS)UniSpeechForCTCN) target_langcs~t|t||_t|j|_||_|j durFt d|j dt |dr\|j r\|jn|j}t||j |_|dS)a3 target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`UniSpeechForCTC`] with adapters. Uses 'eng' by default. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `UniSpeechForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration. add_adapter)r0r1r3rrDrrHrr^ vocab_sizerqr6rIr_output_hidden_sizerFrlm_headr:)r3rPr^rar5r,r-r1s    zUniSpeechForCTC.__init__cCsr|j}|dur2t|jdddur2td|dn<|durXt|jdddurXtdn|durn|j|dddS)a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. NrzCannot pass `target_lang`: z- if `config.adapter_attn_dim` is not defined.z)By default `target_lang` is set to 'eng'.T)Z force_load)r^rrPrqloggerinfoZ load_adapter)r3r^r,r,r- tie_weightss  zUniSpeechForCTC.tie_weightscCstdt|dS)rTrKNrLrQr,r,r-rR s z(UniSpeechForCTC.freeze_feature_extractorcCs|jjdSrSrUrQr,r,r-rPsz&UniSpeechForCTC.freeze_feature_encodercCs|jD] }d|_q dSz Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated. FNrrvrwrxr,r,r-freeze_base_modelsz!UniSpeechForCTC.freeze_base_modelr|rrrrlabelsrc Cs|dur |n|jj}|dur>||jjkr>td|jj|j|||||d}|d}||}||} d} |dur@|dur|ntj |tj d}| | d tj } |dk} | d} || }tjj| dtjddd}tjjjd d 6tjj||| | |jj|jj|jjd } Wdn1s60Y|sp| f|td}| durl| f|S|St| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nz$Label values must be <= vocab_size: r>rr%rc)rBrrF)Zenabled)blankZ reductionZ zero_infinityrrZr"r#)rPr@r r`rqrrrbr(Z ones_likerrrrZ masked_selectrDrZ log_softmaxZfloat32rQbackendsZcudnnflagsZctc_lossZ pad_token_idZctc_loss_reductionZctc_zero_infinity_HIDDEN_STATES_START_POSITIONrr"r#)r3r|rrrrrjrr"rZrr Z labels_maskZtarget_lengthsZflattened_targetsZ log_probsoutputr,r,r-r8'sL     &  zUniSpeechForCTC.forward)N)NNNNN)r$r%r&rrr1rerRrPrhrr(rrrr+rr8r:r,r,r5r-r]s(  r]z UniSpeech Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. c szeZdZfddZddZddZddZed ee j ee j ee ee ee ee j e e efd d d ZZS)"UniSpeechForSequenceClassificationcst|t|dr$|jr$tdt||_|jd}|jrTt t |||_ t |j|j|_t |j|j|_|dS)Nr_z`Sequence classification does not support the use of UniSpeech adapters (config.add_adapter=True)r)r0r1rIr_rqr3rruse_weighted_layer_sumrDrr(r. layer_weightsrrFZclassifier_proj_size projector num_labels classifierr:)r3rPZ num_layersr5r,r-r1vs   z+UniSpeechForSequenceClassification.__init__cCstdt|dSrJrLrQr,r,r-rRs z;UniSpeechForSequenceClassification.freeze_feature_extractorcCs|jjdSrSrUrQr,r,r-rPsz9UniSpeechForSequenceClassification.freeze_feature_encodercCs|jD] }d|_q dSrfrgrxr,r,r-rhsz4UniSpeechForSequenceClassification.freeze_base_modelNricCs|dur |n|jj}|jjr dn|}|j|||||d}|jjr|t}tj|dd}tjj |j dd} || dddj dd}n|d}| |}|dur|jdd} nV||jd|} | ddd|jd} d || <|j dd| j dd dd} || } d}|dur 1` a classification loss is computed (Cross-Entropy). NTr>rrrcrr/rrl)rPr@rrrror(stackrDrrrsrrrtrrrrrrvrrurr"r#)r3r|rrrrrjrr"Z norm_weightsZ pooled_outputZ padding_maskZexpand_padding_maskrZrZloss_fctrpr,r,r-r8sH     z*UniSpeechForSequenceClassification.forward)NNNNN)r$r%r&r1rRrPrhrrr(rrrr+rr8r:r,r,r5r-rqos&   rq)r]rBrqr3r)NrN)Nr)RrrM dataclassesrtypingrrrnumpyr'r(Ztorch.nnrDrZ activationsrZintegrations.deepspeedr Zintegrations.fsdpr Zmodeling_attn_mask_utilsr r Zmodeling_flash_attention_utilsr Zmodeling_layersrZmodeling_outputsrrrrrZmodeling_utilsrrZprocessing_utilsrrHrrrZconfiguration_unispeechrZintegrations.flex_attentionrZ get_loggerr$rcrModuler.r;rRr_rdrgr}rrrrrrrrrrrrr+rrZndarrayr2rAr3rBror]rq__all__r,r,r,r-s             -) X$].aFM  wvs