a h @sdZddlZddlZddlmZddlmZmZmZddl Z ddl Z ddl Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZmZdd lmZdd lmZddlmZmZm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2ddl3m4Z4dZ5dZ6e0r@ddl7m8Z9e1rTddl:m;Z;e2dZ?ee,ddGddde+Z@ddeAeBeBfeCeBee jDeBe jEdddZFdeeAeBee jEd d!d"ZGGd#d$d$eZHGd%d&d&eZIGd'd(d(eZJGd)d*d*e jKZLGd+d,d,e jKZMGd-d.d.e jKZNGd/d0d0eNZOGd1d2d2e jKZPdfe jKe jQe jQe jQee jQeeCeCee jQd4d5d6ZRGd7d8d8e jKZSGd9d:d:e jKZTGd;d<dd>eZVGd?d@d@e jKZWGdAdBdBe jKZXGdCdDdDe jKZYGdEdFdFe jKZZGdGdHdHe jKZ[GdIdJdJe jKZ\e,GdKdLdLe'Z]e,GdMdNdNe]Z^e,dOdGdPdQdQe]Z_e,GdRdSdSe]Z`e,dTdGdUdVdVe]Zae,dWdGdXdYdYe]Zbe,GdZd[d[e]ZcGd\d]d]e jKZdGd^d_d_e jKZee,d`dGdadbdbe]ZfgdcZgdS)gzPyTorch Wav2Vec2 model.N) dataclass)CallableOptionalUnion)nn)CrossEntropyLoss)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)_prepare_4d_attention_mask#_prepare_4d_attention_mask_for_sdpa)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputCausalLMOutputMaskedLMOutputSequenceClassifierOutputTokenClassifierOutputWav2Vec2BaseModelOutput XVectorOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputauto_docstring cached_filecheck_torch_load_is_safeis_peft_availableis_safetensors_availableis_torch_flex_attn_availablelogging)Wav2Vec2Configzadapter.{}.binzadapter.{}.safetensors) load_file)make_flex_block_causal_maskza Output type of [`Wav2Vec2ForPreTraining`], with potential hidden states and attentions. )Z custom_introc@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ ee ejed<dZee ejed<dZeejed <dZeejed <dS) Wav2Vec2ForPreTrainingOutputa loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the [official paper](https://huggingface.co/papers/2006.11477). projected_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Hidden-states of the model projected to *config.proj_codevector_dim* that can be used to predict the masked projected quantized states. projected_quantized_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Quantized extracted feature vectors projected to *config.proj_codevector_dim* representing the positive target vectors for contrastive loss. codevector_perplexity (`torch.FloatTensor` of shape `(1,)`): The perplexity of the codevector distribution, used to measure the diversity of the codebook. contrastive_loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): The contrastive loss (L_m) as stated in the [official paper](https://huggingface.co/papers/2006.11477). diversity_loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): The diversity loss (L_d) as stated in the [official paper](https://huggingface.co/papers/2006.11477). Nlossprojected_statesprojected_quantized_statescodevector_perplexity hidden_states attentionscontrastive_lossdiversity_loss)__name__ __module__ __qualname____doc__r(rtorch FloatTensor__annotations__r)r*r+r,tupler-r.r/r8r8j/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.pyr'Ns r')shape mask_prob mask_lengthattention_mask min_masksreturncs|\}dkrtdkr6tdddtjdfdd}|durt|d nfd d t|D}tj |ft d }g} |} | d kr|S|D]v} || } tjj t | d| dd} t | d krd}n| d }t| tj| | tjd |g} | | qt| } t| dddddf|| f} | || } t ddddf}t||| f|| }| |} | dkr҈d| | dk<t|| dd |S)an Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. r"z&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `csXt|}t|}|kr2}|d|krTt|dd}|S)z;Given input length, compute how many spans should be maskedr"r)intmax) input_lengthnum_masked_spanepsilonr<r;r>sequence_lengthr8r9compute_num_masked_spans  z6_compute_mask_indices..compute_num_masked_spanNcsg|]}qSr8r8.0_)rGr8r9 z)_compute_mask_indices..dtyperF)replace) ValueErrornprandomranditemdetachsumtolistrangezerosboolchoicearangelenZ concatenateonesint32appendarray broadcast_toreshaperBZput_along_axis)r:r;r<r=r> batch_sizerH input_lengthsZ spec_aug_maskZspec_aug_mask_idxsZmax_num_masked_spanrCrDZspec_aug_mask_idxZdummy_mask_idxoffsetsr8rEr9_compute_mask_indicesqs\    ri)features_shape num_negativesmask_time_indicesc Cs|\}}t|}tj|||ftjd}|dur:|tn tj|td}t|D]}||d}|||} t t|ddddf|d|f} tj j d||d|fd} | | | kd7<| | ||||<||||7<qP|S)z> Sample `num_negatives` vectors from feature vectors. )r:rPNrOr"r)size) rSr^r[raZastyper\r`rZrXrdrTrandint) rjrkrlrfrGZsequence_length_rangesampled_negative_indicesZ batch_idxhighZmapped_masked_indicesZfeature_indicesZsampled_indicesr8r8r9_sample_negative_indicess   *rqcs&eZdZdfdd ZddZZS)Wav2Vec2NoLayerNormConvLayerrcsjt|dkr |j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ dS)Nrr" kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvr feat_extract_activation activationselfconfiglayer_id __class__r8r9rx s  z%Wav2Vec2NoLayerNormConvLayer.__init__cCs||}||}|SN)rrrr,r8r8r9forwards  z$Wav2Vec2NoLayerNormConvLayer.forward)rr0r1r2rxr __classcell__r8r8rr9rr srrcs&eZdZdfdd ZddZZS)Wav2Vec2LayerNormConvLayerrcs|t|dkr |j|dnd|_|j||_tj|j|j|j||j||j d|_ tj |jdd|_ t |j|_dS)Nrr"rsT)Zelementwise_affine)rwrxryrzr{rr|r}r~rr LayerNorm layer_normr rrrrr8r9rx"s  z#Wav2Vec2LayerNormConvLayer.__init__cCs:||}|dd}||}|dd}||}|S)NrI)r transposerrrr8r8r9r1s      z"Wav2Vec2LayerNormConvLayer.forward)rrr8r8rr9r!srcs&eZdZdfdd ZddZZS)Wav2Vec2GroupNormConvLayerrcst|dkr |j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ tj|j|jdd|_dS)Nrr"rsT) num_groupsZ num_channelsZaffine)rwrxryrzr{rr|r}r~rrr rr GroupNormrrrr8r9rx=s   z#Wav2Vec2GroupNormConvLayer.__init__cCs"||}||}||}|Sr)rrrrr8r8r9rMs   z"Wav2Vec2GroupNormConvLayer.forward)rrr8r8rr9r<srcs$eZdZfddZddZZS)Wav2Vec2PositionalConvEmbeddingcs$ttj|j|j|j|jd|jd|_tjj }t tjj drNtjj j }t rddl }|jj|jjdd"||jddd|_Wdn1s0Yt |jdr|jj jj}|jj jj}n|jj}|jj}|j|||j||n||jddd|_t|j|_t|j|_dS) Nr&)rtpaddinggroups weight_normr)Z modifier_rankweight)namedimparametrizations)rwrxrr| hidden_sizenum_conv_pos_embeddingsZnum_conv_pos_embedding_groupsrutilsrhasattrrr deepspeedzeroZGatheredParametersrZ original0Z original1weight_gweight_vZregister_external_parameterWav2Vec2SamePadLayerrr rr)rrrrrrrr8r9rxUs2  0   z(Wav2Vec2PositionalConvEmbedding.__init__cCs:|dd}||}||}||}|dd}|SNr"r&)rrrrrr8r8r9rvs      z'Wav2Vec2PositionalConvEmbedding.forwardrr8r8rr9rTs !rcs$eZdZfddZddZZS)rcs$t|ddkrdnd|_dS)Nr&rr")rwrxnum_pad_remove)rrrr8r9rxs zWav2Vec2SamePadLayer.__init__cCs,|jdkr(|ddddd|j f}|S)Nr)rrr8r8r9rs zWav2Vec2SamePadLayer.forwardrr8r8rr9rs rcs0eZdZdZfddZddZddZZS)Wav2Vec2FeatureEncoderz.Construct the features from raw audio waveformcstjdkr@tddgfddtjdD}n6jdkrdfddtjD}ntd jd t||_ d |_ d |_ dS) Ngrouprrcsg|]}t|ddqS)r"r)rrrKirr8r9rMsz3Wav2Vec2FeatureEncoder.__init__..r"layercsg|]}t|dqS)r)rrrr8r9rMsz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT) rwrxZfeat_extract_normrrZZnum_feat_extract_layersrRr ModuleList conv_layersgradient_checkpointing_requires_grad)rrrrrr9rxs       zWav2Vec2FeatureEncoder.__init__cCs|D] }d|_qd|_dS)NF) parameters requires_gradrrparamr8r8r9_freeze_parameterss z)Wav2Vec2FeatureEncoder._freeze_parameterscCs:|dddf}|jr"|jr"d|_|jD] }||}q(|S)NT)rtrainingrr)r input_valuesr,Z conv_layerr8r8r9rs    zWav2Vec2FeatureEncoder.forward)r0r1r2r3rxrrrr8r8rr9rs rcseZdZfddZZS)Wav2Vec2FeatureExtractorcs8t|td|jjd|jjdjdtdS)Nz The class `zD` has been depreciated and will be removed in Transformers v5. Use `rz ` instead.)rwrxwarningswarnrr0 __bases__ FutureWarningrrrr8r9rxs   z!Wav2Vec2FeatureExtractor.__init__)r0r1r2rxrr8r8rr9rsrcs$eZdZfddZddZZS)Wav2Vec2FeatureProjectioncsJttj|jd|jd|_t|jd|j|_ t |j |_ dS)NrIeps) rwrxrrrylayer_norm_epsrLinearr projectionDropoutZfeat_proj_dropoutdropoutrrr8r9rxs z"Wav2Vec2FeatureProjection.__init__cCs&||}||}||}||fSr)rrr)rr,Znorm_hidden_statesr8r8r9rs   z!Wav2Vec2FeatureProjection.forwardrr8r8rr9rs r)modulequerykeyvaluer=scalingr head_maskc Ks|dur|dd}t||dd|} |dur>| |} tjj| dd} |durj| |dddd} tjj| ||j d} t| |} | dd } | | fS)NrIr&rrr")pr) rmr4matmulrr functionalsoftmaxviewrr contiguous) rrrrr=rrrkwargs attn_weights attn_outputr8r8r9eager_attention_forwards  rc seZdZdZd eeeeeeeedfdd Z d e j ee j ee j ee j eee e ee j ee j eee j fd d d ZZS)Wav2Vec2Attentionz=Multi-headed attention from 'Attention Is All You Need' paperrFTN) embed_dim num_headsr is_decoderrv is_causalrcst||_||_||_|||_||_|j||jkrTtd|jd|d|jd|_||_ ||_ t j |||d|_ t j |||d|_t j |||d|_t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r)rv)rwrxrrrhead_dimrrRrrrrrk_projv_projq_projout_proj)rrrrrrvrrrr8r9rxs&    zWav2Vec2Attention.__init__)r,key_value_statesr=layer_head_maskoutput_attentionsrr?cKs|du}|jdd\}} |r(|jdn| } || d|jf} || d|jf} ||j| dd} |rh|n|}||j| dd}||j| dd}t}|jj dkrt |jj }||| |||f|j sdn|j |j ||d|\}}||| d}||}||dfS)z#Input shape: Batch x Time x ChannelNrIr"r&eagerr)rrrr)r:rrrrrrrr_attn_implementationrrrrrerr)rr,rr=rrrZis_cross_attentionZbszZtgt_lenZsrc_lenZ q_input_shapeZkv_input_shapeZ query_statesZcurrent_statesZ key_statesZ value_statesZattention_interfacerrr8r8r9rs:      zWav2Vec2Attention.forward)rFTFN)NNNF)r0r1r2r3rAfloatr\rr#rxr4Tensorrrr7rrr8r8rr9rs8"rcs$eZdZfddZddZZS)Wav2Vec2FeedForwardcsptt|j|_t|j|j|_ t |j t rDt |j |_n|j |_t|j|j|_t|j|_dSr)rwrxrrZactivation_dropoutintermediate_dropoutrrZintermediate_sizeintermediate_dense isinstanceZ hidden_actstrr intermediate_act_fn output_densehidden_dropoutoutput_dropoutrrr8r9rxFs  zWav2Vec2FeedForward.__init__cCs6||}||}||}||}||}|Sr)rrrrrrr8r8r9rSs      zWav2Vec2FeedForward.forwardrr8r8rr9rEs rcs&eZdZfddZdddZZS)Wav2Vec2EncoderLayercshtt|j|j|jd|d|_t|j |_ tj |j|j d|_ t||_tj |j|j d|_dS)NFrrrrrr)rwrxrrnum_attention_headsattention_dropout attentionrrrrrrrr feed_forwardfinal_layer_normrrr8r9rx^s  zWav2Vec2EncoderLayer.__init__NFcCsf|}|j|||d\}}}||}||}||}|||}||}|f}|rb||f7}|SNr=r)rrrrrrr,r=rZ attn_residualrrLoutputsr8r8r9rms     zWav2Vec2EncoderLayer.forward)NFrr8r8rr9r]s rcs8eZdZfddZdejeejedddZZ S) #Wav2Vec2EncoderLayerStableLayerNormcstt|j|j|jd|d|_t|j |_ tj |j|j d|_ t||_tj |j|j d|_t|dddurt||_nd|_dS)NFrradapter_attn_dim)rwrxrrrrrrrrrrrrrrrgetattrWav2Vec2AttnAdapterLayer adapter_layerrrr8r9rxs   z,Wav2Vec2EncoderLayerStableLayerNorm.__init__NF)r,r=rcCsz|}||}|j|||d\}}}||}||}||||}|jdurb|||}|f}|rv||f7}|Sr)rrrrrrrr8r8r9rs     z+Wav2Vec2EncoderLayerStableLayerNorm.forward)NF) r0r1r2rxr4rrr\rrr8r8rr9rs rcsXeZdZfddZd ejeejeeedddZ e ejdfejd d d Z Z S) Wav2Vec2Encodercsft|_t|_tjjjd|_ t j |_ t fddtjD|_d|_dS)Nrcsg|] }tqSr8)rrJrr8r9rMrNz,Wav2Vec2Encoder.__init__..Frwrxrrpos_conv_embedrrrrrrrrrrZnum_hidden_layerslayersrrrrr9rxs   zWav2Vec2Encoder.__init__NFT)r,r=routput_hidden_states return_dictcCs.|rdnd}|rdnd}|durD|ddd|jd}d||<|||}||} || }||}||}tpt|} |j D]f} |r||f}t g} |j o| |j jk} | r| r| |||d}|d}| rd}|r||df}q|r||f}|s tdd |||fDSt|||d S) Nr8rIr"r&rrNNcss|]}|dur|VqdSrr8rKvr8r8r9 rNz*Wav2Vec2Encoder.forward..last_hidden_stater,r-) unsqueezerepeatr:_update_full_maskrrrr r r r4rUrr layerdropr7rrr,r=rr r Zall_hidden_statesZall_self_attentionsZexpand_attention_maskZposition_embeddingsZ synced_gpusrZdropout_probabilityZskip_the_layerZ layer_outputsr8r8r9rsJ          zWav2Vec2Encoder.forwardr= inputs_embedscCsv|durr|jjdkr&d|vr |nd}nL|jjdkr@t||j}n2|jjdkrft|tjrrt|dd}n t||j}|SNZflash_attention_2rZsdpaZflex_attentionF)r rrr rPrr4rr%r rr=rr8r8r9rs     z!Wav2Vec2Encoder._update_full_mask)NFFT) r0r1r2rxr4tensorrrr\rrrrr8r8rr9rs  ? rcsBeZdZfddZd ddZeejdfejdd d ZZ S) Wav2Vec2EncoderStableLayerNormcsft|_t|_tjjjd|_ t j |_ t fddtjD|_d|_dS)Nrcsg|] }tqSr8)rrJrr8r9rMrNz;Wav2Vec2EncoderStableLayerNorm.__init__..Frrrrr9rxs  z'Wav2Vec2EncoderStableLayerNorm.__init__NFTcCs.|rdnd}|rdnd}|durD|ddd|jd}d||<|||}||} || }||}tpxt|} |jD]f} |r||f}t g} |j o| |j j k} | r| r| |||d}|d}| rd}|r||df}q||}|r||f}|s tdd |||fDSt|||d S) Nr8rIr"r&rrr css|]}|dur|VqdSrr8rr8r8r9rPrNz9Wav2Vec2EncoderStableLayerNorm.forward..r)rrr:rrrr r r r4rUrrrrr7rrr8r8r9rsJ          z&Wav2Vec2EncoderStableLayerNorm.forwardrcCsv|durr|jjdkr&d|vr |nd}nL|jjdkr@t||j}n2|jjdkrft|tjrrt|dd}n t||j}|Srrrr8r8r9rXs     z0Wav2Vec2EncoderStableLayerNorm._update_full_mask)NFFT) r0r1r2rxrrr4rrrr8r8rr9r s  A rcs8eZdZdZfddZed ddZd ddZZS) Wav2Vec2GumbelVectorQuantizerz Vector quantization using gumbel softmax. See `[CATEGORICAL REPARAMETERIZATION WITH GUMBEL-SOFTMAX](https://huggingface.co/papers/1611.01144) for more information. cst|j|_|j|_|j|jdkrDtd|jd|jdt t d|j|j|j|j|_ t |jd|j|j|_d|_dS)Nrz`config.codevector_dim z5 must be divisible by `config.num_codevector_groups` z for concatenationr"rIr&)rwrxnum_codevector_groupsrnum_codevectors_per_groupnum_varscodevector_dimrRr Parameterr4r5 codevectorsrry weight_proj temperaturerrr8r9rxus  z&Wav2Vec2GumbelVectorQuantizer.__init__NcCs|durP|ddddf|j}t||t|}|jdd|}n |jdd}ttj|t |ddd }|S)NrrgHz>rI) flattenexpandr:r4whereZ zeros_likerXmeanexplog)ZprobsmaskZ mask_extendedZmarginal_probs perplexityr8r8r9_compute_perplexitys (z1Wav2Vec2GumbelVectorQuantizer._compute_perplexityc Cs|j\}}}||}||||jd}|jrtjj||j dd |}t j ||||jddd}| ||}nJ|jdd} ||jd| ddd}||||jd}| ||}|||d}|d|j} | |||j|jd} | d||d} | |fS)NrIT)tauhardrr"?r)r:r&rrrrrZgumbel_softmaxrr'type_asr4rr0argmaxZ new_zerosZscatter_rr%r"rX) rr,rlrfrGrZcodevector_probsZcodevector_soft_distr/Zcodevector_idxZcodevectors_per_groupr%r8r8r9rs0      z%Wav2Vec2GumbelVectorQuantizer.forward)N)N) r0r1r2r3rx staticmethodr0rrr8r8rr9ros   rcs$eZdZfddZddZZS)Wav2Vec2Adaptercsptjjkr8tjj|_tj|_n d|_|_t fddt j D|_ j |_ dS)Nc3s|]}tVqdSr)Wav2Vec2AdapterLayerrJrr8r9rrNz+Wav2Vec2Adapter.__init__..)rwrxoutput_hidden_sizerrrprojrproj_layer_normrrZnum_adapter_layersr rrrrr9rxs    zWav2Vec2Adapter.__init__cCsr|jdur(|jdur(||}||}|dd}|jD]&}tj}|jrX||jkr:||}q:|dd}|Sr)r:r;rr rSrTrr)rr,rZlayerdrop_probr8r8r9rs       zWav2Vec2Adapter.forwardrr8r8rr9r7s r7cs$eZdZfddZddZZS)r8cs0ttj|jd|j|j|jdd|_dS)Nr&r")rur)rwrxrr|r9Zadapter_kernel_sizeadapter_striderrrr8r9rxs zWav2Vec2AdapterLayer.__init__cCs||}tjj|dd}|S)Nr"r)rrrZglurr8r8r9rs zWav2Vec2AdapterLayer.forwardrr8r8rr9r8s r8cs,eZdZfddZejdddZZS)rcsZt|j|_|j|_t|j|_t |j|j|_ t |_ t |j|j|_ dS)z Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed up training throughput. N)rwrxr input_dimrZ hidden_dimrrnormrlinear_1ReLUact_fnlinear_2rrr8r9rxs  z!Wav2Vec2AttnAdapterLayer.__init__)r,cCs,||}||}||}||}|Sr)r?r@rBrCrr8r8r9rs     z Wav2Vec2AttnAdapterLayer.forward)r0r1r2rxr4r5rrr8r8rr9rs rc@seZdZUeed<dZdZdZdZdZ dZ ddZ de e jefeedd d Zdee jd d d ZddZddZdedddZdS)Wav2Vec2PreTrainedModelrwav2vec2rTc Cst|tr2|j|jd|j_d|j_nt|trp|jjj j ddd|jj j t j|jnlt|trt jj |jjddtd|jjd|jjdt j|jj dnt|trtd|jj}t jj|jj| |dt jj|jj | |dnt|t jrR|jj j d|jjd|j dur|j j nt|t jt jfr|j j |jj d nZt|t j!rt j"|j|j durt|j#|j|jd}t jj|j | |ddS) zInitialize the weightsTrr")r+stdrr&)abNr3)$rWav2Vec2ForPreTraining project_hidZreset_parameters project_qZ_is_hf_initializedrr&rdataZnormal_rvZzero_rinituniform_r%rrmathsqrtrtZ in_channelsZ constant_rrZ in_featuresrrZinitializer_rangerrZfill_r|Zkaiming_normal_r)rrkr8r8r9 _init_weightss@           z%Wav2Vec2PreTrainedModel._init_weightsN)rg add_adaptercCsn|dur|jjn|}dd}t|jj|jjD]\}}||||}q.|rjt|jjD]}||d|jj}qT|S)zH Computes the output length of the convolutional layers NcSstj|||dddS)Nfloor)Z rounding_moder")r4divrCrtrur8r8r9_conv_out_length?szRWav2Vec2PreTrainedModel._get_feat_extract_output_lengths.._conv_out_lengthr")rrSzipr}r~rZr<r=)rrgrSrWrtrurLr8r8r9 _get_feat_extract_output_lengths6sz8Wav2Vec2PreTrainedModel._get_feat_extract_output_lengths)feature_vector_lengthr=cCs|jdddddf}|j||d}|tj}|jd}tj||f|j|jd}d|tj |jd|jd|df<| dgd dg }|S)NrIrrSr)rPdevicer")r\) ZcumsumrYtor4longr:r[rPr\r^flipr\)rrZr=rSZnon_padded_lengthsZoutput_lengthsrfr8r8r9"_get_feature_vector_attention_maskMs  "z:Wav2Vec2PreTrainedModel._get_feature_vector_attention_maskcCs|jjdurt|jdi}|D]6\}}t|tr(|D]\}}||d||g<qBq(t|t r|j D]\}}||dd|g<qt|S)NzF has no adapter layers. Make sure to define `config.adapter_attn_dim`..lm_head) rrrRrZ named_modulesrrZnamed_parametersjoinWav2Vec2ForCTCrb)radapter_weightsrr param_namerr8r8r9 _get_adaptersas   z%Wav2Vec2PreTrainedModel._get_adapterscCs<|D]}t|tr||qt|tr8||jdS)zc (Re-)initialize attention adapter layers and lm head for adapter-only fine-tuning N)modulesrrrRrdrb)rrr8r8r9init_adapter_layersqs     z+Wav2Vec2PreTrainedModel.init_adapter_layers target_langc s|jjdurtd|d||jkr@|s@td|ddS|dd}|dd}|d d}|d d}|d d}|d d} |d d} |dd} |dtrdnd} | durt dt | durtd| } |jj } d}| durvt |}z&t| |||||| | |d }t|}WnPty@| r<Yn6tyt| rptd| d| d|dYn0|durt |}z4t| |||||| | |d }ttj|ddd}WnXtyYnDtyYn0tytd| d| d|dYn0|t|t}tt|}t|dkr|td|dd|dn*t|dkrtd|dd|d|djd}||jjkrtj|jj||j |j!d|_"||j_fd d!|#D}|j$|dd"||_dS)#a Load a language adapter model from a pre-trained adapter model. Parameters: target_lang (`str`): Has to be a language id of an existing adapter weight. Adapter weights are stored in the format adapter..safetensors or adapter..bin force_load (`bool`, defaults to `True`): Whether the weights shall be loaded even if `target_lang` matches `self.target_lang`. cache_dir (`Union[str, os.PathLike]`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. local_files_only(`bool`, *optional*, defaults to `False`): Whether or not to only look at local files (i.e., do not try to download the model). token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `hf auth login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`. mirror (`str`, *optional*): Mirror source to accelerate downloads in China. If you are from China and have an accessibility problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety. Please refer to the mirror site for more information. Activate the special ["offline-mode"](https://huggingface.co/transformers/installation.html#offline-mode) to use this method in a firewalled environment. Examples: ```python >>> from transformers import Wav2Vec2ForCTC, AutoProcessor >>> ckpt = "facebook/mms-1b-all" >>> processor = AutoProcessor.from_pretrained(ckpt) >>> model = Wav2Vec2ForCTC.from_pretrained(ckpt, target_lang="eng") >>> # set specific language >>> processor.tokenizer.set_target_lang("spa") >>> model.load_adapter("spa") ``` NzCannot load_adapter for - if `config.adapter_attn_dim` is not defined.z#Adapter weights are already set to ra cache_dirforce_downloadFresume_downloadproxieslocal_files_onlytokenuse_auth_tokenrevisionuse_safetensorszrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.zV`token` and `use_auth_token` are both specified. Please set only the argument `token`.)filenamernrorprqrrrtrmzCan't load the model for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z=' is the correct path to a directory containing a file named cpuT)Z map_locationZ weights_onlyrzThe adapter weights z has unexpected keys: z, z has missing keys: zlm_head.weightr\rPcs i|]\}}|||qSr8)r])rKrQrrer8r9 :rNz8Wav2Vec2PreTrainedModel.load_adapter..)strict)%rrrRrkloggerwarningpoprrrrZ _name_or_pathWAV2VEC2_ADAPTER_SAFE_FILEformatrsafe_load_fileOSError ExceptionWAV2VEC2_ADAPTER_PT_FILErr4loadrgsetkeysr_rcr: vocab_sizerrr9r\rPrbitemsZload_state_dict)rrk force_loadrrmrnrorprqrrrsrtruZmodel_path_or_idZ state_dictfilepathZ weight_pathZunexpected_keysZ missing_keysZtarget_vocab_sizer8ryr9 load_adapter~s?                  z$Wav2Vec2PreTrainedModel.load_adapter)N)N)T)r0r1r2r#r6Zbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingZ_supports_flash_attnZ_supports_sdpaZ_supports_flex_attnrRrr4 LongTensorrArr\rYr`rgrirrr8r8r8r9rDs& (   rDc seZdZedfdd ZddZddZdeje eje ej d d d Z e de ej e ej e eje ee ee eeeefd d dZZS) Wav2Vec2Modelrcst|||_t||_t||_|jdks:|jdkrRt t |j |_|jrdt||_n t||_|jr|t|nd|_|dS)Nr)rwrxrrfeature_extractorrfeature_projectionmask_time_probmask_feature_probrr$r4rrrNmasked_spec_embedZdo_stable_layer_normrencoderrrSr7adapter post_initrrr8r9rxCs     zWav2Vec2Model.__init__cCstdt|dSz Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training. The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. Please use the equivalent `freeze_feature_encoder` method instead.Nrrrfreeze_feature_encoderrr8r8r9freeze_feature_extractorWs z&Wav2Vec2Model.freeze_feature_extractorcCs|jdS Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)rrrr8r8r9rcsz$Wav2Vec2Model.freeze_feature_encoderN)r,rlr=cCst|jdds|S|\}}}|dur<|j|j||<nZ|jjdkr|jrt||f|jj|jj ||jj d}t j ||j t jd}|j|j||<|jjdkr|jrt||f|jj|jj|jjd}t j ||j t jd}|dddfd|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://huggingface.co/papers/1904.08779). Zapply_spec_augmentTNr)r;r<r=r>rx)r;r<r>rI)rrrmrr]rPrrriZmask_time_lengthZmask_time_min_masksr4rr\r\rZmask_feature_lengthZmask_feature_min_masksr))rr,rlr=rfrGrZmask_feature_indicesr8r8r9_mask_hidden_statesjs4 z!Wav2Vec2Model._mask_hidden_states)rr=rlrr r r?c Cs|dur |n|jj}|dur |n|jj}|dur4|n|jj}||}|dd}|durp|j|jd|dd}||\}}|j |||d}|j |||||d} | d}|j dur| |}|s||f| ddSt ||| j | jd S) a/ mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. Nr"r&Fr[)rlr=r=rr r r)rextract_featuresr,r-)rrr use_return_dictrrr`r:rrrrrr,r-) rrr=rlrr r rr,Zencoder_outputsr8r8r9rs@     zWav2Vec2Model.forward)NN)NNNNN)r0r1r2r#rxrrr4r5rrrrrr\rr7rrrr8r8rr9rAs2   . rz? Wav2Vec2 Model with a quantizer and `VQ` head on top. c seZdZedfdd ZedddZddZd d Ze de j e j e j ed d dZ e dee jee jee jee jeeeeeeeeefdddZZS)rIrcs^t|t||_t|j|_t||_ t |j |j |_ t |j|j |_|dSr)rwrxrrErrZfeat_quantizer_dropoutdropout_featuresr quantizerrrZproj_codevector_dimrJr#rKrrrr8r9rxs   zWav2Vec2ForPreTraining.__init__)r'cCs ||j_dS)zb Set the Gumbel softmax temperature to a given value. Only necessary for training N)rr')rr'r8r8r9set_gumbel_temperaturesz-Wav2Vec2ForPreTraining.set_gumbel_temperaturecCstdt|dSrrrr8r8r9rs z/Wav2Vec2ForPreTraining.freeze_feature_extractorcCs|jjdSrrErrrr8r8r9rsz-Wav2Vec2ForPreTraining.freeze_feature_encoder皙?)target_featuresnegative_featurespredicted_featuresr'cCs<tj||gdd}tj||dd|}||}|S)z Compute logits for contrastive loss based using cosine similarity as the distance measure between `[positive_feature, negative_features]` and `[predicted_features]`. Additionally, temperature can be applied. rrrI)r4catZcosine_similarityrr4)rrrr'logitsr8r8r9compute_contrastive_logitss z1Wav2Vec2ForPreTraining.compute_contrastive_logitsN)rr=rlrorr r r?c Cs"|dur |n|jj}|dur(|tj}|j||||||d}||d} ||d} |durx|j| j d|dd}|j | |d\} } | |j j j } | | } d} }}|dur| j \}}}| d||d}|||d|d ddd }|| dddf|| |jj}| |kd}|rHtd |dd|<|dd d|d}d|d dd}tjj||d d}|jj|jj}|| ||}||jj |} |s| dur| | | | f|d dS| | | f|d dSt!| | | | |j"|j#||dS)a mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. sampled_negative_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length, num_negatives)`, *optional*): Indices indicating which quantized target vectors are used as negative sampled vectors in contrastive loss. Required input for pre-training. Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, Wav2Vec2ForPreTraining >>> from transformers.models.wav2vec2.modeling_wav2vec2 import _compute_mask_indices, _sample_negative_indices >>> from datasets import load_dataset >>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base") >>> model = Wav2Vec2ForPreTraining.from_pretrained("facebook/wav2vec2-base") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> input_values = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt").input_values # Batch size 1 >>> # compute masked indices >>> batch_size, raw_sequence_length = input_values.shape >>> sequence_length = model._get_feat_extract_output_lengths(raw_sequence_length).item() >>> mask_time_indices = _compute_mask_indices( ... shape=(batch_size, sequence_length), mask_prob=0.2, mask_length=2 ... ) >>> sampled_negative_indices = _sample_negative_indices( ... features_shape=(batch_size, sequence_length), ... num_negatives=model.config.num_negatives, ... mask_time_indices=mask_time_indices, ... ) >>> mask_time_indices = torch.tensor(data=mask_time_indices, device=input_values.device, dtype=torch.long) >>> sampled_negative_indices = torch.tensor( ... data=sampled_negative_indices, device=input_values.device, dtype=torch.long ... ) >>> with torch.no_grad(): ... outputs = model(input_values, mask_time_indices=mask_time_indices) >>> # compute cosine similarity between predicted (=projected_states) and target (=projected_quantized_states) >>> cosine_sim = torch.cosine_similarity(outputs.projected_states, outputs.projected_quantized_states, dim=-1) >>> # show that cosine similarity is much higher than random >>> cosine_sim[mask_time_indices.to(torch.bool)].mean() > 0.5 tensor(True) >>> # for contrastive loss training model should be put into train mode >>> model = model.train() >>> loss = model( ... input_values, mask_time_indices=mask_time_indices, sampled_negative_indices=sampled_negative_indices ... ).loss ```N)r=rr rlr rr"Fr[)rlrIr&rz-infirX) reduction)r(r)r*r+r,r-r.r/)$rrr]r4r\rErJrr`r:rrKrrPrr^ZpermuterZcontrastive_logits_temperatureallanyrrrermr(rrZ cross_entropyr!r rXZdiversity_loss_weightr'r,r-)rrr=rlrorr r rZtransformer_featuresrZquantized_featuresr+r(r.r/rfrGrZnegative_quantized_featuresrZ neg_is_postargetZnum_codevectorsr8r8r9rs|B             zWav2Vec2ForPreTraining.forward)r)NNNNNN)r0r1r2r#rxrArrrr6r4r5rrrrZ BoolTensorr\rr7r'rrr8r8rr9rIs:   rIc s^eZdZfddZedejeejee ee ee eej e e e fdddZZS)Wav2Vec2ForMaskedLMcsNt|tdtt||_t|j |_ t |j |j |_|dS)NzSThe class `Wav2Vec2ForMaskedLM` is deprecated. Please use `Wav2Vec2ForCTC` instead.)rwrxrrrrrErr final_dropoutrrrrrbrrrr8r9rxs  zWav2Vec2ForMaskedLM.__init__Nrr=rr r labelsr?c Csn|dur |n|jj}|j||||d}|d}||}||} |s\| f|dd} | St| |j|jdS)N)rr r rr&)rr,r-)rrrErrbrr,r-) rrr=rr r rrr,routputr8r8r9rs   zWav2Vec2ForMaskedLM.forward)NNNNN)r0r1r2rxrr4r5rrr\rrr7rrrr8r8rr9rs   rzp Wav2Vec2 Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). c seZdZdeedfdd ZddZddZd d Zd d Z e dee j ee j ee ee ee ee j eeefd ddZZS)rdNrjcs~t|t||_t|j|_||_|j durFt d|j dt |dr\|j r\|jn|j}t||j |_|dS)a2 target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`Wav2Vec2ForCTC`] with adapters. Uses 'eng' by default. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `Wav2Vec2ForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.rS)rwrxrrErrrrrkrrRrrrSr9rrrbr)rrrkr9rr8r9rxs    zWav2Vec2ForCTC.__init__cCsr|j}|dur2t|jdddur2td|dn<|durXt|jdddurXtdn|durn|j|dddS)a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. NrzCannot pass `target_lang`: rlz)By default `target_lang` is set to 'eng'.T)r)rkrrrRr|infor)rrkr8r8r9 tie_weightss  zWav2Vec2ForCTC.tie_weightscCstdt|dSrrNrrr8r8r9rs z'Wav2Vec2ForCTC.freeze_feature_extractorcCs|jjdSrrrr8r8r9r&sz%Wav2Vec2ForCTC.freeze_feature_encodercCs|jD] }d|_q dSz Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated. FNrErrrr8r8r9freeze_base_model-sz Wav2Vec2ForCTC.freeze_base_modelrc Cs|dur |n|jj}|dur>||jjkr>td|jj|j|||||d}|d}||}||} d} |dur@|dur|ntj |tj d}| | d tj } |dk} | d} || }tjj| dtjddd}tjjjd d 6tjj||| | |jj|jj|jjd } Wdn1s60Y|sp| f|td}| durl| f|S|St| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nz$Label values must be <= vocab_size: rrrOrI)rrPr"F)Zenabled)blankrZ zero_infinityr(rr,r-)rrrBrrRrErrbr4Z ones_liker^rYrXr]Z masked_selectrrZ log_softmaxZfloat32rbackendsZcudnnflagsZctc_lossZ pad_token_idZctc_loss_reductionZctc_zero_infinity_HIDDEN_STATES_START_POSITIONrr,r-)rrr=rr r rrr,rr(rgZ labels_maskZtarget_lengthsZflattened_targetsZ log_probsrr8r8r9r5sL     &  zWav2Vec2ForCTC.forward)N)NNNNN)r0r1r2rrrxrrrrrr4rr\rr7rrrr8r8rr9rds(  rdz Wav2Vec2 Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. c szeZdZfddZddZddZddZed ee j ee j ee ee ee ee j e e efd d d ZZS)!Wav2Vec2ForSequenceClassificationcst|t|dr$|jr$tdt||_|jd}|jrTt t |||_ t |j|j|_t |j|j|_|dS)NrSz_Sequence classification does not support the use of Wav2Vec2 adapters (config.add_adapter=True)r")rwrxrrSrRrrEr use_weighted_layer_sumrr$r4r` layer_weightsrrZclassifier_proj_size projector num_labels classifierrrr num_layersrr8r9rxs   z*Wav2Vec2ForSequenceClassification.__init__cCstdt|dSrrrr8r8r9rs z:Wav2Vec2ForSequenceClassification.freeze_feature_extractorcCs|jjdSrrrr8r8r9rsz8Wav2Vec2ForSequenceClassification.freeze_feature_encodercCs|jD] }d|_q dSrrrr8r8r9rsz3Wav2Vec2ForSequenceClassification.freeze_base_modelNrcCs|dur |n|jj}|jjr dn|}|j|||||d}|jjr|t}tj|dd}tjj |j dd} || dddj dd}n|d}| |}|dur|jdd} nV||jd|} | ddd|jd} d || <|j dd| j dd dd} || } d}|dur 1` a classification loss is computed (Cross-Entropy). NTrr"rrIrr&rr)rrrrErr4stackrrrrrrXrr+r`r:rrrrrrr,r-)rrr=rr r rrr, norm_weightsZ pooled_outputZ padding_maskZexpand_padding_maskrr(loss_fctrr8r8r9rsH     z)Wav2Vec2ForSequenceClassification.forward)NNNNN)r0r1r2rxrrrrrr4rr\rr7rrrr8r8rr9r}s&   rc szeZdZfddZddZddZddZed ee j ee j ee j ee ee ee e e efd d d ZZS)#Wav2Vec2ForAudioFrameClassificationcszt|t|dr$|jr$tdt||_|jd}|jrTt t |||_ t |j|j|_|j|_|dS)NrSzbAudio frame classification does not support the use of Wav2Vec2 adapters (config.add_adapter=True)r")rwrxrrSrRrrEr rrr$r4r`rrrrr init_weightsrrr8r9rxs   z,Wav2Vec2ForAudioFrameClassification.__init__cCstdt|dSrrrr8r8r9rs z@皙?csBt||_||_||_tjt||dd|_ t |_ dS)NT)r) rwrxscalemarginrrr$r4Zrandnrrr()rr>rrrrr8r9rxas  zAMSoftmaxLoss.__init__c Csx|}tjj|jdd}tjj|dd}t||}||j}tj||j }|j t | ||}| ||}|S)Nrrr")r(rr normalizerr4mmrZone_hotrrr*r\r() rr,rrZ cos_thetapsiZonehotrr(r8r8r9ris   zAMSoftmaxLoss.forward)rrrr8r8rr9r`srcs2eZdZdfdd ZejejdddZZS) TDNNLayerrcsvt|dkr |j|dn|j||_|j||_|j||_|j||_t |j|j|j|_ t |_ dS)Nrr")rwrxtdnn_dimrzr{ tdnn_kernelrtZ tdnn_dilationdilationrrkernelrArrrr8r9rxxs "   zTDNNLayer.__init__)r,r?cCstrddlm}tr.t|j|r.td|dd}|jj |j |j |j dd}t jj|||jj|jd}|dd}||}|S)Nr) LoraLayerzDetected LoRA on TDNNLayer. LoRA weights won't be applied due to optimization. You should exclude TDNNLayer from LoRA's target modules.r"r&)r)rZpeft.tuners.lorarrrrrrrrr{rtrzrrZconv1drvrr)rr,rrr8r8r9rs      zTDNNLayer.forward)r)r0r1r2rxr4rrrr8r8rr9rws rzl Wav2Vec2 Model with an XVector feature extraction head on top for tasks like Speaker Verification. c seZdZfddZddZddZddZeej e fd d d Z e de eje eje ee ee ee ejeeefd ddZZS)Wav2Vec2ForXVectorcstt|_jd}jr.rIr&)rwrxrrEr rrr$r4r`rrrrrrZr_rtdnnZxvector_output_dimrrrr objectiver)rrrZ tdnn_layersrrr9rxs    zWav2Vec2ForXVector.__init__cCstdt|dSrrrr8r8r9rs z+Wav2Vec2ForXVector.freeze_feature_extractorcCs|jjdSrrrr8r8r9rsz)Wav2Vec2ForXVector.freeze_feature_encodercCs|jD] }d|_q dSrrrr8r8r9rsz$Wav2Vec2ForXVector.freeze_base_model)rgcCs&dd}|jjD]}|||d}q|S)z? Computes the output length of the TDNN layers cSs|||dS)Nr"r8rVr8r8r9rWszEWav2Vec2ForXVector._get_tdnn_output_lengths.._conv_out_lengthr")rr)rrgrWrtr8r8r9_get_tdnn_output_lengthss z+Wav2Vec2ForXVector._get_tdnn_output_lengthsNrcCs|dur |n|jj}|jjr dn|}|j|||||d}|jjr|t}tj|dd}tjj |j dd} || dddj dd}n|d}| |}|jD] } | |}q|dur|jdd} |jdd} n||j dd} || }g} g} t|D]D\}}| ||d|fjdd| ||d|fjddqt| } t| } tj| | gdd}||}||}d}|dur|||}|s||f|td}|dur|f|S|St||||j|jdS) rNTrr"rrIr)r(rZ embeddingsr,r-)rrrrErr4rrrrrrrXrrr+rFrYr enumeraterbrrrrrr,r-)rrr=rr r rrr,rZ tdnn_layerZ mean_featuresZ std_featuresZfeat_extract_output_lengthsZtdnn_output_lengthsrlengthZstatistic_poolingZoutput_embeddingsrr(rr8r8r9rs\            zWav2Vec2ForXVector.forward)NNNNN)r0r1r2rxrrrrr4rrArrrrr\r7rrrr8r8rr9rs(   r)rrdrrIrrrrD)Nr)N)NrN)hr3rOr dataclassesrtypingrrrnumpyrSr4Ztorch.utils.checkpointrZtorch.nnrZ activationsr Zintegrations.deepspeedr Zintegrations.fsdpr Zmodeling_attn_mask_utilsr r Zmodeling_flash_attention_utilsrZmodeling_layersrZmodeling_outputsrrrrrrrZmodeling_utilsrrZprocessing_utilsrrrrrrrrr r!Zconfiguration_wav2vec2r#rrZsafetensors.torchr$rZintegrations.flex_attentionr%Z get_loggerr0r|rr'r7rArrZndarrayrirqrrrrModulerrrrrrrrrrrrrrr7r8rrDrrIrrdrrrrr__all__r8r8r8r9s        $  (    !  x $- (  X$.^bL=]-si