a hcB@sdZddlZddlmZmZddlZddlZddlZddlm Z ddl m Z m Z m Z ddlmZddlmZmZmZdd lmZdd lmZdd lmZdd lmZmZdd lmZddlm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)m*Z*ddl+m,Z,ddl-m.Z.m/Z/e*0e1Z2dZ3ej4e5e5dddZ6dwej4e5eej4dddZ7dxe8e5e5fe9e5eej:e5ej;dddZGd!d"d"eZ?Gd#d$d$e j@ZAGd%d&d&e j@ZBGd'd(d(e j@ZCGd)d*d*ej j@ZDGd+d,d,e j@ZEGd-d.d.e j@ZFGd/d0d0e j@ZGGd1d2d2e j@ZHGd3d4d4e j@ZIGd5d6d6e j@ZJGd7d8d8e j@ZKGd9d:d:e j@e&ZLGd;d<dd>e j@e&ZNGd?d@d@e j@ZOGdAdBdBe j@ZPGdCdDdDeZQGdEdFdFeZRe)GdGdHdHe'ZSGdIdJdJeSZTGdKdLdLeSZUGdMdNdNeSZVGdOdPdPeSZWGdQdRdReSZXGdSdTdTeSZYGdUdVdVeSZZGdWdXdXeSZ[GdYdZdZe j@Z\Gd[d\d\e j@Z]e)d]d^Gd_d`d`eSZ^e)dad^GdbdcdceSeZ_dyeSej`eej`eej:e9e9e9ee j@eaeaeej`e8ej`ej`ffdh didjZbe)dkd^GdldmdmeSZce)dnd^GdodpdpeSZdGdqdrdre j@Zee)dsd^Gdtdudue'ZfgdvZgdS)zzPyTorch SpeechT5 model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossL1Loss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)is_deepspeed_zero3_enabled)is_fsdp_managed_module)_prepare_4d_attention_mask!_prepare_4d_causal_attention_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutputSeq2SeqSpectrogramOutput)EmbeddingAccessMixinPreTrainedModel)auto_docstringlogging)deprecate_kwarg)SpeechT5ConfigSpeechT5HifiGanConfig) input_ids pad_token_iddecoder_start_token_idcCsh||j}|ddddf|ddddf<||dddf<|durTtd||dk||S)z1 Shift input ids one token to the right. Nrrz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)r r!r"Zshifted_input_idsr)j/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/speecht5/modeling_speecht5.pyshift_tokens_right7s (r+) input_valuesreduction_factorattention_maskcCs|dkrD|dd|dd|f}|durD|dd|dd|f}||j}|ddddf|ddddf<||dkd||fS)zw Shift input spectrograms one timestep to the right. Also applies the reduction factor to the sequence length. rNr#Y)r$r%r&r()r,r-r.Zshifted_input_valuesr)r)r*shift_spectrograms_rightGs (r1)r% mask_prob mask_lengthr. min_masksreturncs|\}dkrtdkr6tdddtjdfdd}|durt|d nfd d t|D}tj |ft d }g} |} | d kr|S|D]v} || } tjj t | d| dd} t | d krd}n| d }t| tj| | tjd |g} | | qt| } t| dddddf|| f} | || } t ddddf}t||| f|| }| |} | dkr҈d| | dk<t|| dd |S)an Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `csXt|}t|}|kr2}|d|krTt|dd}|S)z;Given input length, compute how many spans should be maskedrr)intmax) input_lengthnum_masked_spanepsilonr3r2r4sequence_lengthr)r*compute_num_masked_spans  z6_compute_mask_indices..compute_num_masked_spanNr#csg|]}qSr)r).0_)r=r)r* z)_compute_mask_indices..dtyperF)replace)r'nprandomranditemdetachsumtolistrangezerosboolchoicearangelenZ concatenateonesZint32appendarrayZ broadcast_toreshaper8Zput_along_axis)r%r2r3r.r4 batch_sizer> input_lengthsZ spec_aug_maskZspec_aug_mask_idxsZmax_num_masked_spanr9r:Zspec_aug_mask_idxZdummy_mask_idxoffsetsr)r;r*_compute_mask_indices]s\    r[cs&eZdZdfdd ZddZZS)SpeechT5NoLayerNormConvLayerrcsjt|dkr |j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ dS)Nrr kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvr feat_extract_activation activationselfconfiglayer_id __class__r)r*rbs  z%SpeechT5NoLayerNormConvLayer.__init__cCs||}||}|SN)rjrlrn hidden_statesr)r)r*forwards  z$SpeechT5NoLayerNormConvLayer.forward)r__name__ __module__ __qualname__rbrv __classcell__r)r)rqr*r\sr\cs&eZdZdfdd ZddZZS)SpeechT5LayerNormConvLayerrcs|t|dkr |j|dnd|_|j||_tj|j|j|j||j||j d|_ tj |jdd|_ t |j|_dS)Nrrr]T)Zelementwise_affine)rarbrcrdrerrfrgrhrirj LayerNorm layer_normr rkrlrmrqr)r*rbs  z#SpeechT5LayerNormConvLayer.__init__cCs:||}|dd}||}|dd}||}|S)Nr#)rj transposer~rlrtr)r)r*rvs      z"SpeechT5LayerNormConvLayer.forward)rrwr)r)rqr*r|sr|cs&eZdZdfdd ZddZZS)SpeechT5GroupNormConvLayerrcst|dkr |j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ tj|j|jdd|_dS)Nrrr]T)Z num_groupsZ num_channelsZaffine)rarbrcrdrerrfrgrhrirjr rkrl GroupNormr~rmrqr)r*rbs   z#SpeechT5GroupNormConvLayer.__init__cCs"||}||}||}|Srs)rjr~rlrtr)r)r*rvs   z"SpeechT5GroupNormConvLayer.forward)rrwr)r)rqr*rsrcseZdZdZdeeeedfdd ZdeeeedddZedeeeedd d Z e de j ed d dZ de j eeedddZZS)%SpeechT5SinusoidalPositionalEmbeddingzDThis module produces sinusoidal positional embeddings of any length.N) num_positions embedding_dim padding_idxcs4td|_||_||_|||j||dSN)rarboffsetrr make_weights)rnrrrrqr)r*rb#s  z.SpeechT5SinusoidalPositionalEmbedding.__init__)num_embeddingsrrcCsB||||}t|dr.|j|jj|jjd}|jd|dddS)NweightsrEdeviceF persistent) get_embeddinghasattrtorrErregister_buffer)rnrrrZ emb_weightsr)r)r*r*s z2SpeechT5SinusoidalPositionalEmbedding.make_weightscCs|d}td|d}ttj|tjd| }tj|tjdd|d}tjt |t |gdd |d}|ddkrtj|t |dgdd}|durd||ddf<| tS) z Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.5 of "Attention Is All You Need". ri'rrDrdimr#N)mathlogtorchexprRint64float unsqueezecatsincosviewrOrZget_default_dtype)rrrZhalf_dimembr)r)r*r2s $& z3SpeechT5SinusoidalPositionalEmbedding.get_embeddingr)r past_key_values_lengthcCs||\}}|||j||j}|jd|}||jdkrZ|||j|j|j|j d| d ||d S)Nrrr#) size"create_position_ids_from_input_idsrrrrrrrZ index_selectrrK)rnr rbszseq_lenZ position_idsZmax_posr)r)r*rvDs z-SpeechT5SinusoidalPositionalEmbedding.forward)r rrcCs6||}tj|dd|||}||S)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rr)ner7rcumsumZtype_aslong)rnr rrmaskZincremental_indicesr)r)r*rSs zHSpeechT5SinusoidalPositionalEmbedding.create_position_ids_from_input_ids)N)N)N)r)r)rxryrz__doc__r7rrbr staticmethodrrno_gradTensorrvrr{r)r)rqr*r s rcs$eZdZfddZddZZS)SpeechT5PositionalConvEmbeddingcs$ttj|j|j|j|jd|jd|_tjj }t tjj drNtjj j }t rddl }|jj|jjdd"||jddd|_Wdn1s0Yt |jdr|jj jj}|jj jj}n|jj}|jj}|j|||j||n||jddd|_t|j|_t|j|_dS) Nr)r^paddinggroups weight_normr)Z modifier_rankweight)namerparametrizations)rarbrrf hidden_sizenum_conv_pos_embeddingsZnum_conv_pos_embedding_groupsrjutilsrrrr deepspeedzeroZGatheredParametersrZ original0Z original1weight_gweight_vZregister_external_parameterSpeechT5SamePadLayerrr rkrl)rnrorrrrrqr)r*rbfs2  0   z(SpeechT5PositionalConvEmbedding.__init__cCs:|dd}||}||}||}|dd}|SNrr)rrjrrlrtr)r)r*rvs      z'SpeechT5PositionalConvEmbedding.forwardrwr)r)rqr*res !rcs*eZdZdZdfdd ZddZZS) SpeechT5ScaledPositionalEncodingu[ Scaled positional encoding, see §3.2 in https://huggingface.co/papers/1809.08895 cst||}td|d}ttjd|dtjdtd| }t |||dddddf<t |||dddddf<|d}t |j d|ddtj|d |_||_ttd |_dS) NrrrrDg@peFrp?)rrOrRrrrrrrrrrarbrrDropoutdropoutr Parametertensoralpha)rnrrmax_lenrpositionZdiv_termrqr)r*rbs .$$  z)SpeechT5ScaledPositionalEncoding.__init__cCs4||j|jddd|df}||}|S)Nr)rrrr)rnrr)r)r*rvs& z(SpeechT5ScaledPositionalEncoding.forward)r)rxryrzrrbrvr{r)r)rqr*rs rcs&eZdZdfdd ZddZZS)"SpeechT5RelativePositionalEncodingcs.t||_||_tjd|||_dSr)rarbr max_lengthrr Embeddingpe_k)rnrrrqr)r*rbs z+SpeechT5RelativePositionalEncoding.__init__cCs|jd}td|j|jtjd}|dddf|dddf}|j |||j k<|jd|||jk<||j}||S)NrrrrE)r%rrRrrrrr)rnrurZpos_seqr)r)r*rvs   z*SpeechT5RelativePositionalEncoding.forward)rrwr)r)rqr*rsrcs$eZdZfddZddZZS)rcs$t|ddkrdnd|_dS)Nrrr)rarbnum_pad_remove)rnrrqr)r*rbs zSpeechT5SamePadLayer.__init__cCs,|jdkr(|ddddd|j f}|S)Nr)rrtr)r)r*rvs zSpeechT5SamePadLayer.forwardrwr)r)rqr*rs rcs0eZdZdZfddZddZddZZS)SpeechT5FeatureEncoderz.Construct the features from raw audio waveformcstjdkr@tddgfddtjdD}n6jdkrdfddtjD}ntd jd t||_ d |_ d |_ dS) Ngrouprrpcsg|]}t|ddqS)rr)r\r@iror)r*rBsz3SpeechT5FeatureEncoder.__init__..rlayercsg|]}t|dqS)r)r|rrr)r*rBsz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT) rarbZfeat_extract_normrrNZnum_feat_extract_layersr'r ModuleList conv_layersgradient_checkpointing_requires_grad)rnrorrqrr*rbs       zSpeechT5FeatureEncoder.__init__cCs|D] }d|_qd|_dS)NF) parameters requires_gradr)rnparamr)r)r*_freeze_parameterss z)SpeechT5FeatureEncoder._freeze_parameterscCs:|dddf}|jr"|jr"d|_|jD] }||}q(|SNT)rtrainingrr)rnr,ruZ conv_layerr)r)r*rvs    zSpeechT5FeatureEncoder.forward)rxryrzrrbrrvr{r)r)rqr*rs rcs$eZdZfddZddZZS)SpeechT5FeatureProjectioncsJttj|jd|jd|_t|jd|j|_ t |j |_ dS)Nr#eps) rarbrr}rclayer_norm_epsr~Linearr projectionrZfeat_proj_dropoutrrnrorqr)r*rbs z"SpeechT5FeatureProjection.__init__cCs&||}||}||}||fSrs)r~rr)rnruZnorm_hidden_statesr)r)r*rvs   z!SpeechT5FeatureProjection.forwardrwr)r)rqr*rs rcseZdZfddZddZdejeejeej dddZ e ejd d d Z e eje fd d dZdej eej eejdddZZS)SpeechT5SpeechEncoderPrenetcs|t||_t||_t||_|jdks8|jdkrPt t |j |_t||_t|j|jd|j |j|_dS)Nr0r)rarbrorfeature_encoderrfeature_projectionmask_time_probmask_feature_probrrrrruniform_masked_spec_embedrpos_conv_embedrmax_speech_positionsr!pos_sinusoidal_embedrrqr)r*rbs    z$SpeechT5SpeechEncoderPrenet.__init__cCs|jdSrs)rrrnr)r)r*freeze_feature_encodersz2SpeechT5SpeechEncoderPrenet.freeze_feature_encoderN)r,r.mask_time_indicesc Cs||}|dd}|dur0||jd|}||\}}|j|||d}||}||}|durx|d}nt j |jddt j|j d}| |}||}||fS)Nrr)rr.r) rr"_get_feature_vector_attention_maskr%r_mask_hidden_statesrrrrrOrr) rnr,r.rZextract_featuresruZpositional_conv_embedding padding_maskZ positional_sinusoidal_embeddingsr)r)r*rvs&    z#SpeechT5SpeechEncoderPrenet.forward)feature_vector_lengthr.cCs|jdddddf}||tj}|jd}tj||f|j|jd}d|tj |jd|jd|df<| dgd dg }|S)Nr#rrrrr) r _get_feat_extract_output_lengthsrrrr%rOrErrRfliprP)rnrr.Znon_padded_lengthsoutput_lengthsrXr)r)r*r:s "z>SpeechT5SpeechEncoderPrenet._get_feature_vector_attention_mask)rYcCs4dd}t|jj|jjD]\}}||||}q|S)zH Computes the output length of the convolutional layers cSstj|||dddS)Nfloor)Z rounding_moder)rdiv)r9r^r_r)r)r*_conv_out_lengthOszVSpeechT5SpeechEncoderPrenet._get_feat_extract_output_lengths.._conv_out_length)ziprorgrh)rnrYrr^r_r)r)r*rJszeZdZfddZddZd ejeejdddZZ S) SpeechT5SpeechDecoderPrenetcsrt|_tfddtjD|_tj j |_ t j j j|_tjj j |_dS)Ncs*g|]"}t|dkrjnjjqS)r)rr num_mel_binsspeech_decoder_prenet_unitsrrr)r*rBs z8SpeechT5SpeechDecoderPrenet.__init__..)rarbrorrrNZspeech_decoder_prenet_layerslayersrr r final_layerrpositional_dropoutrencode_positionsZspeaker_embedding_dimspeaker_embeds_layerrrqrr*rbs   z$SpeechT5SpeechDecoderPrenet.__init__cCsJtj|d|d}|d|ddd}t|dk|ddd|S)Nrrr)rZ bernoullirrepeatrwhere)rn inputs_embedsrrZ all_masksr)r)r*_consistent_dropoutsz/SpeechT5SpeechDecoderPrenet._consistent_dropoutN)r,speaker_embeddingscCs|}|jD]$}tj||}|||jj}q ||}||}|durtj |}| d d| dd}t j||gdd}tj||}|S)Nrr#r)r r functionalZrelurroZspeech_decoder_prenet_dropoutr r normalizerrrrrr)rnr,rrrr)r)r*rvs    z#SpeechT5SpeechDecoderPrenet.forward)N) rxryrzrbrrrrrvr{r)r)rqr*r s r cs&eZdZdfdd ZddZZS)SpeechT5BatchNormConvLayerrcst|dkr|j}n|j}||jdkr6|j}n|j}tj|||jd|jdddd|_t ||_ ||jdkrt |_ nd|_ t |j|_dS)NrrrF)r^r_rr`)rarbr Zspeech_decoder_postnet_unitsspeech_decoder_postnet_layersrrfZspeech_decoder_postnet_kernelrj BatchNorm1d batch_normZTanhrlrZspeech_decoder_postnet_dropoutr)rnrorprdrerqr)r*rbs(    z#SpeechT5BatchNormConvLayer.__init__cCs6||}||}|jdur(||}||}|Srs)rjrrlrrtr)r)r*rvs      z"SpeechT5BatchNormConvLayer.forward)rrwr)r)rqr*rsrcs<eZdZfddZejdddZejdddZZS)SpeechT5SpeechDecoderPostnetcs^t|_tjjj|_tjj|_ t fddt j D|_ dS)Ncsg|]}t|qSr))rrrr)r*rBrCz9SpeechT5SpeechDecoderPostnet.__init__..)rarbrorrrr r-feat_outprob_outrrNrr rrqrr*rbs z%SpeechT5SpeechDecoderPostnet.__init__rucCsJ|||dd|jj}||}|||dd}|||fS)Nrr#)rrrror postnetr)rnruoutputs_before_postnetoutputs_after_postnetlogitsr)r)r*rvs z$SpeechT5SpeechDecoderPostnet.forwardcCs0|dd}|jD] }||}q||ddSr)rr )rnruZ layer_outputrr)r)r*r s   z$SpeechT5SpeechDecoderPostnet.postnet) rxryrzrbrrrvr r{r)r)rqr*rs rcs,eZdZfddZejdddZZS)SpeechT5TextEncoderPrenetcs>t||_t|j|j|j|_t |j |j|j |_ dSrs) rarbrorr vocab_sizerr! embed_tokensrrmax_text_positionsrrrqr)r*rbs z"SpeechT5TextEncoderPrenet.__init__)r cCs||}||}|Srs)r&r)rnr rr)r)r*rv s  z!SpeechT5TextEncoderPrenet.forward)rxryrzrbrrrvr{r)r)rqr*r$s r$cs<eZdZfddZdejeejeedddZ Z S)SpeechT5TextDecoderPrenetcsnt||_t|j|_|jr0t |j nd|_ t |j |j |j|_t|j|jd|j |j|_dS)Nrr)rarbrorrrrZscale_embeddingrsqrtr embed_scalerr%r!r&rr'embed_positionsrrqr)r*rbs z"SpeechT5TextDecoderPrenet.__init__N)r r.past_key_valuescCs|dur"|}|d|d}ntdd}|durZt|tsR|ddjdn|}|||}|||j }||7}| |}||fS)Nr#z'You have to specify `decoder_input_ids`rr) rrr' isinstancer r%get_seq_lengthr+r&r*r)rnr r.r, input_shaperZ positionsrr)r)r*rvs  z!SpeechT5TextDecoderPrenet.forward)NN) rxryrzrbrrrrr rvr{r)r)rqr*r(s r(cs<eZdZfddZejdddZddZdd ZZ S) SpeechT5TextDecoderPostnetcs*t||_tj|j|jdd|_dS)NFr`)rarbrorrrr%lm_headrrqr)r*rb<s z#SpeechT5TextDecoderPostnet.__init__rcCs ||Srsr2rtr)r)r*rvAsz"SpeechT5TextDecoderPostnet.forwardcCs|jSrsr3rr)r)r*get_output_embeddingsDsz0SpeechT5TextDecoderPostnet.get_output_embeddingscCs ||_dSrsr3rnZnew_embeddingsr)r)r*set_output_embeddingsIsz0SpeechT5TextDecoderPostnet.set_output_embeddings) rxryrzrbrrrvr4r6r{r)r)rqr*r0;s r0cseZdZdZdeeeeeeeeeedfdd Ze d d d d de j ee j ee ee j ee j ee j eee j e e j ee j ee fd ddZZS)SpeechT5Attentionz Multi-headed attention from 'Attention Is All You Need' paper with relative position bias (see https://aclanthology.org/N18-2074.pdf) r0FTN) embed_dim num_headsr is_decoderr` layer_idxcst||_||_||_|||_|j||jkrNtd|jd|d|jd|_||_||_ t j |||d|_ t j |||d|_ t j |||d|_t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gr1)rarbr8r9rhead_dimr'scalingr:r;rrk_projv_projq_projout_proj)rnr8r9rr:r`r;rqr)r*rbSs$    zSpeechT5Attention.__init__past_key_valuer,4.58new_nameversion) rukey_value_statesr,r.layer_head_mask position_biasoutput_attentionscache_positionr5c Cs|du} |\} } } |||j} |dur^t|trZ|j|j}| rR|j}q^|j }n|}| rf|n|}| r|dur|r|j |jj }|j |jj }n| |}||}|| d|j|jdd}|| d|j|jdd}|dur&| s|nd}||||jd|i\}}| r&d|j|j<| |jd|jf}| | | |j|jdd} | j|} |j|}|j|}|d}t| |dd}|| |j| |fkrtd| |j| |fd||dur>| | |jd|jd d}t||d d}|d d| |j|d |d}||7}|dur|| d| |fkr~td | d| |fd||| |j| ||}|| |j| |}tjj|dd }|dur&||jfkrtd |jfd||dddd|| |j| |}|| |j| |}|rT|| |j| |}|| |j| |}nd}tjj||j|jd}t||}|| |j| |jfkrtd| |j| |jfd||| |j| |j}|dd}|| | |j}| |}||fS)z#Input shape: Batch x Time x ChannelNr#rrrKTz$Attention weights should be of size z , but is rrz!Attention mask should be of size rz/Head mask for a single layer should be of size )rrz `attn_output` should be of size )!rr@r=r-r is_updatedgetr;Zcross_attention_cacheZself_attention_cacher keysvaluesr>r?rr9r<rupdaterWrZbmmr' contiguousmatmulrrZsoftmaxrrr8rA)rnrurGr,r.rHrIrJrKZis_cross_attentionrtgt_lenrAZ query_statesrLZcurr_past_key_valueZcurrent_statesZ key_statesZ value_statesZ proj_shapeZsrc_len attn_weightsZ reshape_qZ rel_pos_biasZattn_weights_reshapedZ attn_probsZ attn_outputr)r)r*rvps            "    "   zSpeechT5Attention.forward)r0FTN)NNNNNFN)rxryrzrr7rrrPrbrrrr tuplervr{r)r)rqr*r7Ms@  r7cs$eZdZfddZddZZS)SpeechT5FeedForwardcsltt|j|_t|j||_t |j t rBt |j |_ n|j |_ t||j|_t|j|_dSrs)rarbrrZactivation_dropoutintermediate_dropoutrrintermediate_denser-Z hidden_actstrr intermediate_act_fn output_densehidden_dropoutoutput_dropout)rnroZintermediate_sizerqr)r*rbs  zSpeechT5FeedForward.__init__cCs6||}||}||}||}||}|Srs)rXrZrWr[r]rtr)r)r*rvs      zSpeechT5FeedForward.forwardrwr)r)rqr*rVs rVcsNeZdZedfdd Zd ejeejeejeejedddZ Z S) SpeechT5EncoderLayerrcsjtt|j|j|jdd|_t|j |_ tj |j|j d|_ t||j|_tj |j|j d|_dS)NF)r8r9rr:r)rarbr7rencoder_attention_headsattention_dropout attentionrrr\rr}rr~rVZencoder_ffn_dim feed_forwardfinal_layer_normrrqr)r*rb s zSpeechT5EncoderLayer.__init__NFrur.rHrIrJc Csh|}|j|||||d\}}||}||}||}|||}||}|f}|rd||f7}|S)as Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, hidden_size)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(config.encoder_attention_heads,)`. position_bias (`torch.FloatTensor`): relative position embeddings of size `(seq_len, seq_len, hidden_size // encoder_attention_heads)` output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. rd)rarr~rbrc) rnrur.rHrIrJresidualrToutputsr)r)r*rvs"     zSpeechT5EncoderLayer.forward)NNNF) rxryrzrrbrrrrPrvr{r)r)rqr*r^ sr^c seZdZdedfdd Zedddddejeejeejeejeejeejee ee ee eejd d d Z Z S)SpeechT5DecoderLayerNrcstt|j|j|jd|d|_t|j |_ tj |j|j d|_ t|j|j|jd|d|_tj |j|j d|_t||j|_tj |j|j d|_dS)NT)r8r9rr:r;r)rr:r;)rarbr7rZdecoder_attention_headsr` self_attnrrr\rr}rself_attn_layer_norm encoder_attnencoder_attn_layer_normrVZdecoder_ffn_dimrbrc)rnror;rqr)r*rbHs( zSpeechT5DecoderLayer.__init__rBr,rCrDFT) rur.encoder_hidden_statesencoder_attention_maskrHcross_attn_layer_head_maskr,rJ use_cacherKc  Cs|} |j|||||| d\}} ||}| |}||}d} |dur|} |j||||||| d\}} ||}| |}||}|||}||}|f}|r|| | f7}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, hidden_size)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, hidden_size)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size `(decoder_attention_heads,)`. past_key_values (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rur,r.rHrJrKN)rurGr.rHr,rJrK)rhrrirjrkrbrc)rnrur.rlrmrHrnr,rJrorKreZself_attn_weightsZcross_attn_weightsrfr)r)r*rv`sB        zSpeechT5DecoderLayer.forward)N) NNNNNNFTN) rxryrzrrbrrrrr rPrvr{r)r)rqr*rgGs. rgc@s2eZdZUeed<dZdZdZej dddZ dS) SpeechT5PreTrainedModelrospeecht5r,Tmodulec Cs|jj}t|tr\tjj|jjddt d|jj d|jj dtj |jjdnpt|trx|jjdnTt|trt d|jj}tjj|jj| |dtjj|jj| |dnt|tjr|jjjd|d|jdur|jjnt|tjtjtjfr2|jj|jjdnt|tjrtj|j|jdurt |j|j |j d}tjj|j| |dn>t|tjr|jjjd|d|j dur|jj|j t!|d rtj|j"dS) zInitialize the weightsrrrmeanstdr)abr0Nr)#roinitializer_ranger-rrinitnormal_rjrrr)r^Z in_channelsZ constant_r`rrdataZfill_rrZ in_featuresrrzero_r}rrrfZkaiming_normal_rrrrr)rnrsrvkr)r)r* _init_weightss@          z%SpeechT5PreTrainedModel._init_weightsN) rxryrzr__annotations__Zbase_model_prefixmain_input_nameZsupports_gradient_checkpointingrModulerr)r)r)r*rps rpc sdeZdZdZedfdd Zd ejeej eej ee ee ee e e e fdddZZS) SpeechT5Encoderzu Transformer encoder consisting of *config.encoder_layers* layers. Each layer is a [`SpeechT5EncoderLayer`]. rcs~ttjjjd|_tj|_ j |_ t fddt jD|_tjjj|_d|_|dS)Nrcsg|] }tqSr))r^r?rr)r*rBrCz,SpeechT5Encoder.__init__..F)rarbrr}rrr~rr\rZencoder_layerdrop layerdroprrNZencoder_layersr rr_Zencoder_max_relative_positionr+r post_initrrqrr*rbs  zSpeechT5Encoder.__init__N)rur. head_maskrJoutput_hidden_states return_dictr5cCs|dur |n|jj}|dur |n|jj}|dur4|n|jj}|durPt||j}||}||}||}t pzt |}|rdnd} |rdnd} |dur| dt |j krtdt |j d| ddt|j D]\} } |r| |f} d} |jrtg}||jk} | r"|rN| ||||dur<|| nd|d}|d}| rXd }|r| |d f} q|r|| |f} |std d || | fDSt|| | d S)a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, feature_size)`): Features extracted from the speech or text input by the encoder prenet. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr)rz&The head_mask should be specified for  layers, but it is for .F)r.rIrHrJ)NNrcss|]}|dur|VqdSrsr)r@vr)r)r* OrCz*SpeechT5Encoder.forward..last_hidden_stateru attentions)rorJruse_return_dictrrEr~rr+rrrrSr r' enumeraterrrIrrUr)rnrur.rrJrrrI synced_gpusall_hidden_statesall_self_attentionsidxZ encoder_layerskip_the_layerdropout_probability layer_outputsr)r)r*rvs`$            zSpeechT5Encoder.forward)NNNNNrxryrzrrrbrrrrrPrrUrrvr{r)r)rqr*rs  rc sdeZdZdZedfdd Zd ejeej eej ee ee ee e e e fdddZZS) SpeechT5EncoderWithSpeechPrenetz Wrapper around SpeechT5Encoder that applies SpeechT5SpeechEncoderPrenet to convert the audio waveform data to hidden features. rcs,t|t||_t||_|dSrs)rarbrprenetrwrapped_encoderrrrqr)r*rb^s   z(SpeechT5EncoderWithSpeechPrenet.__init__Nr,r.rrJrrr5c Cs*|||\}}|j||||||d}|SN)rur.rrJrrrr rnr,r.rrJrrrurfr)r)r*rvfs  z'SpeechT5EncoderWithSpeechPrenet.forward)NNNNNrr)r)rqr*rXs   rc steZdZdZedfdd ZddZddZd ej e ej e ej e e e e e e e eefd d d ZZS)SpeechT5EncoderWithTextPrenetz| Wrapper around SpeechT5Encoder that applies SpeechT5TextEncoderPrenet to convert the input_ids to hidden features. rcs,t|t||_t||_|dSrs)rarbr$rrrrrrqr)r*rbs   z&SpeechT5EncoderWithTextPrenet.__init__cCs |jSrsrget_input_embeddingsrr)r)r*rsz2SpeechT5EncoderWithTextPrenet.get_input_embeddingscCs|j|dSrsrset_input_embeddingsrnvaluer)r)r*rsz2SpeechT5EncoderWithTextPrenet.set_input_embeddingsNrc Cs$||}|j||||||d}|Srrrr)r)r*rvs  z%SpeechT5EncoderWithTextPrenet.forward)NNNNN)rxryrzrrrbrrrrrrrPrrUrrvr{r)r)rqr*r}s$ rc sdeZdZdZedfdd Zd ejeej eej ee ee ee e e e fdddZZS) SpeechT5EncoderWithoutPrenet This wrapper class is a helper class to correctly load pretrained checkpoints when used in combination with [`SpeechT5Model`]. rcs"t|t||_|dSrs)rarbrrrrrqr)r*rbs  z%SpeechT5EncoderWithoutPrenet.__init__NrcCs|j||||||dSr)r)rnr,r.rrJrrr)r)r*rvs z$SpeechT5EncoderWithoutPrenet.forward)NNNNNrr)r)rqr*rs   rcseZdZdZedfdd Zd eejeej eejeej eej eej ee ejee ee ee ee eej e eefd ddZZS) SpeechT5Decoderzt Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`SpeechT5DecoderLayer`] rcsFtj|_tfddtjD|_d|_ | dS)Ncsg|]}t|dqS))r;)rgrrr)r*rBrCz,SpeechT5Decoder.__init__..F) rarbZdecoder_layerdroprrrrNZdecoder_layersr rrrrqrr*rbs   zSpeechT5Decoder.__init__N) rur.rlrmrcross_attn_head_maskr,rorJrrrKr5c  Cs| dur | n|jj} | dur | n|jj} |dur4|n|jj}| durH| n|jj} |dd} |jr~|jr~|r~t dd}|r|durt t |jdt |jd}|rt |t rt dt |}|dur|nd}t|| ||}|dur|durt||j| dd}tpt|}| r*d nd}| r8d nd}| rP|durPd nd}t||gd d gD]V\}}|durf|dt|jkrftd |d t|jd|ddqft|jD]\}}| r||f}d}|jrtg}||jk}|r|sq||||||dur,||nd|dur@||nd|| || d }|d}| r||df}|dur||df}q| r||f}| st dd|||||fDSt|||||dS)a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, feature_size)`): Features extracted from the speech or text input by the decoder prenet. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr#zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FrzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.r)rSr)rrzThe `z` should be specified for rr)rmrHrnr,rJrorKrrcss|]}|dur|VqdSrsr)rr)r)r*r{sz*SpeechT5Decoder.forward..)rr,rurcross_attentions)rorJrrorrrrloggerZ warning_oncer r r-rUZfrom_legacy_cacher.rrrErrrrSr r'rrrIrr)rnrur.rlrmrrr,rorJrrrKr/rrrrZall_cross_attentionsZ attn_maskZ mask_namerZ decoder_layerrrrr)r)r*rvsJ              zSpeechT5Decoder.forward) NNNNNNNNNNNNrxryrzrrrbrrrrrlistrPrrUrrvr{r)r)rqr*rs:   rcseZdZdZedfdd Zd eejeej eejeej eej eej eej ee ejee ee ee ee eej e eefdddZZS) SpeechT5DecoderWithSpeechPrenetz Wrapper around SpeechT5Decoder that applies SpeechT5SpeechDecoderPrenet to convert log-mel filterbanks to hidden features. rcs,t|t||_t||_|dSrs)rarbr rrwrapped_decoderrrrqr)r*rbs   z(SpeechT5DecoderWithSpeechPrenet.__init__N)r,r.rlrmrrrr,rorJrrrKr5cCs2|||}|j|||||||| | | | | d }|SN) rur.rlrmrrr,rorJrrrKrr)rnr,r.rlrmrrrr,rorJrrrKdecoder_hidden_statesrfr)r)r*rvs  z'SpeechT5DecoderWithSpeechPrenet.forward) NNNNNNNNNNNNNrr)r)rqr*rs>   rcseZdZdZedfdd ZddZddZd ee j ee j ee j ee j ee j ee j ee e j eeeeeeeeee j eeefd d d ZZS)SpeechT5DecoderWithTextPrenetz{ Wrapper around SpeechT5Decoder that applies SpeechT5TextDecoderPrenet to convert input tokens to hidden features. rcs,t|t||_t||_|dSrs)rarbr(rrrrrrqr)r*rbs   z&SpeechT5DecoderWithTextPrenet.__init__cCs |jSrsrrr)r)r*rsz2SpeechT5DecoderWithTextPrenet.get_input_embeddingscCs|j|dSrsrrr)r)r*rsz2SpeechT5DecoderWithTextPrenet.set_input_embeddingsN r,r.rlrmrrr,rorJrrrKr5c Cs8||||\} }|j| |||||||| | | | d }|Srr)rnr,r.rlrmrrr,rorJrrrKrrfr)r)r*rvs z%SpeechT5DecoderWithTextPrenet.forward) NNNNNNNNNNNN)rxryrzrrrbrrrrrrrrrPrrUrrvr{r)r)rqr*rs>  rcseZdZdZedfdd Zd eejeej eejeej eej eej ee ejee ee ee ee eej e eefd ddZZS) SpeechT5DecoderWithoutPrenetrrcs"t|t||_|dSrs)rarbrrrrrqr)r*rbs  z%SpeechT5DecoderWithoutPrenet.__init__Nrc Cs&|j||||||||| | | | d } | Sr)r)rnr,r.rlrmrrr,rorJrrrKrfr)r)r*rvsz$SpeechT5DecoderWithoutPrenet.forward) NNNNNNNNNNNNrr)r)rqr*rs:   rcsVeZdZdZedfdd Zejejejej dddZ dd Z e d d Z ZS) $SpeechT5GuidedMultiheadAttentionLossz Guided attention loss from the paper [Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention](https://huggingface.co/papers/1710.08969), adapted for multi-head attention. rcst|j|_|j|_dSrs)rarbZguided_attention_loss_sigmasigmaZguided_attention_loss_scalescalerrqr)r*rb%s z-SpeechT5GuidedMultiheadAttentionLoss.__init__)r input_masks output_masksr5cCsX||||j}|d|d@}||jd}||}t||}|j|S)aY Compute the attention loss. Args: attentions (`torch.FloatTensor` of shape `(batch_size, layers * heads, output_sequence_length, input_sequence_length)`): Batch of multi-head attention weights input_masks (`torch.BoolTensor` of shape `(batch_size, input_sequence_length)`): Input attention mask as booleans. output_masks (`torch.BoolTensor` of shape `(batch_size, output_sequence_length)`): Target attention mask as booleans. Returns: `torch.Tensor` with the loss value r#rr)_make_guided_attention_masksrrrrru masked_selectr)rnrrrguided_attn_masksmasksZlosseslossr)r)r*rv*s z,SpeechT5GuidedMultiheadAttentionLoss.forwardc Cs|d}|d}tjt||jd|jdf|d}tt||D]0\}\}} ||| |j|||d| d|f<qF| dS)Nr#rr) rLrrOrSr%rr_make_guided_attention_maskrr) rnrrrrYrrrZilenZolenr)r)r*rCs   $&zASpeechT5GuidedMultiheadAttentionLoss._make_guided_attention_maskscCsdtjtj||dtj||ddd\}}||}||}dt||d d|dS)NrZxy)Zindexingrr)rZmeshgridrRrr)r9Z output_lengthrrZgrid_yZgrid_xr)r)r*rNs     z@SpeechT5GuidedMultiheadAttentionLoss._make_guided_attention_mask)rxryrzrrrbrrZ BoolTensorrrvrrrr{r)r)rqr*rs   rc sTeZdZdZedfdd Zd ejejejejeje ejej dddZ Z S) SpeechT5SpectrogramLossz; Loss computation used by SpeechT5ForTextToSpeech. rcsPt|j|_|j|_|j|_t|_tt dd|_ |jrLt ||_ dS)Ng@)Z pos_weight) rarbuse_guided_attention_lossguided_attention_loss_num_headsr-r l1_criterionrrr bce_criterionrattn_criterionrrqr)r*rb_s z SpeechT5SpectrogramLoss.__init__N)r.r!r"r#labelsrr5cs@|dk}||}||}||}||||}|dddddf} tj| dt| dd| jgdd} | ddddf| } || }|| } || } j r.) rrrrrTrrrrrr-r)rnr.r!r"r#rrrZl1_lossr stop_labelsZbce_lossrZattnrrZ attn_lossr)rr*rvks(    .   zSpeechT5SpectrogramLoss.forward)N) rxryrzrrrbrrrrrrvr{r)r)rqr*rZsrzv The bare SpeechT5 Encoder-Decoder Model outputting raw hidden-states without any specific pre- or post-nets. Z custom_introcseZdZdeeejeejdfdd ZddZddZ d d Z d d Z e dee jee jee jee jee jee jee jeeee jeeee jeeee jeeeeeeee jeee jefd ddZZS) SpeechT5ModelN)roencoderdecodercsJt|||_|dur"t|n||_|dur8t|n||_|dS)z encoder (`PreTrainedModel`, *optional*): The encoder model to use. decoder (`PreTrainedModel`, *optional*): The decoder model to use. N)rarbrorrrrr)rnrorrrqr)r*rbs zSpeechT5Model.__init__cCs4t|jtr|jSt|jtr,|jStdSrs)r-rrrrrNotImplementedErrorrr)r)r*rs     z"SpeechT5Model.get_input_embeddingscCs4t|jtr|j|t|jtr0|j|dSrs)r-rrrrrrr)r)r*rs   z"SpeechT5Model.set_input_embeddingscCs|jSrs)rrr)r)r* get_encoderszSpeechT5Model.get_encodercCst|jtr|jjdSz Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)r-rrrrrr)r)r*rs z$SpeechT5Model.freeze_feature_encoder)r,r.decoder_input_valuesdecoder_attention_maskrdecoder_head_maskrencoder_outputsr,rorrJrrrKr5cCsl| dur | n|jj} | dur | n|jj} | dur4| n|jj} |durH|n|jj}|durp|j|||| | |d}nH|rt|tst|dt|dkr|dndt|dkr|dndd}|durt|jt r|jj |dj d|}n|}t|j trd| i}ni}|j f|||d|||| | | | ||d |}|sD||St|j|j|j|j|j|j|j|jd S) a input_values (`torch.Tensor` of shape `(batch_size, sequence_length)`): Depending on which encoder is being used, the `input_values` are either: float values of the input raw speech waveform, or indices of input sequence tokens in the vocabulary, or hidden states. decoder_input_values (`torch.Tensor` of shape `(batch_size, target_sequence_length)`, *optional*): Depending on which decoder is being used, the `decoder_input_values` are either: float values of log-mel filterbank features extracted from the raw speech waveform, or indices of decoder input sequence tokens in the vocabulary, or hidden states. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. N)r,r.rrJrrrrrrr) r,r.rlrmrrr,rorJrrrK)rr,rdecoder_attentionsrencoder_last_hidden_staterlencoder_attentions)rorJrrorrr-rrSrrrr%rrrrr,rurr)rnr,r.rrrrrrr,rorrJrrrKrmZ decoder_argsZdecoder_outputsr)r)r*rvsp*   zSpeechT5Model.forward)NN)NNNNNNNNNNNNNNN)rxryrzrrrrrbrrrrrrrrrrUrPrrrvr{r)r)rqr*rsZ rzB SpeechT5 Model with a speech encoder and a text decoder. cseZdZdgZedfdd ZddZddZd d Zd d Z d dZ e de e je e je e je e je e je e je e je eee je eee je ee ee ee ee e je e jeeefdddZZS)SpeechT5ForSpeechToTextz#text_decoder_postnet.lm_head.weightrcs\t||jdur(td|jdt|}t|}t||||_t ||_ | dS)NYou are trying to instantiate a with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `SpeechT5ForSpeechToText.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.) rarbr%r'rrrrrrqr0text_decoder_postnetr)rnrospeech_encoderZ text_decoderrqr)r*rb@s    z SpeechT5ForSpeechToText.__init__cCs |jSrsrqrrr)r)r*rTsz#SpeechT5ForSpeechToText.get_encodercCs |jSrsrq get_decoderrr)r)r*rWsz#SpeechT5ForSpeechToText.get_decodercCs|jdSrrrrrr)r)r*rZsz.SpeechT5ForSpeechToText.freeze_feature_encodercCs |jSrs)rr4rr)r)r*r4asz-SpeechT5ForSpeechToText.get_output_embeddingscCs|j|dSrs)rr6r5r)r)r*r6dsz-SpeechT5ForSpeechToText.set_output_embeddingsN)r,r.decoder_input_idsrrrrrr,rorJrrrrKr5cCs| dur | n|jj} |dur8|dur8t||jj|jj}|j||||||||| | | | d|d}||d}d}|durt}||d|jj |d}| s|f|dd}|dur|f|S|St |||j |j |j |j|j|j|jd S)a( input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a *.flac* or *.wav* audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`SpeechT5Processor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`SpeechT5Processor.__call__`] for details. decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`SpeechT5Tokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) SpeechT5 uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Label indices can be obtained using [`SpeechT5Tokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. Example: ```python >>> from transformers import SpeechT5Processor, SpeechT5ForSpeechToText >>> from datasets import load_dataset >>> dataset = load_dataset( ... "hf-internal-testing/librispeech_asr_demo", "clean", split="validation" ... ) # doctest: +IGNORE_RESULT >>> dataset = dataset.sort("id") >>> sampling_rate = dataset.features["audio"].sampling_rate >>> processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_asr") >>> model = SpeechT5ForSpeechToText.from_pretrained("microsoft/speecht5_asr") >>> # audio file is decoded on the fly >>> inputs = processor(audio=dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt") >>> predicted_ids = model.generate(**inputs, max_length=100) >>> # transcribe speech >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True) >>> transcription[0] 'mister quilter is the apostle of the middle classes and we are glad to welcome his gospel' ``` ```python >>> inputs["labels"] = processor(text_target=dataset[0]["text"], return_tensors="pt").input_ids >>> # compute loss >>> loss = model(**inputs).loss >>> round(loss.item(), 2) 19.68 ``` NT)r,r.rrrrrrr,rorJrrrKrr#r) rr#r,rrrrrlr)rorr+r!r"rqrrrr%rr,rrrrrlr)rnr,r.rrrrrrr,rorJrrrrKrfr#rZloss_fctoutputr)r)r*rvgsR[zSpeechT5ForSpeechToText.forward)NNNNNNNNNNNNNNN)rxryrzZ_tied_weights_keysrrbrrrr4r6rrrrrrrUrPrrrvr{r)r)rqr*r8sR r?r04@F) modelr,rr. threshold minlenratio maxlenratiovocoderoutput_cross_attentionsreturn_output_lengthsr5c " s|durtd|dur.d||jjk} n|} |d} |jj|| dd} | j} t|jjt r~|jjj | dj d| } t| d||jj }t| d||jj }| | d|jj}g}g}d}d}i|d7}|jj ||}|jjj|ddddfd| | |d|dd}|r:|tj|jdd|jd}|j}|j|}|| |jj |jj}|||dddddf| d|jj}tj||fdd}t|j|}||krqq||krtj|dd|k}t|d}n t t!|}fd d |D}t!|dkrlt"|}|#dd$dd }|j%|}|D]}|||<qXt!| krؐq~q؇fd d t t!D}| s2| dkr|dntj&j'j(j)|dd }|dur||}n|}|rtj|d d}| dkr(|j| t|d| g|ddR}||f}ngt | D]} || dq>|durtj&j'j(j)|dd }|f}n.rcsg|] }|qSr)r)rrr)r*rB] rC)Z batch_firstcs&g|]}tdt|qS)r)r7rr8r)spectrogram_lengths waveformsr)r*rBw rC)*r'ror!r7rrqrrr-rrrr%r-r$r rrrUrrrsqueezer,speech_decoder_postnetrrZsigmoidrrLrrMrNrSstackrflattenr rrZrnnZ pad_sequence)"rr,rr.rrrrrrrmrZ encoder_outrmaxlenZminlenZoutput_sequence spectrogramrr,rrZ decoder_outZlast_decoder_outputZspectrumZnew_spectrogramZprobZmeet_thresholdsZ meet_indexesZ spectrogramsZ meet_indexrfrZwaveform_lengthsr))rrrr*_generate_speechs       $      &       rzB SpeechT5 Model with a text encoder and a speech decoder. cseZdZdZedfdd ZeedddZdd Z d d Z e de e je e je e je e je e je e je e je eee je eee je ee ee ee ee e je e je e je e jeeefd ddZe de je e je e jeeee ejeeee jee je jffd ddZe de je e je e jeeee ejeeee jee je jffd ddZZS)SpeechT5ForTextToSpeechr rcs\t||jdur(td|jdt|}t|}t||||_t ||_ | dS)Nra with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `SpeechT5ForTextToSpeech.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.) rarbr%r'rrrrrrqrrr)rnroZ text_encoderspeech_decoderrqr)r*rb s    z SpeechT5ForTextToSpeech.__init__)r5cCsdSrr))clsr)r)r* can_generate sz$SpeechT5ForTextToSpeech.can_generatecCs |jSrsrrr)r)r*r sz#SpeechT5ForTextToSpeech.get_encodercCs |jSrsrrr)r)r*r sz#SpeechT5ForTextToSpeech.get_decoderN)r r.rrrrrrr,rorJrrrrrrKr5cCs| dur | n|jj} |durD|dur8t||jj|\}}|jjrDd} |j||||||||| | || | d|d}||d\}}}d}|durt|j}|||||||j}| s|f|dd}|dur|f|S|St |||j |j |j |j|j |j|jd S)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`SpeechT5Tokenizer`]. See [`~PreTrainedTokenizer.encode`] and [`~PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) decoder_input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`): Float values of input mel spectrogram. SpeechT5 uses an all-zero spectrum as the starting token for `decoder_input_values` generation. If `past_key_values` is used, optionally only the last `decoder_input_values` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. labels (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`, *optional*): Float values of target mel spectrogram. Timesteps set to `-100.0` are ignored (masked) for the loss computation. Spectrograms can be obtained using [`SpeechT5Processor`]. See [`SpeechT5Processor.__call__`] for details. stop_labels (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Binary tensor indicating the position of the stop token in the sequence. Example: ```python >>> from transformers import SpeechT5Processor, SpeechT5ForTextToSpeech, SpeechT5HifiGan, set_seed >>> import torch >>> processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_tts") >>> model = SpeechT5ForTextToSpeech.from_pretrained("microsoft/speecht5_tts") >>> vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan") >>> inputs = processor(text="Hello, my dog is cute", return_tensors="pt") >>> speaker_embeddings = torch.zeros((1, 512)) # or load xvectors from a file >>> set_seed(555) # make deterministic >>> # generate speech >>> speech = model.generate(inputs["input_ids"], speaker_embeddings=speaker_embeddings, vocoder=vocoder) >>> speech.shape torch.Size([15872]) ``` NTr,r.rrrrrrr,rorrJrrrKrr rrr,rrrrrlr)rorr1r-rrqrrrrr,rrrrlr)rnr r.rrrrrrr,rorJrrrrrrKrfr!r"r#r criterionrr)r)r*rv sfM   zSpeechT5ForTextToSpeech.forwardrr0rF) r r.rrrrrrrr5c Ks^|durD|d} |d| krD|ddkr<|| d}ntdt|||||||||| S)aE Converts a sequence of input tokens into a sequence of mel spectrograms, which are subsequently turned into a speech waveform using a vocoder. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`SpeechT5Tokenizer`]. See [`~PreTrainedTokenizer.encode`] and [`~PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Attention mask from the tokenizer, required for batched inference to signal to the model where to ignore padded tokens from the input_ids. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. threshold (`float`, *optional*, defaults to 0.5): The generated sequence ends when the predicted stop token probability exceeds this value. minlenratio (`float`, *optional*, defaults to 0.0): Used to calculate the minimum required length for the output sequence. maxlenratio (`float`, *optional*, defaults to 20.0): Used to calculate the maximum allowed length for the output sequence. vocoder (`nn.Module`, *optional*): The vocoder that converts the mel spectrogram into a speech waveform. If `None`, the output is the mel spectrogram. output_cross_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of the decoder's cross-attention layers. return_output_lengths (`bool`, *optional*, defaults to `False`): Whether or not to return the concrete spectrogram/waveform lengths. Returns: `tuple(torch.FloatTensor)` comprising various elements depending on the inputs: - when `return_output_lengths` is False - **spectrogram** (*optional*, returned when no `vocoder` is provided) `torch.FloatTensor` of shape `(output_sequence_length, config.num_mel_bins)` -- The predicted log-mel spectrogram. - **waveform** (*optional*, returned when a `vocoder` is provided) `torch.FloatTensor` of shape `(num_frames,)` -- The predicted speech waveform. - **cross_attentions** (*optional*, returned when `output_cross_attentions` is `True`) `torch.FloatTensor` of shape `(config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length)` -- The outputs of the decoder's cross-attention layers. - when `return_output_lengths` is True - **spectrograms** (*optional*, returned when no `vocoder` is provided) `torch.FloatTensor` of shape `(batch_size, output_sequence_length, config.num_mel_bins)` -- The predicted log-mel spectrograms that are padded to the maximum length. - **spectrogram_lengths** (*optional*, returned when no `vocoder` is provided) `list[Int]` -- A list of all the concrete lengths for each spectrogram. - **waveforms** (*optional*, returned when a `vocoder` is provided) `torch.FloatTensor` of shape `(batch_size, num_frames)` -- The predicted speech waveforms that are padded to the maximum length. - **waveform_lengths** (*optional*, returned when a `vocoder` is provided) `list[Int]` -- A list of all the concrete lengths for each waveform. - **cross_attentions** (*optional*, returned when `output_cross_attentions` is `True`) `torch.FloatTensor` of shape `(batch_size, config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length)` -- The outputs of the decoder's cross-attention layers. NrrzUThe first dimension of speaker_embeddings must be either 1 or the same as batch_size.rrr'r) rnr r.rrrrrrrkwargsrXr)r)r*generate2 s(E z SpeechT5ForTextToSpeech.generate) r rr.rrrrrrr5c Cs^|durD|d} |d| krD|ddkr<|| d}ntdt|||||||||| S)a Converts a sequence of input tokens into a sequence of mel spectrograms, which are subsequently turned into a speech waveform using a vocoder. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`SpeechT5Tokenizer`]. See [`~PreTrainedTokenizer.encode`] and [`~PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) threshold (`float`, *optional*, defaults to 0.5): The generated sequence ends when the predicted stop token probability exceeds this value. minlenratio (`float`, *optional*, defaults to 0.0): Used to calculate the minimum required length for the output sequence. maxlenratio (`float`, *optional*, defaults to 20.0): Used to calculate the maximum allowed length for the output sequence. vocoder (`nn.Module`, *optional*, defaults to `None`): The vocoder that converts the mel spectrogram into a speech waveform. If `None`, the output is the mel spectrogram. output_cross_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of the decoder's cross-attention layers. return_output_lengths (`bool`, *optional*, defaults to `False`): Whether or not to return the concrete spectrogram/waveform lengths. Returns: `tuple(torch.FloatTensor)` comprising various elements depending on the inputs: - when `return_output_lengths` is False - **spectrogram** (*optional*, returned when no `vocoder` is provided) `torch.FloatTensor` of shape `(output_sequence_length, config.num_mel_bins)` -- The predicted log-mel spectrogram. - **waveform** (*optional*, returned when a `vocoder` is provided) `torch.FloatTensor` of shape `(num_frames,)` -- The predicted speech waveform. - **cross_attentions** (*optional*, returned when `output_cross_attentions` is `True`) `torch.FloatTensor` of shape `(config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length)` -- The outputs of the decoder's cross-attention layers. - when `return_output_lengths` is True - **spectrograms** (*optional*, returned when no `vocoder` is provided) `torch.FloatTensor` of shape `(batch_size, output_sequence_length, config.num_mel_bins)` -- The predicted log-mel spectrograms that are padded to the maximum length. - **spectrogram_lengths** (*optional*, returned when no `vocoder` is provided) `list[Int]` -- A list of all the concrete lengths for each spectrogram. - **waveforms** (*optional*, returned when a `vocoder` is provided) `torch.FloatTensor` of shape `(batch_size, num_frames)` -- The predicted speech waveforms that are padded to the maximum length. - **waveform_lengths** (*optional*, returned when a `vocoder` is provided) `list[Int]` -- A list of all the concrete lengths for each waveform. - **cross_attentions** (*optional*, returned when `output_cross_attentions` is `True`) `torch.FloatTensor` of shape `(batch_size, config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length)` -- The outputs of the decoder's cross-attention layers. NrrzUThe first dimension of speaker_embeddings must be either 1 or the same as batch size.r) rnr rr.rrrrrrrXr)r)r*generate_speech s(I z'SpeechT5ForTextToSpeech.generate_speech)NNNNNNNNNNNNNNNNN)NNrr0rNFF)NNrr0rNFF)rxryrzrrrb classmethodrPrrrrrrrrrrUrrrvrrrrrrr{r)r)rqr*r s [rzD SpeechT5 Model with a speech encoder and a speech decoder. cs&eZdZedfdd ZddZddZdd Zede e j e e j e e j e e j e e j e e j e e j e eee j e eee j e ee ee ee ee e j e e j e e j e e j eeefd d d Ze de j e e j e e j eeee ejeee j d ddZZS)SpeechT5ForSpeechToSpeechrcs@t|t|}t|}t||||_t||_|dSrs) rarbrrrrqrrr)rnrorrrqr)r*rb s   z"SpeechT5ForSpeechToSpeech.__init__cCs |jSrsrrr)r)r*r sz%SpeechT5ForSpeechToSpeech.get_encodercCs |jSrsrrr)r)r*r sz%SpeechT5ForSpeechToSpeech.get_decodercCs|jdSrrrr)r)r*r sz0SpeechT5ForSpeechToSpeech.freeze_feature_encoderN)r,r.rrrrrrr,rorJrrrrrrKr5cCs| dur | n|jj} |dur8|dur8t||jj|\}}|j||||||||| | || | d|d}||d\}}}d}| s|f|dd}|dur|f|S|St|||j|j|j |j |j |j |j d S)a input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a *.flac* or *.wav* audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`SpeechT5Processor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`SpeechT5Processor.__call__`] for details. decoder_input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`): Float values of input mel spectrogram. SpeechT5 uses an all-zero spectrum as the starting token for `decoder_input_values` generation. If `past_key_values` is used, optionally only the last `decoder_input_values` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. labels (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`, *optional*): Float values of target mel spectrogram. Spectrograms can be obtained using [`SpeechT5Processor`]. See [`SpeechT5Processor.__call__`] for details. stop_labels (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Binary tensor indicating the position of the stop token in the sequence. Example: ```python >>> from transformers import SpeechT5Processor, SpeechT5ForSpeechToSpeech, SpeechT5HifiGan, set_seed >>> from datasets import load_dataset >>> import torch >>> dataset = load_dataset( ... "hf-internal-testing/librispeech_asr_demo", "clean", split="validation" ... ) # doctest: +IGNORE_RESULT >>> dataset = dataset.sort("id") >>> sampling_rate = dataset.features["audio"].sampling_rate >>> processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_vc") >>> model = SpeechT5ForSpeechToSpeech.from_pretrained("microsoft/speecht5_vc") >>> vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan") >>> # audio file is decoded on the fly >>> inputs = processor(audio=dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt") >>> speaker_embeddings = torch.zeros((1, 512)) # or load xvectors from a file >>> set_seed(555) # make deterministic >>> # generate speech >>> speech = model.generate_speech(inputs["input_values"], speaker_embeddings, vocoder=vocoder) >>> speech.shape torch.Size([77824]) ``` NTrrrr)rorr1r-rqrrr,rrrrrlr)rnr,r.rrrrrrr,rorJrrrrrrKrfrArr#rrr)r)r*rv sNT z!SpeechT5ForSpeechToSpeech.forwardrr0rF) r,rr.rrrrrrr5c Cs2|durtjd|jd}t|||||||||| S)a' Converts a raw speech waveform into a sequence of mel spectrograms, which are subsequently turned back into a speech waveform using a vocoder. Args: input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a *.flac* or *.wav* audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`SpeechT5Processor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`SpeechT5Processor.__call__`] for details. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) threshold (`float`, *optional*, defaults to 0.5): The generated sequence ends when the predicted stop token probability exceeds this value. minlenratio (`float`, *optional*, defaults to 0.0): Used to calculate the minimum required length for the output sequence. maxlenratio (`float`, *optional*, defaults to 20.0): Used to calculate the maximum allowed length for the output sequence. vocoder (`nn.Module`, *optional*, defaults to `None`): The vocoder that converts the mel spectrogram into a speech waveform. If `None`, the output is the mel spectrogram. output_cross_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of the decoder's cross-attention layers. return_output_lengths (`bool`, *optional*, defaults to `False`): Whether or not to return the concrete spectrogram/waveform lengths. Returns: `tuple(torch.FloatTensor)` comprising various elements depending on the inputs: - when `return_output_lengths` is False - **spectrogram** (*optional*, returned when no `vocoder` is provided) `torch.FloatTensor` of shape `(output_sequence_length, config.num_mel_bins)` -- The predicted log-mel spectrogram. - **waveform** (*optional*, returned when a `vocoder` is provided) `torch.FloatTensor` of shape `(num_frames,)` -- The predicted speech waveform. - **cross_attentions** (*optional*, returned when `output_cross_attentions` is `True`) `torch.FloatTensor` of shape `(config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length)` -- The outputs of the decoder's cross-attention layers. - when `return_output_lengths` is True - **spectrograms** (*optional*, returned when no `vocoder` is provided) `torch.FloatTensor` of shape `(batch_size, output_sequence_length, config.num_mel_bins)` -- The predicted log-mel spectrograms that are padded to the maximum length. - **spectrogram_lengths** (*optional*, returned when no `vocoder` is provided) `list[Int]` -- A list of all the concrete lengths for each spectrogram. - **waveforms** (*optional*, returned when a `vocoder` is provided) `torch.FloatTensor` of shape `(batch_size, num_frames)` -- The predicted speech waveforms that are padded to the maximum length. - **waveform_lengths** (*optional*, returned when a `vocoder` is provided) `list[Int]` -- A list of all the concrete lengths for each waveform. - **cross_attentions** (*optional*, returned when `output_cross_attentions` is `True`) `torch.FloatTensor` of shape `(batch_size, config.decoder_layers, config.decoder_attention_heads, output_sequence_length, input_sequence_length)` -- The outputs of the decoder's cross-attention layers. N)rir)rrOrr) rnr,rr.rrrrrrr)r)r*r sJz)SpeechT5ForSpeechToSpeech.generate_speech)NNNNNNNNNNNNNNNNN)NNrr0rNFF)rxryrzrrbrrrrrrrrrrUrPrrrvrrrrrr{r)r)rqr*r s  rcs@eZdZdfdd ZdddZd d Zd d Zd dZZS)HifiGanResidualBlockrrr皙?csbt|_tfddttD_tfddttD_dS)Nc s2g|]*}tjd||dqSr)r_dilationrrrf get_paddingrchannelsr r^rnr)r*rB s z1HifiGanResidualBlock.__init__..c s*g|]"}tjddddqSr r r?)rr^rnr)r*rB s  ) rarbleaky_relu_sloperrrNrSconvs1convs2)rnrr^r rrqrr*rb s    zHifiGanResidualBlock.__init__rcCs|||dSrr))rnr^r r)r)r*r sz HifiGanResidualBlock.get_paddingcCsLtjj}ttjjdr tjjj}|jD] }||q&|jD] }||q:dSNr)rrrrrrrrnrrr)r)r*apply_weight_norm s    z&HifiGanResidualBlock.apply_weight_normcCs4|jD]}tj|q|jD]}tj|qdSrs)rrrremove_weight_normrrnrr)r)r*r s  z'HifiGanResidualBlock.remove_weight_normcCsXt|j|jD]D\}}|}tj||j}||}tj||j}||}||}q|Srs)rrrrr leaky_relur)rnruZconv1Zconv2rer)r)r*rv s zHifiGanResidualBlock.forward)rrr )r) rxryrzrbrrrrvr{r)r)rqr*r s   rz HiFi-GAN vocoder. csneZdZUeed<dZedfdd ZejdddZ d d Z d d Z e d de je jdddZZS)SpeechT5HifiGanrorrc sNt|t|j|_t|j|_tj|j |j dddd|_ t |_ tt|j|jD]H\}\}}|j tj|j d||j d|d||||ddqZt |_tt|j D]F}|j d|d}t|j|jD] \}}|jt||||jqqtj|ddddd|_|dt|j |dt|j |dS)Nrr)r^r_rrrur)rarbrSZresblock_kernel_sizes num_kernelsZupsample_rates num_upsamplesrrfZ model_in_dimZupsample_initial_channelconv_prer upsamplerrrZupsample_kernel_sizesrUConvTranspose1d resblocksrNZresblock_dilation_sizesrr conv_postrrrOrTr)rnrorZ upsample_rater^rr rqr)r*rb2 s<       zSpeechT5HifiGan.__init__rrcCsBt|tjtjfr>|jjjd|jjd|j dur>|j j dS)zInitialize the weights.r0rtN) r-rrfr rr|r{roryr`r})rnrsr)r)r*rX s zSpeechT5HifiGan._init_weightscCs`tjj}ttjjdr tjjj}||j|jD] }||q0|jD] }|qD||j dSr) rrrrrrrr!rr"rr)r)r*r_ s      z!SpeechT5HifiGan.apply_weight_normcCsLtj|j|jD]}tj|q|jD] }|q,tj|jdSrs)rrrrrr!r"rr)r)r*rk s    z"SpeechT5HifiGan.remove_weight_norma Converts a log-mel spectrogram into a speech waveform. Passing a batch of log-mel spectrograms returns a batch of speech waveforms. Passing a single, un-batched log-mel spectrogram returns a single, un-batched speech waveform. r)rr5cCs|jjr||j|j}|dk}|s2|d}|dd}||}t|j D]p}t j ||jj }|j||}|j||j|}td|jD] }||j||j||7}q||j}qRt j |}||}t|}|s|dddd}n |d}|S)a spectrogram (`torch.FloatTensor`): Tensor containing the log-mel spectrograms. Can be batched and of shape `(batch_size, sequence_length, config.model_in_dim)`, or un-batched and of shape `(sequence_length, config.model_in_dim)`. Returns: `torch.FloatTensor`: Tensor containing the speech waveform. If the input spectrogram is batched, will be of shape `(batch_size, num_frames,)`. If un-batched, will be of shape `(num_frames,)`. rrrrr#)roZnormalize_beforerurrrrrrNrrrrrrr!rr"rtanhrr)rnrZ is_batchedrurZ res_statejZwaveformr)r)r*rvs s*         zSpeechT5HifiGan.forward)rxryrzrrrrbrrrrrrrrrvr{r)r)rqr*r) s & r)rrrrrpr)rN)Nr)NNrr0rNFF)hrrtypingrrnumpyrGrZtorch.utils.checkpointrZtorch.nnrrrZ activationsr Z cache_utilsr r r Z generationr Zintegrations.deepspeedrZintegrations.fsdprZmodeling_attn_mask_utilsrrZmodeling_layersrZmodeling_outputsrrrrrZmodeling_utilsrrrrrZutils.deprecationrZconfiguration_speecht5rrZ get_loggerrxrZ_HIDDEN_STATES_START_POSITIONrr7r+r1rUrrZndarrayr[r\r|rrrrrrrrrrr rrr$r(r0r7rVr^rgrprrrrrrrrrrrrrrPrrrrr__all__r)r)r)r*s           xE- )4(,&=f+%* D26-;=: jx>w