a h@p@sdZddlZddlmZddlmZddlZddlZddl m Z ddl m m Z ddlmZddlmZmZdd lmZeeGd d d eZeeGd d d eZeeGdddeZGddde jZGddde jZGddde jZGddde jZGddde jZGddde jZGddde jZGddde jZ eGd d!d!eZ!ed"d#Gd$d%d%e!Z"d%d!gZ#dS)&zTransformers DAC model.N) dataclass)Optional)PreTrainedAudioTokenizerBase) ModelOutputauto_docstring) DacConfigc@sleZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eej ed<dZ eejed<dS) DacOutputa loss (`torch.Tensor`): Loss from the encoder model, comprising the weighted combination of the commitment and codebook losses. audio_values (`torch.Tensor` of shape `(batch_size, input_length)`): Reconstructed audio data. quantized_representation (`torch.Tensor` of shape `(batch_size, dimension, time_steps)`): Quantized continuous representation of input. audio_codes (`torch.LongTensor` of shape `(batch_size, num_codebooks, time_steps)`): Codebook indices for each codebook (quantized discrete representation of input). projected_latents (`torch.Tensor` of shape `(batch_size, num_codebooks * dimension, time_steps)`): Projected latents (continuous representation of input before quantization). Nloss audio_valuesquantized_representation audio_codesprojected_latents)__name__ __module__ __qualname____doc__r rtorch FloatTensor__annotations__r r rZ LongTensorrrr`/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/dac/modeling_dac.pyr s  r c@sZeZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dS)DacEncoderOutputa loss (`torch.Tensor`): Loss from the encoder model, comprising the weighted combination of the commitment and codebook losses. quantized_representation (`torch.Tensor` of shape `(batch_size, dimension, time_steps)`, *optional*): Quantized continuous representation of input. audio_codes (`torch.Tensor` of shape `(batch_size, num_codebooks, time_steps)`, *optional*): Codebook indices for each codebook (quantized discrete representation of input). projected_latents (`torch.Tensor` of shape `(batch_size, num_codebooks * dimension, time_steps)`, *optional*): Projected latents (continuous representation of input before quantization). Nr r rr) rrrrr rrrrr rrrrrrr6s  rc@s$eZdZUdZdZeejed<dS)DacDecoderOutputz audio_values (`torch.FloatTensor` of shape `(batch_size, input_length)`, *optional*): Decoded audio values, obtained using the decoder part of Dac. Nr ) rrrrr rrrrrrrrrJs rcs(eZdZdZfddZddZZS)Snake1dz; A 1-dimensional Snake activation function module. cs$tttd|d|_dS)Nr)super__init__nn Parameterronesalpha)self hidden_dim __class__rrr[s zSnake1d.__init__cCsR|j}||d|dd}||jdt|j|d}||}|S)Nrrg& .>)shapereshaper!Z reciprocalrsinpow)r" hidden_statesr(rrrforward_s ( zSnake1d.forward)rrrrrr- __classcell__rrr$rrVs rcs6eZdZdZedfdd ZddZddZZS) DacVectorQuantizea Implementation of VQ similar to Karpathy's repo (https://github.com/karpathy/deep-vector-quantization) Additionally uses following tricks from improved VQGAN (https://huggingface.co/papers/2110.04627): 1. Factorized codes: Perform nearest neighbor lookup in low-dimensional space for improved codebook usage 2. l2-normalized codes: Converts euclidean distance to cosine similarity which improves training stability configcsLttj|j|jdd|_tj|j|jdd|_t|j |j|_ dS)Nr kernel_size) rrrConv1d hidden_size codebook_dimin_projout_proj Embedding codebook_sizecodebookr"r1r$rrrss zDacVectorQuantize.__init__cCsh||}||\}}tj||dd}tj||dd}|||}||}|||||fS)aJ Quantizes the input tensor using a fixed codebook and returns the corresponding codebook vectors. Args: hidden_state (`torch.FloatTensor` of shape `(batch_size, dimension, time_steps)`): Input tensor. Returns: quantized_representation (`torch.Tensor`of shape `(batch_size, dimension, time_steps)`): Quantized continuous representation of input. commitment_loss (`torch.FloatTensor`of shape `(1)`): Commitment loss to train encoder to predict vectors closer to codebook entries. codebook_loss (`torch.FloatTensor`of shape `(1)`): Codebook loss to update the codebook. audio_codes (`torch.LongTensor` of shape `(batch_size, time_steps)`): Codebook indices for each codebook, quantized discrete representation of input. projected_latents (torch.FloatTensor of shape `(batch_size, num_codebooks * dimension, time_steps)`): Projected latents (continuous representation of input before quantization). mean)Z reduction)r7decode_latentsFZmse_lossdetachr8)r" hidden_staterr rcommitment_loss codebook_lossrrrr-zs  zDacVectorQuantize.forwardc Cs|j\}}}|ddd|||}|jj}t|}t|}|djddd}|d||  |djddd }| dd} | | dd} ||  dd} | | fS)Nrr'rT)Zkeepdimr&) r(Zpermuter)r;weightr? normalizer+sumtmaxsize transpose) r"r,Z batch_sizer#Zsequence_length encodingsr;Zl2_normdistindicesr rrrr>s   .z DacVectorQuantize.decode_latents) rrrrr rr-r>r.rrr$rr/gs  r/cs2eZdZdZd eedfdd ZddZZS) DacResidualUnitza A residual unit composed of Snake1d and weight-normalized Conv1d layers with dilations. r) dimensiondilationcsVtd|d}t||_tj||d||d|_t||_tj||dd|_dS)Nr')r3rQpaddingrr2) rrrsnake1rr4conv1snake2conv2)r"rPrQpadr$rrrs     zDacResidualUnit.__init__cCsb|}|||}|||}|jd|jdd}|dkrV|d|| f}||}|S)ar Forward pass through the residual unit. Args: hidden_state (`torch.Tensor` of shape `(batch_size, channels, time_steps)`): Input tensor . Returns: output_tensor (`torch.Tensor` of shape `(batch_size, channels, time_steps)`): Input tensor after passing through the residual unit. r&r'r.)rVrUrXrWr()r"rAZ output_tensorrTrrrr-s zDacResidualUnit.forward)rOr)rrrrintrr-r.rrr$rrNs rNcs4eZdZdZdeeedfdd ZddZZS) DacEncoderBlockz"Encoder block used in DAC encoder.rr1stride stride_indexc st|jd|}t|ddd|_t|ddd|_t|ddd|_t|d|_t j |d|d||t |dd|_ dS)Nr'rrQr r3r]rT)rrencoder_hidden_sizerN res_unit1 res_unit2 res_unit3rrUrr4mathceilrV)r"r1r]r^rPr$rrrs zDacEncoderBlock.__init__cCs2||}||}|||}||}|SN)rcrdrUrerVr"rArrrr-s    zDacEncoderBlock.forward)rr rrrrr rZrr-r.rrr$rr[s r[cs4eZdZdZdeeedfdd ZddZZS) DacDecoderBlockz"Decoder block used in DAC decoder.rr\c st|jd|}|jd|d}t||_tj||d||t|dd|_ t |dd|_ t |dd|_ t |dd|_ dS)Nr'rrar_rr`)rrdecoder_hidden_sizerrUrConvTranspose1drfrgconv_t1rNrcrdre)r"r1r]r^Z input_dim output_dimr$rrrs   zDacDecoderBlock.__init__cCs6||}||}||}||}||}|Srh)rUrnrcrdrerirrrr-s      zDacDecoderBlock.forward)rrrjrrr$rrksrkcsZeZdZdZedfdd ZdeedddZe j d d d Z e j d d dZ Z S)DacResidualVectorQuantizez ResidualVectorQuantize block - Introduced in SoundStream: An end2end neural audio codec (https://huggingface.co/papers/2107.03312) r0csFtj}j}||_tfddtjD|_||_dS)Ncsg|] }tqSr)r/).0ir0rr z6DacResidualVectorQuantize.__init__..)rr n_codebooksquantizer_dropoutr ModuleListrange quantizers)r"r1rurvr$r0rrs   z"DacResidualVectorQuantize.__init__N) n_quantizerscCspd}|}d}d}g}g}|dur$|n|j}|jrt|jdf|jd}td|jd|jdf} t|jd|j} | d| |d| <||j }t |j D]\} } |jdur| |krqF| |\} }}}}tj |jdf| |j d|k}|| |ddddf}|| }|||7}|||7}| || |qtj|dd}tj|dd}|||||fS)aQ Quantizes the input tensor using a fixed set of codebooks and returns corresponding codebook vectors. Args: hidden_state (`torch.Tensor` of shape `(batch_size, dimension, time_steps)`): Input tensor to be quantized. n_quantizers (`int`, *optional*): Number of quantizers to use. If specified and `self.quantizer_dropout` is True, this argument is ignored during training, and a random number of quantizers is used. Returns: quantized_representation (`torch.Tensor` of shape `(batch_size, dimension, time_steps)`): Quantized continuous representation of input. audio_codes (`torch.Tensor` of shape `(batch_size, num_codebooks, time_steps)`): Codebook indices for each codebook (quantized discrete representation of input). projected_latents (`torch.Tensor` of shape `(batch_size, num_codebooks * dimension, time_steps)`): Projected latents (continuous representation of input before quantization). commitment_loss (`torch.Tensor` of shape `(1)`): Commitment loss to train the encoder to predict vectors closer to codebook entries. codebook_loss (`torch.Tensor` of shape `(1)`): Codebook loss to update the codebook. rNrF)Z fill_valuedevicedim)ruZtrainingrr r(randintrZrvtor{ enumerateryfullappendstackcat)r"rArzr ZresidualrBrCrrZdropoutZ n_dropoutrr quantizerquantized_representation_iZcommitment_loss_iZcodebook_loss_iZ indices_iprojected_latents_imaskrrrr-s:     z!DacResidualVectorQuantize.forward)rcCs|d}g}|jd}t|D]L}|j||dd|ddfdd}||||j||7}q|tj|dd|fS)a Reconstructs the continuous representation from quantized codes. Args: audio_codes (`torch.Tensor` of shape `(batch_size, num_codebooks, time_steps)`): Quantized discrete representation of input. Returns: quantized_representation (`torch.Tensor`): Quantized continuous representation of input. projected_latents (`torch.Tensor`): List of projected latents (continuous representations of input before quantization) for each codebook. audio_codes (`torch.Tensor`): Codebook indices for each codebook. rNr'r|) r(rxryr;rJrr8rr)r"rr rrurrrrrr from_codesYs  * z$DacResidualVectorQuantize.from_codes)latentscCsd}g}g}tdgdd|jD}tj|dd}t||jdkdjdddd}t|D]p}||||d} } |j| |dd| | ddf\} } | | | | |j| | } || }qd|tj |ddfS) aReconstructs the quantized representation from unquantized latents. Args: latents (`torch.Tensor` of shape `(batch_size, total_latent_dimension, time_steps)`): Continuous representation of input after projection. Returns: quantized_representation (`torch.Tensor` of shape `(batch_size, dimension, time_steps)`): Quantized representation of the full-projected space. quantized_latents (`torch.Tensor` of shape `(batch_size, dimension, time_steps)`): Quantized representation of the latent space (continuous representation before quantization). rcSsg|] }|jqSr)r6)rqqrrrrsrtz:DacResidualVectorQuantize.from_latents..r|rT)ZaxisZkeepdimsN) rZtensorryZcumsumnpwherer(rHrxr>rr8r)r"rr Zquantized_latentscodesZcodebook_dims_tensordimsrurrZ hidden_dim_jZ hidden_dim_kZquantized_latents_iZcodes_irrrr from_latentsss & *   z&DacResidualVectorQuantize.from_latents)N)rrrrr rrrZr-rTensorrrr.rrr$rrp s  @rpcs.eZdZdZedfdd ZddZZS) DacDecoderz DAC Decoderr0c st|j}|j}|j}tj||ddd|_g}t|D]\}}|t |||g7}q>> from datasets import load_dataset, Audio >>> from transformers import DacModel, AutoProcessor >>> librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> model = DacModel.from_pretrained("descript/dac_16khz") >>> processor = AutoProcessor.from_pretrained("descript/dac_16khz") >>> librispeech_dummy = librispeech_dummy.cast_column("audio", Audio(sampling_rate=processor.sampling_rate)) >>> audio_sample = librispeech_dummy[-1]["audio"]["array"] >>> inputs = processor(raw_audio=audio_sample, sampling_rate=processor.sampling_rate, return_tensors="pt") >>> encoder_outputs = model.encode(inputs["input_values"]) >>> # Get the intermediate audio codes >>> audio_codes = encoder_outputs.audio_codes >>> # Reconstruct the audio from its quantized representation >>> audio_values = model.decode(encoder_outputs.quantized_representation) >>> # or the equivalent with a forward pass >>> audio_values = model(inputs["input_values"]).audio_values ```Nr&F)rr.)r1rr(rrr ) r"rrzrlengthr r rrr rrrr-{s# zDacModel.forward)NN)NNN)NN)rrrr rrrrrrZboolrrr-r.rrr$rr)s4 r)$rrf dataclassesrtypingrnumpyrrZtorch.nnrZtorch.nn.functionalZ functionalr?Zmodeling_utilsrrrrZconfiguration_dacr r rrModulerr/rNr[rkrprrrr__all__rrrrsF      F% %!O~