a h@@sddlZddlmZddlmZmZmZddlZddl m Z m Z e rNddl Z e r\ddl ZddlmZmZddlmZddlmZmZmZmZdd lmZmZGd d d ed d ZGddded d ZGdddeZdgZdS)N)Path)AnyOptionalUnion)is_soundfile_availableis_torch_available) AudioInputmake_list_of_audio) BatchFeature) AudioKwargsProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInputc@s"eZdZUeeeefed<dS)CsmAudioKwargsencoded_length_kwargsN)__name__ __module__ __qualname__rdictstrr__annotations__rrb/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/csm/processing_csm.pyr%s rF)totalc@sJeZdZUeed<ddddgdgdgddd d d d d idZdS)CsmProcessorKwargs audio_kwargsTleftF)paddingZ padding_sideZadd_special_tokens)rrr" rr" rr"r)r"r"r"r'r"r"r"r"r"r"r#r")r"r"r"r"r"r"r"r"r"r"r"r"r"r"r") kernel_sizesstrides dilationsuse_causal_convi])r sampling_ratereturn_tensorspt) text_kwargsr common_kwargsN)rrrrr _defaultsrrrrr)s  rcseZdZdZddgZdZdZdfdd Zedd d Z e e e e ee e e ffeed d d Zdee eeeeeefee eeeeeedddZeddZZS) CsmProcessora  Constructs a Csm processor which wraps [`EncodecFeatureExtractor`] and [`PretrainedTokenizerFast`] into a single processor that inherits both the audio feature extraction and tokenizer functionalities. See the [`~CsmProcessor.__call__`] for more information. The preferred way of passing kwargs is as a dictionary per modality, see usage example below. ```python from transformers import CsmProcessor from datasets import load_dataset ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") audio = ds[0]["audio"]["array"] processor = CsmProcessor.from_pretrained("sesame/csm-1b") processor( text=["<|begin_of_text|>[0]What are you working on?<|end_of_text|><|AUDIO|><|audio_eos|><|begin_of_text|>[1]I'm figuring out my budget.<|end_of_text|>"], audio=audio, text_kwargs = {"padding": False}, audio_kwargs = {"sampling_rate": 16000}, common_kwargs = {"return_tensors": "pt"}, ) # this should error out because EncodecFeatureExtractor expects a 24kHz audio :) ``` Args: feature_extractor ([`EncodecFeatureExtractor`]): The feature extractor is a required input. tokenizer ([`PreTrainedTokenizer`, `PreTrainedTokenizerFast`]): The tokenizer is a required input. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. feature_extractor tokenizerZEncodecFeatureExtractorZPreTrainedTokenizerFastNcsvt|ds d|_||j|_n|j|_|j|_t|dsPd|_||j|_n|j|_|j|_tj|||ddS)N audio_tokenz <|AUDIO|>audio_eos_tokenz <|audio_eos|>) chat_template)hasattrr8Zconvert_tokens_to_idsaudio_token_idr9audio_eos_token_idsuper__init__)selfr6r7r: __class__rrr?fs  zCsmProcessor.__init__cCs|}|dus$|dus$|dus$|dur(|St|||D]\}}}|d|d} ||} | d} | | } || | |d} t| d} | ||| }||}|r| } |} n | } | |} || | }|||dd|d}q4|S)a| Compute the length of the encoded audio sequence. Args: audio_length (int): The length of the audio sequence. kernel_sizes (list[int]): The kernel sizes for the convolutional layers. strides (list[int]): The strides for the convolutional layers. use_causal_conv (bool): Whether to use causal convolutions. Nr"r*)zipmathceil)Z audio_lengthr+r,r-r.Z cur_lengthZ kernel_sizeZstrideZdilationZeffective_kernel_sizeZ padding_total padding_right padding_leftZn_framesZ ideal_lengthZ extra_paddingrrr_get_encoded_length|s(   z CsmProcessor._get_encoded_length)audio saving_pathkwargsc Kststdt|}t|ttfr,|g}n(t|ttfrLtdd|DsTt dt |t |krlt d|j t fi|}|d}|d}t ||D]2\}}t|tjr|}t|||qdS)Nz/Please install `soundfile` to save audio files.css|]}t|ttfVqdSN) isinstancerr).0prrr z*CsmProcessor.save_audio..zAInvalid input path. Please provide a string, or a list of stringsz5The number of audio and saving paths must be the samerr/)r ImportErrorr rMrrlisttupleall ValueErrorlen _merge_kwargsrrCtorchTensorcpufloatnumpysfwrite) r@rIrJrK output_kwargsrr/Z audio_valuerOrrr save_audios(  zCsmProcessor.save_audioF?)textrI output_labelsdepth_decoder_labels_ratiorKc sjtfdjji|}|d}|d}|d} | dd} | dkrXtjjdt|t rj|g}n(t|t t frt d d |Dstd fd d |D} d} |durt |}t|} t| dkr| t| kr|durtdntd| d| d|dur|difdd |D} | }g}|D]t}g}j|vr|d}j|}|||jdd}qFd|vr|d|dd}q||q>|}j|fi|}i}|||dur|ddgg}}d}| D]} | dkr2|td|tdgn`|tjdd |||| Ddd|tdd |||| Djdd|| 7}qj|fi|}|dd||tdd |Dfdd |D}tj|dd|d <|r|d!jk}|j d}|d"krHt!|dt"|d|}||}n|}t#|d!jk|d!j$kB|d!d#}d$||dddf|dddff<||d%<t%|| d&S)'a Main method to prepare text(s) and audio to be fed as input to the model. This method forwards the `text` arguments to PreTrainedTokenizerFast's [`~PreTrainedTokenizerFast.__call__`] to encode the text. To prepare the audio, this method forwards the `audio` arguments to EncodecFeatureExtractor's [`~EncodecFeatureExtractor.__call__`]. Please refer to the docstring of the above two methods for more information. Args: audio (`np.ndarray`, `torch.Tensor`, `list[np.ndarray]`, `list[torch.Tensor]`): The audio or batch of audio to be prepared. Each audio can be a NumPy array or PyTorch tensor. text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). output_labels (bool, *optional*, default=False): Whether to return labels for training. Indices will be in `[config.audio_token_id, -100, -101]`. - `config.audio_token_id` indicates an audio frame (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) depth_decoder_labels_ratio (float, *optional*, default=1.0): The ratio of audio frames to keep for the depth decoder labels. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **input_values** -- List of audio values to be fed to a model. Returned when `audio` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **labels** -- List of labels for the audio frames. Returned when `output_labels=True`. Ztokenizer_init_kwargsr2rr3r0Nr1z% only supports `return_tensors='pt'`.css|]}t|tVqdSrL)rMrrNtrrrrPrQz(CsmProcessor.__call__..zAInvalid input text. Please provide a string, or a list of stringscsg|]}|jqSr)countr8rf)r@rr rQz)CsmProcessor.__call__..rz@No audio were provided, but there are audio tokens in the promptz)The number of audio tokens in each text (z7) should be the same as the number of provided audios (z).rcs$g|]}j|jdfiqS)rHshape)rNZ audio_array)rr@rrrisz r"Zreturn_attention_maskrkcSs(g|] }t|tjr |n|qSr)rMrYrZr[r]rNelrrrri>s)ZaxiscSsg|]}|jdqSrjrlrmrrrriFrQ)dim padding_maskcss|]}|jdVqdS)rkNrorNZcut_idxsrrrrPOrQcs.g|]&}tjjj|d|jdfddqS)rrk)value)rYnnZ functionalpadrlrr)max_lenrrriPsinput_values_cutoffsZ input_idsrbiilabels)dataZ tensor_type)&rXrr7Z init_kwargspoprVrBrrMrrSrTrUr rWsumcopyr8appendreplaceupdatenpZzerosrYZtensorZ concatenateZcumsumr6maxstackr<ZnonzerorlZrandpermintwherer=r )r@rcrIrdrerKr`r2rr3r0Zn_audio_in_textZn_audioZnum_audio_tokens_listZnum_audio_tokens_list_copyZ expanded_textsampleZ replace_strZnum_audio_tokensZexpanded_audio_tokenencodingryZconcatenated_audiorwoffsetZ audio_inputsZaudio_frame_idxsZn_audio_framesZ rand_idxsZskip_frames_idxsrxr)rrvr@r__call__s/                  &       $zCsmProcessor.__call__cCs0|jj}|jj}dd|D}t||dgS)NcSsg|]}|dkr|qS)rqr)rNnamerrrrirrQz2CsmProcessor.model_input_names..rw)r7model_input_namesr6rS)r@Ztokenizer_input_namesZfeature_extractor_input_namesrrrrkszCsmProcessor.model_input_names)N)NNNN)NFrb)rrr__doc__ attributesZfeature_extractor_classZtokenizer_classr? staticmethodrHr rrrrSrrrarrrboolr\rpropertyr __classcell__rrrArr5>s2# ( % 'r5) rDpathlibrtypingrrrr]rutilsrrrYZ soundfiler^Z audio_utilsr r Zfeature_extraction_utilsr Zprocessing_utilsr r rrZtokenization_utils_baserrrrr5__all__rrrrs$  :