a h@sddlmZddlmZmZddlZddlmZddlm Z ddl m Z m Z ddl mZddlmZdd lmZmZdd lmZdd lmZdd lmZmZmZmZd dlmZd dlm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*ddl+m,Z,e-e.Z/eeddGdddeZ0Gddde%Z1Gddde&Z2Gddde#Z3Gddde Z4Gdd d e!Z5ed!deGd"d#d#eZ6eGd$d%d%e$e6Z7Gd&d'd'ej8Z9ed(dGd)d*d*e"eZ:Gd+d,d,ej8Z;eGd-d.d.e$ZdS)3) dataclass)OptionalUnionN)check_model_inputs)Cache DynamicCache)GenerationMixin)create_causal_mask)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)Unpack) ModelOutputauto_docstringcan_return_tuplelogging) AutoModel)LlamaAttentionLlamaDecoderLayerLlamaForCausalLMLlamaMLP LlamaModel LlamaRMSNormLlamaRotaryEmbeddingTransformersKwargs) CsmConfigCsmDepthDecoderConfig)CsmGenerationMixinz: Base class for the model autoregressive outputs. )Z custom_introc@seZdZUdZdZeejed<dZ ejed<dZ ee e ejed<dZ ee ejdfed<dZ ee ejdfed<dZeejed <dZejed <dZee e ejed <dZee ejdfed <dZee ejdfed <dZeejed<dS)CsmOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_decoder_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the depth decoder model. depth_decoder_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the depth decoder (scores for each vocabulary token before SoftMax). depth_decoder_past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) depth_decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. depth_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. backbone_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the backbone model. Nlosslogitspast_key_values. hidden_states attentionsdepth_decoder_lossdepth_decoder_logitsdepth_decoder_past_key_valuesdepth_decoder_hidden_statesdepth_decoder_attentions backbone_loss)__name__ __module__ __qualname____doc__r"rtorch FloatTensor__annotations__r#r$tupler%r&r'r(r)r*r+r,r5r5_/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/csm/modular_csm.pyr!1s r!c@s eZdZdS) CsmRMSNormNr-r.r/r5r5r5r6r7dsr7c@s eZdZdS)CsmRotaryEmbeddingNr8r5r5r5r6r9hsr9c@s eZdZdS)CsmMLPNr8r5r5r5r6r:lsr:c@s eZdZdS) CsmAttentionNr8r5r5r5r6r;psr;c@s eZdZdS)CsmDecoderLayerNr8r5r5r5r6r<tsr<z[ The bare Csm Model outputting raw hidden-states without any specific head on top. csTeZdZUeed<dZdZdgZdgZdZ dZ dZ dZ e edZfddZZS) CsmPreTrainedModelconfigmodelTr<r$)r%r&csLt|t|trH|j}t|dD]}|jj|jd|j j dq(dS)Nrg)meanZstd) super _init_weights isinstanceCsmCodebooksHead num_codebooksrangeweightdataZnormal_r>Zinitializer_range)selfmodulerEi __class__r5r6rBs   z CsmPreTrainedModel._init_weights)r-r.r/rr3Zbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_skip_keys_device_placementZ_supports_flash_attnZ_supports_sdpaZ_can_compile_fullgraphZ_supports_attention_backendr<r;Z_can_record_outputsrB __classcell__r5r5rLr6r=xs r=cseZdZUeed<fddZeedej e ej e ej e ej e e e ej e ee ej eeeeefd ddZZS) CsmDepthDecoderModelr>cs>t|t|j|j|j|_tj|j|j dd|_ dSNF)Zbias) rA__init__nn EmbeddingrE vocab_sizeZbackbone_hidden_size embed_tokensLinear hidden_sizeinputs_embeds_projectorrIr>rLr5r6rQs zCsmDepthDecoderModel.__init__N) input_idsbackbone_last_hidden_stateattention_mask position_idsr$ inputs_embeds use_cachecache_positionkwargsreturnc  Ks|dur tjs tdd}|du|duAr8td|rP|durPt|jd}|dur|durh|nd} |dur~|j dn|j d} |dur|j n|j } tj | | | | d}|dur(tj |ddd} | |j }|||}|ddk}|dur ||dddf<ntjs(|r(td ||}t|j|||||d }|}|d}|||}|jd|jjD]$}||f||||||d | }qt||}t||r|ndd S) aJ backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. NzCustom `position_ids` were provided but will be ignored. CSM depth decoder automatically determines position_ids from `cache_position` and as it requires them to be identical across the batch, the provided position_ids will be ignored.z;You must specify exactly one of input_ids or inputs_embeds.)r>rrdevice)minzvWhen the first codebook token is provided, `backbone_last_hidden_state` should also be provided for correct inference.)r> input_embedsr\r`r$r])r\r]r$r_r`position_embeddings)Zlast_hidden_stater$)r1compilerZ is_compilingloggerZ warning_once ValueErrorrr>Zget_seq_lengthshaperdarangeclamprTrUwarningrXr unsqueezeZ rotary_embZlayersZnum_hidden_layersZnormr )rIrZr[r\r]r$r^r_r`raZpast_seen_tokensZinputs_seq_lengthrd codebook_idxsoffsetZinput_ids_are_first_codebookZ causal_maskr%rgZ decoder_layerr5r5r6forwardsn            zCsmDepthDecoderModel.forward)NNNNNNNN)r-r.r/rr3rQrrr1 LongTensorrr2Tensorrboolrrrr4r rrrNr5r5rLr6rOs0   rOcs&eZdZfddZdddZZS)rDcs0t||_tt|jd|||_dS)Nr)rArQrErR Parameterr1emptyrG)rIrWrErTrLr5r6rQs zCsmCodebooksHead.__init__Ncsf|dur$jd}|jt|n|d}|j|fddtjdDtjddS)Nrc s2g|]*}tjdd|ddf|jqSN)rR functionalZlinearT).0Z codebook_idxZcodebook_weightr%r5r6 sz,CsmCodebooksHead.forward..rdim)rkrGr1rlrFstack)rIr%r`Z seq_lengthrpr5r|r6rrs    zCsmCodebooksHead.forward)Nr-r.r/rQrrrNr5r5rLr6rDs rDa$ The CsmDepthDecoder Model transformer, with a [`CsmCodebooksHead`] on top, which can be seen a position-specific language modeling head, allowing to use a different linear layer for each codebook (e.g. position 0 is the first codebook and uses the first codebook head, etc.) cseZdZdZdZdZfddZd eje e e eje ej e ejdfdd Z e ed eje ej e eje eje ee eej fe ej e eje ee ejeeejfeeeeefd d d ZZS) CsmDepthDecoderForCausalLMNcs2t||`t|j|j|j|_t||_ dSrx) rArQlm_headrDrWrErTcodebooks_headrOr?rYrLr5r6rQs z#CsmDepthDecoderForCausalLM.__init__rZr$r\r^r`c  sHtj|||||fi|}|dddk}|s:|d|d|S)Nr`rr[r])rAprepare_inputs_for_generationpop) rIrZr$r\r^r`ra model_inputsZis_first_generation_steprLr5r6r!s    z8CsmDepthDecoderForCausalLM.prepare_inputs_for_generationr) rZr[r\r]r$r^labelsr_r`logits_to_keeprarbc  Ks|jf|||||||| d| } | d} t| trV| dkrHtdd}qZt| d}n| }|| dd|ddf| dur| |nd}|}d}|dur|dddf}|jf|d|jj|d| }t ||| j | j | j dS)a backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. )rZr[r\r]r$r^r_r`rrN.)r#rrT shift_labels)r"r#r$r%r&) r?rCintslicer contiguous loss_functionr>rTr r$r%r&)rIrZr[r\r]r$r^rr_r`rraoutputsr% slice_indicesr#r"rr5r5r6rr7sJ    & z"CsmDepthDecoderForCausalLM.forward)NNNN) NNNNNNNNNr)r-r.r/_tied_weights_keysZ_tp_planZ_pp_planrQr1rsrrr2rrrrtrlistrurrrr4r rrrNr5r5rLr6rsR   rcs$eZdZfddZddZZS)CsmBackboneModelEmbeddingscsDtt|j|j|j|_|jdt |j|jdddS)Naudio_tokens_offsetsF) persistent) rArQrRrSrErTrWembed_audio_tokensZregister_bufferr1rlrYrLr5r6rQ}s  z#CsmBackboneModelEmbeddings.__init__cCs |||j}|jdd}|S)Nrr~)rrsum)rIrZrfr5r5r6rrs z"CsmBackboneModelEmbeddings.forwardrr5r5rLr6r|s rcs0eZdZfddZeefddZZS)CsmBackboneModelcst|t||_dSrx)rArQrrUrYrLr5r6rQs zCsmBackboneModel.__init__c stjfi|S)a& input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) )rArr)rIZ super_kwargsrLr5r6rrszCsmBackboneModel.forward)r-r.r/rQrrrrrNr5r5rLr6rs rz The Csm model consists of two llama-like auto-regressive transformer models: a backbone model that predicts the first codebook token and a depth decoder that predicts the other codebook tokens. csJeZdZddgZfddZddZddZd d Zefd d Z fd dZ de e j e e j e e j e e j e e j dddZde je ee e je e je e jdfdd Zeede je e j e e j e e j e e je eeee jfe e je e je ee e jeee j feeeeefd ddZZS)CsmForConditionalGenerationz5backbone_model.embed_tokens.embed_audio_tokens.weightz'depth_decoder.model.embed_tokens.weightcspt||j|_tj|j|jdd|_t|j|j|_ t ||_ t |j|_t|j|_|dSrP)rArQrTrRrVrWrrSZtext_vocab_sizeembed_text_tokensr _from_configbackbone_modelrZdepth_decoder_config depth_decoderr from_configZ codec_config codec_modelZ post_initrYrLr5r6rQs  z$CsmForConditionalGeneration.__init__cCs|jjSrxrrUrIr5r5r6get_input_embeddingssz0CsmForConditionalGeneration.get_input_embeddingscCs ||j_dSrxr)rIvaluer5r5r6set_input_embeddingssz0CsmForConditionalGeneration.set_input_embeddingscCs$|jjr ||jjj|jjjdSrx)r>Ztie_codebooks_embeddingsZ_tie_or_clone_weightsrrUrrr?rr5r5r6 _tie_weightss z(CsmForConditionalGeneration._tie_weightscs|ddr$tj|i|\}}ntj|i|}dtfddt|jD}t|jjddi||D]}t |j|q~d|vr||fS|SdS)NZoutput_loading_infoFdepth_decoder_cs(i|] \}}|r|d|qSrx) startswith)r{attrrprefix prefix_lenr5r6 s z?CsmForConditionalGeneration.from_pretrained..Z_from_model_config) getrAfrom_pretrainedlenvarsgeneration_configitemsrupdatedelattr)clsargsrar?Z loading_infodepth_decoder_attrsrrLrr6rs   z+CsmForConditionalGeneration.from_pretrainedcsVd}|jj}|dd|D]\}}t|j|||q$tj|i|dS)NrZtransformers_version)rrZ to_diff_dictrrsetattrrAsave_pretrained)rIrrarrrrrLr5r6rs   z+CsmForConditionalGeneration.save_pretrainedN)rZ input_valuesinput_values_cutoffsrrbcsH||}|dur>tj|d}||dk}||dk}tj||jd t |d}|| dk}t g}t ||D]~\} } | | dk} t| jddD]V} | | } | | d} | d| | f}|j| d}|jdd}||dqqtdd |Dtfd d |D}|j|}Wdn1sP0Y|jj}||k}|j|}||||<tjdd|jjf|jtjd |jj}|j|d}||jj k}|!|"d||<|dur>| d!dd|jj}||||<|||<|d kj#dd}d||d|dddf<|}||dS)a Merges the input_ids and input_values to produce a single inputs_embeds tensor: 1 - Infers the codec model on the input_values to retreive codebook token. 2 - Embeds codebook tokens and places them at the correct positions in the inputs_embeds tensor. 3 - If labels are provided, expands them to match codebook dimensions and position the target codebook tokens in the inputs_embeds tensor. Args: input_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`): The input ids to embed. input_values (`torch.Tensor` of shape `(batch_size, channels, audio_sequence_length)`): The audio input values to embed. input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`): The cutoffs of the audio input values relative to its batch index, padded with -1 when no audio. Nrrrrcr.css|]}|jdVqdS)rN)rkr{elr5r5r6 zQCsmForConditionalGeneration._merge_input_ids_with_input_values..c s,g|]$}tj|ddd|jdfqS)r)rRrypadrkrZmax_audio_framesr5r6r}rzRCsmForConditionalGeneration._merge_input_ids_with_input_values..)rdZdtypeiTas_tuple)r^r)$rrRryrdiffr1rlmaxrdexpandrroZno_gradziprFrkrencodeZ audio_codesZ transposeappendrZget_audio_codes_maskr>audio_token_idrrUZonesrElongZcodebook_eos_token_idZsqueezeZaudio_eos_token_idrepeatrnonzero)rIrZrrrr^Z audio_lengthsZinput_values_maskZaudio_tokens_listZbatch_input_valuesZbatch_input_values_cutoffsrKZ start_idxZend_idxZ audio_batchZ codec_outputsZ codebook_idsZbatched_audio_token_idsZaudio_codes_maskrZaudio_token_maskZ audio_embedsZaudio_eos_frame_idsZaudio_eos_embedsZaudio_eos_token_maskZlabels_expandedZ depth_decoder_ignore_frames_idxsr5rr6"_merge_input_ids_with_input_valuessX      ,     z>CsmForConditionalGeneration._merge_input_ids_with_input_valuesrc  stjf|||||d|}|dur||jdkr||ddur||j||d|d|dd}||d|ddd|S) Nrrr^rrr)rZrrr)r^rrZ)rArndimrrr) rIrZr$r\r^r`rar merged_inputsrLr5r6r;s(  z9CsmForConditionalGeneration.prepare_inputs_for_generationr) rZrr\rr]r$r^rr_r`rrarbc  Ks|dur6|jdkr6|||||} | d}| d}d}|jf|||||| | d| }|d}t| trtt| dn| }||dd|ddf}d}d}d}d}|dur|dddddf}|jf|||jj d| }|ddddddfd kj d d }||d d|jj df}t j j|d dd}|jdd}||d|ddddf}||}|jf||| d|d| }|j}||}t|||||j|j|j|dur|jnd|dur|jnd|dur|jnd|dur|jndd S)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`, *optional*): Specify the end positions of audio segments within each batch entry, relative to the concatenated audio input. If a batch entry has fewer segments than the maximum, it is padded with -1. For example, in a batch of 2 sequences where the first contains 2 audio segments of length l1, and the second contains 1 audio segment of length l2, the input_values_cutoffs would be: [[l1, 2 * l1], [l2, -1]]. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[config.audio_token_id, -100, -101]`. Requires targeted `input_values` to be provided as audio tokens will be inferred from it using the `codec_model`. - `config.audio_token_id` indicates an audio frames (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) Such labels can be prepared using `output_labels=True` when calling [`CsmProcessor`]. logits_to_keep (`int` or `torch.Tensor`, *optional*): Kept for compatibility. Does not support another value than: 1. `0`, which is equivalent to keeping all logits, used in the training regime 2. `1`, which is equivalent to keeping only the last logit, used in the generation regime Example: ```python >>> import torch >>> from transformers import CsmForConditionalGeneration, AutoProcessor >>> from datasets import load_dataset, Audio >>> model_id = "sesame/csm-1b" >>> torch_device = "cuda" if torch.cuda.is_available() else "cpu" >>> processor = AutoProcessor.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") >>> # ensure the audio is 24kHz >>> ds = ds.cast_column("audio", Audio(sampling_rate=24000)) >>> conversation = [] >>> # prepare a conversation with text and corresponding audio >>> for text, audio, speaker_id in zip(ds[:4]["text"], ds[:4]["audio"], ds[:4]["speaker_id"]): ... conversation.append( ... { ... "role": f"{speaker_id}", ... "content": [{"type": "text", "text": text}, {"type": "audio", "path": audio["array"]}], ... } ... ) >>> inputs = processor.apply_chat_template( ... conversation, ... tokenize=True, ... return_dict=True, ... output_labels=True, ... ).to(torch_device) >>> model = CsmForConditionalGeneration.from_pretrained(model_id, device_map=torch_device) >>> output = model(**inputs) >>> output.loss.backward() ```Nrr^r)rZr\r]r$r^r_r`r)r#rrTrrrr~.r)rTr)rZr[r_Z return_dictr) r"r,r'r#r$r%r&r(r)r*r+)rrrrCrrrrr>rTallrErRryrrrr"r!r$r%r&r#)rIrZrr\rr]r$r^rr_r`rrarZbackbone_outputsZbackbone_hidden_statesrZbackbone_logitsr"r,r'Zdepth_decoder_outputsZbackbone_labelsZ train_maskZdepth_decoder_input_idsZ train_idxsZbackbone_last_hidden_statesZdepth_decoder_labelsr5r5r6rrZsS   (    z#CsmForConditionalGeneration.forward)NNNN)NNNN) NNNNNNNNNNr)r-r.r/rrQrrr classmethodrrrr1rtrrsrr2rrrrrrurrrr4r!rrrNr5r5rLr6rsx   U  r)r=rrOrr)? dataclassesrtypingrrr1Ztorch.nnrRZtransformers.utils.genericrZ cache_utilsrrZ generationr Z masking_utilsr Zmodeling_outputsr r Zmodeling_utilsr Zprocessing_utilsrutilsrrrrautorZllama.modeling_llamarrrrrrrrZconfiguration_csmrrZgeneration_csmr Z get_loggerr-rir!r7r9r:r;r<r=rOModulerDrrrr__all__r5r5r5r6s`        (   -_fU