a hI @sddlmZddlmZmZmZddlZddlmZddl m Z ddl m Z ddl mZmZddlmZdd lmZdd lmZdd lmZdd lmZmZdd lmZmZddlmZm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.m/Z/ddl0m1Z1e(2e3Z4ee&ddGddde$Z5edGdddej6Z7Gdddej6Z8Gd d!d!ej6Z9d"d#Z:dCd$d%Z;ejdDej6ejd>eCZHe&d?dGd@dAdAeCe1ZIgdBZJdS)E) dataclass)CallableOptionalUnionN)check_model_inputs)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging)deprecate_kwarg) AutoModel) CsmConfigCsmDepthDecoderConfig)CsmGenerationMixinz: Base class for the model autoregressive outputs. )Z custom_introc@seZdZUdZdZeejed<dZ ejed<dZ ee e ejed<dZ ee ejdfed<dZ ee ejdfed<dZeejed <dZejed <dZee e ejed <dZee ejdfed <dZee ejdfed <dZeejed<dS)CsmOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_decoder_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the depth decoder model. depth_decoder_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the depth decoder (scores for each vocabulary token before SoftMax). depth_decoder_past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) depth_decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. depth_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. backbone_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the backbone model. Nlosslogitspast_key_values. hidden_states attentionsdepth_decoder_lossdepth_decoder_logitsdepth_decoder_past_key_valuesdepth_decoder_hidden_statesdepth_decoder_attentions backbone_loss)__name__ __module__ __qualname____doc__r#rtorch FloatTensor__annotations__r$r%tupler&r'r(r)r*r+r,r-r6r6`/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/csm/modeling_csm.pyr"2s r"ZRMSNormcs.eZdZdfdd ZddZddZZS) CsmRMSNormư>cs&ttt||_||_dS)z9 CsmRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parameterr2onesweightvariance_epsilon)self hidden_sizeeps __class__r6r7r;fs zCsmRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)NrT)Zkeepdim) dtypetor2float32powmeanZrsqrtr@r?)rAr&Z input_dtypeZvariancer6r6r7forwardns  zCsmRMSNorm.forwardcCst|jjd|jS)Nz, eps=)r5r?shaper@rAr6r6r7 extra_repruszCsmRMSNorm.extra_repr)r9)r.r/r0r;rLrO __classcell__r6r6rDr7r8dsr8csDeZdZUejed<dedfdd Zee ddZ Z S) CsmRotaryEmbeddinginv_freqNconfigcstt|dr:t|jtr:|jd|jd|_nd|_|j|_ |j|_ ||_ t |j|_ | |j |\}|_|jd|dd|j|_dS)N rope_scaling rope_typetypedefaultrRF persistent)r:r;hasattr isinstancerUdictgetrVZmax_position_embeddingsZmax_seq_len_cachedZoriginal_max_seq_lenrTrZ rope_init_fnattention_scalingregister_bufferrRZoriginal_inv_freq)rArTdevicerRrDr6r7r;|s  zCsmRotaryEmbedding.__init__c Cs|jddddf|jddd|j}|dddddf}t|jjtrl|jjdkrl|jjnd}t j |ddV|| dd}t j ||fdd }| |j}||j} Wdn1s0Y|j|jd | j|jd fS) NrrFrZmpscpuF) device_typeZenabledrdim)rG)rRfloatexpandrMrHrar\rWstrr2Zautocast transposecatcosr_sinrG) rAx position_idsZinv_freq_expandedZposition_ids_expandedrcZfreqsZembrkrlr6r6r7rLs0&,zCsmRotaryEmbedding.forward)N) r.r/r0r2Tensorr4rr;no_gradrrLrPr6r6rDr7rQys  rQcs$eZdZfddZddZZS)CsmMLPcsxt||_|j|_|j|_tj|j|j|jd|_tj|j|j|jd|_ tj|j|j|jd|_ t |j |_ dS)NZbias)r:r;rTrBZintermediate_sizer<LinearZmlp_bias gate_projup_proj down_projrZ hidden_actact_fnrArTrDr6r7r;s zCsmMLP.__init__cCs$||||||}|SN)rvrwrtru)rArmrvr6r6r7rLs zCsmMLP.forwardr.r/r0r;rLrPr6r6rDr7rqs rqcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..NrFrrd)rMr2rj)rmx1Zx2r6r6r7 rotate_halfsr|cCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer|)qkrkrlrnZ unsqueeze_dimZq_embedZk_embedr6r6r7apply_rotary_pos_embs   r)r&n_repreturncCs^|j\}}}}|dkr|S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rMrgreshape)r&rbatchnum_key_value_headsslenhead_dimr6r6r7 repeat_kvs 0r)modulequerykeyvalueattention_maskscalingdropoutkwargsc Kst||j}t||j} t||dd|} |durf|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)NrrrF)rerG)ptrainingr)rnum_key_value_groupsr2matmulrirMr< functionalZsoftmaxrIrHrGrr contiguous) rrrrrrrr key_states value_states attn_weights causal_mask attn_outputr6r6r7eager_attention_forwards  & rc seZdZdZeedfdd Zeddddd ej e ej ej fe ej e e e ej eee ej ej fd d d ZZS) CsmAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrT layer_idxcst||_||_t|d|j|j|_|j|j|_ |jd|_ |j |_ d|_ t j|j|j|j|jd|_t j|j|j|j|jd|_t j|j|j|j|jd|_t j|j|j|j|jd|_dS)NrgTrr)r:r;rTrgetattrrBZnum_attention_headsrrrrattention_dropoutZ is_causalr<rsZattention_biasq_projk_projv_projo_projrArTrrDr6r7r;s(  zCsmAttention.__init__past_key_valuer%4.58new_nameversionN)r&position_embeddingsrr%cache_positionrrcKs$|jdd}g|d|jR}|||dd} |||dd} |||dd} |\} } t| | | | \} } |dur| | |d}|| | |j |\} } t }|j j dkrt |j j }||| | | |f|jsdn|j|jd|\}}|jg|dR}||}||fS)NrFrr)rlrkreagerr)rr)rMrrviewrirrrupdaterrrTZ_attn_implementationrrrrrrr)rAr&rrr%rrZ input_shapeZ hidden_shapeZ query_statesrrrkrlZ cache_kwargsZattention_interfacerrr6r6r7rLs8      zCsmAttention.forward)NN)r.r/r0r1rintr;rr2ror5rr LongTensorrrrLrPr6r6rDr7rs rc seZdZeedfdd Zeddddd eje eje ej e e e e e ej e e ejejfeeejd d d ZZS)CsmDecoderLayerrcsRt|j|_t||d|_t||_t|j|jd|_ t|j|jd|_ dS)NrrC) r:r;rBr self_attnrqmlpr8 rms_norm_epsinput_layernormpost_attention_layernormrrDr6r7r;=s   zCsmDecoderLayer.__init__rr%rrNF) r&rrnr% use_cacherrrrc Ks^|} ||}|jf|||||||d|\}} | |}|} ||}||}| |}|S)N)r&rrnr%rrr)rrrr) rAr&rrnr%rrrrZresidual_r6r6r7rLGs&     zCsmDecoderLayer.forward)NNNFNN)r.r/r0rrr;rr2rorrr boolr5rrrLrPr6r6rDr7r<s& rz[ The bare Csm Model outputting raw hidden-states without any specific head on top. csTeZdZUeed<dZdZdgZdgZdZ dZ dZ dZ e edZfddZZS) CsmPreTrainedModelrTmodelTrr%)r&r'csLt|t|trH|j}t|dD]}|jj|jd|j j dq(dS)Nrr)rKZstd) r: _init_weightsr\CsmCodebooksHead num_codebooksranger?dataZnormal_rTZinitializer_range)rArrirDr6r7rs   z CsmPreTrainedModel._init_weights)r.r/r0rr4Zbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_skip_keys_device_placementZ_supports_flash_attnZ_supports_sdpaZ_can_compile_fullgraphZ_supports_attention_backendrrZ_can_record_outputsrrPr6r6rDr7rjs rcseZdZUeed<fddZeedej e ej e ej e ej e e e ej e ee ej eeeeefd ddZZS) CsmDepthDecoderModelrTcstj|_j|_tjjj|_ t fddt j D|_ tjjd|_td|_d|_tjjjdd|_|dS)Ncsg|]}t|qSr6r.0rrSr6r7 z1CsmDepthDecoderModel.__init__..rrSFrr)r:r; pad_token_id padding_idx vocab_sizer< EmbeddingrZbackbone_hidden_size embed_tokens ModuleListrnum_hidden_layerslayersr8rBrnormrQ rotary_embgradient_checkpointingrsinputs_embeds_projector post_initrxrDrSr7r;s  zCsmDepthDecoderModel.__init__N) input_idsbackbone_last_hidden_staterrnr% inputs_embedsrrrrc  Ks|dur tjs tdd}|du|duAr8td|rP|durPt|jd}|dur|durh|nd} |dur~|j dn|j d} |dur|j n|j } tj | | | | d}|dur(tj |ddd} | |j }|||}|ddk}|dur ||dddf<ntjs(|r(td ||}t|j|||||d }|}|d}|||}|jd|jjD]$}||f||||||d | }qt||}t||r|ndd S) aJ backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. NzCustom `position_ids` were provided but will be ignored. CSM depth decoder automatically determines position_ids from `cache_position` and as it requires them to be identical across the batch, the provided position_ids will be ignored.z;You must specify exactly one of input_ids or inputs_embeds.rSrrra)minzvWhen the first codebook token is provided, `backbone_last_hidden_state` should also be provided for correct inference.rT input_embedsrrr%rn)rrnr%rrrZlast_hidden_stater%)r2compilerZ is_compilingloggerZ warning_once ValueErrorr rTget_seq_lengthrMraarangeclamprrwarningrr r}rrrrr)rArrrrnr%rrrrpast_seen_tokensZinputs_seq_lengthra codebook_idxsoffsetZinput_ids_are_first_codebookrr&r decoder_layerr6r6r7rLsn            zCsmDepthDecoderModel.forward)NNNNNNNN)r.r/r0r r4r;rrr2rrr3ror rrrrr5rrLrPr6r6rDr7rs0   rcs&eZdZfddZdddZZS)rcs0t||_tt|jd|||_dS)Nr)r:r;rr<r=r2emptyr?)rArBrrrDr6r7r;s zCsmCodebooksHead.__init__Ncsf|dur$jd}|jt|n|d}|j|fddtjdDtjddS)Nrc s2g|]*}tjdd|ddf|jqSry)r<rZlinearT)rZ codebook_idxZcodebook_weightr&r6r7rsz,CsmCodebooksHead.forward..rrd)rMr?r2rrstack)rAr&rZ seq_lengthrr6rr7rLs    zCsmCodebooksHead.forward)Nrzr6r6rDr7rs ra$ The CsmDepthDecoder Model transformer, with a [`CsmCodebooksHead`] on top, which can be seen a position-specific language modeling head, allowing to use a different linear layer for each codebook (e.g. position 0 is the first codebook and uses the first codebook head, etc.) cseZdZdZdZdZfddZeed e j e e j e e j e e j e eeee j fe e j e e j e ee e j eee j feeeeefd ddZd e j e ee e j e e j e e j dfd d ZZS) CsmDepthDecoderForCausalLMNcs>t|t||_|j|_t|j|j|j|_| dSry) r:r;rrrrrBrcodebooks_headrrxrDr6r7r;s   z#CsmDepthDecoderForCausalLM.__init__r) rrrrnr%rlabelsrrlogits_to_keeprrc  Ks|jf|||||||| d| } | d} t| trV| dkrHtdd}qZt| d}n| }|| dd|ddf| dur| |nd}|}d}|dur|dddf}|jf|d|jj|d| }t ||| j | j | j dS)a backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. )rrrrnr%rrrrrN.)r$rr shift_labels)r#r$r%r&r') rr\rslicerr loss_functionrTrrr%r&r')rArrrrnr%rrrrrroutputsr& slice_indicesr$r#rr6r6r7rL!sJ    & z"CsmDepthDecoderForCausalLM.forwardrr%rrrc  sHtj|||||fi|}|dddk}|s:|d|d|S)Nrrrrn)r:prepare_inputs_for_generationpop) rArr%rrrr model_inputsZis_first_generation_steprDr6r7res    z8CsmDepthDecoderForCausalLM.prepare_inputs_for_generation) NNNNNNNNNr)NNNN)r.r/r0_tied_weights_keysZ_tp_planZ_pp_planr;rrr2rrr3rorr listrrrrr5rrLrrPr6r6rDr7r sR   Ercs$eZdZfddZddZZS)CsmBackboneModelEmbeddingscsDtt|j|j|j|_|jdt |j|jdddS)Naudio_tokens_offsetsFrY) r:r;r<rrrrBembed_audio_tokensr`r2rrxrDr6r7r;}s  z#CsmBackboneModelEmbeddings.__init__cCs |||j}|jdd}|S)Nrrd)rrsum)rArrr6r6r7rLs z"CsmBackboneModelEmbeddings.forwardrzr6r6rDr7r|s rc sneZdZfddZeedeejeej eejee eej eejee e eed ddZZS)CsmBackboneModelcsvtj|_j|_t|_tfddt j D|_ t j jd|_td|_d|_|dS)Ncsg|]}t|qSr6rrrSr6r7rrz-CsmBackboneModel.__init__..rrSF)r:r;rrrrrr<rrrrr8rBrrrQrrrrxrDrSr7r;s   zCsmBackboneModel.__init__N) rrrnr%rrrrrc Ks|du|duArtd|dur*||}|rB|durBt|jd}|durz|durZ|nd} tj| | |jd|jd}|dur| d}t |j|||||d} |} | | |} |j d|jj D] } | | f| |||| d|} q|| } t| |d S) a& input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Nz:You must specify exactly one of input_ids or inputs_embedsrSrrrr)rrnr%rrr)rrr rTrr2rrMrar}r rrrrr)rArrrnr%rrrrrrr&rrr6r6r7rLsP       zCsmBackboneModel.forward)NNNNNNN)r.r/r0r;rrrr2rror r3rrrrrLrPr6r6rDr7rs* rz The Csm model consists of two llama-like auto-regressive transformer models: a backbone model that predicts the first codebook token and a depth decoder that predicts the other codebook tokens. csJeZdZddgZfddZddZddZd d Zefd d Z fd dZ de e j e e j e e j e e j e e j dddZde je ee e je e je e jdfdd Zeede je e j e e j e e j e e je eeee jfe e je e je ee e jeee j feeeeefd ddZZS)CsmForConditionalGenerationz5backbone_model.embed_tokens.embed_audio_tokens.weightz'depth_decoder.model.embed_tokens.weightcspt||j|_tj|j|jdd|_t|j|j|_ t ||_ t |j|_t|j|_|dS)NFrr)r:r;rr<rsrBlm_headrZtext_vocab_sizeembed_text_tokensr _from_configbackbone_modelrZdepth_decoder_config depth_decoderr from_configZ codec_config codec_modelrrxrDr6r7r;s  z$CsmForConditionalGeneration.__init__cCs|jjSryr rrNr6r6r7get_input_embeddingssz0CsmForConditionalGeneration.get_input_embeddingscCs ||j_dSryr )rArr6r6r7set_input_embeddingssz0CsmForConditionalGeneration.set_input_embeddingscCs$|jjr ||jjj|jjjdSry)rTZtie_codebooks_embeddingsZ_tie_or_clone_weightsr rrr rrNr6r6r7 _tie_weightss z(CsmForConditionalGeneration._tie_weightscs|ddr$tj|i|\}}ntj|i|}dtfddt|jD}t|jjddi||D]}t |j|q~d|vr||fS|SdS)NZoutput_loading_infoFdepth_decoder_cs(i|] \}}|r|d|qSry) startswith)rattrrprefix prefix_lenr6r7 s z?CsmForConditionalGeneration.from_pretrained..Z_from_model_config) r^r:from_pretrainedlenvarsgeneration_configitemsr rdelattr)clsargsrrZ loading_infodepth_decoder_attrsrrDrr7rs   z+CsmForConditionalGeneration.from_pretrainedcsVd}|jj}|dd|D]\}}t|j|||q$tj|i|dS)NrZtransformers_version)r rZ to_diff_dictrrsetattrr:save_pretrained)rArrrr rrrDr6r7r"!s   z+CsmForConditionalGeneration.save_pretrainedN)r input_valuesinput_values_cutoffsrrcsH||}|dur>tj|d}||dk}||dk}tj||jd t |d}|| dk}t g}t ||D]~\} } | | dk} t| jddD]V} | | } | | d} | d| | f}|j| d}|jdd}||dqqtdd |Dtfd d |D}|j|}Wdn1sP0Y|jj}||k}|j|}||||<tjdd|jjf|jtjd |jj}|j|d}||jj k}|!|"d||<|dur>| d!dd|jj}||||<|||<|d kj#dd}d||d|dddf<|}||dS)a Merges the input_ids and input_values to produce a single inputs_embeds tensor: 1 - Infers the codec model on the input_values to retreive codebook token. 2 - Embeds codebook tokens and places them at the correct positions in the inputs_embeds tensor. 3 - If labels are provided, expands them to match codebook dimensions and position the target codebook tokens in the inputs_embeds tensor. Args: input_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`): The input ids to embed. input_values (`torch.Tensor` of shape `(batch_size, channels, audio_sequence_length)`): The audio input values to embed. input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`): The cutoffs of the audio input values relative to its batch index, padded with -1 when no audio. NrrrrrFr.css|]}|jdVqdS)rN)rMrelr6r6r7 [rzQCsmForConditionalGeneration._merge_input_ids_with_input_values..c s,g|]$}tj|ddd|jdfqS)r)r<rpadrMr&Zmax_audio_framesr6r7r]rzRCsmForConditionalGeneration._merge_input_ids_with_input_values..)rarGiTas_tuple)rr)$rr<rr)diffr2rmaxrargrr}rpziprrMr encodeZ audio_codesriappendrZget_audio_codes_maskrTaudio_token_idr rr>rlongZcodebook_eos_token_idZsqueezeZaudio_eos_token_idrepeatrnonzero)rArr#r$rrZ audio_lengthsZinput_values_maskZaudio_tokens_listZbatch_input_valuesZbatch_input_values_cutoffsrZ start_idxZend_idxZ audio_batchZ codec_outputsZ codebook_idsZbatched_audio_token_idsZaudio_codes_maskr3Zaudio_token_maskZ audio_embedsZaudio_eos_frame_idsZaudio_eos_embedsZaudio_eos_token_maskZlabels_expandedZ depth_decoder_ignore_frames_idxsr6r*r7"_merge_input_ids_with_input_values+sX      ,     z>CsmForConditionalGeneration._merge_input_ids_with_input_valuesrc  stjf|||||d|}|dur||jdkr||ddur||j||d|d|dd}||d|ddd|S) Nrrrr#r$r)rr#r$r)rrr)r:rndimr^r7r) rArr%rrrrr merged_inputsrDr6r7r}s(  z9CsmForConditionalGeneration.prepare_inputs_for_generationr) rr#rr$rnr%rrrrrrrc  Ks|dur6|jdkr6|||||} | d}| d}d}|jf|||||| | d| }|d}t| trtt| dn| }||dd|ddf}d}d}d}d}|dur|dddddf}|jf|||jj d| }|ddddddfd kj d d }||d d|jj df}t j j|d dd}|jdd}||d|ddddf}||}|jf||| d|d| }|j}||}t|||||j|j|j|dur|jnd|dur|jnd|dur|jnd|dur|jndd S)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`, *optional*): Specify the end positions of audio segments within each batch entry, relative to the concatenated audio input. If a batch entry has fewer segments than the maximum, it is padded with -1. For example, in a batch of 2 sequences where the first contains 2 audio segments of length l1, and the second contains 1 audio segment of length l2, the input_values_cutoffs would be: [[l1, 2 * l1], [l2, -1]]. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[config.audio_token_id, -100, -101]`. Requires targeted `input_values` to be provided as audio tokens will be inferred from it using the `codec_model`. - `config.audio_token_id` indicates an audio frames (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) Such labels can be prepared using `output_labels=True` when calling [`CsmProcessor`]. logits_to_keep (`int` or `torch.Tensor`, *optional*): Kept for compatibility. Does not support another value than: 1. `0`, which is equivalent to keeping all logits, used in the training regime 2. `1`, which is equivalent to keeping only the last logit, used in the generation regime Example: ```python >>> import torch >>> from transformers import CsmForConditionalGeneration, AutoProcessor >>> from datasets import load_dataset, Audio >>> model_id = "sesame/csm-1b" >>> torch_device = "cuda" if torch.cuda.is_available() else "cpu" >>> processor = AutoProcessor.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") >>> # ensure the audio is 24kHz >>> ds = ds.cast_column("audio", Audio(sampling_rate=24000)) >>> conversation = [] >>> # prepare a conversation with text and corresponding audio >>> for text, audio, speaker_id in zip(ds[:4]["text"], ds[:4]["audio"], ds[:4]["speaker_id"]): ... conversation.append( ... { ... "role": f"{speaker_id}", ... "content": [{"type": "text", "text": text}, {"type": "audio", "path": audio["array"]}], ... } ... ) >>> inputs = processor.apply_chat_template( ... conversation, ... tokenize=True, ... return_dict=True, ... output_labels=True, ... ).to(torch_device) >>> model = CsmForConditionalGeneration.from_pretrained(model_id, device_map=torch_device) >>> output = model(**inputs) >>> output.loss.backward() ```Nrrr)rrrnr%rrrr)r$rrrr-rFrd.r%)rTr+)rrrZ return_dictr) r#r-r(r$r%r&r'r)r*r+r,)r8r7r r\rrrrrTrallrr<rr)r6r r#r"r%r&r'r$)rArr#rr$rnr%rrrrrrr9Zbackbone_outputsZbackbone_hidden_statesrZbackbone_logitsr#r-r(Zdepth_decoder_outputsZbackbone_labelsZ train_maskZdepth_decoder_input_idsZ train_idxsZbackbone_last_hidden_statesZdepth_decoder_labelsr6r6r7rLsS   (    z#CsmForConditionalGeneration.forward)NNNN)NNNN) NNNNNNNNNNr)r.r/r0rr;rrr classmethodrr"rr2ror7rr r3rrrrrrrrrr5r"rLrPr6r6rDr7rsx   U  r)rrrrr)Nr)r)K dataclassesrtypingrrrr2Ztorch.nnr<Ztransformers.utils.genericrZ activationsrZ cache_utilsr r Z generationr Z integrationsr Z masking_utilsr Zmodeling_layersrZmodeling_outputsrrZmodeling_rope_utilsrrZmodeling_utilsrrZprocessing_utilsrutilsrrrrrZutils.deprecationrautorZconfiguration_csmrr Zgeneration_csmr!Z get_loggerr.rr"Moduler8rQrqr|rrorrrfrrrrrrrrrr__all__r6r6r6r7s             ,$  G.jiYU