a hW-@s@ddlmZmZddlZddlmZddlmZddlmZddl m Z ddl m Z m Z mZdd lmZdd lmZmZmZdd lmZd d lmZmZd dlmZmZmZmZddlm Z GdddeZ!GdddeZ"GdddeZ#eddGdddeZ$Gdddej%Z&eddGddde#e Z'gd Z(dS)!)OptionalUnionN)nn)ACT2FN)Cache)GenerationMixin)BaseModelOutputBaseModelOutputWithPastCausalLMOutputWithPast)Unpack)TransformersKwargsauto_docstringcan_return_tuple)check_model_inputs) AutoModelAutoModelForCausalLM)Qwen2AudioAttentionQwen2AudioEncoderQwen2AudioEncoderLayerQwen2AudioPreTrainedModel) VoxtralConfigc@s eZdZdS)VoxtralAttentionN__name__ __module__ __qualname__rrg/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/voxtral/modular_voxtral.pyr&src@s eZdZdS)VoxtralEncoderLayerNrrrrr r!*sr!c@s$eZdZdZdZdZdZdZdZdS)VoxtralPreTrainedModelTN)rrrZ_supports_flex_attnZ_supports_cache_classZ_supports_attention_backendZ_can_compile_fullgraphZ_no_split_modulesrrrr r".s r"z: The Voxtral encoder, which is a Whisper encoder. )Z custom_introc@s.eZdZeedZedeedddZ dS)VoxtralEncoder)Z attentions hidden_statesN)kwargsc Ks|jj|jjd|jjd}|jd|krPtd|d|jdd|d|j|jjj |jjj d}t j ||}t j ||}|ddd }|jj}|||j }t j j||j|jd }t|jD]\}} | ||d d } | d}q||}t|d S)a Args: input_features (`torch.LongTensor` of shape `(batch_size, feature_size, sequence_length)`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] attention_mask (`torch.Tensor`)`, *optional*): Voxtral does not support masking of the `input_features`, this argument is preserved for compatibility, but it is not used. By default the silence in the input log mel spectrogram are ignored. rz:Qwen2Audio expects the mel input features to be of length z , but found z-. Make sure to pad the input mel features to .)dtypedevicerr)ptrainingN)attention_maskZlayer_head_mask)last_hidden_state)configZmax_source_positionsZconv1ZstrideZconv2shape ValueErrortoweightr(r)rZ functionalZgeluZpermuteembed_positionsZdropoutr+ enumerateZlayersZ layer_normr ) selfinput_featuresr,r%Zexpected_seq_length inputs_embedsZ embed_posr$idxZ encoder_layerZ layer_outputsrrr forwardCs.   zVoxtralEncoder.forward)N) rrrrr!Z_can_record_outputsrr r r9rrrr r#8sr#cs*eZdZedfdd ZddZZS)VoxtralMultiModalProjector)r.csNttj|jj|jjdd|_t |j |_ tj|jj|jjdd|_ dS)NF)Zbias) super__init__rZLinear audio_configintermediate_size text_configZ hidden_sizelinear_1rZprojector_hidden_actactlinear_2r5r. __class__rr r<us  z#VoxtralMultiModalProjector.__init__cCs"||}||}||}|SN)r@rArB)r5Zaudio_featuresr$rrr r9{s   z"VoxtralMultiModalProjector.forward)rrrrr<r9 __classcell__rrrDr r:tsr:zs The Voxtral model, which consists of Whisper encoder, a multi-modal projector and a LLama language model. cseZdZdgZddiZddgdgfiZdgZfddZd d Zd d Z d dZ ddZ ddZ ddZ ejdddZeedeejeejeejeejeeeejeejeeeejeeejfeeed ddZfddZZS) VoxtralForConditionalGenerationzlm_head.weightZlm_headZ colwise_repr$Zlogitsr3csHt||jj|_t|j|_t|j|_ t ||_ | dSrF) r;r<r?Z vocab_sizer from_configr= audio_towerrlanguage_modelr:multi_modal_projectorZ post_initrCrDrr r<s    z(VoxtralForConditionalGeneration.__init__cCs |jSrF)rKget_input_embeddingsr5rrr rMsz4VoxtralForConditionalGeneration.get_input_embeddingscCs|j|dSrF)rKset_input_embeddings)r5valuerrr rOsz4VoxtralForConditionalGeneration.set_input_embeddingscCs |jSrF)rKget_output_embeddingsrNrrr rQsz5VoxtralForConditionalGeneration.get_output_embeddingscCs|j|dSrF)rKset_output_embeddings)r5Znew_embeddingsrrr rRsz5VoxtralForConditionalGeneration.set_output_embeddingscCs|j|dSrF)rK set_decoder)r5decoderrrr rSsz+VoxtralForConditionalGeneration.set_decodercCs |jSrF)rK get_decoderrNrrr rUsz+VoxtralForConditionalGeneration.get_decoder)r6cCs0||}|j}|d|jjj}||}|S)a This method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector. Args: input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] Returns: `torch.FloatTensor`: The audio embeddings. r&)rJr-Zreshaper.r=r>rL)r5r6Z audio_outputsZaudio_hidden_states audio_embedsrrr get_audio_embedss   z0VoxtralForConditionalGeneration.get_audio_embedsNr) input_idsr6r, position_idspast_key_valuesr7labels use_cachecache_positionlogits_to_keepr%returnc  Ks`|dur||}|dur:||} ||jjk} | || <|jf||||||| | d| }|S)aj Example: ```python >>> from transformers import VoxtralForConditionalGeneration, AutoProcessor >>> import torch >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> repo_id = "mistralai/Voxtral-Mini-3B-2507" >>> processor = AutoProcessor.from_pretrained(repo_id) >>> model = VoxtralForConditionalGeneration.from_pretrained(repo_id, dtype=torch.bfloat16, device_map=device) >>> conversation = [ { "role": "user", "content": [ { "type": "audio", "url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/dude_where_is_my_car.wav", }, {"type": "text", "text": "What can you tell me about this audio?"}, ], } ] >>> inputs = processor.apply_chat_template(conversation) >>> inputs = inputs.to(device, dtype=torch.bfloat16) >>> outputs = model.generate(**inputs, max_new_tokens=30) >>> processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True) ["This audio is a humorous conversation between two friends, likely in English, where one of them is trying to figure out what the other's tattoo says."] ```N)r,rYrZr7r[r\r]r^)rMrWr.Zaudio_token_idrK)r5rXr6r,rYrZr7r[r\r]r^r%rVZaudio_token_maskoutputsrrr r9s&1     z'VoxtralForConditionalGeneration.forwardcsH|dd}|d}tj|i|}|durD|ddkrD||d<|S)Nr6r]r)popgetr;prepare_inputs_for_generation)r5argsr%r6r]Z model_inputsrDrr rcs   z=VoxtralForConditionalGeneration.prepare_inputs_for_generation) NNNNNNNNNr) rrrZ_tied_weights_keysZ_tp_planZ_pp_planZ_keep_in_fp32_modules_strictr<rMrOrQrRrSrUtorchZ FloatTensorrWrrrZ LongTensorZTensorrboolrintr r r r9rcrGrrrDr rHsN  FrH)r"r#rH))typingrrrerZ activationsrZ cache_utilsrZ generationrZmodeling_outputsr r r Zprocessing_utilsr utilsr rrZ utils.genericrautorrZ qwen2_audio.modeling_qwen2_audiorrrrZconfiguration_voxtralrrr!r"r#Moduler:rH__all__rrrr s4        7