a ¾ Àh!ã@sHddlmZddlmZmZGdd„deƒZGdd„deƒZddgZdS) é)ÚPretrainedConfigé)ÚCONFIG_MAPPINGÚ AutoConfigc s6eZdZdZdZddddddœZd‡fdd„ Z‡ZS)ÚVoxtralEncoderConfiga` This is the configuration class to store the configuration of a [`VoxtralEncoder`]. It is used to instantiate a Voxtral audio encoder according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the audio encoder of the Voxtral architecture. e.g. [mistralai/Voxtral-Mini-3B-2507](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507) Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 51866): Vocabulary size of the model. hidden_size (`int`, *optional*, defaults to 1280): Dimensionality of the hidden representations. intermediate_size (`int`, *optional*, defaults to 5120): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 32): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 20): Number of attention heads for each attention layer in the Transformer encoder. scale_embedding (`bool`, *optional*, defaults to `False`): Scale embeddings by dividing by sqrt(hidden_size) if True. activation_function (`str`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu", num_mel_bins (`int`, *optional*, defaults to 128): Number of mel features used per input features. Should correspond to the value used in the `VoxtralProcessor` class. max_source_positions (`int`, *optional*, defaults to 1500): The maximum sequence length of log-mel filter-bank features that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. ```python >>> from transformers import VoxtralEncoderConfig, VoxtralEncoder >>> # Initializing a VoxtralEncoderConfig >>> configuration = VoxtralEncoderConfig() >>> # Initializing a VoxtralEncoder (with random weights) >>> model = VoxtralEncoder(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```Úvoxtral_encoderÚ hidden_sizeÚnum_hidden_layersÚnum_attention_headsÚintermediate_sizeÚ layerdrop)Zd_modelZencoder_layersZencoder_attention_headsZencoder_ffn_dimZencoder_layerdropéšÊééé éFÚgelué€éÜç{®Gáz”?çc  sjtƒjfi| ¤Ž||_||_||_||_||_||_||_||_ | |_ | |_ d|_ d|_ d|_| |_dS)Nr)ÚsuperÚ__init__Ú vocab_sizerr r r Úscale_embeddingÚactivation_functionÚ num_mel_binsÚmax_source_positionsÚinitializer_rangeZdropoutr Zactivation_dropoutÚattention_dropout) Úselfrrr r r rrrrrrÚkwargs©Ú __class__©úm/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/voxtral/configuration_voxtral.pyrPszVoxtralEncoderConfig.__init__) r rrrrFrrrrr)Ú__name__Ú __module__Ú __qualname__Ú__doc__Ú model_typeZ attribute_maprÚ __classcell__r$r$r"r%rs(1û ôrc sJeZdZdZdZeedœZddddddd d d d d œ Zd‡fdd„ Z‡Z S)Ú VoxtralConfiga3 This is the configuration class to store the configuration of a [`VoxtralForConditionalGeneration`]. It is used to instantiate an Voxtral model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Voxtral-Mini-3B. e.g. [mistralai/Voxtral-Mini-3B-2507](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507) Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: audio_config (`Union[AutoConfig, dict]`, *optional*): The config object or dictionary of the audio encoder. text_config (`Union[AutoConfig, dict]`, *optional*): The config object or dictionary of the text model. audio_token_id (`int`, *optional*): The image token index to encode the image prompt. projector_hidden_act (`str`, *optional*, defaults to `"gelu"`): The activation function (function or string) in the multi-modal projector. ```python >>> from transformers import VoxtralForConditionalGeneration, VoxtralConfig >>> # Initializing a Voxtral configuration >>> configuration = VoxtralConfig(audio_token_id=24, projector_hidden_act="gelu") >>> # Initializing a 3B model with random weights >>> model = VoxtralForConditionalGeneration(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```Zvoxtral)Ú text_configÚ audio_configii i éégñh㈵øä>Tg„×—Ar) rrr r Znum_key_value_headsZmax_position_embeddingsZ rms_norm_epsZ use_cacheZ rope_thetaZhead_dimNrc sÚt|tƒr2| dd¡|d<t|dfi|¤Ž}n|durDtdƒ}||_t|tƒr†| dd¡|d<t|dfii|j¥|¥¤Ž}n|dur¢tdfi|j¤Ž}||_|j|_|j|_||_ ||_ t ƒj fi|¤ŽdS)Nr*rÚllama) Ú isinstanceÚdictÚgetrr.Ú_default_text_config_kwargsr-rrÚaudio_token_idÚprojector_hidden_actrr)r r.r-r6r7r!r"r$r%r©s&    ÿzVoxtralConfig.__init__)NNNr) r&r'r(r)r*rZ sub_configsr5rr+r$r$r"r%r,ws&! öûr,N)Zconfiguration_utilsrÚautorrrr,Ú__all__r$r$r$r%Ús cR