a ½ Àhù5ã@s\dZddlmZddlmZddlmZe e¡Z Gdd„deƒZ Gdd„deƒZ dgZ d S) zEvolla model configurationé)ÚPretrainedConfig)Úrope_config_validation)Úloggingcs"eZdZdZd‡fdd„ Z‡ZS)Ú SaProtConfigaÊ This is the configuration class to store the configuration of a [`EvollaSaProtProteinEncoder`]. It is used to instantiate a SaProt model according to the specified arguments, defining the model architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 446): Vocabulary size of the protein sequence model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`EvollaModel`]. mask_token_id (`int`, *optional*, defaults to 4): The id of the *mask* token in the protein sequence model. pad_token_id (`int`, *optional*, defaults to 1): The id of the *padding* token in the protein sequence model. hidden_size (`int`, *optional*, defaults to 1280): Dimensionality of the protein sequence model layers and the pooler layer. num_hidden_layers (`int`, *optional*, defaults to 33): Number of hidden layers in the protein sequence model. num_attention_heads (`int`, *optional*, defaults to 20): Number of attention heads for each attention layer in the protein sequence model. intermediate_size (`int`, *optional*, defaults to 5120): Dimensionality of the intermediate layers in the protein sequence model. hidden_dropout_prob (`float`, *optional*, defaults to 0.1): The dropout ratio for the hidden layers in the protein sequence model. attention_probs_dropout_prob (`float`, *optional*, defaults to 0.1): The dropout ratio for the attention probabilities in the protein sequence model. max_position_embeddings (`int`, *optional*, defaults to 1026): The maximum sequence length that the protein sequence model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). layer_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon value for the layer normalization layer in the protein sequence model. position_embedding_type (`str`, *optional*, defaults to `"rotary"`): The type of position embedding to use in the protein sequence model. Currently only `"rotary"` is supported. emb_layer_norm_before (`bool`, *optional*, defaults to `False`): Whether to apply layer normalization before the position embedding in the protein sequence model. token_dropout (`bool`, *optional*, defaults to `True`): Whether to apply dropout to the tokens in the protein sequence model.é¾éééé!ééçš™™™™™¹?éç{®Gáz”?çñh㈵øä>ÚrotaryFTc sjtƒjf||dœ|¤Ž||_||_||_||_||_||_| |_| |_ | |_ | |_ | |_ ||_ ||_dS)N)Ú pad_token_idÚ mask_token_id)ÚsuperÚ__init__Ú vocab_sizeÚ hidden_sizeÚnum_hidden_layersÚnum_attention_headsÚintermediate_sizeÚhidden_dropout_probÚattention_probs_dropout_probÚmax_position_embeddingsÚinitializer_rangeÚlayer_norm_epsÚposition_embedding_typeÚemb_layer_norm_beforeÚ token_dropout)Úselfrrrrrrrrrrrrr r!r"Úkwargs©Ú __class__©úk/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/evolla/configuration_evolla.pyr@szSaProtConfig.__init__)rrrr r r r r r rrrrFT)Ú__name__Ú __module__Ú __qualname__Ú__doc__rÚ __classcell__r'r'r%r(rs"(ðrcs.eZdZdZdZdeiZd‡fdd„ Z‡ZS)Ú EvollaConfiga¹ This is the configuration class to store the configuration of a [`EvollaModel`]. It is used to instantiate an Evolla model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Evolla-10B. e.g. [westlake-repl/Evolla-10B-hf](https://huggingface.co/westlake-repl/Evolla-10B-hf) Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: protein_encoder_config (`dict`, *optional*): Dictionary of configuration options used to initialize [`SaProtConfig`]. vocab_size (`int`, *optional*, defaults to 128256): Vocabulary size of the Evolla llama model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`EvollaModel`]. hidden_size (`int`, *optional*, defaults to 4096): Dimensionality of the llama layers and the pooler layer. intermediate_size (`int`, *optional*, defaults to 14336): Dimensionality of the intermediate layers in the llama model. num_hidden_layers (`int`, *optional*, defaults to 32): Number of hidden layers in the llama model. num_attention_heads (`int`, *optional*, defaults to 32): Number of attention heads for each attention layer in the llama model. num_key_value_heads (`int`, *optional*, defaults to 8): Number of key-value pairs for each attention layer in the llama model. hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the llama model. If string, `"gelu"`, `"relu"`, `"selu"` and `"silu"` are supported. max_position_embeddings (`int`, *optional*, defaults to 8192): The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). rms_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon value for the RMS-norm layer in the llama model. rope_theta (`float`, *optional*, defaults to 500000.0): The threshold value for the RoPE layer in the llama model. rope_scaling (`float`, *optional*): The scaling factor for the RoPE layer in the llama model. attention_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the attention layer. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention layer. mlp_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the MLP layer. aligner_ffn_mult (`int`, *optional*, defaults to 4): The FFN multiplier for the aligner layer. aligner_enable_bias (`bool`, *optional*, defaults to `True`): Whether to use bias in the aligner layer. aligner_attention_probs_dropout_prob (`float`, *optional*, defaults to 0.1): The dropout ratio for the attention probabilities in the aligner layer. aligner_num_add_layers (`int`, *optional*, defaults to 8): The number of additional layers for the aligner layer. resampler_depth (`int`, *optional*, defaults to 6): The depth of the resampler layer in the llama model. resampler_dim_head (`int`, *optional*, defaults to 64): The dimension of the heads in the resampler layer in the llama model. resampler_heads (`int`, *optional*, defaults to 8): The number of heads in the resampler layer in the llama model. resampler_num_latents (`int`, *optional*, defaults to 64): The number of latents in the resampler layer in the llama model. resampler_ff_mult (`int`, *optional*, defaults to 4): The FFN multiplier for the resampler layer. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. pad_token_id (`int`, *optional*): The id of the *padding* token. bos_token_id (`int`, *optional*, defaults to 128000): The id of the *beginning-of-sequence* token. eos_token_id (`int`, *optional*, defaults to 128009): The id of the *end-of-sequence* token. use_cache (`bool`, *optional*, defaults to `False`): Whether or not the model should return the last key/values attentions (not used by all models). tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether or not to tie the input and output word embeddings. Example: ```python >>> from transformers import EvollaModel, EvollaConfig >>> # Initializing a Evolla evolla-10b style configuration >>> configuration = EvollaConfig() >>> # Initializing a model from the evolla-10b style configuration >>> model = EvollaModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```Z EvollaModelÚprotein_encoder_configNéõéé8é éÚsilué r瀄AFçrTr éé@réôé ôc  s||_||_||_||_||_||_||_| |_| |_||_ | |_ ||_ ||_ ||_ ||_||_||_||_||_||_||_||_||_||_| |_| |_|jdurÀd|jvrÀ|jd|jd<t|ƒ|durÞi}t d¡tfi|¤Ž|_tƒj f||||dœ|¤ŽdS)NÚtypeZ rope_typezX`protein_encoder_config` is `None`. Initializing the `SaProtConfig` with default values.)rÚ bos_token_idÚ eos_token_idÚtie_word_embeddings)!rrrrrÚnum_key_value_headsÚ hidden_actrÚ rms_norm_epsr@Úattention_biasÚattention_dropoutÚmlp_biasÚaligner_ffn_multÚaligner_enable_biasÚ$aligner_attention_probs_dropout_probÚaligner_num_add_layersÚ use_cacherÚresampler_depthÚresampler_dim_headÚresampler_headsÚresampler_num_latentsÚresampler_ff_multÚ rope_thetaÚ rope_scalingrÚloggerÚinforr/rr) r#r/rrrrrrArBrrCrQrRrDrErFrGrHrIrJrLrMrNrOrPrrr>r?rKr@r$r%r'r(rÂsR" üûzEvollaConfig.__init__)Nr0r1r2r3r3r4r5r6rr7NFr8FrTr r4r9r:r4r:rrNr;r<FF) r)r*r+r,Z model_typerZ sub_configsrr-r'r'r%r(r.dsDZár.N) r,Zconfiguration_utilsrZmodeling_rope_utilsrÚutilsrZ get_loggerr)rSrr.Ú__all__r'r'r'r(Ús    K2