a hs.@sddlmZddlZddlZddlZddlZddlmZddlm Z zddlm Z Wne yrddl m Z Yn0ddl ZddlZddlmZddlmZddlmZdd lmZdd lmZdd lmZeeZGd d d eZdS)) annotationsN)Path)Any)Self) save_file) Tokenizer)nn)PreTrainedTokenizerFast) InputModule)get_device_namecseZdZd3dddddfdd Zd d d d d Zd d dddZeddddZddddZddddddddZ e d4ddd d!d!dd"d#d$d%Z e d5dd*d!ddd+d!ddd,dd- d.d/Z e ddd0d1d2Z ZS)6StaticEmbeddingNz#Tokenizer | PreTrainedTokenizerFastz np.ndarray | torch.Tensor | Nonez int | NoneNone) tokenizerembedding_weights embedding_dimreturnc stt|tr|j}nt|ts.td|dur`t|tjrLt |}t j j |dd|_n$|dur|t |||_ntd|jj|_|jj|_||_|j|dd|_dS)a4 Initializes the StaticEmbedding model given a tokenizer. The model is a simple embedding bag model that takes the mean of trained per-token embeddings to compute text embeddings. Args: tokenizer (Tokenizer | PreTrainedTokenizerFast): The tokenizer to be used. Must be a fast tokenizer from ``transformers`` or ``tokenizers``. embedding_weights (np.ndarray | torch.Tensor | None, optional): Pre-trained embedding weights. Defaults to None. embedding_dim (int | None, optional): Dimension of the embeddings. Required if embedding_weights is not provided. Defaults to None. .. tip:: Due to the extremely efficient nature of this module architecture, the overhead for moving inputs to the GPU can be larger than the actual computation time. Therefore, consider using a CPU device for inference and training. Example:: from sentence_transformers import SentenceTransformer from sentence_transformers.models import StaticEmbedding from tokenizers import Tokenizer # Pre-distilled embeddings: static_embedding = StaticEmbedding.from_model2vec("minishlab/potion-base-8M") # or distill your own embeddings: static_embedding = StaticEmbedding.from_distillation("BAAI/bge-base-en-v1.5", device="cuda") # or start with randomized embeddings: tokenizer = Tokenizer.from_pretrained("FacebookAI/xlm-roberta-base") static_embedding = StaticEmbedding(tokenizer, embedding_dim=512) model = SentenceTransformer(modules=[static_embedding]) embeddings = model.encode(["What are Pandas?", "The giant panda, also known as the panda bear or simply the panda, is a bear native to south central China."]) similarity = model.similarity(embeddings[0], embeddings[1]) # tensor([[0.8093]]) (If you use potion-base-8M) # tensor([[0.6234]]) (If you use the distillation method) # tensor([[-0.0693]]) (For example, if you use randomized embeddings) Raises: ValueError: If the tokenizer is not a fast tokenizer. ValueError: If neither `embedding_weights` nor `embedding_dim` is provided. zThe tokenizer must be fast (i.e. Rust-backed) to use this class. Use Tokenizer.from_pretrained() from `tokenizers` to load a fast tokenizer.NF)freezez?Either `embedding_weights` or `embedding_dim` must be provided. base_model)super__init__ isinstancer _tokenizerr ValueErrornpndarraytorch from_numpyrZ EmbeddingBagfrom_pretrained embeddingZget_vocab_sizeZnum_embeddingsrrZ no_paddinggetr)selfrrrkwargs __class__h/var/www/html/assistant/venv/lib/python3.9/site-packages/sentence_transformers/models/StaticEmbedding.pyrs&3        zStaticEmbedding.__init__z list[str]zdict[str, torch.Tensor])textsrc Ksj|jj|dd}dd|D}ttdgdd|ddD}tjdd|Dtjd }||d S) NF)Zadd_special_tokenscSsg|] }|jqSr$)Zids).0encodingr$r$r% oz,StaticEmbedding.tokenize..rcSsg|] }t|qSr$)len)r' token_idsr$r$r%r)qr*cSsg|]}|D]}|q qSr$r$)r'r,Ztoken_idr$r$r%r)rr*)Zdtype) input_idsoffsets)rZ encode_batchrrrZcumsumZtensorlong)r r&r! encodingsZ encodings_idsr/r.r$r$r%tokenizems (zStaticEmbedding.tokenize)featuresrcKs||d|d|d<|S)Nr.r/Zsentence_embedding)r)r r3r!r$r$r%forwarduszStaticEmbedding.forwardint)rcCstjSN)mathinfr r$r$r%max_seq_lengthyszStaticEmbedding.max_seq_lengthcCs|jSr6)rr9r$r$r% get_sentence_embedding_dimension}sz0StaticEmbedding.get_sentence_embedding_dimensionT)safe_serializationstrbool) output_pathr<rcOsT|rt|tj|dnt|tj|d|jtt |ddS)Nzmodel.safetensorszpytorch_model.bintokenizer.json) save_safetensors_fileZ state_dictospathjoinrsaverr=r)r r?r<argsr!r$r$r%rEszStaticEmbedding.saveFzbool | str | Nonez str | Noner)model_name_or_path subfoldertoken cache_folderrevisionlocal_files_onlyrc Ksx|||||d}|j|fddi|} t| } |jfd|i|} z | d} Wntyj| d} Yn0t| | dS)N)rIrJrKrLrMfilenamer@rHzembedding.weightZ embeddings)r)Zload_file_pathr from_fileZload_torch_weightsKeyErrorr ) clsrHrIrJrKrLrMr!Z hub_kwargsZtokenizer_pathrweightsr$r$r%loads    zStaticEmbedding.load-C6? \[unused\d+\]float32zlist[str] | Nonez float | Noner) model_name vocabularydevicepca_dims apply_zipfsif_coefficienttoken_remove_pattern quantize_to use_subwordr!rc  szddlm} Wnty*tdYn0t| } t| jdh||||| |||d| } t| } rt dd t t | dfd d | D} t}| |fi| }t|jtjrt|j}n|jj}|j}||||d S) ak Creates a StaticEmbedding instance from a distillation process using the `model2vec` package. Args: model_name (str): The name of the model to distill. vocabulary (list[str] | None, optional): A list of vocabulary words to use. Defaults to None. device (str): The device to run the distillation on (e.g., 'cpu', 'cuda'). If not specified, the strongest device is automatically detected. Defaults to None. pca_dims (int | None, optional): The number of dimensions for PCA reduction. Defaults to 256. apply_zipf (bool): Whether to apply Zipf's law during distillation. Defaults to True. sif_coefficient (float | None, optional): The coefficient for SIF weighting. Defaults to 1e-4. token_remove_pattern (str | None, optional): A regex pattern to remove tokens from the vocabulary. Defaults to r"\[unused\d+\]". quantize_to (str): The data type to quantize the weights to. Defaults to 'float32'. use_subword (bool): Whether to use subword tokenization. Defaults to True. Returns: StaticEmbedding: An instance of StaticEmbedding initialized with the distilled model's tokenizer and embedding weights. Raises: ImportError: If the `model2vec` package is not installed. r)distillz\To use this method, please install the `model2vec` package: `pip install model2vec[distill]`rX)rYrZr[r\r`r_r]r^z1Your version of `model2vec` does not support the z, zh arguments for the `distill` method. Consider updating `model2vec` to take advantage of these arguments.csi|]\}}|vr||qSr$r$)r'keyvalueZdistill_kwargsr$r% r*z5StaticEmbedding.from_distillation..rr)Zmodel2vec.distillra ImportErrorinspect signatureset parameterskeysloggerwarningrDmaprepritemsr rrrrrrweightr)rQrXrYrZr[r\r]r^r_r`r!raZdistill_signatureZ leftovers static_modelrrr$rdr%from_distillations>&     z!StaticEmbedding.from_distillation)model_id_or_pathrcCsnzddlm}Wnty*tdYn0||}t|jtjrRt |j}n|jj }|j }||||dS)aH Create a StaticEmbedding instance from a model2vec model. This method loads a pre-trained model2vec model and extracts the embedding weights and tokenizer to create a StaticEmbedding instance. Args: model_id_or_path (str): The identifier or path to the pre-trained model2vec model. Returns: StaticEmbedding: An instance of StaticEmbedding initialized with the tokenizer and embedding weights the model2vec model. Raises: ImportError: If the `model2vec` package is not installed. r) StaticModelzSTo use this method, please install the `model2vec` package: `pip install model2vec`rf) Z model2vecrvrgrrrrrrrrrr)rQrurvrsrrr$r$r%from_model2vecs  zStaticEmbedding.from_model2vec)NN)rGNNNF)NNrTTrUrVrWT)__name__ __module__ __qualname__rr2r4propertyr:r;rE classmethodrSrtrw __classcell__r$r$r"r%r s8P&Jr ) __future__rrhloggingr7rBpathlibrtypingrrrgZtyping_extensionsnumpyrrZsafetensors.torchrrAZ tokenizersrrZ transformersr Z(sentence_transformers.models.InputModuler Zsentence_transformers.utilr getLoggerrxrmr r$r$r$r%s(