a hC@s@ddlmZddlZddlZddlZddlmZmZddlZ ddl Z ddl m Z ddl m Z ddlmZddlmZeeZerdd lmZd d d d ddedddf dddddddddddddd ddZd d ddefddddddddddZdd d!d"Zddd#efdddddddd$d%d&Zd/d)d*dddd+d,d-d.ZdS)0) annotationsN) TYPE_CHECKINGCallable)Tensor)tqdm)cos_sim)normalize_embeddings)SentenceTransformerF iii dr z list[str]boolintz"Callable[[Tensor, Tensor], Tensor]z int | Nonez str | Nonezlist[list[float | int]]) model sentencesshow_progress_bar batch_sizequery_chunk_sizecorpus_chunk_size max_pairstop_kscore_function truncate_dim prompt_namepromptreturnc Cs,|j|||d| | | d} t| |||||dS)a@ Given a list of sentences / texts, this function performs paraphrase mining. It compares all sentences against all other sentences and returns a list with the pairs that have the highest cosine similarity score. Args: model (SentenceTransformer): SentenceTransformer model for embedding computation sentences (List[str]): A list of strings (texts or sentences) show_progress_bar (bool, optional): Plotting of a progress bar. Defaults to False. batch_size (int, optional): Number of texts that are encoded simultaneously by the model. Defaults to 32. query_chunk_size (int, optional): Search for most similar pairs for #query_chunk_size at the same time. Decrease, to lower memory footprint (increases run-time). Defaults to 5000. corpus_chunk_size (int, optional): Compare a sentence simultaneously against #corpus_chunk_size other sentences. Decrease, to lower memory footprint (increases run-time). Defaults to 100000. max_pairs (int, optional): Maximal number of text pairs returned. Defaults to 500000. top_k (int, optional): For each sentence, we retrieve up to top_k other sentences. Defaults to 100. score_function (Callable[[Tensor, Tensor], Tensor], optional): Function for computing scores. By default, cosine similarity. Defaults to cos_sim. truncate_dim (int, optional): The dimension to truncate sentence embeddings to. If None, uses the model's ones. Defaults to None. prompt_name (Optional[str], optional): The name of a predefined prompt to use when encoding the sentence. It must match a key in the model `prompts` dictionary, which can be set during model initialization or loaded from the model configuration. Ignored if `prompt` is provided. Defaults to None. prompt (Optional[str], optional): A raw prompt string to prepend directly to the input sentence during encoding. For instance, `prompt="query: "` transforms the sentence "What is the capital of France?" into: "query: What is the capital of France?". Use this to override the prompt logic entirely and supply your own prefix. This takes precedence over `prompt_name`. Defaults to None. Returns: List[List[Union[float, int]]]: Returns a list of triplets with the format [score, id1, id2] T)rrZconvert_to_tensorrrr)rrrrr)encodeparaphrase_mining_embeddings) rrrrrrrrrrrr embeddingsr`/var/www/html/assistant/venv/lib/python3.9/site-packages/sentence_transformers/util/retrieval.pyparaphrase_minings". r!r)rrrrrrrc Cs|d7}t}d}d}tdt||D]} tdt||D]} ||| | ||| | |} tj| t|t| ddddd\} } | } | } tt| D]z}t | |D]h\}}| |}| |}||kr| |||kr| | ||||f|d7}||kr| }|d}qqq>q(t }g}| s| \}}}t||g\}}||kr8||f|vr8|||f||||gq8t|dddd }|S) a Given a list of sentences / texts, this function performs paraphrase mining. It compares all sentences against all other sentences and returns a list with the pairs that have the highest cosine similarity score. Args: embeddings (Tensor): A tensor with the embeddings query_chunk_size (int): Search for most similar pairs for #query_chunk_size at the same time. Decrease, to lower memory footprint (increases run-time). corpus_chunk_size (int): Compare a sentence simultaneously against #corpus_chunk_size other sentences. Decrease, to lower memory footprint (increases run-time). max_pairs (int): Maximal number of text pairs returned. top_k (int): For each sentence, we retrieve up to top_k other sentences score_function (Callable[[Tensor, Tensor], Tensor]): Function for computing scores. By default, cosine similarity. Returns: List[List[Union[float, int]]]: Returns a list of triplets with the format [score, id1, id2] rrTFdimlargestsortedcSs|dS)Nrrxrrr z.paraphrase_mining_embeddings..keyreverse)queue PriorityQueuerangelentorchtopkmincputolist enumerateputgetsetemptyr&addappend)rrrrrrpairsZ min_scoreZ num_addedcorpus_start_idxquery_start_idxZscoresZscores_top_k_valuesZscores_top_k_idx query_itrZ top_k_idxZ corpus_itrijentryZ added_pairsZ pairs_listscoreZsorted_iZsorted_jrrr rXsF    rz"list[list[dict[str, int | float]]])rcOst|i|S)z8This function is deprecated. Use semantic_search instead)semantic_search)argskwargsrrr information_retrievalsrI )query_embeddingscorpus_embeddingsrrrrrc Cst|tjtjfrt|}nt|tr2t|}t|j dkrJ| d}t|tjtjfrht|}nt|tr|t|}|j |j kr| |j }ddt t|D}t dt||D]n}t||t|}|jrtj|||j d} |d| } n |||} t dt||D]} t| |t|} |jrTtj| | |j d} |d| } n || | } || | }tj|t|t|ddddd\}}|}|}t t|D]p}t||||D]V\}}| |}||}t|||kr t||||fnt||||fqʐqqqt t|D]Z}t t||D](}|||\}}||d |||<qLt||d d dd ||<q8|S) a3 This function performs by default a cosine similarity search between a list of query embeddings and a list of corpus embeddings. It can be used for Information Retrieval / Semantic Search for corpora up to about 1 Million entries. Args: query_embeddings (:class:`~torch.Tensor`): A 2 dimensional tensor with the query embeddings. Can be a sparse tensor. corpus_embeddings (:class:`~torch.Tensor`): A 2 dimensional tensor with the corpus embeddings. Can be a sparse tensor. query_chunk_size (int, optional): Process 100 queries simultaneously. Increasing that value increases the speed, but requires more memory. Defaults to 100. corpus_chunk_size (int, optional): Scans the corpus 100k entries at a time. Increasing that value increases the speed, but requires more memory. Defaults to 500000. top_k (int, optional): Retrieve top k matching entries. Defaults to 10. score_function (Callable[[:class:`~torch.Tensor`, :class:`~torch.Tensor`], :class:`~torch.Tensor`], optional): Function for computing scores. By default, cosine similarity. Returns: List[List[Dict[str, Union[int, float]]]]: A list with one entry for each query. Each entry is a list of dictionaries with the keys 'corpus_id' and 'score', sorted by decreasing cosine similarity scores. rrcSsg|]}gqSrr).0_rrr r*z#semantic_search..deviceTFr#) corpus_idrEcSs|dS)NrErr'rrr r)r*z!semantic_search..r+) isinstancenpZndarrayZgenericr2Z from_numpyliststackr1shapeZ unsqueezerQtor0r4Z is_sparseZarangeZ index_selectr3r5r6zipheapqheappush heappushpopr&)rKrLrrrrZqueries_result_listr@Z query_end_idxindicesZ query_chunkr?Zcorpus_end_idxZ corpus_chunk cos_scoresZcos_scores_top_k_valuesZcos_scores_top_k_idxrAZ sub_corpus_idrErRZquery_idZdoc_itrrrr rFs\                "rF?ztorch.Tensor | np.ndarrayfloatzlist[list[int]])r thresholdmin_community_sizerrrcCsRt|tjst|}tj||jd}t|}g}t|t|}ttd|dt|}t t dt||d| dD]N}|||||j }|jj dvr||k} | d} | |k} | sqt| | } || }| } |j| d d \} }t| |D]\}}||d |qqt|j|d d \}} t t|D]}||d |kr6||j|d d \}}|d |kr|t|krtd|t|}||j|d d \}}qb||||kq6qtt|d dd d}g}t}t|D]P\}}g}|D]}||vr||qt||kr||||qt|ddd d}|S)a Function for Fast Community Detection. Finds in the embeddings all communities, i.e. embeddings that are close (closer than threshold). Returns only communities that are larger than min_community_size. The communities are returned in decreasing order. The first element in each list is the central point in the community. Args: embeddings (torch.Tensor or numpy.ndarray): The input embeddings. threshold (float): The threshold for determining if two embeddings are close. Defaults to 0.75. min_community_size (int): The minimum size of a community to be considered. Defaults to 10. batch_size (int): The batch size for computing cosine similarity scores. Defaults to 1024. show_progress_bar (bool): Whether to show a progress bar during computation. Defaults to False. Returns: List[List[int]]: A list of communities, where each community is represented as a list of indices. rP2rzFinding clusters)descdisable)cudaZnpurT)kr%Nr"cSst|SNr1r'rrr r)Rr*z%community_detection..r+cSst|Srjrkr'rrr r)br*)rSr2rtensorrQr r4r1maxrr0Ttypesumanyr3rYr=r6r&r:r7update)rrbrcrrZextracted_communitiesZ sort_max_sizeZ start_idxr^Zthreshold_maskZrow_wise_countZlarge_enough_maskrirNZ top_k_indicescountr]Z top_k_valuesrBZ top_val_largeZ top_idx_largeZunique_communitiesZ extracted_idsZ cluster_idZ communityZnon_overlapped_communityidxrrr community_detectionsX      ru)r_rJr`F) __future__rrZloggingr.typingrrnumpyrTr2rZtqdm.autonotebookrZ similarityrrlr getLogger__name__loggerZ)sentence_transformers.SentenceTransformerr r!rrIrFrurrrr sR       (DI]