a h:@sddlmZddlZddlZddlZddlmZmZddlZ ddl Z ddl Z ddl m Z mZddlmZddlmZerddl mZddlmZeeZGd d d eZdS) ) annotationsN) TYPE_CHECKINGCallable)average_precision_score ndcg_score)SentenceEvaluator)cos_sim)Tensor)SentenceTransformerc seZdZdZdddedddddf dd d d d d d d d d d fdd Zd'ddd d ddddZddddZddddZddddZ d(dddd d!d"d#d$Z d%d&Z Z S))RerankingEvaluatoraD This class evaluates a SentenceTransformer model for the task of re-ranking. Given a query and a list of documents, it computes the score [query, doc_i] for all possible documents and sorts them in decreasing order. Then, MRR@10, NDCG@10 and MAP is compute to measure the quality of the ranking. Args: samples (list): A list of dictionaries, where each dictionary represents a sample and has the following keys: - 'query': The search query. - 'positive': A list of positive (relevant) documents. - 'negative': A list of negative (irrelevant) documents. at_k (int, optional): Only consider the top k most similar documents to each query for the evaluation. Defaults to 10. name (str, optional): Name of the evaluator. Defaults to "". write_csv (bool, optional): Write results to CSV file. Defaults to True. similarity_fct (Callable[[torch.Tensor, torch.Tensor], torch.Tensor], optional): Similarity function between sentence embeddings. By default, cosine similarity. Defaults to cos_sim. batch_size (int, optional): Batch size to compute sentence embeddings. Defaults to 64. show_progress_bar (bool, optional): Show progress bar when computing embeddings. Defaults to False. use_batched_encoding (bool, optional): Whether or not to encode queries and documents in batches for greater speed, or 1-by-1 to save memory. Defaults to True. truncate_dim (Optional[int], optional): The dimension to truncate sentence embeddings to. `None` uses the model's current truncation dimension. Defaults to None. mrr_at_k (Optional[int], optional): Deprecated parameter. Please use `at_k` instead. Defaults to None. Example: :: from sentence_transformers import SentenceTransformer from sentence_transformers.evaluation import RerankingEvaluator from datasets import load_dataset # Load a model model = SentenceTransformer("all-MiniLM-L6-v2") # Load a dataset with queries, positives, and negatives eval_dataset = load_dataset("microsoft/ms_marco", "v1.1", split="validation") samples = [ { "query": sample["query"], "positive": [text for is_selected, text in zip(sample["passages"]["is_selected"], sample["passages"]["passage_text"]) if is_selected], "negative": [text for is_selected, text in zip(sample["passages"]["is_selected"], sample["passages"]["passage_text"]) if not is_selected], } for sample in eval_dataset ] # Initialize the evaluator reranking_evaluator = RerankingEvaluator( samples=samples, name="ms-marco-dev", ) results = reranking_evaluator(model) ''' RerankingEvaluator: Evaluating the model on the ms-marco-dev dataset: Queries: 9706 Positives: Min 1.0, Mean 1.1, Max 5.0 Negatives: Min 1.0, Mean 7.1, Max 9.0 MAP: 56.07 MRR@10: 56.70 NDCG@10: 67.08 ''' print(reranking_evaluator.primary_metric) # => ms-marco-dev_ndcg@10 print(results[reranking_evaluator.primary_metric]) # => 0.6708042171399308 T@FNz list[dict[str, str | list[str]]]intstrboolz4Callable[[torch.Tensor, torch.Tensor], torch.Tensor]z int | None) samplesat_kname write_csvsimilarity_fct batch_sizeshow_progress_baruse_batched_encoding truncate_dimmrr_at_kc st||_||_| dur8td| d| |_n||_||_||_||_ ||_ | |_ t |jt rxt|j|_dd|jD|_d|rd|ndd|jd |_d d d d |jd|jg|_||_d|j|_dS)Nz?The `mrr_at_k` parameter has been deprecated; please use `at_k=z ` instead.cSs0g|](}t|ddkrt|ddkr|qS)positivernegativelen.0sampler#o/var/www/html/assistant/venv/lib/python3.9/site-packages/sentence_transformers/evaluation/RerankingEvaluator.py zsz/RerankingEvaluator.__init__..r _r z _results_@z.csvepochstepsZMAPMRR@NDCG@ndcg@)super__init__rrloggerwarningrrrrrr isinstancedictlistvaluescsv_file csv_headersrZprimary_metric) selfrrrrrrrrrr __class__r#r$r-Ys4  $  zRerankingEvaluator.__init__r z str | Nonezdict[str, float])model output_pathr'r(returncCsD|dkr0|dkrd|}q4d|d|d}nd}|jdurP|d|jd 7}td |jd |d ||}|d }|d}|d} dd|jD} dd|jD} tdt|jdt| ddt | ddt | ddt| ddt | ddt | dtd|ddtd|j d|ddtd|j d| dd|dur|j rt j||j} t j| } t| d| rdnd d!d"@}t|}| s||j|||||| gWdn1s0Yd |d#|j |d$|j | i}|||j}||||||S)%a Evaluates the model on the dataset and returns the evaluation metrics. Args: model (SentenceTransformer): The SentenceTransformer model to evaluate. output_path (str, optional): The output path to write the results. Defaults to None. epoch (int, optional): The current epoch number. Defaults to -1. steps (int, optional): The current step number. Defaults to -1. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. r9z after epoch z in epoch z after z stepsr Nz (truncated to )z0RerankingEvaluator: Evaluating the model on the z dataset:mapmrrndcgcSsg|]}t|dqS)rrr r#r#r$r%z/RerankingEvaluator.__call__..cSsg|]}t|dqS)rrr r#r#r$r%rBz Queries: z Positives: Min z.1fz, Mean z, Max z Negatives: Min zMAP: dz.2fr)z: r*awzutf-8)newlinemodeencodingzmrr@r+)rr.inforcompute_metricesrrnpminmeanmaxrrospathjoinr4isfileopencsvwriterwriterowr5Zprefix_name_to_metricsZ store_metrics_in_model_card_data)r6r:r;r'r(Zout_txtZscoresmean_apmean_mrr mean_ndcgZ num_positivesZ num_negativesZcsv_pathZoutput_file_existsfrUZmetricsr#r#r$__call__sF   b   4  zRerankingEvaluator.__call__)r:cCs|jr||S||S)a Computes the evaluation metrics for the given model. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. )rcompute_metrices_batchedcompute_metrices_individual)r6r:r#r#r$rJs  z#RerankingEvaluator.compute_metricescCsg}g}g}|j|dd|jDd|jd}g}|jD] }||d||dq6|j||d|jd}d\} } |jD]} || } | d 7} t| d} t| d}|| | | |}| | |7} | d ksz|d krqz|| |}t|jd kr|d }t| }| }d g| d g|}d }t |d |j D]&\}}||r>d |d }qfq>| || t|g|g|j d | t||qzt|}t|}t|}|||d S) aE Computes the evaluation metrics in a batched way, by batching all queries and all documents together. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. cSsg|] }|dqS)queryr#r r#r#r$r%rBz?RerankingEvaluator.compute_metrices_batched..r^encode_fn_namerrrdocument)rrrkr?r@rA) embed_inputsrrextendrrshapetorchargsortcputolist enumeraterappendrrrKrM)r6r:all_mrr_scoresall_ndcg_scores all_ap_scoresZall_query_embsZall_docsr"Z all_docs_embsZ query_idxZdocs_idxinstance query_embnum_posZnum_negdocs_emb pred_scorespred_scores_argsort is_relevant mrr_scorerankindexrWrXrYr#r#r$r\sX                 z+RerankingEvaluator.compute_metrices_batchedcCsg}g}g}tj|j|j ddD]0}|d}t|d}t|d}t|dks"t|dkrbq"||} dgt|dgt|} |j||gddd } |j|| d dd } || | } t| jdkr| d} t | }| } d}t |d|j D]"\}}| |rd|d}q q|||t| g| g|j d |t| | q"t|}t|}t|}|||d S) aO Computes the evaluation metrics individually by embedding every (query, positive, negative) tuple individually. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. ZSamples)disabledescr^rrrrbFr_rarcre)tqdmrrr2rrfrrhrirjrkrlrmrrnrrrKrM)r6r:rorprqrrr^rrdocsrxrsrurvrwryrzr{rWrXrYr#r#r$r]!s<           z.RerankingEvaluator.compute_metrices_individualzstr | list[str] | np.ndarrayz bool | Noner )r: sentencesr`rr<cKsL|dur|j}n|dkr |j}n|dkr.|j}||f|j|d|jd|S)Nr^raT)rrZconvert_to_tensorr)encodeZ encode_queryZencode_documentrr)r6r:rr`rkwargsZ encode_fnr#r#r$rfXs zRerankingEvaluator.embed_inputscCs"d|ji}|jdur|j|d<|S)Nrr)rr)r6Z config_dictr#r#r$get_config_dictos   z"RerankingEvaluator.get_config_dict)Nr9r9)NN) __name__ __module__ __qualname____doc__rr-r[rJr\r]rfr __classcell__r#r#r7r$r s(B&1?I;r ) __future__rrTloggingrOtypingrrnumpyrKrir~Zsklearn.metricsrrZ2sentence_transformers.evaluation.SentenceEvaluatorrZsentence_transformers.utilrr Z)sentence_transformers.SentenceTransformerr getLoggerrr.r r#r#r#r$s