a hx@sddlZddlZddlmZmZmZmZddlZddl m Z ddl m Z m Z mZmZddlmZmZmZmZerddlZddlmZerddlZdd lmZGd d d eZGd d d e Ze edddGdddeZeZdS)N)AnyOptionalUnionoverload)BasicTokenizer) ExplicitEnumadd_end_docstringsis_tf_availableis_torch_available)ArgumentHandler ChunkPipelineDatasetbuild_pipeline_init_args)/TF_MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING_NAMES),MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING_NAMESc@s*eZdZdZeeeefdddZdS)"TokenClassificationArgumentHandlerz5 Handles arguments for token classification. )inputscKs|dd}|d}|durJt|ttfrJt|dkrJt|}t|}nHt|tr`|g}d}n2tdurrt|ts~t|tjr||d|fSt d|d}|rt|trt|dtr|g}t||krt d||||fS) Nis_split_into_wordsF delimiterrr zAt least one input is required.offset_mappingz;offset_mapping should have the same batch size as the input) get isinstancelisttuplelenstrrtypes GeneratorType ValueError)selfrkwargsrrZ batch_sizerr#g/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/pipelines/token_classification.py__call__ s$  "     z+TokenClassificationArgumentHandler.__call__N)__name__ __module__ __qualname____doc__rrrr%r#r#r#r$rsrc@s$eZdZdZdZdZdZdZdZdS)AggregationStrategyzDAll the valid aggregation strategies for TokenClassificationPipelinenonesimplefirstZaveragemaxN) r&r'r(r)NONESIMPLEFIRSTAVERAGEMAXr#r#r#r$r*8s r*T)Z has_tokenizera ignore_labels (`list[str]`, defaults to `["O"]`): A list of labels to ignore. grouped_entities (`bool`, *optional*, defaults to `False`): DEPRECATED, use `aggregation_strategy` instead. Whether or not to group the tokens corresponding to the same entity together in the predictions or not. stride (`int`, *optional*): If stride is provided, the pipeline is applied on all the text. The text is split into chunks of size model_max_length. Works only with fast tokenizers and `aggregation_strategy` different from `NONE`. The value of this argument defines the number of overlapping tokens between chunks. In other words, the model will shift forward by `tokenizer.model_max_length - stride` tokens each step. aggregation_strategy (`str`, *optional*, defaults to `"none"`): The strategy to fuse (or not) tokens based on the model prediction. - "none" : Will simply not do any aggregation and simply return raw results from the model - "simple" : Will attempt to group entities following the default schema. (A, B-TAG), (B, I-TAG), (C, I-TAG), (D, B-TAG2) (E, B-TAG2) will end up being [{"word": ABC, "entity": "TAG"}, {"word": "D", "entity": "TAG2"}, {"word": "E", "entity": "TAG2"}] Notice that two consecutive B tags will end up as different entities. On word based languages, we might end up splitting words undesirably : Imagine Microsoft being tagged as [{"word": "Micro", "entity": "ENTERPRISE"}, {"word": "soft", "entity": "NAME"}]. Look for FIRST, MAX, AVERAGE for ways to mitigate that and disambiguate words (on languages that support that meaning, which is basically tokens separated by a space). These mitigations will only work on real words, "New york" might still be tagged with two different entities. - "first" : (works only on word based models) Will use the `SIMPLE` strategy except that words, cannot end up with different tags. Words will simply use the tag of the first token of the word when there is ambiguity. - "average" : (works only on word based models) Will use the `SIMPLE` strategy except that words, cannot end up with different tags. scores will be averaged first across tokens, and then the maximum label is applied. - "max" : (works only on word based models) Will use the `SIMPLE` strategy except that words, cannot end up with different tags. Word entity will simply be the token with the maximum score.c seZdZdZdZdZdZdZdZe ffdd Z d+e e e e e e e eeeefe e e ee edd d Zeeeeeeefd d d Zeeeeeeeeefd dd Zeeeefeeeeeefeeeeeffd fdd Zd,ddZddZe jdfddZddZd-eejeje eeeefeje e ee ee eeeefeed ddZeee eedddZ eee eddd Z!eee eedd!d"Z"eeed#d$d%Z#eeeefd&d'd(Z$eeeed#d)d*Z%Z&S).TokenClassificationPipelineuv Named Entity Recognition pipeline using any `ModelForTokenClassification`. See the [named entity recognition examples](../task_summary#named-entity-recognition) for more information. Example: ```python >>> from transformers import pipeline >>> token_classifier = pipeline(model="Jean-Baptiste/camembert-ner", aggregation_strategy="simple") >>> sentence = "Je m'appelle jean-baptiste et je vis à montréal" >>> tokens = token_classifier(sentence) >>> tokens [{'entity_group': 'PER', 'score': 0.9931, 'word': 'jean-baptiste', 'start': 12, 'end': 26}, {'entity_group': 'LOC', 'score': 0.998, 'word': 'montréal', 'start': 38, 'end': 47}] >>> token = tokens[0] >>> # Start and end provide an easy way to highlight words in the original text. >>> sentence[token["start"] : token["end"]] ' jean-baptiste' >>> # Some models use the same idea to do part of speech. >>> syntaxer = pipeline(model="vblagoje/bert-english-uncased-finetuned-pos", aggregation_strategy="simple") >>> syntaxer("My name is Sarah and I live in London") [{'entity_group': 'PRON', 'score': 0.999, 'word': 'my', 'start': 0, 'end': 2}, {'entity_group': 'NOUN', 'score': 0.997, 'word': 'name', 'start': 3, 'end': 7}, {'entity_group': 'AUX', 'score': 0.994, 'word': 'is', 'start': 8, 'end': 10}, {'entity_group': 'PROPN', 'score': 0.999, 'word': 'sarah', 'start': 11, 'end': 16}, {'entity_group': 'CCONJ', 'score': 0.999, 'word': 'and', 'start': 17, 'end': 20}, {'entity_group': 'PRON', 'score': 0.999, 'word': 'i', 'start': 21, 'end': 22}, {'entity_group': 'VERB', 'score': 0.998, 'word': 'live', 'start': 23, 'end': 27}, {'entity_group': 'ADP', 'score': 0.999, 'word': 'in', 'start': 28, 'end': 30}, {'entity_group': 'PROPN', 'score': 0.999, 'word': 'london', 'start': 31, 'end': 37}] ``` Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial) This token recognition pipeline can currently be loaded from [`pipeline`] using the following task identifier: `"ner"` (for predicting the classes of tokens in a sequence: person, organisation, location or miscellaneous). The models that this pipeline can use are models that have been fine-tuned on a token classification task. See the up-to-date list of available models on [huggingface.co/models](https://huggingface.co/models?filter=token-classification). sequencesFTcs@tj|i|||jdkr$tnttdd|_||_dS)NtfF)Z do_lower_case) super__init__Zcheck_model_type frameworkrrrZ_basic_tokenizer _args_parser)r!Z args_parserargsr" __class__r#r$r8s z$TokenClassificationPipeline.__init__N)grouped_entitiesignore_subwordsaggregation_strategyrrstriderc Csli} || d<|r$|durdn|| d<|dur4|| d<i} |dusH|dur|rX|rXtj}n|rh|shtj}ntj}|durtd|d|durtd|d|durt|trt|}|tjtj tj hvr|j j st d|| d <|dur|| d <|durb||j jkrt d |tjkr:t d |d n(|j j rZdd|d} | | d<nt d| i| fS)Nr rrzl`grouped_entities` is deprecated and will be removed in version v5.0.0, defaulted to `aggregation_strategy="z "` instead.zk`ignore_subwords` is deprecated and will be removed in version v5.0.0, defaulted to `aggregation_strategy="z{Slow tokenizers cannot handle subwords. Please set the `aggregation_strategy` option to `"simple"` or use a fast tokenizer.r@ ignore_labelszl`stride` must be less than `tokenizer.model_max_length` (or even lower if the tokenizer adds special tokens)zI`stride` was provided to process all the text but `aggregation_strategy="z&"`, please select another one instead.T)Zreturn_overflowing_tokenspaddingrAtokenizer_paramszm`stride` was provided to process all the text but you're using a slow tokenizer. Please use a fast tokenizer.)r*r1r0r/warningswarnrrupperr3r2 tokenizeris_fastr model_max_length) r!rCr>r?r@rrrArpreprocess_paramsZpostprocess_paramsrEr#r#r$_sanitize_parameterssz       z0TokenClassificationPipeline._sanitize_parameters)rr"returncKsdSNr#r!rr"r#r#r$r%sz$TokenClassificationPipeline.__call__cKsdSrOr#rPr#r#r$r%sc sv|j|fi|\}}}}||d<||d<|rVtdd|DsVtj|gfi|S|rb||d<tj|fi|S)a Classify each token of the text(s) given as inputs. Args: inputs (`str` or `List[str]`): One or several texts (or one list of texts) for token classification. Can be pre-tokenized when `is_split_into_words=True`. Return: A list or a list of list of `dict`: Each result comes as a list of dictionaries (one for each token in the corresponding input, or each entity if this pipeline was instantiated with an aggregation_strategy) with the following keys: - **word** (`str`) -- The token/word classified. This is obtained by decoding the selected tokens. If you want to have the exact string in the original sentence, use `start` and `end`. - **score** (`float`) -- The corresponding probability for `entity`. - **entity** (`str`) -- The entity predicted for that token/word (it is named *entity_group* when *aggregation_strategy* is not `"none"`. - **index** (`int`, only present when `aggregation_strategy="none"`) -- The index of the corresponding token in the sentence. - **start** (`int`, *optional*) -- The index of the start of the corresponding entity in the sentence. Only exists if the offsets are available within the tokenizer - **end** (`int`, *optional*) -- The index of the end of the corresponding entity in the sentence. Only exists if the offsets are available within the tokenizer rrcss|]}t|tVqdSrO)rr).0inputr#r#r$ z7TokenClassificationPipeline.__call__..r)r:allr7r%)r!rr"Z_inputsrrrr<r#r$r%sc+s|di}|jjo|jjdk}d}|d}|r|d}t|tsJtd|} || }g}t|} d} | D]*} || | t| f| t| | 7} ql| } d|d<nt|t std|} |j| f|j |d|jj d|}|r|jj std |d dt|d }t |D]|j d kr:fd d| D}nfdd| D}|durb||d<dkrp|nd|d<|dk|d<|dur||d<||d<|VqdS)NrErrrzEWhen `is_split_into_words=True`, `sentence` must be a list of tokens.TzKWhen `is_split_into_words=False`, `sentence` must be an untokenized string.)Zreturn_tensors truncationZreturn_special_tokens_maskZreturn_offsets_mappingz@is_split_into_words=True is only supported with fast tokenizers.Zoverflow_to_sample_mapping input_idsr6cs"i|]\}}|t|dqSr)r6Z expand_dimsrQkvir#r$ FrTz:TokenClassificationPipeline.preprocess..cs i|]\}}||dqSrX)Z unsqueezerYr\r#r$r^HrTrsentencer is_lastword_idsword_to_chars_map)poprIrKrrr joinrappendrr9rJrangeitemsra)r!r_rrLrErVrbrrwordsZ delimiter_lenZ char_offsetwordZtext_to_tokenizer num_chunks model_inputsr#r\r$ preprocesss^            z&TokenClassificationPipeline.preprocessc Cs|d}|dd}|d}|d}|dd}|dd}|jdkrb|jfi|d}n*|jfi|} t| tr| d n| d}|||||||d |S) Nspecial_tokens_maskrr_r`rarbr6rlogits)rnrmrr_r`rarb)rcr9modelrdict) r!rkrmrr_r`rarbrnoutputr#r#r$_forwardTs(       z$TokenClassificationPipeline._forwardc sdurdgg}|dd}|D]<}|jdkrh|ddjtjtjfvrh|ddtj}n|dd}|dd}|dd} |ddur|ddnd} |d d} |d } t j |d d d } t || }||j d d d }|jdkr | } | dur| nd} |j || || | || |d}|||}fdd|D}||q$t|}|dkr||}|S)NOrrbptrnr_rWrrmraT)axisZkeepdimsr6)rarbcs0g|](}|ddvr|ddvr|qS)entityN entity_group)rrQrwrCr#r$ sz;TokenClassificationPipeline.postprocess..r )rr9ZdtypetorchZbfloat16Zfloat16toZfloat32numpynpr.expsumgather_pre_entities aggregateextendraggregate_overlapping_entities)r!Z all_outputsr@rCZ all_entitiesrbZ model_outputsrnr_rWrrmraZmaxesZ shifted_expscores pre_entitiesr>entitiesrjr#rzr$ postprocessnsN $         z'TokenClassificationPipeline.postprocesscCst|dkr|St|ddd}g}|d}|D]|}|d|dkrT|dkrnnF|d|d}|d|d}||ks||kr|d|dkr|}q0|||}q0|||S)NrcSs|dS)Nstartr#)xr#r#r$rTzLTokenClassificationPipeline.aggregate_overlapping_entities..keyrendscore)rsortedre)r!rZaggregated_entitiesZprevious_entityrwZcurrent_lengthZprevious_lengthr#r#r$rs( $  z:TokenClassificationPipeline.aggregate_overlapping_entities) r_rWrrrmr@rarbrNc Cszg} t|D]f\} } || r q |jt|| } |durL|| \} }|dur|dur|| }|dur||\}}| |7} ||7}t| ts|jdkr| } |}|| |}t|jddrt|jjj ddrt | t |k}nD|t j t j t jhvr tdt| dko*d|| d| dv}t|| |jjkrX|} d }n d} d}d }| | | || |d }| |q | S) zTFuse various numpy arrays into dicts with all the information needed for aggregationNrt _tokenizerZcontinuing_subword_prefixz?Tokenizer does not support real words, using fallback heuristicrrBr F)rirrrindex is_subword) enumeraterIZconvert_ids_to_tokensintrr9itemgetattrrrorr*r1r2r3rFrG UserWarningZ unk_token_idre)r!r_rWrrrmr@rarbridxZ token_scoresriZ start_indZend_indZ word_indexZ start_char_Zword_refr pre_entityr#r#r$rs^        " z/TokenClassificationPipeline.gather_pre_entities)rr@rNcCs|tjtjhvrng}|D]R}|d}|d|}|jjj|||d|d|d|dd}||qn |||}|tjkr|S| |S)Nrrrirr)rwrrrirr) r*r/r0argmaxroconfigid2labelreaggregate_wordsgroup_entities)r!rr@rr entity_idxrrwr#r#r$rs"     z%TokenClassificationPipeline.aggregate)rr@rNc Cs|jdd|D}|tjkrL|dd}|}||}|jjj|}n|tjkrt |ddd}|d}|}||}|jjj|}nT|tj krt dd|D}t j |dd } | } |jjj| }| | }ntd ||||dd |d d d} | S)NcSsg|] }|dqSrir#ryr#r#r$r{rTz>TokenClassificationPipeline.aggregate_word..rrcSs |dS)Nr)r.)rwr#r#r$r!rTz.rcSsg|] }|dqS)rr#ryr#r#r$r{'rT)rvzInvalid aggregation_strategyrrur)rwrrirr)rIconvert_tokens_to_stringr*r1rrorrr3r.r2rstacknanmeanr ) r!rr@rirrrrwZ max_entityZaverage_scoresrZ new_entityr#r#r$aggregate_words4       z*TokenClassificationPipeline.aggregate_wordcCs|tjtjhvrtdg}d}|D]@}|dur8|g}q$|drL||q$|||||g}q$|dur|||||S)z Override tokens from a given word that disagree to force agreement on word boundaries. Example: micro|soft| com|pany| B-ENT I-NAME I-ENT I-ENT will be rewritten with first strategy as microsoft| company| B-ENT I-ENT z;NONE and SIMPLE strategies are invalid for word aggregationNr)r*r/r0r rer)r!rr@Z word_entitiesZ word_grouprwr#r#r$r7s" z+TokenClassificationPipeline.aggregate_words)rrNcCsl|ddddd}tdd|D}dd|D}|t||j||dd |dd d }|S) z Group together the adjacent tokens with the same entity predicted. Args: entities (`dict`): The entities predicted by the pipeline. rrw-r rucSsg|] }|dqS)rr#ryr#r#r$r{\rTzBTokenClassificationPipeline.group_sub_entities..cSsg|] }|dqSrr#ryr#r#r$r{]rTrr)rxrrirr)splitrrmeanrIr)r!rrwrtokensrxr#r#r$group_sub_entitiesSs   z.TokenClassificationPipeline.group_sub_entities) entity_namerNcCsH|drd}|dd}n$|dr8d}|dd}nd}|}||fS)NzB-BrzI-I) startswith)r!rbitagr#r#r$get_taghs  z#TokenClassificationPipeline.get_tagc Csg}g}|D]n}|s ||q ||d\}}||dd\}}||krd|dkrd||q ||||g}q |r||||S)z Find and group together the adjacent tokens with the same entity predicted. Args: entities (`dict`): The entities predicted by the pipeline. rwrur)rerr) r!rZ entity_groupsZentity_group_disaggrwrrZlast_biZlast_tagr#r#r$rvs  z*TokenClassificationPipeline.group_entities)NNNNNFNN)N)NN)'r&r'r(r)Zdefault_input_namesZ_load_processorZ_load_image_processorZ_load_feature_extractorZ_load_tokenizerrr8rboolr*rrrrrMrrrpr%rrlrrr/rrrZndarrayrrrrrrr __classcell__r#r#r<r$r4Bsl#$ P (&' ;6 Ir4) rrFtypingrrrrr~rZmodels.bert.tokenization_bertrutilsrr r r baser rrrZ tensorflowr6Zmodels.auto.modeling_tf_autorr|Zmodels.auto.modeling_autorrr*r4Z NerPipeliner#r#r#r$s0    "<