a h!9@sddlZddlZddlZddlmZddlmZmZmZm Z m Z m Z m Z ddl mZmZejeZejedZeeZeZWdn1s0YGdddZe eZe e eZGdd d e ZGd d d ZGd d d ZGdddZ efe e!e!dddZ"dS)N)Template)AnyCallableDictList NamedTupleOptionalTuple)Encoding Tokenizerzvisualizer-styles.cssc@s8eZdZUeed<eed<eed<eeedddZdS) Annotationstartendlabelr rrcCs||_||_||_dSNr)selfr rrrW/var/www/html/assistant/venv/lib/python3.9/site-packages/tokenizers/tools/visualizer.py__init__szAnnotation.__init__N)__name__ __module__ __qualname__int__annotations__strrrrrrr s r c@s&eZdZUeeed<eeed<dS) CharStateKeytoken_ixanno_ixN)rrrrrrrrrrrs  rc@sHeZdZUeeed<ddZeddZeddZ e dd d Z d S) CharStatechar_ixcCs||_d|_g|_dSr)r rtokens)rr rrrr'szCharState.__init__cCst|jdkr|jdSdS)Nrlenr!rrrrr-szCharState.token_ixcCst|jdkS)zJ BPE tokenizers can output more than one token for a char r"r$rrr is_multitoken1szCharState.is_multitoken)returncCst|j|jdS)N)rr)rrrr$rrr partition_key8szCharState.partition_keyN) rrrrrrrpropertyrr&rr(rrrrr$s    rc@s eZdZdS)AlignedN)rrrrrrrr*?sr*c@seZdZdZejdejdZdee e e e ge fdddZgdfeee e e ed d d Zeeeeefd d dZeeeeedddZeeeeedddZeeeedddZeeeeeedddZdS)EncodingVisualizera Build an EncodingVisualizer Args: tokenizer (:class:`~tokenizers.Tokenizer`): A tokenizer instance default_to_notebook (:obj:`bool`): Whether to render html output in a notebook by default annotation_converter (:obj:`Callable`, `optional`): An optional (lambda) function that takes an annotation in any format and returns an Annotation object z(.{1})?(unk|oov)(.{1})?)flagsTN) tokenizerdefault_to_notebookannotation_convertercCsJ|r4zddlm}m}Wnty2tdYn0||_||_||_dS)NrHTMLdisplayzWe couldn't import IPython utils for html display. Are you running in a notebook? You can also pass `default_to_notebook=False` to get back raw HTML )IPython.core.displayr1r2 ImportError Exceptionr-r.annotation_coverter)rr-r.r/r1r2rrrrVs  zEncodingVisualizer.__init__)text annotationsr.r'c Cs|j}|dur|}|rFzddlm}m}WntyDtdYn0|jdur`tt|j|}|j |}t |||}|r|||n|SdS)a Build a visualization of the given text Args: text (:obj:`str`): The text to tokenize annotations (:obj:`List[Annotation]`, `optional`): An optional list of annotations of the text. The can either be an annotation class or anything else if you instantiated the visualizer with a converter function default_to_notebook (:obj:`bool`, `optional`, defaults to `False`): If True, will render the html in a notebook. Otherwise returns an html string. Returns: The HTML string if default_to_notebook is False, otherwise (default) returns None and renders the HTML in the notebook Nrr0zeWe couldn't import IPython utils for html display. Are you running in a notebook?) r.r3r1r2r4r5r6listmapr-encoder+_EncodingVisualizer__make_html) rr7r8r.Zfinal_default_to_notebookr1r2encodinghtmlrrr__call__ls"    zEncodingVisualizer.__call__)r8r'c Cst|dkriSttdd|}t|}td|}|dkrBd}d}d}d}i}t|D](}d |d |d |d ||<||7}qZ|S) a Generates a color palette for all the labels in a given set of annotations Args: annotations (:obj:`Annotation`): A list of annotations Returns: :obj:`dict`: A dictionary mapping labels to colors in HSL format rcSs|jSr)r)xrrrz;EncodingVisualizer.calculate_label_colors.. @ zhsl(,z%,z%))r#setr:rsorted) r8labelsZ num_labelsZh_stepslhcolorsrrrrcalculate_label_colorss    z)EncodingVisualizer.calculate_label_colors)consecutive_chars_listr7r=cCs.|d}|jdur*|j|j}d|dS|d}|j}|jd}|||}g} i} |jdur| d|jrz| d|jd r| d n | d tj|j|jdur| d |j|j| d <n | ddd| d} d} | D]\} }| d| d|d7} qd| d| d|dS)a Converts a list of "consecutive chars" into a single HTML element. Chars are consecutive if they fall under the same word, token and annotation. The CharState class is a named tuple with a "partition_key" method that makes it easy to compare if two chars are consecutive. Args: consecutive_chars_list (:obj:`List[CharState]`): A list of CharStates that have been grouped together text (:obj:`str`): The original text being processed encoding (:class:`~tokenizers.Encoding`): The encoding returned from the tokenizer Returns: :obj:`str`: The HTML span for a set of consecutive chars rNz(r%tokenz multi-tokenz odd-tokenz even-tokenz special-tokenZstokz non-tokenzclass=" "z data-z="z) r r!rappendr&r+unk_token_regexsearchjoinitems)rQr7r=firstZstokenlastr rZ span_textZ css_classesZ data_itemscssdatakeyvalrrrconsecutive_chars_to_htmls4             z,EncodingVisualizer.consecutive_chars_to_html)r7r=r8r'cCsXt|||}|dg}|dj}g}t|}|dj}|durp||} | j} || } |d| d| d|ddD]} | j}||kr|tj|||d| g}|dur|d|dur||} | j} || } |d| d| d|}| |dkr|| q||tj|||d| g}q||tj|||dt|} | S)Nrz&r%)r7r=rX) r+%_EncodingVisualizer__make_char_statesrrPrrYrdr(HTMLBody)r7r=r8 char_statesZcurrent_consecutive_charsZ prev_anno_ixspansZlabel_colors_dictZ cur_anno_ixannorcolorcsresrrrZ __make_htmlsb      zEncodingVisualizer.__make_html)r7r8r'cCs@dgt|}t|D]$\}}t|j|jD] }|||<q,q|S)a Args: text (:obj:`str`): The raw text we want to align to annotations (:obj:`AnnotationList`): A (possibly empty) list of annotations Returns: A list of length len(text) whose entry at index i is None if there is no annotation on character i or k, the index of the annotation that covers index i where k is with respect to the list of annotations N)r# enumerateranger r)r7r8annotation_maprairrrZ__make_anno_map<s  z"EncodingVisualizer.__make_anno_mapc Cst||}ddtt|D}t|jD]B\}}||}|dur,|\}} t|| D]} || j|qXq,t|D]\} } | || _qx|S)a For each character in the original text, we emit a tuple representing it's "state": * which token_ix it corresponds to * which word_ix it corresponds to * which annotation_ix it corresponds to Args: text (:obj:`str`): The raw text we want to align to annotations (:obj:`List[Annotation]`): A (possibly empty) list of annotations encoding: (:class:`~tokenizers.Encoding`): The encoding returned from the tokenizer Returns: :obj:`List[CharState]`: A list of CharStates, indicating for each char in the text what it's state is cSsg|] }t|qSr)r).0r rrr jrBz9EncodingVisualizer.__make_char_states..N) r+"_EncodingVisualizer__make_anno_maprnr#rmr!Ztoken_to_charsrYr) r7r=r8rorgrrSoffsetsr rrqr rrrrZ__make_char_statesQs   z%EncodingVisualizer.__make_char_states)TN)rrr__doc__recompile IGNORECASErZr boolrrrr rrAnnotationListr? staticmethodrrPrrr rdr<PartialIntListrtrerrrrr+Cs<  -CAr+)childrenr'cCsd|}d|d|dS)a[ Generates the full html with css from a list of html spans Args: children (:obj:`List[str]`): A list of strings, assumed to be html elements css_styles (:obj:`str`, `optional`): Optional alternative implementation of the css Returns: :obj:`str`: An HTML string with style markup rWz?
z4
)r\)r~Z css_stylesZ children_textrrrrfws   rf)# itertoolsosrwstringrtypingrrrrrrr Z tokenizersr r pathdirname__file__r\Z css_filenameopenfreadr`r r{rr}rrr*r+rrfrrrrs& $  &  6