a h @sUdZddlmZddlmZmZddlmZee Z dZ dZ dZ dZd Zd Zd Ze d ed edede dediZeeefed<ddeDZeeefed<GdddeZdgZdS)z Tokenization classes for CANINE.)Optional) AddedTokenPreTrainedTokenizer)loggingiiiiiiz[CLS]z[SEP]z[BOS]z[MASK]z[PAD]z [RESERVED]SPECIAL_CODEPOINTScCsi|]\}}||qSr).0 codepointnamerrj/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/canine/tokenization_canine.py 7r SPECIAL_CODEPOINTS_BY_NAMEcseZdZdZeeeeeeeeeeeeddffdd Z e e dddZ d d Z eeed d d Zee dddZe edddZddZd ee eee ee dddZd!ee eee eee dfdd Zd"eeedddZZS)#CanineTokenizera Construct a CANINE tokenizer (i.e. a character splitter). It turns text into a sequence of characters, and then converts each character into its Unicode code point. [`CanineTokenizer`] inherits from [`PreTrainedTokenizer`]. Refer to superclass [`PreTrainedTokenizer`] for usage examples and documentation concerning parameters. Args: model_max_length (`int`, *optional*, defaults to 2048): The maximum sentence length the model accepts. Fic st|trt|dddn|}t|tr4t|dddn|}t|trPt|dddn|}t|trlt|dddn|}t|trt|dddn|}t|trt|dddn|}i|_tD]\} } | |j| <qdd|jD|_t|_t |j|_ t j f||||||||d| dS)NF)lstriprstripTcSsi|]\}}||qSrr)r r r rrr r csz,CanineTokenizer.__init__..) bos_token eos_token sep_token cls_token pad_token mask_tokenadd_prefix_spacemodel_max_length) isinstancestrrZ_special_codepointsritemsZ_special_codepoint_stringsUNICODE_VOCAB_SIZE_unicode_vocab_sizelenZ_num_special_tokenssuper__init__) selfrrrrrrrrkwargsr r  __class__rr r"Hs4    zCanineTokenizer.__init__)returncCs|jS)N)r)r#rrr vocab_sizevszCanineTokenizer.vocab_sizecCs$ddt|jD}||j|S)NcSsi|]}t||qSr)chr)r irrr r {rz-CanineTokenizer.get_vocab..)ranger(updateZadded_tokens_encoder)r#Zvocabrrr get_vocabzs zCanineTokenizer.get_vocab)textr'cCst|S)z5Tokenize a string (i.e. perform character splitting).)list)r#r.rrr _tokenizeszCanineTokenizer._tokenize)tokenr'cCs2z t|WSty,td|dYn0dS)zaConverts a token (i.e. a Unicode character) in an id (i.e. its integer Unicode code point value).zinvalid token: ''N)ord TypeError ValueError)r#r1rrr _convert_token_to_ids  z$CanineTokenizer._convert_token_to_id)indexr'cCsBz|tvrt|WSt|WSty<td|Yn0dS)z Converts a Unicode code point (integer) in a token (str). In case it's a special code point, convert to human-readable format. z invalid id: N)rr)r4r5)r#r7rrr _convert_id_to_tokens    z$CanineTokenizer._convert_id_to_tokencCs d|S)N)join)r#tokensrrr convert_tokens_to_stringsz(CanineTokenizer.convert_tokens_to_stringN) token_ids_0 token_ids_1r'cCs4|jg}|jg}|||}|dur0|||7}|S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A CANINE sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences: `[CLS] A [SEP] B [SEP]` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N)Z sep_token_idZ cls_token_id)r#r=r>sepclsresultrrr build_inputs_with_special_tokenss   z0CanineTokenizer.build_inputs_with_special_tokens)r=r>already_has_special_tokensr'csT|rtj||ddSdgdgt|dg}|durP|dgt|dg7}|S)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)r=r>rCrN)r!get_special_tokens_maskr )r#r=r>rCrAr%rr rEsz'CanineTokenizer.get_special_tokens_mask)save_directoryfilename_prefixcCsdS)Nrr)r#rFrGrrr save_vocabularyszCanineTokenizer.save_vocabulary)N)NF)N)__name__ __module__ __qualname____doc__r)CLSSEPPADMASKr"propertyintr(r-rr/r0r6r8r<rrBboolrErH __classcell__rrr%r r:s8.  rN)rLtypingrZtokenization_utilsrrutilsrZ get_loggerrIloggerrrOrMrNZBOSrPZRESERVEDrdictrRr__annotations__rrr__all__rrrr s,    "