a h8@sdZddlZddlZddlmZddlZddlmZddl m Z e e Z ddd Zd d d d ddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d6d7d8d9d:d;dd?d@dA7ZdBdCZGdDdEdEeZdEgZdS)Fz)Tokenization classes for Salesforce CTRL.N)Optional)PreTrainedTokenizer)loggingz vocab.jsonz merges.txt) vocab_file merges_fileiiiiii#jivi~i6iivii.iiiךiͨii%iii3iR-iniS.iKiiwii[i*iii/i?iin1iipiiiiϒi i)i-i(iiKiiiǢiihi)7Z PregnancyZ ChristianityZExplainZFitnessZSavingZAskZAssZJokeZ QuestionsZThoughtsZRetailZFeminismZWritingZAtheismZNetflixZ ComputingZOpinionZAloneFunnyZGamingZHumanZIndiaZJokerZDietZLegalZNormanZTipZWeightZMoviesZRunningZScienceZHorrorZ ConfessionZFinanceZPoliticsZScaryZSupportZ TechnologiesZTeenageEventZLearnedZNotionZ WikipediaZBooksZExtractZ ConfessionsZ ConspiracyZLinksZ NarcissusZ RelationshipZ RelationshipsZReviewsZNewsZ TranslationZ multilingualcCs>t}|d}|ddD]}|||f|}qt|}|S)z Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rN)setadd)wordpairsZ prev_charcharrf/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/ctrl/tokenization_ctrl.py get_pairs^srcseZdZdZeZeZdfdd Ze ddZ ddZ d d Z d d Z d dZddZddZdeeeeedddZZS) CTRLTokenizera` Construct a CTRL tokenizer. Based on Byte-Pair-Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. c st|dd}t||_Wdn1s.0Ydd|jD|_t|dd&}|ddd}Wdn1s0Ydd |D}tt |t t ||_ i|_ tjfd |i|dS) Nutf-8encodingcSsi|]\}}||qSrr).0kvrrr z*CTRLTokenizer.__init__.. r cSsg|]}t|qSr)tuplesplit)rmergerrr rz*CTRLTokenizer.__init__.. unk_token)openjsonloadencoderitemsdecoderreadr dictziprangelen bpe_rankscachesuper__init__)selfrrr#kwargsZ vocab_handleZ merges_handleZmerges __class__rrr2s*4zCTRLTokenizer.__init__cCs t|jSN)r.r'r3rrr vocab_sizeszCTRLTokenizer.vocab_sizecCst|jfi|jSr7)r+r'Zadded_tokens_encoderr8rrr get_vocabszCTRLTokenizer.get_vocabc s|jvrj|St|}tt|dd|ddg}t|}|sN|St|fddd}|jvrpql|\}}g}d}|t|krBz|||} Wn*ty| ||dYqBYn0| ||| | }|||kr*|t|dkr*||d|kr*| |||d7}q| |||d7}qt|}|}t|dkrbqlqNt|}qNd |}|dd }|j|<|S) Nrzcsj|tdS)Ninf)r/getfloat)pairr8rrrz#CTRLTokenizer.bpe..keyrr @@ ) r0rlistrminr/r.index ValueErrorextendappendjoin) r3tokenr rZbigramfirstsecondZnew_wordijrr8rbpesF  "   2      zCTRLTokenizer.bpecCs8g}td|}|D]}|t||dq|S)zTokenize a string.z\S+\n? )refindallrIrErQr )r3textZ split_tokenswordsrLrrr _tokenizes  zCTRLTokenizer._tokenizecCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)r'r<r#)r3rLrrr_convert_token_to_idsz"CTRLTokenizer._convert_token_to_idcCs|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)r)r<r#)r3rGrrr_convert_id_to_tokensz"CTRLTokenizer._convert_id_to_tokencCsd|dd}|S)z:Converts a sequence of tokens (string) in a single string.rRrC)rKreplacestrip)r3tokensZ out_stringrrrconvert_tokens_to_stringsz&CTRLTokenizer.convert_tokens_to_stringN)save_directoryfilename_prefixreturnc CsTtj|s"td|ddStj||r6|dndtd}tj||rX|dndtd}t|ddd .}|t j |j d d d d dWdn1s0Yd}t|ddd v}|dt |j dddD]D\}} || krtd|d| }|d|d|d7}qWdn1sB0Y||fS)NzVocabulary path (z) should be a directory-rZrrwrrrBTF)indent sort_keys ensure_asciirrz#version: 0.2 cSs|dS)Nr r)kvrrrr?rz/CTRLTokenizer.save_vocabulary..r@zSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rRr )ospathisdirloggererrorrKVOCAB_FILES_NAMESr$writer%dumpsr'sortedr/r(warning) r3r_r`rZ merge_filefrGwriterZ bpe_tokensZ token_indexrrrsave_vocabularys. <   *zCTRLTokenizer.save_vocabulary)r)N)__name__ __module__ __qualname____doc__rmZvocab_files_names CONTROL_CODES control_codesr2propertyr9r:rQrWrXrYr^strrrrt __classcell__rrr5rrns  , r)rxr%rhtypingrregexrSZtokenization_utilsrutilsrZ get_loggerrurkrmryrr__all__rrrrs    ;