a h&@sdZddlZddlZddlmZddlmZmZer>ddlZddl m Z ddl m Z e eZdd iZd d ZGd d d ZGddde ZdgZdS)z Tokenization classes for CPMAnt.N)Optional)is_jieba_availablerequires_backends)PreTrainedTokenizer)logging vocab_filez vocab.txtcCsft}t|ddd}|}Wdn1s40Yt|D]\}}|d}|||<qF|S)z*Loads a vocabulary file into a dictionary.rutf-8encodingN ) collections OrderedDictopen readlines enumeraterstrip)rvocabreadertokensindextokenrj/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/cpmant/tokenization_cpmant.py load_vocab$s&  rc@seZdZdddZddZdS) WordpieceTokenizercCs||_||_||_dSN)r unk_tokenmax_input_chars_per_word)selfrr r!rrr__init__0szWordpieceTokenizer.__init__cCst|}t||jkr|jgSd}g}|t|krt|}d}||krrd|||}||jvrh|}qr|d8}q>|dur||j|d7}q&|||}q&|S)Nr)listlenr!r joinrappend)r"rcharsstartZ sub_tokensendZ cur_substrsubstrrrrtokenize5s(      zWordpieceTokenizer.tokenizeN)rr)__name__ __module__ __qualname__r#r.rrrrr/s rc seZdZdZeZddgZdZd2fdd Ze ddZ e ddZ e ddZ e e dddZddZddZfddZdd Zeeed!d"d#Zd$d%Zd&d'Zd3eeeeed)d*d+Zd4ee eee ee d,d-d.Zd5ee eee eee d/fd0d1 ZZS)6CpmAntTokenizera Construct a CPMAnt tokenizer. Based on byte-level Byte-Pair-Encoding. Args: vocab_file (`str`): Path to the vocabulary file. bod_token (`str`, *optional*, defaults to `""`): The beginning of document token. eod_token (`str`, *optional*, defaults to `""`): The end of document token. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str`, *optional*, defaults to `""`): The token used for padding. unk_token (`str`, *optional*, defaults to `""`): The unknown token. line_token (`str`, *optional*, defaults to `""`): The line token. space_token (`str`, *optional*, defaults to `""`): The space token. Z input_idsZattention_maskFrleftc st|dg||_||_t||_|j| |jd<|j||jd<|j| =|j|=tt|jddd|_dd|jD|_ t |j|d |_ t j f|||||||| | d | dS) Njieba r cSs|dSNr%rxrrrz*CpmAntTokenizer.__init__..keycSsi|]\}}||qSrr).0kvrrr rAz,CpmAntTokenizer.__init__..)rr ) bod_token eod_token bos_token eos_token pad_tokenr line_token space_token padding_side)rrHrIrencoderrrsorteditemsdecoderrwordpiece_tokenizersuperr#) r"rrHrIrJrKrLr rMrNrOkwargs __class__rrr#ls0   zCpmAntTokenizer.__init__cCs |j|jSr)rPrHr"rrr bod_token_idszCpmAntTokenizer.bod_token_idcCs |j|jSr)rPrIrYrrr eod_token_idszCpmAntTokenizer.eod_token_idcCs |jdS)Nr rPrYrrr newline_idszCpmAntTokenizer.newline_id)returncCs t|jSr)r'rPrYrrr vocab_sizeszCpmAntTokenizer.vocab_sizecCst|jfi|jSr)dictrPZadded_tokens_encoderrYrrr get_vocabszCpmAntTokenizer.get_vocabcCs.g}tj|ddD]}||j|q|S)zTokenize a string.F)Zcut_all)r;cutextendrTr.)r"textZ output_tokensr?rrr _tokenizeszCpmAntTokenizer._tokenizec s4dd|D}fdd|D}tj|fi|S)zDecode ids into a string.cSsg|]}|dkr|qS)rr)rDirrr rAz+CpmAntTokenizer._decode..cs.g|]&}|jkr|jkr|jkr|qSr)Z pad_token_idZ eos_token_id bos_token_id)rDr?rYrrrgs)rU_decode)r"Z token_idsrVrWrYrris  zCpmAntTokenizer._decodecCs ||jvSrr\r"rrrrcheckszCpmAntTokenizer.check)rr^cCs d|S)Nr$)r()r"rrrrconvert_tokens_to_stringsz(CpmAntTokenizer.convert_tokens_to_stringcCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)rPgetr rjrrr_convert_token_to_idsz$CpmAntTokenizer._convert_token_to_idcCs|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)rSrmr )r"rrrr_convert_id_to_tokensz$CpmAntTokenizer._convert_id_to_tokenN)save_directoryfilename_prefixr^cCs(tj|r0tj||r |dndtd}n|r<|dnd|}d}d|jvrj|jd|jd<|jd=d|jvr|jd|jd<|jd=tt|j d d d |_t |d d dX}|j D]<\}}||krt d|d|}| |d|d7}qWdn1s0Y|fS)N-r$rrr<r9r r8cSs|dSr=rr>rrrr@rAz1CpmAntTokenizer.save_vocabulary..rBwr r zSaving vocabulary to z\: vocabulary indices are not consecutive. Please check that the vocabulary is not corrupted!r%)ospathisdirr(VOCAB_FILES_NAMESrPrrrQrRrloggerwarningwrite)r"rprqrrwriterrZ token_indexrrrsave_vocabularys.    *zCpmAntTokenizer.save_vocabulary) token_ids_0 token_ids_1r^cCs,|dur|jg|S|jg||jg|S)a1 Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A CPMAnt sequence has the following format: - single sequence: `[BOS] Sequence`. Args: token_ids_0 (`list[int]`): The first tokenized sequence that special tokens will be added. token_ids_1 (`list[int]`): The optional second tokenized sequence that special tokens will be added. Returns: `list[int]`: The model input with special tokens. N)rh)r"r}r~rrr build_inputs_with_special_tokenss z0CpmAntTokenizer.build_inputs_with_special_tokens)r}r~already_has_special_tokensr^csZ|rtj||ddS|durFdgdgt|dgdgt|Sdgdgt|S)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)r}r~rNr%r)rUget_special_tokens_maskr')r"r}r~rrWrrrs(z'CpmAntTokenizer.get_special_tokens_mask) r3r4r5r6r7rr8r9r:)N)N)NF)r/r0r1__doc__rwZvocab_files_namesZmodel_input_namesZadd_prefix_spacer#propertyrZr[r]intr_rarerirkr&strrlrnrortupler|rboolr __classcell__rrrWrr2OsP*     r2)rrrttypingrZtransformers.utilsrrr;Ztokenization_utilsrutilsrZ get_loggerr/rxrwrrr2__all__rrrrs      B