a h@s^dZddlZddlZddlZddlmZddlmZddlm Z m Z m Z ddl m ZddlmZddlmZddlmZdd lmZmZmZmZd d lmZd d lmZd d lm Z d dl!m"Z"d dl#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-d dl.m/Z/m0Z0m1Z1e12e3Z4dZ5dZ6dZ7dZ8dZ9e$d7Z$eeeedZ:e5e8dZ;e0e$Gddde)Zd?Z7dd e-j0e.j1ddZ?S)UPreTrainedTokenizerFastaQ Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handles all the shared methods for tokenization and special tokens, as well as methods for downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary. This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). Nslow_tokenizer_classcs|dd}|dd}|dd}|dd}|dd}|di}|dd|_|rt|durt|jdurttd |durt|} n|dur|st|} n|rt |} n|durt |d } | d d } | d } | d} t | | \} }| | t |dkr~| |nj|jdurD|durD|j|i|}t |} n:|sv|d |_|dg|_t |dd} d}ntd| |_|dur| |jd|_|jj}|dur|jjfi||d|d|d|d|d|d|d|dn |j|jj}|dur|jjfi||d|d|d|d|d|d|d|d|d|dtjfi||j|j_d d!|jDfd"d#t|d$d%d&Dt |j!"d'd#Dfd(d#|j#D7t dkrg}|j$}D]\}t%|t&rH|j'pRt(||vn t(||v}t%|t(rnt&||d)}n||_'|)|q$|r|*|zXt+,|j-j./}|d|j|jkrt0t1|d*}|j|d<|fi||j-_.Wnt2yYn0dS)+Ntokenizer_objectZ__slow_tokenizer gguf_filer$ from_slowFadded_tokens_decoderadd_prefix_spacezCannot instantiate this tokenizer from a slow version. If it's based on sentencepiece, make sure you have sentencepiece installed.r%configZ model_type tokenizertokenizer_configradditional_special_tokensT)Z from_tiktokena9Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one. max_lengthtruncation_side directionstridetruncation_strategystrategy pad_tokenpad_token_type_id pad_type_id padding_sidelengthpad_to_multiple_ofcSsh|]}tt|qShashrepr.0tokenr=r=`/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py z3PreTrainedTokenizerFast.__init__..cs$g|]\}}tt|vr|qSr=r>)rBindexrC)added_tokens_decoder_hashr=rD sz4PreTrainedTokenizerFast.__init__..cSs|dSNrr=)xr=r=rDrFz2PreTrainedTokenizerFast.__init__..keycSsg|] }t|qSr=)strrAr=r=rDrIrFcs g|]}|vr|vr|qSr=r=rA)encoder tokens_to_addr=rDrIs)specialtype)3popgetr,r' ValueErrorcopydeepcopy TokenizerFast from_filerrrupdatelenr%r0 _tokenizer init_kwargs_decode_use_source_tokenizer truncationenable_truncation setdefault no_truncationpaddingenable_paddingsuper__init__split_special_tokensencode_special_tokensr+sorteditemslistadded_tokens_encoderkeysZall_special_tokens_extendedZall_special_tokens isinstancerrRrOappend add_tokensjsonloadsbackend_tokenizer pre_tokenizer __getstate__getattrpre_tokenizers_fast Exception)selfargskwargsr(Zslow_tokenizerr)Zfast_tokenizer_filer*r+Zfast_tokenizerZ gguf_param architectureZtokenizer_dictr/Zadditional_kwargs _truncation_paddingtokensspecial_tokensrCZ is_specialZ pre_tok_stateZ pre_tok_class __class__)rHrPrQrDrgbs                            z PreTrainedTokenizerFast.__init__)returncCsdS)NTr=rzr=r=rDis_fastszPreTrainedTokenizerFast.is_fastcCsDd|jvr<|jddrrFz@PreTrainedTokenizerFast.added_tokens_encoder..cSs|dSrJr=itemr=r=rDrLrFz>PreTrainedTokenizerFast.added_tokens_encoder..rMrjr+rkrr=r=rDrmsz,PreTrainedTokenizerFast.added_tokens_encodercCs |jS)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `dict[str, int]`: The added tokens. )r]Zget_added_tokens_decoderrr=r=rDr+sz,PreTrainedTokenizerFast.added_tokens_decodercCs ddt|jdddDS)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `dict[str, int]`: The added tokens. cSsi|]\}}|j|qSr=rrr=r=rDrrFz;PreTrainedTokenizerFast.get_added_vocab..cSs|dSrJr=rr=r=rDrLrFz9PreTrainedTokenizerFast.get_added_vocab..rMrrr=r=rDget_added_vocabsz'PreTrainedTokenizerFast.get_added_vocabcCsdS)zN Returns True, to avoid expensive `assert tokenizer` gotchas. Tr=rr=r=rD__bool__sz PreTrainedTokenizerFast.__bool__cCs|jjddS)zD Size of the full vocabulary with the added tokens. Trrrr=r=rD__len__szPreTrainedTokenizerFast.__len__cCs|jS)zc `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )r]rr=r=rDrt%sz)PreTrainedTokenizerFast.backend_tokenizercCs|jjS)zU `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )r]decoderrr=r=rDr,szPreTrainedTokenizerFast.decoderFT) encodingreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverboserc Cs|durd|jv}|dur$d|jv}|r@|jdur@|g|j} n|g} tt} | D]|} | d| j|rz| d| j|r| d| j|r| d| j|r| d| j |rR| dt | jqR| | fS)a Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) NZtoken_type_idsattention_mask input_idsspecial_tokens_maskZoffset_mappingr;) Zmodel_input_namesZ overflowingrrlrpidsZtype_idsrroffsetsr\) rzrrrrrrrr encodingsZ encoding_dicter=r=rD_convert_encoding3s*  z)PreTrainedTokenizerFast._convert_encoding)rrcs&t|tr|Sfdd|DS)aX Converts a token string (or a sequence of tokens) in a single integer id (or a Iterable of ids), using the vocabulary. Args: tokens (`str` or `Iterable[str]`): One or several token(s) to convert to token id(s). Returns: `int` or `list[int]`: The token id or list of token ids. csg|]}|qSr=)#_convert_token_to_id_with_added_vocrArr=rDrIprFzAPreTrainedTokenizerFast.convert_tokens_to_ids..)rorOrrzrr=rrDconvert_tokens_to_idsbs  z-PreTrainedTokenizerFast.convert_tokens_to_ids)rCrcCs|j|}|dur|jS|Sr)r] token_to_idZ unk_token_id)rzrCrGr=r=rDrrs z;PreTrainedTokenizerFast._convert_token_to_id_with_added_voc)rGrcCs|jt|Sr)r] id_to_tokenint)rzrGr=r=rD_convert_id_to_tokenxsz,PreTrainedTokenizerFast._convert_id_to_token) new_tokensrcCs|r|j|S|j|Sr)r]add_special_tokensrq)rzrrr=r=rD _add_tokens{s z#PreTrainedTokenizerFast._add_tokens)pairrcCs |j|S)aG Returns the number of added tokens when encoding a sequence with special tokens. This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )r]num_special_tokens_to_add)rzrr=r=rDrsz1PreTrainedTokenizerFast.num_special_tokens_to_add)rskip_special_tokensrcCs`t|tr|j|Sg}|r(t|jnt}|D](}t|}||vrHq2||j|q2|S)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `list[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `list[str]`: The decoded token(s). )rorr]rsetZall_special_idsrp)rzrrrZ ids_to_skiprGr=r=rDconvert_ids_to_tokenss  z-PreTrainedTokenizerFast.convert_ids_to_tokens)textrrrcKs|jf|||d|S)N)r text_pairr)Z encode_plusr)rzrrrr|r=r=rDtokenizesz PreTrainedTokenizerFast.tokenizepadding_strategyr5r1r4r<r:c s|jj|jj}|tjkr.durz|jnL|||j|jd}durNd} nfdd|D} | |krz|jjfi||t j kr|dur|j nR|t j kr|nd} | |dur|n|j |j|j|j|d}||kr|jjfi|dS)a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. N)r1r4r6r3csi|]}||dqSrrU)rBrr~r=rDrrFzFPreTrainedTokenizerFast.set_truncation_and_padding..)r;r3Zpad_idr7r9r<)r]r`rdrDO_NOT_TRUNCATErcvaluer2rar DO_NOT_PADZ no_paddingZ MAX_LENGTHr:Z pad_token_idr7r8re) rzrr5r1r4r<r:rtargetcurrentr;r=rrDset_truncation_and_paddings8!     z2PreTrainedTokenizerFast.set_truncation_and_paddingr)batch_text_or_text_pairsrrr5r1r4is_split_into_wordsr<r:return_tensorsrrrrrrrrhrcs$t|ttfs"tdt|dj|||||| djj|krL|j_jj|||d}fdd|D}i}|ddD]fdd|D}||<qd d|D}rg}t |D]"\}\}}||gt |d 7}q||d <|d D]} ||qt ||| d S) Nz:batch_text_or_text_pairs has to be a list or a tuple (got )r)rZis_pretokenizedc s&g|]}j|dqS))rrrrrrrr)r)rBr)rrrrrrrzrr=rDrI5s z>PreTrainedTokenizerFast._batch_encode_plus..rcs"g|]\}}|D]}|qqSr=r=)rBr_rrMr=rDrIKrFcSsg|]\}}|D]}|qqSr=r=)rBrrrr=r=rDrIMrFroverflow_to_sample_mapping)Z tensor_type) rotuplerl TypeErrorrSrr]riZ encode_batch enumerater\&_eventual_warn_about_too_long_sequencer)rzrrrr5r1r4rr<r:rrrrrrrrrhrZtokens_and_encodingsZsanitized_tokensstackZsanitized_encodingsritoksrrr=) rNrrrrrrrzrrD_batch_encode_plussF     z*PreTrainedTokenizerFast._batch_encode_plus)rrrrr5r1r4rr<r:rrrrrrrrrhrcKs|r||fgn|g}|j|f||||||| | | | | ||||||d|}| durp|sptdd|D|j}||d|||S)N)rrrr5r1r4r<r:rrrrrrrrrhcSs8i|]0\}}|t|dkr0t|dtr0|dn|qS)r)r\rorl)rBrNrr=r=rDrsz8PreTrainedTokenizerFast._encode_plus..r)rrrkrr)rzrrrrr5r1r4rr<r:rrrrrrrrrhr|Z batched_inputZbatched_outputr=r=rD _encode_plus[sB z$PreTrainedTokenizerFast._encode_pluscCs$|jjdur|jj|Sd|S)N )rtrdecodejoinrr=r=rDconvert_tokens_to_strings z0PreTrainedTokenizerFast.convert_tokens_to_string) token_idsrclean_up_tokenization_spacesrcKsZ|dd|_t|tr|g}|jj||d}|dur:|n|j}|rR||}|S|SdS)NZuse_source_tokenizerF)r)rTr_rorr]rrZclean_up_tokenization)rzrrrr|rZ clean_textr=r=rD_decodes  zPreTrainedTokenizerFast._decode)save_directory file_names legacy_formatfilename_prefixrc s:t|}jdur"|dur"td|dus2|duo@jduo@j}|dupP|du}|rtj||rj|dndt}fddj D}|rt |d d d 0} t j |d ddd d} | | Wdn1s0Yj||d} || |f}|r6tj||r|dndt} j| || f}|S)z Save a tokenizer using the slow-tokenizer/legacy format: vocabulary + added tokens as well as in a unique JSON file containing {config + vocab + added-tokens}. NTzYour tokenizer does not have a legacy version defined and therefore cannot register this version. You might consider leaving the legacy_format at `None` or setting it to `False`.F-cs i|]\}}|jkr||qSr=)r)rBtokrGrr=rDrrFz.wzutf-8)r)indent sort_keys ensure_ascii )r)rOr'rVrrrrADDED_TOKENS_FILErmrkopenrrdumpswriteZsave_vocabularyTOKENIZER_FILErtsave) rzrrrrZ save_slowZ save_fastZadded_tokens_fileZ added_vocabfZout_strZ vocab_filesr$r=rrD_save_pretraineds: (  z(PreTrainedTokenizerFast._save_pretrainedc s>t|j}|d}|d} d} |dddkrRi|dd<g|dd<n|ddd kr|dd dur|dd } |dd| d } dur| vr| } d |dd <| d gg|dd<n6|ddd vri|dd<ntd|ddddurBd|dvrB|ddvrB|dd|dd<tt|g} |D]v} | dd}| dd}|ddd kr|sqZdur| dvr| d| d<| t fi| qZ|dur| ||dddkr$d|vr$|dddur$|dd|d<|dddkrbd|vrb|dddurb|dd|d<|ddd kr| dur| |d<|ddur|dddks|dddkrd|dvrt dd|ddDrt j|d<t|dd}|f|| d|}j|||d| durVt}d| vr| dD]}| d|d }dur~fd!d"|D}|| d|d <|D]"}|}|durtd#qfd$d"|D| d|d%<qNd&D]`}|| vr| |\}}dur|vr|}|}|dur.td#||g| |<q| |d<tt||j}tj}|d'|D]}t||durxt||}dur|vr|}|j|d}t|t rt ||j|j|j|jd(d)||<n|||<qx|j }|dur| |t!|d kr*||d'<|j"fd*i|S)+uf Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `list[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs (`dict[str, Any]`, *optional*): Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. added_tokenspost_processorNmodelrSr rZmergesr!unk_idrg)r"r#z;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece. unk_tokenrRidrZcontinuing_subword_prefixZend_of_word_suffixru ByteLevelSequenceZ pretokenizerscss|]}|ddkVqdS)rSrNr=)rBZ pretokenizerr=r=rD PszBPreTrainedTokenizerFast.train_new_from_iterator..Zinitial_alphabet)rr)r;trainerrrcsg|]}||qSr=rrA)special_tokens_mapr=rDrIbrFzCPreTrainedTokenizerFast.train_new_from_iterator..zQAttempted to set a token in the post processor that does not exist in the mappingcsg|]}|qSr=)rrA)r.r=rDrIkrFr)clssepr0T) single_wordlstriprstrip normalizedrRr()#rrrsr]Zto_strrTrVrYZfrom_strrrprextendanyrxralphabetMODEL_TO_TRAINER_MAPPINGZtrain_from_iteratorrr^rWrZSPECIAL_TOKENS_ATTRIBUTESremoverwZ_special_tokens_maprUrorrrrr0r\r)rzZ text_iteratorrr;Znew_special_tokensrr|Ztokenizer_jsonrrrrrZ added_tokenrRrZ trainer_classrZtrained_tokenizer_jsonrNrrCZtoken_idZ special_tokenZspecial_tokens_listZspecial_token_fullr0r=)rr.rDtrain_new_from_iterators"                   "            z/PreTrainedTokenizerFast.train_new_from_iterator)NNFFFFT)F)F)F)NF)FN)NN)NNN)@__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESrr'r__annotations__rgpropertyboolrrrrdictrOrrrmrr+rrrrYrt DecoderFastr EncodingFastrrrrlrrrrrrrrrrrrrrrrrrrrrrrrrPathLikerr __classcell__r=r=rrDr&QsD    |     /(  P ^  =    5r&)=rrWrrr collectionsrcollections.abcrtypingrrrZtokenizers.pre_tokenizersZpre_tokenizersrxZ tokenizersrrrrYZtokenizers.decodersr rZtokenizers.trainersr r r r rZintegrations.ggmlrZmodeling_gguf_pytorch_utilsrZtokenization_utilsrZtokenization_utils_baserrrrrrrrrrutilsrrrZ get_loggerrloggerrZSPECIAL_TOKENS_MAP_FILEZTOKENIZER_CONFIG_FILEZTIKTOKEN_VOCAB_FILErr rr&r=r=r=rDs@          0