a h7@sddlZddlmZddlmZmZmZddlZddl m Z m Z ddl m Z ddlmZerhddlmZe eZd d iZd Zed d Gddde ZdgZdS)N)copyfile) TYPE_CHECKINGAnyOptional) AddedTokenPreTrainedTokenizer)logging)requires) TextInput vocab_fileztokenizer.modelu▁) sentencepiece)backendsc s(eZdZdZeZddgZd0eee e fd fd d Z ddZ ddZ eddZddZdee dfdd ZddZddZddZd d!Zd1ee ee d"d#d$Zd2d%d&Zd3eeeeeeeed'fd(d) Zd4eeeeeeed*d+d,Zd5eeeee d-d.d/ZZS)6GemmaTokenizera Construct a Gemma tokenizer. Based on byte-level Byte-Pair-Encoding. The default padding token is unset as there is no padding token in the original model. Args: vocab_file (`str`): Path to the vocabulary file. unk_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. eos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. sp_model_kwargs (`dict[str, Any]`, `Optional`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. add_bos_token (`bool`, *optional*, defaults to `True`): Whether or not to add an `bos_token` at the start of sequences. add_eos_token (`bool`, *optional*, defaults to `False`): Whether or not to add an `eos_token` at the end of sequences. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`): Whether or not to cleanup spaces after decoding, cleanup consists in removing potential artifacts like extra spaces. use_default_system_prompt (`bool`, *optional*, defaults to `False`): Whether or not the default system prompt for Gemma should be used. spaces_between_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to add spaces between special tokens. Z input_idsZattention_maskNTF)sp_model_kwargsc s|dur in||_t|tr*t|dddn|}t|trFt|dddn|}t|trbt|dddn|}t|tr~t|dddn|}||_||_||_| |_tj fi|j|_ |j |t j f|||||||| | | d | dS)NFT) normalizedZspecial) bos_token eos_token unk_token pad_token add_bos_token add_eos_tokenrclean_up_tokenization_spacesuse_default_system_promptspaces_between_special_tokens)r isinstancestrrr rrrspmSentencePieceProcessorsp_modelLoadsuper__init__) selfr rrrrrrrrrrkwargs __class__h/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/gemma/tokenization_gemma.pyr&^s2  zGemmaTokenizer.__init__cCs$|j}d|d<|j|d<|S)Nr#sp_model_proto)__dict__copyr#serialized_model_proto)r'stater+r+r, __getstate__s zGemmaTokenizer.__getstate__cCs2|j|tjfi|j|_|j|jdSN)r.updater!r"rr#ZLoadFromSerializedProtor-)r'dr+r+r, __setstate__s zGemmaTokenizer.__setstate__cCs |jS)zReturns vocab size)r#Zget_piece_sizer'r+r+r, vocab_sizeszGemmaTokenizer.vocab_sizecs(fddtjD}|j|S)zReturns vocab as a dictcsi|]}||qSr+)Zconvert_ids_to_tokens).0ir7r+r, z,GemmaTokenizer.get_vocab..)ranger8r4Zadded_tokens_encoder)r'Zvocabr+r7r, get_vocabs zGemmaTokenizer.get_vocabr )textreturnc stj|fi|S)ze Args: text: TextInput Simply calls PreTrainedTokenizer's method )r%tokenizer'r?r(r)r+r,rAszGemmaTokenizer.tokenizecKs|jj|tdS)z Args: text: TextInput Returns a tokenized string. The Gemma tokenizer never adds a prefix space. )Zout_type)r#encoder rBr+r+r, _tokenizeszGemmaTokenizer._tokenizecCs |j|S)z0Converts a token (str) in an id using the vocab.)r#Z piece_to_id)r'tokenr+r+r,_convert_token_to_idsz#GemmaTokenizer._convert_token_to_idcCs|j|}|S)z=Converts an index (integer) in a token (str) using the vocab.)r#Z IdToPiece)r'indexrEr+r+r,_convert_id_to_tokens z#GemmaTokenizer._convert_id_to_tokencCsTg}d}|D]2}||jvr4||j||7}g}q ||q ||j|7}|S)z:Converts a sequence of tokens (string) in a single string.)Z_added_tokens_encoderr#decodeappend)r'tokensZcurrent_sub_tokensZ out_stringrEr+r+r,convert_tokens_to_strings  z'GemmaTokenizer.convert_tokens_to_string)filename_prefixr@cCstj|s"td|ddStj||r6|dndtd}tj|jtj|krztj |jrzt |j|nLtj |jst |d$}|j }||Wdn1s0Y|fS)a Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. Returns: `Tuple(str)`: Paths to the files saved. zVocabulary path (z) should be a directoryN-rIr wb)ospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathr isfileropenr#r0write)r'Zsave_directoryrNZout_vocab_filefiZcontent_spiece_modelr+r+r,save_vocabularys (  (zGemmaTokenizer.save_vocabularycCsL|jr|jgng}|jr |jgng}|||}|durH||||}|Sr3)r bos_token_idr eos_token_idr' token_ids_0 token_ids_1r^r_outputr+r+r, build_inputs_with_special_tokenss  z/GemmaTokenizer.build_inputs_with_special_tokens)rarbalready_has_special_tokensr@cs|rtj||ddS|jr"dgng}|jr2dgng}|durT|dgt||S|dgt|||dgt||S)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rarbreNr)r%get_special_tokens_maskrrlen)r'rarbrer^r_r)r+r,rgs(  z&GemmaTokenizer.get_special_tokens_mask)rarbr@cCs`|jr|jgng}|jr |jgng}dgt|||}|dur\|dgt|||7}|S)a Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`list[int]`): List of ids. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). rNrf)rr^rr_rhr`r+r+r,$create_token_type_ids_from_sequencess z3GemmaTokenizer.create_token_type_ids_from_sequences) token_idsskip_special_tokensrr@cKsg}g}|D]V}|r ||jvr q ||jvrX|r@||j|||j|jg}q ||q |rz||j||rd|}n d|}|tdS)N rI) Zall_special_idsZ_added_tokens_decoderrKr#rJcontentrVreplaceSPIECE_UNDERLINE)r'rjrkrr(Z sub_textsZcurrent_sub_textZidsr+r+r,_decode1s"    zGemmaTokenizer._decode) rrrrNTFFFF)N)N)NF)N)FF) __name__ __module__ __qualname____doc__rWZvocab_files_namesZmodel_input_namesrdictr rr&r2r6propertyr8r>listrArDrFrHrMtupler]rdintboolrgrirp __classcell__r+r+r)r,r+s\.*  & $r)rQshutilrtypingrrrr r!Ztokenization_utilsrrutilsr Zutils.import_utilsr Ztokenization_utils_baser Z get_loggerrqrTrWror__all__r+r+r+r,s      %