a h1#@sdZddlZddlmZddlmZmZddlZddl m Z ddl m Z ddl mZd d lmZe eZd d iZed dGddde ZdgZdS)z Tokenization class for SpeechT5.N)copyfile)AnyOptional)PreTrainedTokenizer)logging)requires)EnglishNumberNormalizer vocab_filezspm_char.model) sentencepiece)backendscseZdZdZeZddgZd,eee e fd d fd d Z d-d dZ e ddZe ddZejddZddZddZddZe ee dddZddZdd Zd!d"Zd.eed#d$d%Zd/eeeeeeeed&fd'd( Zd0e ee ee d)d*d+ZZS)1SpeechT5Tokenizera Construct a SpeechT5 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that contains the vocabulary necessary to instantiate a tokenizer. bos_token (`str`, *optional*, defaults to `""`): The begin of sequence token. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. normalize (`bool`, *optional*, defaults to `False`): Whether to convert numeric quantities in the text to their spelt-out english counterparts. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Attributes: sp_model (`SentencePieceProcessor`): The *SentencePiece* processor that is used for every conversion (string, tokens and IDs). Z input_idsZattention_maskFN)sp_model_kwargsreturnc sj|dur in||_||_||_d|_tjfi|j|_|j|tj f||||||jd|dS)N) bos_token eos_token unk_token pad_token normalizer) rr r _normalizerspmSentencePieceProcessorsp_modelLoadsuper__init__) selfr rrrrrrkwargs __class__n/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/speecht5/tokenization_speecht5.pyr Qs  zSpeechT5Tokenizer.__init__cKs0|d|j}|rd|}|r(||}||fS)Nr )popr normalizer)r!textZis_split_into_wordsr"rr%r%r&prepare_for_tokenizationns  z*SpeechT5Tokenizer.prepare_for_tokenizationcCs |jSN)rZget_piece_sizer!r%r%r& vocab_sizevszSpeechT5Tokenizer.vocab_sizecCs|jdurt|_|jSr,)rr r-r%r%r&r)zs zSpeechT5Tokenizer.normalizercCs ||_dSr,)r)r!valuer%r%r&r)scs(fddtjD}|j|S)Ncsi|]}||qSr%)Zconvert_ids_to_tokens).0ir-r%r& z/SpeechT5Tokenizer.get_vocab..)ranger.updateZadded_tokens_encoder)r!Zvocabr%r-r& get_vocabs zSpeechT5Tokenizer.get_vocabcCs|j}d|d<|S)Nr)__dict__copy)r!stater%r%r& __getstate__s zSpeechT5Tokenizer.__getstate__cCs<||_t|dsi|_tjfi|j|_|j|jdS)Nr)r7hasattrrrrrrr )r!dr%r%r& __setstate__s  zSpeechT5Tokenizer.__setstate__)r*rcCs|jj|tdS)zPTake as input a string and return a list of strings (tokens) for words/sub-words)Zout_type)rencodestr)r!r*r%r%r& _tokenizeszSpeechT5Tokenizer._tokenizecCs |j|S)z0Converts a token (str) in an id using the vocab.)rZ piece_to_id)r!tokenr%r%r&_convert_token_to_idsz&SpeechT5Tokenizer._convert_token_to_idcCs|j|}|S)z=Converts an index (integer) in a token (str) using the vocab.)rZ IdToPiece)r!indexrAr%r%r&_convert_id_to_tokens z&SpeechT5Tokenizer._convert_id_to_tokencCspg}d}d}|D]F}||jvrH|s*|d7}||j||7}d}g}q||d}q||j|7}|S)z:Converts a sequence of tokens (string) in a single string.Fr'T)Zall_special_tokensrdecodeappendstrip)r!tokensZcurrent_sub_tokensZ out_stringZprev_is_specialrAr%r%r&convert_tokens_to_strings  z*SpeechT5Tokenizer.convert_tokens_to_string)rcCs$|dur||jgS|||jgS)z=Build model inputs from a sequence by appending eos_token_id.N)Z eos_token_id)r! token_ids_0 token_ids_1r%r%r& build_inputs_with_special_tokenss z2SpeechT5Tokenizer.build_inputs_with_special_tokens)rKrLalready_has_special_tokensrcsV|rtj||ddSdg}|dur6dgt||Sdgt|dgt||S)NT)rKrLrNr r)rget_special_tokens_masklen)r!rKrLrNZ suffix_onesr#r%r&rOsz)SpeechT5Tokenizer.get_special_tokens_mask)save_directoryfilename_prefixrcCstj|s"td|ddStj||r6|dndtd}tj|jtj|krztj |jrzt |j|nLtj |jst |d$}|j }||Wdn1s0Y|fS)NzVocabulary path (z) should be a directory-rEr wb)ospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathr isfileropenrZserialized_model_protowrite)r!rQrRZout_vocab_filefiZcontent_spiece_modelr%r%r&save_vocabularys (  (z!SpeechT5Tokenizer.save_vocabulary)rrrrFN)F)N)NF)N) __name__ __module__ __qualname____doc__r[Zvocab_files_namesZmodel_input_namesrdictr?rr r+propertyr.r)setterr6r:r=listr@rBrDrJintrMboolrOtuplera __classcell__r%r%r#r&r"sD*      r)rerUshutilrtypingrrr rZtokenization_utilsrutilsrZutils.import_utilsrZnumber_normalizerr Z get_loggerrbrXr[r__all__r%r%r%r&s      =