a h$@sdZddlZddlZddlZddlmZmZmZddlm Z ddl m Z m Z m Z e r^ddlZe rlddlZe eZddiZd d ZGd d d e Zd gZdS) zTokenization class for VITS.N)AnyOptionalUnion)PreTrainedTokenizer)is_phonemizer_availableis_uroman_availablelogging vocab_filez vocab.jsoncCs td}||}|du}|S)Nz [^\x00-\x7F])recompilesearch) input_stringZnon_roman_patternmatchZ has_non_romanrf/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/vits/tokenization_vits.pyhas_non_roman_characters%s  rc seZdZdZeZddgZd$dd fd d Zed d Z ddZ ddZ ddZ d%e eeeee ee effdddZe ee dddZee e dddZddZdd Zd&e ee eee dfd!d"d#ZZS)' VitsTokenizera Construct a VITS tokenizer. Also supports MMS-TTS. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. language (`str`, *optional*): Language identifier. add_blank (`bool`, *optional*, defaults to `True`): Whether to insert token id 0 in between the other tokens. normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the input text by removing all casing and punctuation. phonemize (`bool`, *optional*, defaults to `True`): Whether to convert the input text into phonemes. is_uroman (`bool`, *optional*, defaults to `False`): Whether the `uroman` Romanizer needs to be applied to the input text prior to tokenizing. Z input_idsZattention_maskNTF)returnc st|dd} t| |_Wdn1s.0Ydd|jD|_||_||_||_||_ ||_ t j f|||||||d| dS)Nutf-8encodingcSsi|]\}}||qSrr).0kvrrr Wz*VitsTokenizer.__init__..) pad_token unk_tokenlanguage add_blank normalize phonemize is_uroman) openjsonloadencoderitemsdecoderr!r"r#r$r%super__init__) selfr rr r!r"r#r$r%kwargsZ vocab_handle __class__rrr-Hs& *zVitsTokenizer.__init__cCs t|jSN)lenr)r.rrr vocab_sizejszVitsTokenizer.vocab_sizecs(fddtjD}|j|S)Ncsi|]}||qSr)Zconvert_ids_to_tokens)rir4rrrorz+VitsTokenizer.get_vocab..)ranger5updateadded_tokens_encoder)r.Zvocabrr4r get_vocabns zVitsTokenizer.get_vocabcCst|jt|j}d}d}|t|krd}|D]8}|||t||kr8||7}|t|7}d}qrq8|s$|||7}|d7}q$|S)zfLowercase the input string, respecting any special token ids that may be part or entirely upper-cased.rFT)listr)keysr9r3lower)r.rZall_vocabulary filtered_textr6Z found_matchwordrrrnormalize_textss   zVitsTokenizer.normalize_textcCs|jdkr|dd}|S)z4Special treatment of characters in certain languagesZronuțuţ)r!replace)r.textrrr_preprocess_chars  zVitsTokenizer._preprocess_char)rDis_split_into_wordsr#rc s|dur |nj}|r |}|}t|r\jr\tsJtdnt }| |}j rt spt dtj |dddddd}tdd |}n$|rd ttfd d |}||fS) a Performs any necessary transformations before tokenization. This method should pop the arguments from kwargs and return the remaining `kwargs` as well. We test the `kwargs` at the end of the encoding process to be sure all the arguments have been used. Args: text (`str`): The text to prepare. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. normalize (`bool`, *optional*, defaults to `None`): Whether or not to apply punctuation and casing normalization to the text inputs. Typically, VITS is trained on lower-cased and un-punctuated text. Hence, normalization is used to ensure that the input text consists only of lower-case characters. kwargs (`dict[str, Any]`, *optional*): Keyword arguments to use for the tokenization. Returns: `tuple[str, dict[str, Any]]`: The prepared text and the unused kwargs. NaCText to the tokenizer contains non-Roman characters. To apply the `uroman` pre-processing step automatically, ensure the `uroman` Romanizer is installed with: `pip install uroman` Note `uroman` requires python version >= 3.10Otherwise, apply the Romanizer manually as per the instructions: https://github.com/isi-nlp/uromanzEPlease install the `phonemizer` Python package to use this tokenizer.zen-usZespeakT)r!backendstripZpreserve_punctuationZ with_stressz\s+ r;cs |jvSr2)r))charr4rrrz8VitsTokenizer.prepare_for_tokenization..)r#rBrErr%rloggerwarningurZUromanZromanize_stringr$r ImportError phonemizerr subjoinr=filterrH)r.rDrFr#r/r@uromanrr4rprepare_for_tokenizations4    z&VitsTokenizer.prepare_for_tokenization)rDrcCs@t|}|jr<|dgt|dd}||ddd<|}|S)z]Tokenize a string by inserting the `` token at the boundary between adjacent characters.rr<N)r=r"_convert_id_to_tokenr3)r.rDtokensZ interspersedrrr _tokenizes zVitsTokenizer._tokenize)rXrcCs*|jr t|dkr |ddd}d|S)Nr<rVr;)r"r3rR)r.rXrrrconvert_tokens_to_stringsz&VitsTokenizer.convert_tokens_to_stringcCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)r)getr )r.tokenrrr_convert_token_to_idsz"VitsTokenizer._convert_token_to_idcCs |j|S)z=Converts an index (integer) in a token (str) using the vocab.)r+r[)r.indexrrrrWsz"VitsTokenizer._convert_id_to_token)save_directoryfilename_prefixrc Cstj|s"td|ddStj||r6|dndtd}t|ddd.}|t j |j d d d d d Wdn1s0Y|fS)NzVocabulary path (z) should be a directory-r;r wrrrVTF)indent sort_keys ensure_ascii ) ospathisdirrLerrorrRVOCAB_FILES_NAMESr&writer'dumpsr))r.r_r`r frrrsave_vocabularys <zVitsTokenizer.save_vocabulary)rrNTTTF)FN)N)__name__ __module__ __qualname____doc__rkZvocab_files_namesZmodel_input_namesr-propertyr5r:rBrEstrboolrtupledictrrUr=rYrZr]rWrro __classcell__rrr0rr/s8 "   A r)rsr'rgr typingrrrZtokenization_utilsrutilsrrr rPrTrNZ get_loggerrprLrkrr__all__rrrrs    H