a hx7@sddlZddlmZddlmZmZddlZddlm Z m Z m Z ddl m Z ddlmZe eZdZd d iZgd Zed d Gddde ZdgZdS)N)copyfile)AnyOptional) AddedToken BatchEncodingPreTrainedTokenizer)logging)requiresu▁ vocab_filezsentencepiece.bpe.model)Zar_ARcs_CZde_DEen_XXZes_XXet_EEfi_FIZfr_XXgu_INhi_INit_ITZja_XXkk_KZko_KRlt_LTlv_LVZmy_MMne_NPZnl_XXro_ROru_RUsi_LKtr_TRvi_VNzh_CN) sentencepiece)backendsc seZdZUdZeZddgZgZee e d<gZ ee e d<dBe e eefd fd d ZddZddZeddZeedddZejed dddZdCee e ee eee dfdd ZdDee e ee ee ddd ZdEee e ee ee dd!d"Zee ee ed#d$d%Zd&d'Zeeed(d)d*Zd+d,Zd-d.Zd/d0Z dFee ee!ed1d2d3Z"dGeeee eeee#d6fd7d8 Z$d9d:Z%d;d<Z&d dd=d>Z'ed d?d@dAZ(Z)S)HMBartTokenizeruT Construct an MBART tokenizer. Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on [SentencePiece](https://github.com/google/sentencepiece). The tokenization method is ` ` for source language documents, and ` ` for target language documents. Examples: ```python >>> from transformers import MBartTokenizer >>> tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-en-ro", src_lang="en_XX", tgt_lang="ro_RO") >>> example_english_phrase = " UN Chief Says There Is No Military Solution in Syria" >>> expected_translation_romanian = "Şeful ONU declară că nu există o soluţie militară în Siria" >>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_romanian, return_tensors="pt") ```Z input_idsZattention_mask prefix_tokens suffix_tokensN)sp_model_kwargsc st|trt|dddn|}| dur(in| _tjfij_jt||_ddddd_ d_ t j_ fd d t tD_d d jD_t jt jj j d <j jd d j D_tj| durfdd| Dtjf|||||||d| | jd || durX| nd_jj_| _jdS)NTF)lstrip normalizedrr)r$r'r%r&cs"i|]\}}|j|jqS) sp_model_sizefairseq_offset).0icodeselfr.h/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/mbart/tokenization_mbart.py jsz+MBartTokenizer.__init__..cSsi|]\}}||qSr.r.r1kvr.r.r6r7mr(cSsi|]\}}||qSr.r.r8r.r.r6r7qr;csg|]}|vr|qSr.r.)r1t)_additional_special_tokensr.r6 wr;z+MBartTokenizer.__init__..) bos_token eos_token unk_token sep_token cls_token pad_token mask_tokentokenizer_filesrc_langtgt_langadditional_special_tokensr)r) isinstancestrrr)spmSentencePieceProcessorsp_modelLoadr fairseq_tokens_to_idsr0lenr/ enumerateFAIRSEQ_LANGUAGE_CODESlang_code_to_iditemsZid_to_lang_codeupdatefairseq_ids_to_tokenslistkeysextendsuper__init__ _src_langZcur_lang_code_idrHset_src_lang_special_tokens)r5r r?r@rBrCrArDrErFrGrHr)rIkwargs __class__)r=r5r6r\AsR      zMBartTokenizer.__init__cCs$|j}d|d<|j|d<|S)NrNsp_model_proto)__dict__copyrNserialized_model_proto)r5stater.r.r6 __getstate__s zMBartTokenizer.__getstate__cCs<||_t|dsi|_tjfi|j|_|j|jdS)Nr))rchasattrr)rLrMrNZLoadFromSerializedProtorb)r5dr.r.r6 __setstate__s  zMBartTokenizer.__setstate__cCst|jt|j|jdS)Nr,)rQrNrTr0r4r.r.r6 vocab_sizeszMBartTokenizer.vocab_size)returncCs|jSN)r]r4r.r.r6rGszMBartTokenizer.src_lang) new_src_langrlcCs||_||jdSrm)r]r^)r5rnr.r.r6rGsF) token_ids_0 token_ids_1already_has_special_tokensrlcsx|rtj||ddSdgt|j}dgt|j}|durT|dgt||S|dgt|dgt||S)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rorprqr,Nr)r[get_special_tokens_maskrQr"r#)r5rorprqZ prefix_onesZ suffix_onesr`r.r6rrsz&MBartTokenizer.get_special_tokens_mask)rorprlcCs,|dur|j||jS|j|||jS)ab Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An MBART sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`list[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N)r"r#)r5rorpr.r.r6 build_inputs_with_special_tokenssz/MBartTokenizer.build_inputs_with_special_tokenscCsP|jg}|jg}|dur.t|||dgSt||||||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. mBART does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of zeros. Nr)Z sep_token_idZ cls_token_idrQ)r5rorpsepclsr.r.r6$create_token_type_ids_from_sequencess z3MBartTokenizer.create_token_type_ids_from_sequences)return_tensorsrGrHcKsJ|dus|durtd||_||fd|d|}||}||d<|S)zIUsed by translation pipeline, to prepare inputs for the generate functionNzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)Zadd_special_tokensrwZforced_bos_token_id) ValueErrorrGZconvert_tokens_to_ids)r5Z raw_inputsrwrGrH extra_kwargsinputsZ tgt_lang_idr.r.r6_build_translation_inputss z(MBartTokenizer._build_translation_inputscs(fddtjD}|j|S)Ncsi|]}||qSr.)Zconvert_ids_to_tokens)r1r2r4r.r6r7 r;z,MBartTokenizer.get_vocab..)rangerkrVZadded_tokens_encoder)r5Zvocabr.r4r6 get_vocab s zMBartTokenizer.get_vocab)textrlcCs|jj|tdS)N)Zout_type)rNencoderK)r5r~r.r.r6 _tokenizeszMBartTokenizer._tokenizecCs4||jvr|j|S|j|}|r.||jS|jS)z0Converts a token (str) in an id using the vocab.)rPrNZ PieceToIdr0Z unk_token_id)r5tokenZspm_idr.r.r6_convert_token_to_ids   z#MBartTokenizer._convert_token_to_idcCs&||jvr|j|S|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)rWrNZ IdToPiecer0)r5indexr.r.r6_convert_id_to_tokens  z#MBartTokenizer._convert_id_to_tokencCsd|td}|S)zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)r5tokensZ out_stringr.r.r6convert_tokens_to_string!sz'MBartTokenizer.convert_tokens_to_string)save_directoryfilename_prefixrlcCstj|s"td|ddStj||r6|dndtd}tj|jtj|krztj |jrzt |j|nLtj |jst |d$}|j }||Wdn1s0Y|fS)NzVocabulary path (z) should be a directory-rr wb)ospathisdirloggererrorrVOCAB_FILES_NAMESabspathr isfileropenrNrewrite)r5rrZout_vocab_filefiZcontent_spiece_modelr.r.r6save_vocabulary&s (  (zMBartTokenizer.save_vocabularyrr) src_textsrG tgt_textsrHrlc s"||_||_tj||fi|Srm)rGrHr[prepare_seq2seq_batch)r5rrGrrHr_r`r.r6r7sz$MBartTokenizer.prepare_seq2seq_batchcCs ||jSrm)r^rGr4r.r.r6_switch_to_input_modeCsz$MBartTokenizer._switch_to_input_modecCs ||jSrm)set_tgt_lang_special_tokensrHr4r.r.r6_switch_to_target_modeFsz%MBartTokenizer._switch_to_target_modecCs$|j||_g|_|j|jg|_dS)z_Reset the special tokens to the source lang setting. No prefix and suffix=[eos, src_lang_code].NrTZ cur_lang_coder"Z eos_token_idr#)r5rGr.r.r6r^Is z*MBartTokenizer.set_src_lang_special_tokens)langrlcCs$|j||_g|_|j|jg|_dS)zcReset the special tokens to the target language setting. No prefix and suffix=[eos, tgt_lang_code].Nr)r5rr.r.r6rOs z*MBartTokenizer.set_tgt_lang_special_tokens) r$r%r%r$r&r'r(NNNNN)NF)N)N)N)rNr)*__name__ __module__ __qualname____doc__rZvocab_files_namesZmodel_input_namesr"rXint__annotations__r#rdictrKrr\rgrjpropertyrkrGsetterboolrrrsrvr{r}rrrrtuplerrrrrr^r __classcell__r.r.r`r6r!%s  N        r!)rshutilrtypingrrrrLZtokenization_utilsrrrutilsr Zutils.import_utilsr Z get_loggerrrrrrSr!__all__r.r.r.r6s    2