a h@sddZddlZddlmZddlmZddlmZee Z ddiZ d d Z Gd d d eZ d gZdS) zTokenization classes for ESM.N)Optional)PreTrainedTokenizer)logging vocab_file vocab.txtcCsHt|d*}|}dd|DWdS1s:0YdS)NrcSsg|] }|qS)strip).0lr r d/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/esm/tokenization_esm.py z#load_vocab_file..)openread splitlines)rflinesr r r load_vocab_files  rcseZdZdZeZddgZd&fd d Zee d d d Z e edddZ ddZ ddZ e edddZee d ddZd'eeeeeeedddZd(eeeeeeddd Zd!d"Zeed#d$d%ZZS)) EsmTokenizerz& Constructs an ESM tokenizer. Z input_idsZattention_maskc sft||_tt|j|_ddt|jD|_tjf|||||d||j|_| |jdS)NcSsi|]\}}||qSr r )r indtokr r r 7rz)EsmTokenizer.__init__..) unk_token cls_token pad_token mask_token eos_token) r all_tokensdict enumerate _id_to_token _token_to_idsuper__init__Zunique_no_split_tokensZ _update_trie)selfrrr r!r"r#kwargs __class__r r r*+s  zEsmTokenizer.__init__)indexreturncCs|j||jSNr'getrr+r/r r r _convert_id_to_tokenGsz!EsmTokenizer._convert_id_to_token)tokenr0cCs|j||j|jSr1r(r3rr+r6r r r _convert_token_to_idJsz!EsmTokenizer._convert_token_to_idcKs|Sr1)split)r+textr,r r r _tokenizeMszEsmTokenizer._tokenizecCs|j}||j|Sr1)r(copyupdateZadded_tokens_encoder)r+Z base_vocabr r r get_vocabPs  zEsmTokenizer.get_vocabcCs|j||j|jSr1r7r8r r r token_to_idUszEsmTokenizer.token_to_idcCs|j||jSr1r2r4r r r id_to_tokenXszEsmTokenizer.id_to_tokenN) token_ids_0 token_ids_1r0cCs^|jg}|jg}|dur8|jdur*||S|||Sn|jdurJtd|||||S)Nz=Cannot tokenize multiple sequences when EOS token is not set!)Z cls_token_idZ eos_token_id ValueError)r+rBrCclssepr r r build_inputs_with_special_tokens[s  z-EsmTokenizer.build_inputs_with_special_tokensF)rBrCalready_has_special_tokensr0csd|r&|durtdfdd|DSdgdgt|dg}|dur`|dgt|dg7}|S)a Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods. Args: token_ids_0 (`list[int]`): List of ids of the first sequence. token_ids_1 (`list[int]`, *optional*): List of ids of the second sequence. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. NzYou should not supply a second sequence if the provided sequence of ids is already formatted with special tokens for the model.csg|]}|jvrdndqS)r)Zall_special_ids)r r6r+r r rrz8EsmTokenizer.get_special_tokens_mask..rIr)rDlen)r+rBrCrHmaskr rJr get_special_tokens_maskisz$EsmTokenizer.get_special_tokens_maskcCs`tj||r|dndd}t|d"}|d|jWdn1sP0Y|fS)N-rw )ospathjoinrwriter$)r+Zsave_directoryZfilename_prefixrrr r r save_vocabularys 0zEsmTokenizer.save_vocabulary)r0cCs t|jSr1)rKr$rJr r r vocab_sizeszEsmTokenizer.vocab_size)rrrrr)N)NF)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESZvocab_files_namesZmodel_input_namesr*intstrr5r9r<r?r@rAlistrrGboolrMrVpropertyrW __classcell__r r r-r r#s8   r)r[rRtypingrZtokenization_utilsrutilsrZ get_loggerrXloggerr\rr__all__r r r r s    p