a h}/@sndZddlZddlZddlmZddlmZddlmZe e Z ddd Z d d Z Gd d d eZd gZdS)z Tokenization classes for BioGPT.N)Optional)PreTrainedTokenizer)loggingz vocab.jsonz merges.txt) vocab_file merges_filecCs6t}|d}|ddD]}|||f|}q|S)z Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings) rN)setadd)wordpairsZ prev_charcharrj/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/biogpt/tokenization_biogpt.py get_pairs!s rcseZdZdZeZddgZd+fdd Zed d Z d d Z ddZ ddZ ddZ d,ddZddZddZddZd-eeeeeeeddd Zd.eeeeeeeed!fd"d# Zd/eeeeed$d%d&Zd'd(Zd)d*ZZS)0BioGptTokenizera: Construct an FAIRSEQ Transformer tokenizer. Moses tokenization followed by Byte-Pair Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Merges file. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. Z input_idsZattention_maskc  sz ddl} Wnty&tdYn0d|_| |_i|_i|_t|dd} t| |_ Wdn1sn0Ydd|j D|_ t|dd&} | ddd } Wdn1s0Yd d | D} tt| tt| |_i|_tjf|||||d |dS) NrzqYou need to install sacremoses to use BioGptTokenizer. See https://pypi.org/project/sacremoses/ for installation.enutf-8encodingcSsi|]\}}||qSrr).0kvrrr xz,BioGptTokenizer.__init__.. cSs g|]}t|ddqS)N)tuplesplit)rmergerrr {rz,BioGptTokenizer.__init__..) bos_token eos_token sep_token unk_token pad_token) sacremoses ImportErrorlangsmcache_moses_tokenizercache_moses_detokenizeropenjsonloadencoderitemsdecoderreadr#dictziprangelen bpe_rankscachesuper__init__) selfrrr)r&r'r(r*kwargsr+Z vocab_handleZ merges_handleZmerges __class__rrr?\s6    *4zBioGptTokenizer.__init__cCs t|jS)zReturns vocab size)r;r4r@rrr vocab_sizeszBioGptTokenizer.vocab_sizecCst|jfi|jS)N)r8r4Zadded_tokens_encoderrDrrr get_vocabszBioGptTokenizer.get_vocabcCs:||jvr"|jj|d}||j|<|j|j|ddddS)Nr-TF)Zaggressive_dash_splitsZ return_strescape)r/r.ZMosesTokenizertokenize)r@textr-Zmoses_tokenizerrrrmoses_tokenizes    zBioGptTokenizer.moses_tokenizecCs2||jvr"|jj|d}||j|<|j||S)NrG)r0r.ZMosesDetokenizerZ detokenize)r@tokensr-Zmoses_detokenizerrrrmoses_detokenizes  z BioGptTokenizer.moses_detokenizec st|dd|ddf}|jvr2j|St|}|sF|dSt|fddd}|jvrhqd|\}}g}d}|t|kr:z|||} Wn*ty|||dYq:Yn0|||| | }|||kr"|t|dkr"||d|kr"| |||d7}qx| |||d7}qxt|}|}t|dkrZqdqFt|}qFd |}|d kr|d }|j|<|S) Nr csj|tdS)Ninf)r<getfloat)pairrDrrrz%BioGptTokenizer.bpe..keyrrr! z z ) r"r=rminr<r;index ValueErrorextendappendjoin) r@tokenr r ZbigramfirstsecondZnew_wordijrrDrbpesF     2      zBioGptTokenizer.bpeFcCsL|r|}n|||j}g}|D]"}|r$|t||dq$|S)zReturns a tokenized string.rV)r#rKr-rZlistrb)r@rJZbypass_tokenizerZ split_tokensr]rrr _tokenizes zBioGptTokenizer._tokenizecCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)r4rPr))r@r]rrr_convert_token_to_idsz$BioGptTokenizer._convert_token_to_idcCs|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)r6rPr))r@rXrrr_convert_id_to_tokensz$BioGptTokenizer._convert_id_to_tokencCs.dd|D}d|}|||j}|S)z:Converts a sequence of tokens (string) in a single string.cSs g|]}|ddddqS)rVrN)replace)rtrrrr%rz.rg)r\r#rMr-)r@rLrJrrrconvert_tokens_to_stringsz(BioGptTokenizer.convert_tokens_to_stringN) token_ids_0 token_ids_1returncCs,|dur|jg|S|jg}||||S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BioGPT sequence has the following format: - single sequence: ` X ` - pair of sequences: ` A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N)Z sep_token_id)r@rkrlseprrr build_inputs_with_special_tokenss z0BioGptTokenizer.build_inputs_with_special_tokens)rkrlalready_has_special_tokensrmcsZ|rtj||ddS|durFdgdgt|dgdgt|Sdgdgt|S)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rkrlrpNrr)r>get_special_tokens_maskr;)r@rkrlrprBrrrqs(z'BioGptTokenizer.get_special_tokens_mask)save_directoryfilename_prefixrmc CsJtj|s"td|ddStj||r6|dndtd}tj||rX|dndtd}t|ddd .}|t j |j d d d d dWdn1s0Yd}t|ddd l}t |j dddD]D\}} || krtd|d| }|d|d|d7}qWdn1s80Y||fS)NzVocabulary path (z) should be a directory-rgrrwrrr!TF)indent sort_keys ensure_asciirrcSs|dS)Nrr)kvrrrrS,rz1BioGptTokenizer.save_vocabulary..rTzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rVr)ospathisdirloggererrorr\VOCAB_FILES_NAMESr1writer2dumpsr4sortedr<r5warning) r@rrrsrZ merge_filefrXwriterZ bpe_tokensZ token_indexrrrsave_vocabularys, <  *zBioGptTokenizer.save_vocabularycCs|j}d|d<|S)Nr.)__dict__copy)r@staterrr __getstate__8s zBioGptTokenizer.__getstate__cCs8||_z ddl}Wnty,tdYn0||_dS)NrznYou need to install sacremoses to use XLMTokenizer. See https://pypi.org/project/sacremoses/ for installation.)rr+r,r.)r@dr+rrr __setstate__=s   zBioGptTokenizer.__setstate__)rrrrr)F)N)NF)N)__name__ __module__ __qualname____doc__rZvocab_files_namesZmodel_input_namesr?propertyrErFrKrMrbrdrerfrjrcintrroboolrqstrr"rrr __classcell__rrrBrr.s@*, ,   r)rr2rztypingrZtokenization_utilsrutilsrZ get_loggerrr}rrr__all__rrrrs