a
h73 @ sz d Z ddlZddlZddlmZ ddlmZ ddlmZ ddl m
Z
e
e Z
dd d
Zd d ZG d
d deZdgZdS )z Tokenization classes for PhoBERT N)copyfile)Optional )PreTrainedTokenizer)loggingz vocab.txtz bpe.codes)
vocab_file merges_filec C s> t }| d }| dd D ]}|||f |}qt |}|S )z
Return set of symbol pairs in a word.
Word is represented as tuple of symbols (symbols being variable-length strings).
r N)setadd)wordpairsZ prev_charchar r l/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/phobert/tokenization_phobert.py get_pairs# s r c s e Zd ZdZeZd& fdd Zd'ee e ee ee d
d d Z
d(ee e ee e ee d fdd
Z d)ee e ee ee d
ddZ
edd Zdd Zdd Zdd Zdd Zd d Zd d Zd*ee e ee d!d"d#Zd$d% Z ZS )+PhobertTokenizeraO
Construct a PhoBERT tokenizer. Based on Byte-Pair-Encoding.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
merges_file (`str`):
Path to the merges file.
bos_token (`st`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
mask_token (`str`, *optional*, defaults to `""`):
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.
X `
- pair of sequences: ` A B `
Args:
token_ids_0 (`list[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
N) cls_token_id sep_token_id)r= rA rB clssepr r r build_inputs_with_special_tokens s
z1PhobertTokenizer.build_inputs_with_special_tokensF)rA rB already_has_special_tokensrC c sh |rt j||ddS |du r8dgdgt| dg S dgdgt| ddg dgt| dg S )a
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)rA rB rI Nr r )r; get_special_tokens_maskr8 )r= rA rB rI r? r r rJ s z(PhobertTokenizer.get_special_tokens_maskc C sP | j g}| jg}|du r.t|| | dg S t|| | | | | dg S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. PhoBERT does not
make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of zeros.
Nr )rE rD r8 )r= rA rB rG rF r r r $create_token_type_ids_from_sequences s
z5PhobertTokenizer.create_token_type_ids_from_sequencesc C s
t | jS N)r8 r. r= r r r
vocab_size s zPhobertTokenizer.vocab_sizec C s t | jfi | jS rL )r5 r. Zadded_tokens_encoderrM r r r get_vocab s zPhobertTokenizer.get_vocabc
s | j v r j | S t|}tt|d d |d d g }t|}|sN|S t| fddd}| jvrpql|\}}g }d}|t|k rBz|||} W n* ty | ||d Y qBY n0 | ||| | }|| |kr*|t|d k r*||d |kr*|
|| |d7 }q|
|| |d7 }qt|}|}t|dkrbqlqNt|}qNd |}|d d
}| j |< |S ) Nr" zc s j | tdS )Ninf)r9 getfloat)pairrM r r r z&PhobertTokenizer.bpe..)keyr r r @@ ) r: r# listr minr9 r8 index
ValueErrorextendappendjoin)
r= tokenr r
ZbigramfirstsecondZnew_wordijr rM r bpe sF
"
2
zPhobertTokenizer.bpec C s8 g }t d|}|D ] }|t| |d q|S )zTokenize a string.z\S+\n? )refindallr\ rX rd r$ )r= textZ split_tokenswordsr_ r r r _tokenize s
zPhobertTokenizer._tokenizec C s | j || j | jS )z0Converts a token (str) in an id using the vocab.)r. rQ r) )r= r_ r r r _convert_token_to_id s z%PhobertTokenizer._convert_token_to_idc C s | j || jS )z=Converts an index (integer) in a token (str) using the vocab.)r2 rQ r) )r= rZ r r r _convert_id_to_token! s z%PhobertTokenizer._convert_id_to_tokenc C s d |dd }|S )z:Converts a sequence of tokens (string) in a single string.re rV )r^ replacestrip)r= tokensZ
out_stringr r r convert_tokens_to_string% s z)PhobertTokenizer.convert_tokens_to_string)save_directoryfilename_prefixrC c C s t j|s"td| d d S t j||r6|d ndtd }t j||rX|d ndtd }t j| jt j|krt j | jrt
| j| nLt j | jst |d$}| j
}|| W d n1 s0 Y t j| jt j|krt
| j| ||fS )NzVocabulary path (z) should be a directory-rm r r wb)ospathisdirloggererrorr^ VOCAB_FILES_NAMESabspathr isfiler r3 Zsp_modelZserialized_model_protowriter )r= rr rs Zout_vocab_fileZout_merge_filefiZcontent_spiece_modelr r r save_vocabulary* s$ (
( z PhobertTokenizer.save_vocabularyc
C s t |trz ')
isinstancer/ r3 r0 FileNotFoundError UnicodeError Exception readlinesro rfindr[ r8 r. ) r= ffdZfnfelinesZlineTmplineidxr r r r r0 G s"
,
z PhobertTokenizer.add_from_file)r r r r r r r )N)NF)N)N)__name__
__module__ __qualname____doc__r{ Zvocab_files_namesr< rX intr rH boolrJ rK propertyrN rO rd rj rk rl rq r/ r# r r0
__classcell__r r r? r r 3 sF 0 -
,
r )r rv rf shutilr typingr Ztokenization_utilsr utilsr Z
get_loggerr ry r{ r r __all__r r r r s
.