a hp@sfdZddlZddlZddlZddlZddlmZddlmZm Z m Z ddl m Z ddl mZddlmZmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZd dlm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(erddl)m*Z*ndZ*e+e,Z-ee.e/e e.e e.ffdder0dndffderDdnderRdndffdderhdndffdder~dndffdderdndffd derd!ndffd"derdndffd#d$erd%nderd&ndffd'd(derdndffd)erd*nddffd+d,d-er*d.nder8d/ndffd0d1erNd2ndffd3d4derfd5ndffd6d7d8derdndffd9d:erd;ndffdd?erd@ndffdAderdndffdBdCerdDnderdEndffdFdGerdnder dndffdHder6dndffdId?erLd@ndffdJderbdndffdKderxdndffdLdMerdNnderdOndffdPdQerdRndffdSderd!ndffdTderd!ndffdUderdndffdVdWer dXndffdYdZer d[ndffd\er4d]nderBd^ndffd_d`dadbd?er^d@ndffdcd:ertd;ndffdddeerdfndffdgerdhnderdindffdjerdnderdndffdkerdnderdndffdlerdnderdndffdmer&dnder4dndffdndoerJdnderXdndffdpdqerndrndffdsdterdundffdvdwerdxndffdyd:erd;ndffdzderdndffd{derdndffd|derdndffd}erd~nddffdderd:nder,d;ndffdderBd5ndffdderXdndffderldnddffddderdndffddderdndffderdnderdndffderdnderdndffderdnderdndffderdnder&dndffder:dnderHdndffder\dnderjdndffdderdndffdderd5ndffdderd5ndffdderd5ndffdderd5ndffdderd5ndffderdnddffdd:erd;ndffdd:er0d;ndffdd:erFd;ndffdder\dndffdddertd5ndffdd:erd;ndffddddddderdndffdderdndffdderd5ndffdderdndffddd?e rd@ndffdde rdndffdde r0dndffdde rFdndffdd:e r\d;ndffdd:e rrd;ndffddWe rdXndffde rdnde rdndffdde rdndffde rdnde rdndffdde rdnde rdndffdde rd5ndffdde r2dndffdde rHdndffdde r^dndffdde rtdndffdde rdndffdde rdndffde rdnde rdndffde rdnde rdndffde rdnde rdndffdde rdndffdde r2dndffdde rHdndffdde r^dndffdde rtdndffde rdnde rdndffddde rdndffde rdnddffdde rdndffdde rdndffde rdnddffde rdnde r(dndffde rdndffd.dWerVdXndffd/dWerndXndffd0dWerdXndffd1dWerdXndffd2dWerdXndffd3dWerdXndffd4dWerdXndffd5dWerdXndffd6d7d8erd9ndffd:er4dnderBdndffd;erZderd?nderd@ndffdAdBerdCndffdDd?erd@ndffdEd?erd@ndffdFdGdHerdIndffdJderdndffdKer.dLnder>dMndffdNerVdLnderfdMndffdOer|dnderdndffdPerdQnddffdRerdnderdndffdSderd5ndffdTerdUnddffdVdWerdXnddffdYdZd[er8d\ndffd]derPdndffd^d:erhd;ndffd_er~dnderdndffd`erdnderdndffdaerdnderdndffdbdcdddederdndffdferdgnder dhndffdier6dnderDdndffdjder\dndffdkdertdndffdlderdndffdmderdndffdndoe rdnerdndere sdndffdpdqdrdsdtduer dvndffdwder$dndffdxer<dynderLdzndffd{d|erhd}nddffd~erdnderdndffderdnderdndffderʐdnderڐdndffdderdndffderdnderdndffder,dnder:dndffderPdnder^dndffdertdnderdndffgZ0e"e$e0Z1dde$2DZ3e.e e4edfdddZ5de e.ej6e.fe e e.ej6e.fe7e e7e e8e.e.fe e e7e.fe e.e7e.e8e.efd ddZ9GdddZ:ddgZ;dS(zAuto Tokenizer class.N) OrderedDict)AnyOptionalUnion)is_mistral_common_available)PretrainedConfig)get_class_from_dynamic_moduleresolve_trust_remote_code)load_gguf_checkpoint)PreTrainedTokenizer)TOKENIZER_CONFIG_FILE) cached_fileextract_commit_hashis_g2p_en_availableis_sentencepiece_availableis_tokenizers_availablelogging)EncoderDecoderConfig)_LazyAutoMapping)CONFIG_MAPPING_NAMES AutoConfigconfig_class_to_model_typemodel_type_to_module_name!replace_list_option_in_docstrings)PreTrainedTokenizerFastZaimv2Z CLIPTokenizerZCLIPTokenizerFastZalbertZAlbertTokenizerZAlbertTokenizerFastalignZ BertTokenizerZBertTokenizerFastZarceeZLlamaTokenizerZLlamaTokenizerFastZariaZ aya_visionZCohereTokenizerFastZbark)Zbart)Z BartTokenizerZBartTokenizerFastZbarthezZBarthezTokenizerZBarthezTokenizerFast)Zbartpho)ZBartphoTokenizerNZbertzbert-generationZBertGenerationTokenizer)z bert-japanese)ZBertJapaneseTokenizerN)Zbertweet)ZBertweetTokenizerNZbig_birdZBigBirdTokenizerZBigBirdTokenizerFastZbigbird_pegasusZPegasusTokenizerZPegasusTokenizerFast)Zbiogpt)ZBioGptTokenizerNZbitnetr)Z blenderbot)ZBlenderbotTokenizerZBlenderbotTokenizerFast)zblenderbot-small)ZBlenderbotSmallTokenizerNZblipzblip-2 GPT2TokenizerZGPT2TokenizerFastZbloomZBloomTokenizerFastZ bridgetowerZRobertaTokenizerZRobertaTokenizerFastZbros)Zbyt5)Z ByT5TokenizerNZ camembertZCamembertTokenizerZCamembertTokenizerFast)Zcanine)ZCanineTokenizerNZ chameleonZ chinese_clipclapZclipZclipseg)Zclvp)Z ClvpTokenizerNZ code_llamaZCodeLlamaTokenizerZCodeLlamaTokenizerFastZcodegenZCodeGenTokenizerZCodeGenTokenizerFastZcohereZcohere2ZcolpaliZcolqwen2ZQwen2TokenizerZQwen2TokenizerFastZconvbertZConvBertTokenizerZConvBertTokenizerFastZcpmZ CpmTokenizerZCpmTokenizerFast)Zcpmant)ZCpmAntTokenizerN)Zctrl)Z CTRLTokenizerN)zdata2vec-audioZWav2Vec2CTCTokenizerNz data2vec-textZdbrxZdebertaZDebertaTokenizerZDebertaTokenizerFastz deberta-v2ZDebertaV2TokenizerZDebertaV2TokenizerFastZ deepseek_v2Z deepseek_v3Z deepseek_vlZdeepseek_vl_hybrid)Zdia)Z DiaTokenizerNZ diffllamaZ distilbertZDistilBertTokenizerZDistilBertTokenizerFastZdprZDPRQuestionEncoderTokenizerZDPRQuestionEncoderTokenizerFastZelectraZElectraTokenizerZElectraTokenizerFastZemu3ZernieZernie4_5Z ernie4_5_moeZernie_mZErnieMTokenizer)Zesm)Z EsmTokenizerNZexaone4ZfalconZ falcon_mambaZGPTNeoXTokenizerFastZfastspeech2_conformerZFastSpeech2ConformerTokenizer)Zflaubert)ZFlaubertTokenizerNZfnetZ FNetTokenizerZFNetTokenizerFast)Zfsmt)Z FSMTTokenizerNZfunnelZFunnelTokenizerZFunnelTokenizerFastZgemmaZGemmaTokenizerZGemmaTokenizerFastZgemma2Zgemma3Z gemma3_textZgemma3nZ gemma3n_textgitZglmZglm4Zglm4_moeZglm4vZ glm4v_moezgpt-sw3ZGPTSw3TokenizerZgpt2Z gpt_bigcodeZgpt_neoZgpt_neox)Zgpt_neox_japanese)ZGPTNeoXJapaneseTokenizerNZgpt_ossZgptj)zgptsan-japanese)ZGPTSanJapaneseTokenizerN)ZgraniterN)Z granitemoer#)Zgranitemoehybridr#)Zgranitemoesharedr#zgrounding-dinoZgroupvitZheliumZherbertZHerbertTokenizerZHerbertTokenizerFast)Zhubertr!ZibertZideficsZidefics2Zidefics3Z instructblipZinstructblipvideoZinternvlZjambaZjanusZjetmoe)Zjukebox)ZJukeboxTokenizerNzkosmos-2ZXLMRobertaTokenizerZXLMRobertaTokenizerFastz kosmos-2.5ZlayoutlmZLayoutLMTokenizerZLayoutLMTokenizerFastZ layoutlmv2ZLayoutLMv2TokenizerZLayoutLMv2TokenizerFastZ layoutlmv3ZLayoutLMv3TokenizerZLayoutLMv3TokenizerFastZ layoutxlmZLayoutXLMTokenizerZLayoutXLMTokenizerFastZledZ LEDTokenizerZLEDTokenizerFastZliltllamaZllama4Z llama4_textZllavaZ llava_nextZllava_next_videoZllava_onevisionZ longformerZLongformerTokenizerZLongformerTokenizerFastZlongt5Z T5TokenizerZT5TokenizerFast)Zluke)Z LukeTokenizerNZlxmertZLxmertTokenizerZLxmertTokenizerFastZm2m_100ZM2M100TokenizerZmambaZmamba2ZmarianZMarianTokenizerZmbartZMBartTokenizerZMBartTokenizerFastZmbart50ZMBart50TokenizerZMBart50TokenizerFastmegaz megatron-bertZ metaclip_2)zmgp-str)ZMgpstrTokenizerNZminimaxmistralMistralCommonTokenizermixtralZmllamaZmlukeZMLukeTokenizerzmm-grounding-dinoZ mobilebertZMobileBertTokenizerZMobileBertTokenizerFastZ modernbertZ moonshineZmoshiZmpnetZMPNetTokenizerZMPNetTokenizerFastZmptZmraZmt5Z MT5TokenizerZMT5TokenizerFastZmusicgenZmusicgen_melodyZmvpZ MvpTokenizerZMvpTokenizerFast)Zmyt5)Z MyT5TokenizerNZnemotronZnezhaZnllbZ NllbTokenizerZNllbTokenizerFastznllb-moeZ nystromformerZolmoZolmo2Zolmoez omdet-turboZ oneformerz openai-gptZOpenAIGPTTokenizerZOpenAIGPTTokenizerFastoptZowlv2ZowlvitZ paligemmaZpegasusZ pegasus_x)Z perceiver)ZPerceiverTokenizerNZ persimmonphiZphi3Zphimoe)Zphobert)ZPhobertTokenizerNZ pix2structZpixtralZplbartZPLBartTokenizer)Z prophetnet)ZProphetNetTokenizerNZqdqbertZqwen2Z qwen2_5_omniZ qwen2_5_vlZ qwen2_audioZ qwen2_moeZqwen2_vlZqwen3Z qwen3_moe)Zrag)Z RagTokenizerNrealmZRealmTokenizerZRealmTokenizerFastZrecurrent_gemmaZreformerZReformerTokenizerZReformerTokenizerFastZrembertZRemBertTokenizerZRemBertTokenizerFastZ retribertZRetriBertTokenizerZRetriBertTokenizerFastZrobertazroberta-prelayernorm)Zroc_bert)ZRoCBertTokenizerNZroformerZRoFormerTokenizerZRoFormerTokenizerFastZrwkvZ seamless_m4tZSeamlessM4TTokenizerZSeamlessM4TTokenizerFastZseamless_m4t_v2Z shieldgemma2ZsiglipZSiglipTokenizerZsiglip2Zsmollm3Zspeech_to_textZSpeech2TextTokenizer)Zspeech_to_text_2)ZSpeech2Text2TokenizerNZspeecht5ZSpeechT5Tokenizer)Zsplinter)ZSplinterTokenizerZSplinterTokenizerFastZ squeezebertZSqueezeBertTokenizerZSqueezeBertTokenizerFastZstablelmZ starcoder2Zswitch_transformersZt5Zt5gemma)Ztapas)ZTapasTokenizerN)Ztapex)ZTapexTokenizerN)z transfo-xl)ZTransfoXLTokenizerNZtvpZudopZ UdopTokenizerZUdopTokenizerFastZumt5Z video_llavaZviltZvipllavaZ visual_bert)Zvits)Z VitsTokenizerNZvoxtral)Zwav2vec2r!)z wav2vec2-bertr!)zwav2vec2-conformerr!)Zwav2vec2_phoneme)ZWav2Vec2PhonemeCTCTokenizerNZwhisperZWhisperTokenizerZWhisperTokenizerFastZxclipZxglmZ XGLMTokenizerZXGLMTokenizerFast)Zxlm)Z XLMTokenizerNzxlm-prophetnetZXLMProphetNetTokenizerz xlm-robertazxlm-roberta-xlZxlnetZXLNetTokenizerZXLNetTokenizerFastZxlstmZxmodZyosoZzambaZzamba2cCsi|]\}}||qSr,).0kvr,r,f/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py r1) class_namereturnc Cs|dkr tStD]p\}}||vrt|}|dvrJ|dkrJtdd}ntd|d}zt||WStyYqYq0qtj D]*}|D] }t|dd|kr|Sqqtd}t ||rt||SdS) Nr)r&r(r'z.tokenization_mistral_commonZ transformers.ztransformers.models__name__) rTOKENIZER_MAPPING_NAMESitemsr importlib import_modulegetattrAttributeErrorTOKENIZER_MAPPING_extra_contentvalueshasattr)r3 module_nameZ tokenizersmodule tokenizerZ main_moduler,r,r0tokenizer_class_from_names(     rDF) pretrained_model_name_or_path cache_dirforce_downloadresume_downloadproxiestokenrevisionlocal_files_only subfolderr4c Ks| dd} | dur4tdt|dur0td| }| d} t|t||||||||ddd| d} | durxt diSt | | } t | d d } t | }Wdn1s0Y| |d<|S) a Loads the tokenizer configuration from a pretrained model tokenizer configuration. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained model configuration hosted inside a model repo on huggingface.co. - a path to a *directory* containing a configuration file saved using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force to (re-)download the configuration files and override the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request. token (`str` or *bool*, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `hf auth login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. local_files_only (`bool`, *optional*, defaults to `False`): If `True`, will only try to load the tokenizer configuration from local files. subfolder (`str`, *optional*, defaults to `""`): In case the tokenizer config is located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. Passing `token=True` is required when you want to use a private model. Returns: `dict`: The configuration of the tokenizer. Examples: ```python # Download configuration from huggingface.co and cache. tokenizer_config = get_tokenizer_config("google-bert/bert-base-uncased") # This model does not have a tokenizer config so the result will be an empty dict. tokenizer_config = get_tokenizer_config("FacebookAI/xlm-roberta-base") # Save a pretrained tokenizer locally and you can reload its config from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased") tokenizer.save_pretrained("tokenizer-test") tokenizer_config = get_tokenizer_config("tokenizer-test") ```use_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.V`token` and `use_auth_token` are both specified. Please set only the argument `token`. _commit_hashF) rGrHrIrJrKrLrMrNZ _raise_exceptions_for_gated_repoZ%_raise_exceptions_for_missing_entriesZ'_raise_exceptions_for_connection_errorsrRz\Could not locate the tokenizer configuration file, will try to use the model config instead.zutf-8)encoding)popwarningswarn FutureWarning ValueErrorgetrr loggerinforopenjsonload)rFrGrHrIrJrKrLrMrNkwargsrOZ commit_hashZresolved_config_filereaderresultr,r,r0get_tokenizer_config&sDI    (rbc@s:eZdZdZddZeeeddZe d dd Z dS) AutoTokenizera This is a generic tokenizer class that will be instantiated as one of the tokenizer classes of the library when created with the [`AutoTokenizer.from_pretrained`] class method. This class cannot be instantiated directly using `__init__()` (throws an error). cCs tddS)Nz}AutoTokenizer is designed to be instantiated using the `AutoTokenizer.from_pretrained(pretrained_model_name_or_path)` method.)OSError)selfr,r,r0__init__szAutoTokenizer.__init__cOs|dd}|dur>tdt|ddur6td||d<|dd}d|d<|d d}|d d}|d d}|d } |dur4d} t|d} | durtd |ddddtDd| \} } |r| durt| } n t d| durt| } | durtd| d| j |g|Ri|St |fi|}d|vrZ|d|d<|d}d}d|vrt |dttfr|d}n|ddd}|dur.t |ts| rt|| fi|}t|ddd}tjfi|}ntj |fd |i|}|j}t|dr.d|jvr.|jd}|du}t|tvpj|duojt|dupjt|ddu}|r|r|ddur|d}n|d}d|vr|dd}nd}t|||||}|r|rt||fi|} |d d}| | j |g|Rd |i|S|durd} |rF|dsF|d}t|} | dur\|}t|} | durvtd|d!| j |g|Ri|St |trt|j t|j!urt d"|j!j"d#|j j"d$|j!}t#t|j$}|durVtt|\}}|r,|s|dur,|j |g|Ri|S|durN|j |g|Ri|Std%td&|j"d'dd(dtDddS))a] Instantiate one of the tokenizer classes of the library from a pretrained model vocabulary. The tokenizer class to instantiate is selected based on the `model_type` property of the config object (either passed as an argument or loaded from `pretrained_model_name_or_path` if possible), or when it's missing, by falling back to using pattern matching on `pretrained_model_name_or_path`: List options Params: pretrained_model_name_or_path (`str` or `os.PathLike`): Can be either: - A string, the *model id* of a predefined tokenizer hosted inside a model repo on huggingface.co. - A path to a *directory* containing vocabulary files required by the tokenizer, for instance saved using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`. - A path or url to a single saved vocabulary file if and only if the tokenizer only requires a single vocabulary file (like Bert or XLNet), e.g.: `./my_model_directory/vocab.txt`. (Not applicable to all derived classes) inputs (additional positional arguments, *optional*): Will be passed along to the Tokenizer `__init__()` method. config ([`PretrainedConfig`], *optional*) The configuration object used to determine the tokenizer class to instantiate. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download the model weights and configuration files and override the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. subfolder (`str`, *optional*): In case the relevant files are located inside a subfolder of the model repo on huggingface.co (e.g. for facebook/rag-token-base), specify it here. use_fast (`bool`, *optional*, defaults to `True`): Use a [fast Rust-based tokenizer](https://huggingface.co/docs/tokenizers/index) if it is supported for a given model. If a fast tokenizer is not available for a given model, a normal Python-based tokenizer is returned instead. tokenizer_type (`str`, *optional*): Tokenizer type to be loaded. trust_remote_code (`bool`, *optional*, defaults to `False`): Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. kwargs (additional keyword arguments, *optional*): Will be passed to the Tokenizer `__init__()` method. Can be used to set special tokens like `bos_token`, `eos_token`, `unk_token`, `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`. See parameters in the `__init__()` for more details. Examples: ```python >>> from transformers import AutoTokenizer >>> # Download vocabulary from huggingface.co and cache. >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") >>> # Download vocabulary from huggingface.co (user-uploaded) and cache. >>> tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-german-cased") >>> # If vocabulary files are in a directory (e.g. tokenizer was saved using *save_pretrained('./test/saved_model/')*) >>> # tokenizer = AutoTokenizer.from_pretrained("./test/bert_saved_model/") >>> # Download vocabulary from huggingface.co and define model-specific arguments >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base", add_prefix_space=True) ```rONrPrKrQconfigTZ _from_autouse_fasttokenizer_typetrust_remote_code gguf_filezPassed `tokenizer_type` z3 does not exist. `tokenizer_type` should be one of z, css|] }|VqdSNr,r-cr,r,r0 r2z0AutoTokenizer.from_pretrained..r5zt`use_fast` is set to `True` but the tokenizer class does not have a fast version. Falling back to the slow version.zTokenizer class z is not currently imported.rRtokenizer_classauto_maprcF)Zreturn_tensorsZFastrrz--Z code_revisionz- does not exist or is not currently imported.z The encoder model config class: z3 is different from the decoder model config class: z. It is not recommended to use the `AutoTokenizer.from_pretrained()` method in this case. Please use the encoder and decoder specific tokenizer classes.zzThis tokenizer cannot be instantiated. Please make sure you have `sentencepiece` installed in order to use this tokenizer.z!Unrecognized configuration class z8 to build an AutoTokenizer. Model type should be one of css|] }|jVqdSrl)r6rmr,r,r0ror2)%rTrUrVrWrYrXr7joinrDrZwarningfrom_pretrainedrb isinstancetuplelistrrr rZ for_modelrpr@rqtyper=splitr r Zregister_for_auto_classendswithrdecoderencoder __class__rr6)clsrFinputsr_rOrgrhrirjrkrpZtokenizer_class_tupleZtokenizer_class_nameZtokenizer_fast_class_nameZtokenizer_configZconfig_tokenizer_classZtokenizer_auto_mapZ gguf_pathZ config_dictZhas_remote_codeZhas_local_codeZ class_refZ upstream_repo_Ztokenizer_class_candidateZ model_typeZtokenizer_class_pyZtokenizer_class_fastr,r,r0rtsM                                   zAutoTokenizer.from_pretrainedNFcCs|dur|durtd|dur2t|tr2td|durLt|trLtd|dur|durt|tr|j|krtd|jd|d|tjvrt|\}}|dur|}|dur|}tj|||f|ddS) a Register a new tokenizer in this mapping. Args: config_class ([`PretrainedConfig`]): The configuration corresponding to the model to register. slow_tokenizer_class ([`PretrainedTokenizer`], *optional*): The slow tokenizer to register. fast_tokenizer_class ([`PretrainedTokenizerFast`], *optional*): The fast tokenizer to register. NzKYou need to pass either a `slow_tokenizer_class` or a `fast_tokenizer_classz:You passed a fast tokenizer in the `slow_tokenizer_class`.z:You passed a slow tokenizer in the `fast_tokenizer_class`.zThe fast tokenizer class you are passing has a `slow_tokenizer_class` attribute that is not consistent with the slow tokenizer class you passed (fast tokenizer has z and you passed z!. Fix one of those so they match!)exist_ok)rX issubclassrr slow_tokenizer_classr=r>register)Z config_classrZfast_tokenizer_classrZ existing_slowZ existing_fastr,r,r0rs8  zAutoTokenizer.register)NNF) r6 __module__ __qualname____doc__rf classmethodrr7rt staticmethodrr,r,r,r0rcs crcr=)NFNNNNFrE)s               M "  o!