a
hP @ s4 d Z ddlZddlZddlZddlZddlmZ ddlmZ ddl m
Z
m Z ddl m
Z
ddlmZ dd lmZ dd
lmZ dd lmZmZmZmZmZ dd lmZ d
dl m Z d
dl m Z m Z m!Z!m"Z" d
dl#m$Z$ d
dl%m&Z& d
dl'm(Z( e)e*Z+eg dZ,e e e,Z-e.dddZ/G dd dZ0ddgZ1dS )zAutoProcessor class. N) OrderedDict )PretrainedConfig) get_class_from_dynamic_moduleresolve_trust_remote_code)FeatureExtractionMixin)ImageProcessingMixin)ProcessorMixin)TOKENIZER_CONFIG_FILE)FEATURE_EXTRACTOR_NAMEPROCESSOR_NAMEVIDEO_PROCESSOR_NAME cached_filelogging)BaseVideoProcessor )_LazyAutoMapping)CONFIG_MAPPING_NAMES
AutoConfigmodel_type_to_module_name!replace_list_option_in_docstrings)AutoFeatureExtractor)AutoImageProcessor)
AutoTokenizer)i)Zaimv2
CLIPProcessor)alignZAlignProcessor)ZaltclipZAltCLIPProcessor)ZariaZ
AriaProcessor)Z
aya_visionZAyaVisionProcessor)ZbarkZ
BarkProcessor)ZblipZ
BlipProcessor)zblip-2ZBlip2Processor)Z bridgetowerZBridgeTowerProcessor)Z chameleonZChameleonProcessor)Z chinese_clipZChineseCLIPProcessor)clapZ
ClapProcessor)Zclipr )ZclipsegZCLIPSegProcessor)ZclvpZ
ClvpProcessor)Zcohere2_visionZCohere2VisionProcessor)ZcolpaliZColPaliProcessor)Zcolqwen2ZColQwen2Processor)Z deepseek_vlZDeepseekVLProcessor)Zdeepseek_vl_hybridZDeepseekVLHybridProcessor)ZdiaZ DiaProcessor)Zemu3Z
Emu3Processor)ZevollaZEvollaProcessor)ZflavaZFlavaProcessor)Z florence2ZFlorence2Processor)ZfuyuZ
FuyuProcessor)Zgemma3ZGemma3Processor)Zgemma3nZGemma3nProcessor)gitZ GitProcessor)Zglm4vGlm4vProcessor)Z glm4v_moer )Zgot_ocr2ZGotOcr2Processor)Zgranite_speechZGraniteSpeechProcessor)zgrounding-dinoGroundingDinoProcessor)Zgroupvitr )ZhubertWav2Vec2Processor)ZideficsZIdeficsProcessor)Zidefics2ZIdefics2Processor)Zidefics3ZIdefics3Processor)Z instructblipZInstructBlipProcessor)ZinstructblipvideoZInstructBlipVideoProcessor)ZinternvlZInternVLProcessor)ZjanusZJanusProcessor)zkosmos-2ZKosmos2Processor)z
kosmos-2.5ZKosmos2_5Processor)Zkyutai_speech_to_textZKyutaiSpeechToTextProcessor)Z
layoutlmv2ZLayoutLMv2Processor)Z
layoutlmv3ZLayoutLMv3Processor)Zllama4ZLlama4Processor)ZllavaLlavaProcessor)Z
llava_nextZLlavaNextProcessor)Zllava_next_videoZLlavaNextVideoProcessor)Zllava_onevisionZLlavaOnevisionProcessor)ZmarkuplmZMarkupLMProcessor)ZmctctZMCTCTProcessor)Z
metaclip_2r )zmgp-strZMgpstrProcessor)Zmistral3PixtralProcessor)ZmllamaZMllamaProcessor)zmm-grounding-dinor )Z moonshiner )Z oneformerZOneFormerProcessor)Zovis2ZOvis2Processor)Zowlv2ZOwlv2Processor)ZowlvitZOwlViTProcessor)Z paligemmaZPaliGemmaProcessor)Z
perception_lmZPerceptionLMProcessor)Zphi4_multimodalZPhi4MultimodalProcessor)Z
pix2structZPix2StructProcessor)Zpixtralr" )Z pop2pianoZPop2PianoProcessor)Z qwen2_5_omniZQwen2_5OmniProcessor)Z
qwen2_5_vlZQwen2_5_VLProcessor)Z qwen2_audioZQwen2AudioProcessor)Zqwen2_vlZQwen2VLProcessor)ZsamZ SamProcessor)Zsam2Z
Sam2Processor)Zsam_hqZSamHQProcessor)Z seamless_m4tZSeamlessM4TProcessor)Zsewr )zsew-dr )Z shieldgemma2ZShieldGemma2Processor)ZsiglipZSiglipProcessor)Zsiglip2ZSiglip2Processor)ZsmolvlmZSmolVLMProcessor)Zspeech_to_textZSpeech2TextProcessor)Zspeech_to_text_2ZSpeech2Text2Processor)Zspeecht5ZSpeechT5Processor)ZtrocrZTrOCRProcessor)ZtvltZ
TvltProcessor)ZtvpZ TvpProcessor)ZudopZ
UdopProcessor)Z unispeechr )z
unispeech-satr )Z video_llavaZVideoLlavaProcessor)ZviltZ
ViltProcessor)Zvipllavar! )zvision-text-dual-encoderZ VisionTextDualEncoderProcessor)ZvoxtralZVoxtralProcessor)Zwav2vec2r )z
wav2vec2-bertr )zwav2vec2-conformerr )Zwavlmr )ZwhisperZWhisperProcessor)ZxclipZXCLIPProcessor)
class_namec C s t D ]R\}}| |v rt|}td| d}zt|| W S tyX Y qY q0 qtj D ] }t|dd | krf| S qftd}t
|| rt|| S d S )N.ztransformers.models__name__Z transformers) PROCESSOR_MAPPING_NAMESitemsr importlib
import_modulegetattrAttributeErrorPROCESSOR_MAPPINGZ_extra_contentvalueshasattr)r# module_nameZ
processorsmodule processorZ main_module r2 d/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/auto/processing_auto.pyprocessor_class_from_name s
r4 c @ s: e Zd ZdZdd Zeeedd Ze d
ddZ
d S )
AutoProcessora
This is a generic processor class that will be instantiated as one of the processor classes of the library when
created with the [`AutoProcessor.from_pretrained`] class method.
This class cannot be instantiated directly using `__init__()` (throws an error).
c C s t dd S )Nz}AutoProcessor is designed to be instantiated using the `AutoProcessor.from_pretrained(pretrained_model_name_or_path)` method.)OSError)selfr2 r2 r3 __init__ s zAutoProcessor.__init__c s dd}|dur>tdt ddur6td| d< dd} dd}d d < d}d} fd
d ttj D }|
d d d d
t|t fi |} | durt j
|fi \}
} |
dd}d|
di v r|
d d }|du r&t|tfi |} | durPtj|fi \}
} |
dd}d|
di v rP|
d d }| du rt|tfi |} | durtj|fi \}
} |
dd}d|
di v r|
d d }| du r&t|tfi |} | dur&|du r&tj|fi \}
} |
dd}d|
di v r&|
d d }|du rt|tfi |}
|
durt|
dd}t|}
W d n1 sz0 Y |
dd}d|
di v r|
d d }|du rt|tst j |fd|i }t |dd}t |drd|j v r|j d }|durt!|}|du}|dup6t"|t#v }|rld|v rX|$dd }nd}t%|||||}|r|rt&||fi } dd} |' |j |fd|i S |dur|j |fd|i S t"|t#v rt#t"| j |fi S zt(j |fd|i W S t)y z t*j |fd|i W Y S t)yT Y n0 z t+j |fd|i W Y S t)y Y n0 Y n0 td| ddS )a
Instantiate one of the processor classes of the library from a pretrained model vocabulary.
The processor class to instantiate is selected based on the `model_type` property of the config object (either
passed as an argument or loaded from `pretrained_model_name_or_path` if possible):
List options
Params:
pretrained_model_name_or_path (`str` or `os.PathLike`):
This can be either:
- a string, the *model id* of a pretrained feature_extractor hosted inside a model repo on
huggingface.co.
- a path to a *directory* containing a processor files saved using the `save_pretrained()` method,
e.g., `./my_model_directory/`.
cache_dir (`str` or `os.PathLike`, *optional*):
Path to a directory in which a downloaded pretrained model feature extractor should be cached if the
standard cache should not be used.
force_download (`bool`, *optional*, defaults to `False`):
Whether or not to force to (re-)download the feature extractor files and override the cached versions
if they exist.
resume_download:
Deprecated and ignored. All downloads are now resumed by default when possible.
Will be removed in v5 of Transformers.
proxies (`dict[str, str]`, *optional*):
A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128',
'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request.
token (`str` or *bool*, *optional*):
The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
when running `hf auth login` (stored in `~/.huggingface`).
revision (`str`, *optional*, defaults to `"main"`):
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
identifier allowed by git.
return_unused_kwargs (`bool`, *optional*, defaults to `False`):
If `False`, then this function returns just the final feature extractor object. If `True`, then this
functions returns a `Tuple(feature_extractor, unused_kwargs)` where *unused_kwargs* is a dictionary
consisting of the key/value pairs whose keys are not feature extractor attributes: i.e., the part of
`kwargs` which has not been used to update `feature_extractor` and is otherwise ignored.
trust_remote_code (`bool`, *optional*, defaults to `False`):
Whether or not to allow for custom models defined on the Hub in their own modeling files. This option
should only be set to `True` for repositories you trust and in which you have read the code, as it will
execute code present on the Hub on your local machine.
kwargs (`dict[str, Any]`, *optional*):
The values in kwargs of any keys which are feature extractor attributes will be used to override the
loaded values. Behavior concerning key/value pairs whose keys are *not* feature extractor attributes is
controlled by the `return_unused_kwargs` keyword parameter.
Passing `token=True` is required when you want to use a private model.
Examples:
```python
>>> from transformers import AutoProcessor
>>> # Download processor from huggingface.co and cache.
>>> processor = AutoProcessor.from_pretrained("facebook/wav2vec2-base-960h")
>>> # If processor files are in a directory (e.g. processor was saved using *save_pretrained('./test/saved_model/')*)
>>> # processor = AutoProcessor.from_pretrained("./test/saved_model/")
```use_auth_tokenNzrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.tokenzV`token` and `use_auth_token` are both specified. Please set only the argument `token`.configtrust_remote_codeTZ
_from_autoc s i | ]}| v r| | qS r2 r2 ).0keykwargsr2 r3
z1AutoProcessor.from_pretrained..F)Z _raise_exceptions_for_gated_repoZ%_raise_exceptions_for_missing_entriesZ'_raise_exceptions_for_connection_errorsprocessor_classr5 auto_mapzutf-8)encodingz--r Z
code_revisionz!Unrecognized processing class in z. Can't instantiate a processor, a tokenizer, an image processor or a feature extractor for this model. Make sure the repository contains the files of at least one of those processing classes.),popwarningswarn
FutureWarningget
ValueErrorinspect signaturer
parametersupdater r Zget_processor_dictr r Zget_image_processor_dictr
r Zget_video_processor_dictr Zget_feature_extractor_dictr
openjsonload
isinstancer r from_pretrainedr* r. rD r4 typer, splitr r Zregister_for_auto_classr Exceptionr r )clsZ pretrained_model_name_or_pathr@ r9 r; r< rC Zprocessor_auto_mapZcached_file_kwargsZprocessor_config_fileZ config_dict_Zpreprocessor_config_fileZtokenizer_config_filereaderZhas_remote_codeZhas_local_codeZ
upstream_repor2 r? r3 rT sD E
*
z AutoProcessor.from_pretrainedFc C s t j| ||d dS )a
Register a new processor for this class.
Args:
config_class ([`PretrainedConfig`]):
The configuration corresponding to the model to register.
processor_class ([`ProcessorMixin`]): The processor to register.
)exist_okN)r, register)Z config_classrC r[ r2 r2 r3 r\ s
zAutoProcessor.registerN)F) r%
__module__ __qualname____doc__r8 classmethodr r& rT staticmethodr\ r2 r2 r2 r3 r5 s cr5 r, )2r_ r( rL rQ rG collectionsr Zconfiguration_utilsr Zdynamic_module_utilsr r Zfeature_extraction_utilsr Zimage_processing_utilsr Zprocessing_utilsr Ztokenization_utilsr
utilsr r r
r r Zvideo_processing_utilsr Z auto_factoryr Zconfiguration_autor r r r Zfeature_extraction_autor Zimage_processing_autor Ztokenization_autor Z
get_loggerr% loggerr& r, strr4 r5 __all__r2 r2 r2 r3 s8
n