a
hq" @ s d Z ddlZddlZddlmZ ddlmZmZ ddlm Z ddl
m Z m Z m
Z
ddlmZmZ dd lmZ G d
d d e d d
ZeeZG dd de ZdgZdS )z
Processor class for Donut.
N)contextmanager)OptionalUnion )
ImageInput)ProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)loggingc @ s e Zd Zi ZdS )DonutProcessorKwargsN)__name__
__module__ __qualname__ _defaults r r f/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/donut/processing_donut.pyr
s r
F)totalc s e Zd ZdZddgZdZdZd fdd Zdee e
e e e e
ef ee d d
d Zed d
Zedd ZdddZedd Zedd Z ZS )DonutProcessora
Constructs a Donut processor which wraps a Donut image processor and an XLMRoBERTa tokenizer into a single
processor.
[`DonutProcessor`] offers all the functionalities of [`DonutImageProcessor`] and
[`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. See the [`~DonutProcessor.__call__`] and
[`~DonutProcessor.decode`] for more information.
Args:
image_processor ([`DonutImageProcessor`], *optional*):
An instance of [`DonutImageProcessor`]. The image processor is a required input.
tokenizer ([`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`], *optional*):
An instance of [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. The tokenizer is a required input.
image_processor tokenizerZAutoImageProcessorZ
AutoTokenizerNc sr d }d|v r"t dt |d}|d ur.|n|}|d u rBtd|d u rRtdt || | j| _d| _ d S )Nfeature_extractorzhThe `feature_extractor` argument is deprecated and will be removed in v5, use `image_processor` instead.z)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.F)
warningswarn
FutureWarningpop
ValueErrorsuper__init__r current_processor_in_target_context_manager)selfr r kwargsr __class__r r r 9 s
zDonutProcessor.__init__)imagestextr# c K s | j r| j||fi |S |du r2|du r2td| jtfd| jji|}|durj| j|fi |d }|dur|dur|d dd | j|fi |d }|du r|S |du r|S |d |d < |d |d< |S dS )
a
When used in normal mode, this method forwards all its arguments to AutoImageProcessor's
[`~AutoImageProcessor.__call__`] and returns its output. If used in the context
[`~DonutProcessor.as_target_processor`] this method forwards all its arguments to DonutTokenizer's
[`~DonutTokenizer.__call__`]. Please refer to the docstring of the above two methods for more information.
NzBYou need to specify either an `images` or `text` input to process.Ztokenizer_init_kwargsZ
images_kwargsZ text_kwargsZadd_special_tokensF input_idslabels) r! r r Z
_merge_kwargsr
r Z init_kwargsr
setdefault) r" r& r' ZaudioZvideosr# Z
output_kwargsinputs encodingsr r r __call__M s0 zDonutProcessor.__call__c C s | j j}t|ddg S )Nr( r) )r model_input_nameslist)r" Zimage_processor_input_namesr r r r. w s z DonutProcessor.model_input_namesc c s0 t d d| _| j| _dV | j| _d| _dS )z
Temporarily sets the tokenizer for processing the input. Useful for encoding the labels when fine-tuning TrOCR.
z`as_target_processor` is deprecated and will be removed in v5 of Transformers. You can process your labels by using the argument `text` of the regular `__call__` method (either in the same call as your images inputs, or in a separate call.TNF)r r r! r r r r" r r r as_target_processor} s z"DonutProcessor.as_target_processorFc C sX |du r| j }i }|r,td|tj}|du r8q,|| d }d|vrTq,|d|dd }|tdtd }t|}td| d|tj} | du r| |d}q|
} t|}
t| } t|
d| |tjtj B } | dur|
d } d| v rVd| v rV| j
| d|d }
|
rt|
dkrL|
d
}
|
||< ng ||< | d D ]R}| }||v r|d
d kr|d
d dkr|dd
}|| | qht|| dkr|| d
||< ||| t| d }|dd d kr|g| j
|dd d|d S q|rB|r>|gS |S |rLg S d|iS dS )zS
Convert a (generated) token sequence into an ordered JSON format.
Nz z Z
text_sequence)r Zget_added_vocabresearch
IGNORECASEstartindexlenescapereplacegroupDOTALLstrip
token2jsonsplitappendfind)r" tokensr5 r6 outputZpotential_startZ start_tokenkeyZ key_escapedZ end_tokenZstart_token_escapedZend_token_escapedcontentvalueleafr r r rE sX
* zDonutProcessor.token2jsonc C s t dt | jS )Nzg`feature_extractor_class` is deprecated and will be removed in v5. Use `image_processor_class` instead.)r r r image_processor_classr0 r r r feature_extractor_class s
z&DonutProcessor.feature_extractor_classc C s t dt | jS )Nz[`feature_extractor` is deprecated and will be removed in v5. Use `image_processor` instead.)r r r r r0 r r r r s
z DonutProcessor.feature_extractor)NN)NNNN)FN)r r r __doc__
attributesrO Ztokenizer_classr r r r strr/ r r
r r
r- propertyr. r r1 rE rP r
__classcell__r r r$ r r % s. *
:
r )rQ r: r
contextlibr typingr r Z image_utilsr Zprocessing_utilsr r r Ztokenization_utils_baser
r utilsr r
Z
get_loggerr loggerr __all__r r r r s
4