a hy!@sjdZddlZddlmZmZddlmZddlmZm Z m Z m Z m Z ddl mZGdd d eZd gZdS) z! Processor class for LayoutLMv3. N)OptionalUnion)ProcessorMixin) BatchEncodingPaddingStrategyPreTokenizedInput TextInputTruncationStrategy) TensorTypecseZdZdZddgZdZdZdfdd Zdee e e e e e fe ee e e fe ee e e e e e e fe ee e e e e feeeeefeeeefe e e e e e ee eeeeeee eeefed d dZddZeddZeddZeddZZS)LayoutLMv3Processorav Constructs a LayoutLMv3 processor which combines a LayoutLMv3 image processor and a LayoutLMv3 tokenizer into a single processor. [`LayoutLMv3Processor`] offers all the functionalities you need to prepare data for the model. It first uses [`LayoutLMv3ImageProcessor`] to resize and normalize document images, and optionally applies OCR to get words and normalized bounding boxes. These are then provided to [`LayoutLMv3Tokenizer`] or [`LayoutLMv3TokenizerFast`], which turns the words and bounding boxes into token-level `input_ids`, `attention_mask`, `token_type_ids`, `bbox`. Optionally, one can provide integer `word_labels`, which are turned into token-level `labels` for token classification tasks (such as FUNSD, CORD). Args: image_processor (`LayoutLMv3ImageProcessor`, *optional*): An instance of [`LayoutLMv3ImageProcessor`]. The image processor is a required input. tokenizer (`LayoutLMv3Tokenizer` or `LayoutLMv3TokenizerFast`, *optional*): An instance of [`LayoutLMv3Tokenizer`] or [`LayoutLMv3TokenizerFast`]. The tokenizer is a required input. image_processor tokenizerZLayoutLMv3ImageProcessor)ZLayoutLMv3TokenizerZLayoutLMv3TokenizerFastNc sdd}d|vr"tdt|d}|dur.|n|}|durBtd|durRtdt||dS)Nfeature_extractorzhThe `feature_extractor` argument is deprecated and will be removed in v5, use `image_processor` instead.z)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.)warningswarn FutureWarningpop ValueErrorsuper__init__)selfr rkwargsr __class__p/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/layoutlmv3/processing_layoutlmv3.pyr3s zLayoutLMv3Processor.__init__TFr)text text_pairboxes word_labelsadd_special_tokenspadding truncation max_lengthstridepad_to_multiple_ofreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosereturn_tensorsreturncKs|jjr|durtd|jjr0|dur0td|j||d}|durn|jjrn|durnt|trf|g}|d}|jf|dur|n|d|dur|nd|dur|n|d||||| | | | | ||||||d|}|d}|d ur|||d }||d<|S) aF This method first forwards the `images` argument to [`~LayoutLMv3ImageProcessor.__call__`]. In case [`LayoutLMv3ImageProcessor`] was initialized with `apply_ocr` set to `True`, it passes the obtained words and bounding boxes along with the additional arguments to [`~LayoutLMv3Tokenizer.__call__`] and returns the output, together with resized and normalized `pixel_values`. In case [`LayoutLMv3ImageProcessor`] was initialized with `apply_ocr` set to `False`, it passes the words (`text`/``text_pair`) and `boxes` specified by the user along with the additional arguments to [`~LayoutLMv3Tokenizer.__call__`] and returns the output, together with resized and normalized `pixel_values`. Please refer to the docstring of the above two methods for more information. NzdYou cannot provide bounding boxes if you initialized the image processor with apply_ocr set to True.zaYou cannot provide word labels if you initialized the image processor with apply_ocr set to True.)imagesr.wordsr)rrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r. pixel_valuesToverflow_to_sample_mapping)r Z apply_ocrr isinstancestrrrget_overflowing_images)rr0rrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.rfeaturesZencoded_inputsrrr__call__EsP#  zLayoutLMv3Processor.__call__cCsLg}|D]}|||qt|t|krHtdt|dt||S)Nz`Expected length of images to be the same as the length of `overflow_to_sample_mapping`, but got z and )appendlenr)rr0r3Zimages_with_overflowZ sample_idxrrrr6sz*LayoutLMv3Processor.get_overflowing_imagescCsgdS)N)Z input_idsZbboxZattention_maskr2rrrrrmodel_input_namessz%LayoutLMv3Processor.model_input_namescCstdt|jS)Nzg`feature_extractor_class` is deprecated and will be removed in v5. Use `image_processor_class` instead.)rrrimage_processor_classr;rrrfeature_extractor_classs z+LayoutLMv3Processor.feature_extractor_classcCstdt|jS)Nz[`feature_extractor` is deprecated and will be removed in v5. Use `image_processor` instead.)rrrr r;rrrrs z%LayoutLMv3Processor.feature_extractor)NN)NNNNTFNNrNNNFFFFTN)__name__ __module__ __qualname____doc__ attributesr=Ztokenizer_classrrr rlistrintboolr5rr r rr8r6propertyr<r>r __classcell__rrrrr sf"   T  r )rBrtypingrrZprocessing_utilsrZtokenization_utils_baserrrr r utilsr r __all__rrrrs  "