a ½ Àhýã@sZdZddlmZmZddlmZddlmZddlm Z m Z m Z Gdd„deƒZ dgZ d S) z Processor class for MarkupLM. é)ÚOptionalÚUnioné)Ú TensorType)ÚProcessorMixin)Ú BatchEncodingÚPaddingStrategyÚTruncationStrategyc@sxeZdZdZdZdZdZd eeee e feee e fe e e e e e ee eeeeeee ee efedœd d „ZdS) ÚMarkupLMProcessoraJ Constructs a MarkupLM processor which combines a MarkupLM feature extractor and a MarkupLM tokenizer into a single processor. [`MarkupLMProcessor`] offers all the functionalities you need to prepare data for the model. It first uses [`MarkupLMFeatureExtractor`] to extract nodes and corresponding xpaths from one or more HTML strings. Next, these are provided to [`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`], which turns them into token-level `input_ids`, `attention_mask`, `token_type_ids`, `xpath_tags_seq` and `xpath_subs_seq`. Args: feature_extractor (`MarkupLMFeatureExtractor`): An instance of [`MarkupLMFeatureExtractor`]. The feature extractor is a required input. tokenizer (`MarkupLMTokenizer` or `MarkupLMTokenizerFast`): An instance of [`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`]. The tokenizer is a required input. parse_html (`bool`, *optional*, defaults to `True`): Whether or not to use `MarkupLMFeatureExtractor` to parse HTML strings into nodes and corresponding xpaths. ZMarkupLMFeatureExtractor)ZMarkupLMTokenizerZMarkupLMTokenizerFastTNFr)Úadd_special_tokensÚpaddingÚ truncationÚ max_lengthÚstrideÚpad_to_multiple_ofÚreturn_token_type_idsÚreturn_attention_maskÚreturn_overflowing_tokensÚreturn_special_tokens_maskÚreturn_offsets_mappingÚ return_lengthÚverboseÚreturn_tensorsÚreturncKsê|jrR|durtdƒ‚|dus.|dus.|dur6tdƒ‚| |¡}|d}|d}n(|durbtdƒ‚|dusr|durztdƒ‚|dur˜|jr˜t|tƒr˜|g}|jf|durª|n||dur¸|nd|||||| | | | | ||||||dœ|¤Ž}|S) aÆ This method first forwards the `html_strings` argument to [`~MarkupLMFeatureExtractor.__call__`]. Next, it passes the `nodes` and `xpaths` along with the additional arguments to [`~MarkupLMTokenizer.__call__`] and returns the output. Optionally, one can also provide a `text` argument which is passed along as first sequence. Please refer to the docstring of the above two methods for more information. NzDMake sure to pass HTML strings in case `parse_html` is set to `True`zUPlease don't pass nodes, xpaths nor node labels in case `parse_html` is set to `True`ÚnodesÚxpathsz@You have passed HTML strings but `parse_html` is set to `False`.zIMake sure to pass nodes and xpaths in case `parse_html` is set to `False`)ÚtextZ text_pairrÚ node_labelsr r r rrrrrrrrrrr)Ú parse_htmlÚ ValueErrorZfeature_extractorÚ isinstanceÚstrÚ tokenizer)ÚselfZ html_stringsrrrZ questionsr r r rrrrrrrrrrrÚkwargsÚfeaturesZencoded_inputs©r&úl/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/markuplm/processing_markuplm.pyÚ__call__2sP!ÿ   îízMarkupLMProcessor.__call__)NNNNNTFNNrNNNFFFFTN)Ú__name__Ú __module__Ú __qualname__Ú__doc__Zfeature_extractor_classZtokenizer_classrÚboolrr!rr rÚintrrr(r&r&r&r'r sPì  êr N)r,ÚtypingrrZ file_utilsrZprocessing_utilsrZtokenization_utils_baserrr r Ú__all__r&r&r&r'Ús   i