a
hx
@ s~ d Z ddlZddlZddlZddlmZmZ ddlmZ ddl m
Z
m Z ddl m
Z
mZmZmZmZ ddlmZ dd lmZmZ eeeeef eeeeeef eeeeef eeeeeef f ZG d
d d e
d d
Z G dd ded d
Z G dd ded d
Z G dd deZ eeeeef eeeef dddZ eeedddZ!dd Z"d d Z#d d Z$d#d!d"Z%dgZ&dS )$z Processor class for KOSMOS-2. N)OptionalUnion ) BatchFeature)
ImageInput
is_batched) ImagesKwargsProcessingKwargsProcessorMixin
TextKwargsUnpack)
AddedToken)
BatchEncoding TextInputc @ s6 e Zd ZU eee ed< ee ed< ee ed< dS )Kosmos2ImagesKwargsbboxesnum_image_tokensfirst_image_token_idN)__name__
__module__ __qualname__r listfloat__annotations__int r r j/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/kosmos2/processing_kosmos2.pyr % s
r F)totalc @ s e Zd ZU ee ed< dS )Kosmos2TextKwargs
add_eos_tokenN)r r r r boolr r r r r r + s
r c
@ s@ e Zd ZU eed< eed< dddddddddd ddid Zd
S ) Kosmos2ProcessorKwargs text_kwargs
images_kwargsTFr ) add_special_tokenspaddingZstrideZreturn_overflowing_tokensZreturn_special_tokens_maskZreturn_offsets_mappingreturn_token_type_idsverboser r @ )r" r# N)r r r r r r _defaultsr r r r r! / s
r! c s e Zd ZdZddgZdZdZd" fdd Zd#ee e
e e
f e e
ed
d d Zd
d Zdd Zd$e e
e e
f eeee e ee e f dddZd%ddZd&ddZedd Zee e ee e ee f ed d d Z e eeef ee e e e f f eeef d d d!Z Z S )'Kosmos2Processora,
Constructs an KOSMOS-2 processor which wraps a KOSMOS-2 image processor and a KOSMOS-2 tokenizer into a single
processor.
[`Kosmos2Processor`] offers all the functionalities of [`CLIPImageProcessor`] and some functionalities of
[`XLMRobertaTokenizerFast`]. See the docstring of [`~Kosmos2Processor.__call__`] and [`~Kosmos2Processor.decode`]
for more information.
Args:
image_processor (`CLIPImageProcessor`):
An instance of [`CLIPImageProcessor`]. The image processor is a required input.
tokenizer (`XLMRobertaTokenizerFast`):
An instance of ['XLMRobertaTokenizerFast`]. The tokenizer is a required input.
num_patch_index_tokens (`int`, *optional*, defaults to 1024):
The number of tokens that represent patch indices.
image_processor tokenizer)ZCLIPImageProcessorZCLIPImageProcessorFastZ
AutoTokenizer c s d|_ d| _d| _d| _d| _d| _d| _d| _d | _d
| _ d | _
d | _ | j| j| j| j| j| j| j| j| j | j
| j g | _ || _
d
d t| j
D }g }| j | D ]}|t|dddd q|| t || d S )NFzzzzzzz z z c S s" g | ]}d t |d dqS )
)strzfill.0xr r r
~ z-Kosmos2Processor.__init__..T)lstriprstrip
normalized)r& Z eod_token boi_token eoi_tokenZ eoc_tokenZ eol_tokenZ bop_tokenZ eop_tokenZ boo_tokenZ eoo_tokenZ dom_tokenZ grd_tokenZ
tag_tokensnum_patch_index_tokensrangeappendr
Z
add_tokenssuper__init__)selfr+ r, r> kwargsZpatch_index_tokensZ
tokens_to_addtoken __class__r r rB Z s>
zKosmos2Processor.__init__N)imagestextrD returnc s\ |du r|du rt djtfdjji|}|d dd}|d dd}|d dd} |d d
d }
|d d } |d d
} |d dd}
t }|durȈj|fi |d }| | |durj
||||d}| r,|
s,t |t rjj
| }n t |tr,fdd|D }|d d o<|
|d d < |du rT| nd |d d
< |du rn|
nd|d d< jf d|i|d }| | | |d d < | |d d
< |
|d d< |durX|durX| du rjjd } | }t|d }tt| | | }dgdg| dg }g }g }|d }t |t rR|g}|d g|d< |D ]n}|d| | ||| d }|| t|}|rdg| }|dgt|t| 7 }|| qVt |trtdd t|jD dd d}|d \}} |d \} }|d d o|
|d d < d|d d< jf d|| gi|d }t|jd | krjjd kr fd d|D } fd d|D } fd d|d D |d< nNjjd kr fd!d|D } fd"d|D } fd#d|d D |d< t |t r:|
du r:|d }|d d |d< |d }| t||d |d$|
d% |S )&a
This method uses [`CLIPImageProcessor.__call__`] method to prepare image(s) for the model, and
[`XLMRobertaTokenizerFast.__call__`] to prepare text for the model.
Please refer to the docstring of the above two methods for more information.
The rest of this documentation shows the arguments specific to `Kosmos2Processor`.
Args:
bboxes (`Union[list[tuple[int]], list[tuple[float]], list[list[tuple[int]]], list[list[tuple[float]]]]`, *optional*):
The bounding bboxes associated to `texts`.
num_image_tokens (`int`, *optional* defaults to 64):
The number of (consecutive) places that are used to mark the placeholders to store image information.
This should be the same as `latent_query_num` in the instance of `Kosmos2Config` you are using.
first_image_token_id (`int`, *optional*):
The token id that will be used for the first place of the subsequence that is reserved to store image
information. If unset, will default to `self.tokenizer.unk_token_id + 1`.
add_eos_token (`bool`, defaults to `False`):
Whether or not to include `EOS` token id in the encoding when `add_special_tokens=True`.
Nz*You have to specify either images or text.Ztokenizer_init_kwargsr# r r r( r r" r Fr$ r% return_tensors)r c s g | ]} j j | qS r )r, bos_token)r5 srC r r r7 r8 z-Kosmos2Processor.__call__..rI r input_idsattention_maskc S s g | ]\}}|t |fqS r len)r5 idxr6 r r r r7 r8 c S s | d S Nr )r6 r r r r8 z+Kosmos2Processor.__call__..)keyrV rightc s& g | ] }|j jg t| qS r r, Z pad_token_idrS r4 max_len_paddedrC r r r7 r8 c s" g | ]}|d g t | qS r rR r4 r\ r r r7 s c s" g | ]}|d g t | qS r] rR r4 r^ r r r7 s leftc s& g | ] }j jg t| | qS r rZ r4 r[ r r r7
r8 c s" g | ]}d g t | | qS r] rR r4 r^ r r r7 s c s" g | ]}d g t | | qS r] rR r4 r^ r r r7 s )rP rQ image_embeds_position_mask)dataZ tensor_type)
ValueErrorZ
_merge_kwargsr! r, Z init_kwargspop
setdefaultr r+ updatepreprocess_examples
isinstancer2 rL r Z unk_token_idr r? r@ copyrS sorted enumeraterP Z padding_sider ) rC rH rI ZaudioZvideosrD Z
output_kwargsr r r r r$ r% rK encodingZimage_encodingZ
text_encodingZwith_bos start_indexZimage_token_idsZ base_image_embeds_position_maskrP r` Z
all_input_idsZtext_idsmaskZ
sorted_length_Zmin_len_not_paddedrT r r[ r __call__ s
zKosmos2Processor.__call__c C s |du r dS t |ts td|D ]x}|du r2q"nt |tsB|g}|D ]R}t |trt|dkrrtdd |D sFt|dkrtdd |D sFtdqFq"dS ) a
Check `bboxes` for a single text example. It could be
- `None`: no bounding box associated to a text.
- A list with each element being the bounding boxes associated to one ` ... ` pair found
in a text. This could be:
- `None`: no bounding box associated to a ` ... ` pair.
- A tuple of 2 integers: A single bounding box specified by patch indices.
- A tuple of 4 float point number: A single bounding box specified by (normalized) coordinates.
- A list containing the above 2 tuple types: Multiple bounding boxes for a
` ... ` pair.
Nz@`bboxes` (for a single text example) should be `None` or a list. c s s | ]}t |tV qd S N)rg r r4 r r r B r8 zAKosmos2Processor._check_bboxes_for_single_text..r0 c s s | ]}t |tV qd S rq )rg r r4 r r r rr C r8 a' Each element in `bboxes` (for a single text example) should be either `None`, a tuple containing 2 integers or 4 float point numbers, or a list containing such tuples. Also make sure the arguments `texts` and `bboxes` passed to `preprocess_text` are both in batches or both for a single example.)rg r rb tuplerS all)rC r bboxelementr r r _check_bboxes_for_single_text) s,
z.Kosmos2Processor._check_bboxes_for_single_textc C s. | }|d ur | d| }| ||}|S )N )strip_insert_patch_index_tokens)rC rI imager img_info_tokensr r r _preprocess_single_exampleL s
z+Kosmos2Processor._preprocess_single_exampler( )textsrH r r rJ c sD j g| }dj g| jg d}t|tr>d}|g}|du rVdgt| }nt|sd|g}t|t|krtdt| dt| d|s| |g}n>|durt|t std|D ]}| qndgt| }t|t|krtd t| dt| d fd
d t
|||D }|s@|d }|S )
a- Add image and bounding box information to `texts` as image and patch index tokens.
Args:
texts (`Union[TextInput, list[TextInput]]`): The texts to be processed.
images (`ImageInput`, *optional*): The images associated to `texts`.
bboxes (`Union[list[tuple[int]], list[tuple[float]], list[list[tuple[int]]], list[list[tuple[float]]]]`, *optional*):
The bounding bboxes associated to `texts`.
num_image_tokens (`int`, *optional*, defaults to 64):
The number of image tokens (used as latent queries). This should corresponds to the `latent_query_num`
attribute in `Kosmos2Config`.
Returns:
`Union[TextInput, list[TextInput]]`: The processed texts with image and patch index tokens.
rx TFNzGThe number of examples in `texts` and `images` should be the same. Got v.s. instead.zS`bboxes` should be `None` or a list (as a batch) when `texts` is passed as a batch.zGThe number of examples in `texts` and `bboxes` should be the same. Got c s" g | ]\}}} ||| qS r )r} )r5 rI r{ ru r| rC r r r7 s z8Kosmos2Processor.preprocess_examples..r ) r< joinr= rg r2 rS r rb rw r zip) rC r~ rH r r Z
img_tokensZbatchedr6 resultr r r rf V sB
z$Kosmos2Processor.preprocess_examplesTc C s | | jd }|r t|S |S rU )splitr= +clean_text_and_extract_entities_with_bboxes)rC rI cleanup_and_extractcaptionr r r post_process_generation s z(Kosmos2Processor.post_process_generationc s( j |fd|i|} fdd|D S )a
Post-process the output of the model to decode the text.
Args:
generated_outputs (`torch.Tensor` or `np.ndarray`):
The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)`
or `(sequence_length,)`.
skip_special_tokens (`bool`, *optional*, defaults to `True`):
Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method.
**kwargs:
Additional arguments to be passed to the tokenizer's `batch_decode method`.
Returns:
`list[str]`: The decoded text.
skip_special_tokensc s g | ]} j |d dqS )F)r )r )r5 rI rN r r r7 r8 zDKosmos2Processor.post_process_image_text_to_text..)Z batch_decode)rC Zgenerated_outputsr rD Zgenerated_textsr rN r post_process_image_text_to_text s z0Kosmos2Processor.post_process_image_text_to_textc C s | j j}| jj}|| dg S )Nr` )r, model_input_namesr+ )rC Ztokenizer_input_namesZimage_processor_input_namesr r r r s z"Kosmos2Processor.model_input_names)rI r rJ c C sT |d u st |dkr|S ttjd|d}t |t |krXtdt | dt | dd}g }t||D ]\}}| \}} |||| | }|d u rqjt|t r|g}g }
t
dd |D std |D ]&} | | \} }
|
| d
|
qt |
dkrqjd |
}|d | d
qj|t |k rF|||d d |}|S )Nr z.+?)stringzuThe number of elements in `bboxes` should be the same as the number of ` ... ` pairs in `text`. Got r r c s s | ]}|d uV qd S rq r )r5 boxr r r rr r8 z>Kosmos2Processor._insert_patch_index_tokens..zTThe multiple bounding boxes for a single phrase should not contain any `None` value.rx z z )
rS r refinditerrb r spanr@ rg rs rt #_convert_bbox_to_patch_index_tokensr )rC rI r Zmatched_phrasescurr_posbufferZmatchedru rn endZpatch_index_stringsr Z
patch_index_1Z
patch_index_2Z position_strr r r rz sB
z+Kosmos2Processor._insert_patch_index_tokens)ru rJ c C sh t |dkr|\}}n tt| j}t||\}}dt|d d}dt|d d}||fS )Nrp r/ r0 r1 )rS r mathsqrtr> coordinate_to_patch_indexr2 r3 )rC ru Zidx_1Zidx_2num_patches_per_sideZtoken_1Ztoken_2r r r r s
z4Kosmos2Processor._convert_bbox_to_patch_index_tokens)r- )NNNN)NNr( )T)T) r r r __doc__
attributesZimage_processor_classZtokenizer_classrB r r r r r r! r ro rw r} BboxInputr r r2 rf r r propertyr rs r rz r
__classcell__r r rF r r* D sH / ##
B
*.
r* )ru r rJ c C s | \}}}}||kr ||ks$t dt|| }t|| }t|| d }t|| d } || | }
| | | } |
| fS )a Convert a bounding box to a pair of patch indices.
Args:
bbox (`tuple[float, float, float, float]`):
The 4 coordinates of the bounding box, with the format being (x1, y1, x2, y2) specifying the upper-left and
lower-right corners of the box. It should have x2 > x1 and y2 > y1.
num_patches_per_side (`int`): the number of patches along each side.
Returns:
`tuple[int, int]`: A pair of patch indices representing the upper-left patch and lower-right patch.
zTThe coordinates in `bbox` should be `(x1, y1, x2, y2)` with `x2 > x1` and `y2 > y1`.rO )rb r floorceil) ru r x1y1x2y2ul_xul_ylr_xlr_yul_idxlr_idxr r r r s r )r r r c C s d| }| | }| | }|| }|| }| |krZ|| }|| } || | }
|| | } nz||ksj||kr|| }|| } || | }
|| | } n@|| |d }|| |d } || |d }
|| |d } || |
| fS )a
Given a grid of length `num_patches_per_side` and the indices of the upper-left and lower-right corners of a
bounding box, returns the normalized coordinates of the bounding box, in the form (x1, y1, x2, y2).
Args:
ul_idx (`int`): the index of the grid cell that corresponds to the upper-left corner of the bounding box.
lr_idx (`int`): the index of the grid cell that corresponds to the lower-right corner of the bounding box.
num_patches_per_side (`int`): the number of patches along each side.
Returns:
`tuple[float]`: the normalized coordinates of the bounding box, in the form (x1, y1, x2, y2).
g ?rp r ) r r r cell_sizer r r r r r r r r r r patch_index_to_coordinate s( r c C s6 d}t || }g }|D ]}|d}| \}}}|sZd}|dd |dd f}|d} g }
| D ]v} t d| } t d| dd }
| rl|
rl|r|
t| dt|
df ql|
t| dt|
df ql|r||||
f q|
D ]0}d|d d |d d
}||||gf qq|S ) a Extract entities contained in `text`. The bounding bboxes is given in the form of patch indices.
This functioin is only intended to be used within `clean_text_and_extract_entities_with_bboxes` where further
processing happens, including converting to normalized coordinates and whitespace character cleaning up.
Examples:
```python
>>> text = " An image of a snowman warming himself by a fire."
>>> entities = extract_entities_with_patch_indices(text)
>>> entities
[(' a snowman', (31, 41), [(44, 863)]), (' a fire', (130, 137), [(5, 911)])]
```z(?:(([^<]+)))?rp Nr r. zrO r/ z>r N)rS r sub)r rI entity_namestartr Zadjusted_startZ adjusted_endadjusted_entityr r r adjust_entity_positions| s
r c C s | }t| t| }g }|D ]j\}\}}}t|t| } t|t| }
|| | }|| |
}| }||||f|f q$||fS )z9Remove the spaces around the text and the entities in it.)ry rS r9 r: r@ ) rI entitiesnew_textZleading_spacesZ new_entitiesr r r r Zentity_name_leading_spacesZentity_name_trailing_spacesr r r _cleanup_spaces s r c
sp t dd| }t| }g }|D ]F}|dd |d }}t|| } fdd|D } ||| f q t||S )a Remove the tag tokens from `text`, extract entities in it with some cleaning up of white characters.
Examples:
```python
>>> text = " An image of a snowman warming himself by a fire."
>>> clean_text, entities = clean_text_and_extract_entities_with_bboxes(text)
>>> clean_text
'An image of a snowman warming himself by a fire.'
>>> entities
[('a snowman', (12, 21), [(0.390625, 0.046875, 0.984375, 0.828125)]), ('a fire', (41, 47), [(0.171875, 0.015625, 0.484375, 0.890625)])]
```r r r rp c s g | ]}t |d |d qS )r rO )r )r5 ru r r r r7 r8 z?clean_text_and_extract_entities_with_bboxes..)r r r r r@ r )
rI r Zprocessed_textr r itemr r r Zbboxes_in_coordsr r r r s
r )r )'r rh r r typingr r Zimage_processing_utilsr Z image_utilsr r Zprocessing_utilsr r r
r r Ztokenization_utilsr
Ztokenization_utils_baser r r rs r r r r r r! r* r r r r r r __all__r r r r s< 5& -: