a
h @ s d dl mZmZ d dlZd dlmZ d dlmZ ddlm Z m
Z
ddl m Z ddl
mZmZmZ dd lmZmZmZmZ G d
d d eZeG d d
d
eZG dd deeZeddG dd deeZg dZdS ) )OptionalUnionN) IJepaConfig )BaseModelOutputWithPoolingImageClassifierOutput)Unpack)TransformersKwargsauto_docstring torch_int )
ViTEmbeddingsViTForImageClassificationViTModelViTPreTrainedModelc s` e Zd Zd eedd fdd
ZejeeejdddZ d
eje
ej eejd d
d Z Z
S )IJepaEmbeddingsFN)configuse_mask_tokenreturnc s6 t || | `| jj}ttd||j | _
d S )N ) super__init__Z cls_tokenpatch_embeddings num_patchesnn ParametertorchZrandnZ hidden_sizeposition_embeddings)selfr r r __class__ c/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/ijepa/modular_ijepa.pyr s zIJepaEmbeddings.__init__)
embeddingsheightwidthr c C s |j d }| jj d }tj s6||kr6||kr6| jS | j}|j d }|| j }|| j } t|d }
|d|
|
|}|dddd}t j
j ||| fddd }|dddd dd|}|S )
a
This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution
images. This method is also adapted to support torch.jit tracing.
Adapted from:
- https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and
- https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211
r g ?r r r ZbicubicF)sizemodeZ
align_corners)
shaper r Zjit
is_tracingZ
patch_sizer ZreshapeZpermuter Z
functionalZ interpolateview) r r# r$ r% r Z
num_positionsZpatch_pos_embeddimZ
new_heightZ new_widthZsqrt_num_positionsr! r! r" interpolate_pos_encoding s&
z(IJepaEmbeddings.interpolate_pos_encoding) pixel_valuesbool_masked_posr- r c C s |j \}}}}| j||d}|d urb|j d } | j|| d}
|d|
} |d| |
| }|rz|| ||| }n
|| j }| |}|S )N)r- r r& ?) r) r
mask_tokenexpandZ unsqueezeZtype_asr- r Zdropout) r r. r/ r- Z
batch_size_r$ r% r# Z
seq_lengthZ mask_tokensmaskr! r! r" forward= s
zIJepaEmbeddings.forward)F)NF)__name__
__module__ __qualname__r boolr r Tensorintr- r Z
BoolTensorr5
__classcell__r! r! r r" r s * r c @ s, e Zd Zeejejejf ddddZdS )IJepaPreTrainedModelN)moduler c C s t |tjtjfrZtjj|jjt j
d| j j d|jj
|j_|jdur|jj nxt |tjr|jj |jjd nPt |trtjj|jjt j
d| j j d|jj
|j_|jdur|jj dS )zInitialize the weightsg )meanZstdNr0 )
isinstancer LinearConv2dinitZ
trunc_normal_weightdatator Zfloat32r Zinitializer_rangeZdtypeZbiasZzero_ LayerNormZfill_r r r1 )r r> r! r! r"
_init_weightsZ s*
z"IJepaPreTrainedModel._init_weights) r6 r7 r8 r r rA rB rG rH r! r! r! r" r= X s r= c s( e Zd Zdeeed fdd
Z ZS )
IJepaModelF)r add_pooling_layerr c s$ t | || _t||d| _dS )z
add_pooling_layer (bool, *optional*, defaults to `True`):
Whether to add a pooling layer
use_mask_token (`bool`, *optional*, defaults to `False`):
Whether to use a mask token for masked image modeling.
)r N)r r r r r# )r r rJ r r r! r" r r s zIJepaModel.__init__)FF)r6 r7 r8 r r9 r r< r! r! r r" rI q s rI a
IJepa Model transformer with an image classification head on top (a linear layer on top of the final hidden states)
e.g. for ImageNet.
Note that it's possible to fine-tune IJepa on higher resolution images than the ones it has been trained on, by
setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained
position embeddings to the higher resolution.
)Z custom_introc sV e Zd Zed fdd Zdeej eej eej ee e e
e dddZ Z
S ) IJepaForImageClassification)r c s& t | t|dd| _| d S )NF)rJ )r r rI ijepaZ post_init)r r r r! r" r s z$IJepaForImageClassification.__init__N)r. head_masklabelsr- kwargsr c
K sh | j |f||d|}|j}| |jdd}d} |durT| j||| jfi |} t| ||j|jdS )a
labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
Labels for computing the image classification/regression loss. Indices should be in `[0, ...,
config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
)rM r- r )r, N)losslogits
hidden_states
attentions) rL Zlast_hidden_state
classifierr? Z
loss_functionr r rR rS )
r r. rM rN r- rO outputsZsequence_outputrQ rP r! r! r" r5 s& z#IJepaForImageClassification.forward)NNNN)r6 r7 r8 r r r r r: r9 r r r r5 r< r! r! r r" rK ~ s rK )r= rI rK )typingr r r Ztorch.nnr Z-transformers.models.ijepa.configuration_ijepar Zmodeling_outputsr r Zprocessing_utilsr utilsr r
r Zvit.modeling_vitr
r r r r r= rI rK __all__r! r! r! r" s J
*