a h @sbdZddlZddlmZddlmZmZmZddl Z ddl Z ddl m Z ddl m Z ddlmZdd lmZmZmZmZdd lmZmZdd lmZdd lmZmZdd lmZmZm Z m!Z!m"Z"ddl#m$Z$m%Z%ddl&m'Z'e!(e)Z*Gddde j+Z,Gddde j+Z-d;e j+e j.e j.e j.ee j.e/e/dddZ0Gddde j+Z1Gddde j+Z2Gddde j+Z3Gdd d e j+Z4Gd!d"d"e j+Z5Gd#d$d$eZ6Gd%d&d&e j+Z7e Gd'd(d(eZ8e Gd)d*d*e8Z9Gd+d,d,e j+Z:e d-d.Gd/d0d0e8Z;e d1d.Gd2d3d3e8Zgd:Z?dS)||kr>||kr>|jS|jddddf}|jddddf}|jd}||j} ||j} t|d} |d| | |}|dddd}t j j || | fdd d }|dddd dd|}tj ||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing and 2 class embeddings. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rr N?rrZbicubicF)sizemodeZ align_cornersdim)shaper,r$Zjit is_tracingr0rreshapepermuter functionalZ interpolateviewcat) r1r6r7r8r+Z num_positionsZclass_and_dist_pos_embedZpatch_pos_embedr>Z new_heightZ new_widthZsqrt_num_positionsr4r4r5interpolate_pos_encoding=s(     z'DeiTEmbeddings.interpolate_pos_encoding) pixel_valuesbool_masked_posrFrcCs|j\}}}}||}|\}} }|durb|j|| d} |d| } |d| | | }|j|dd} |j|dd} t j | | |fdd}|j }|r| |||}||}| |}|S)Nr9?rr=)r?r*r;r(expand unsqueezeZtype_asr&r'r$rEr,rFr/)r1rGrHrF_r7r8r6 batch_sizeZ seq_lengthZ mask_tokensmaskZ cls_tokensZdistillation_tokensZposition_embeddingr4r4r5forwardes   zDeiTEmbeddings.forward)F)NF)__name__ __module__ __qualname____doc__rboolr"r$TensorintrFr BoolTensorrO __classcell__r4r4r2r5r,s +rcs4eZdZdZfddZejejdddZZS)r)z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r8|n||f}t|tjj rR|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr) kernel_sizeZstride)r!r" image_sizer0 num_channelsr% isinstance collectionsabcIterabler+rConv2d projection)r1rrZr0r[r%r+r2r4r5r"s  zDeiTPatchEmbeddings.__init__)rGrcCs<|j\}}}}||jkr td||ddd}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r r)r?r[ ValueErrorraflatten transpose)r1rGrMr[r7r8xr4r4r5rOs zDeiTPatchEmbeddings.forward) rPrQrRrSr"r$rUrOrXr4r4r2r5r)s r))modulequerykeyvalueattention_maskscalingr/c Ks|t||dd|}tjj|dtjd|j}tjj |||j d}|durX||}t||} | dd } | |fS)Nr9)r>dtype)ptrainingrr ) r$matmulrdrrCZsoftmaxfloat32tornr/rp contiguous) rgrhrirjrkrlr/kwargsZ attn_weightsZ attn_outputr4r4r5eager_attention_forwards  rvcsJeZdZedfdd ZdejeejeejejfdddZ Z S) DeiTSelfAttentionrcst|j|jdkr>t|ds>td|jd|jd||_|j|_t|j|j|_|j|j|_ |j |_ |jd|_ d|_ tj|j|j |jd|_tj|j|j |jd|_tj|j|j |jd|_dS) NrZembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .gF)bias)r!r"r%num_attention_headshasattrrbrrVattention_head_size all_head_sizeZattention_probs_dropout_prob dropout_probrl is_causalrLinearZqkv_biasrhrirjr1rr2r4r5r"s"   zDeiTSelfAttention.__init__N hidden_states head_maskrc Cs|jd}|d|j|jf}||j|dd}||j|dd}||j|dd}t}|j j dkr~t |j j }|||||||j |j |jsdn|jd\} } | dd|jf} | | } | | fS) Nrr9rr eagerrf)rrlr/rm)r?r{r}rirDrdrjrhrvrZ_attn_implementationrrrlrprr;r~rA) r1rrrMZ new_shapeZ key_layerZ value_layerZ query_layerZattention_interfaceZ context_layerZattention_probsZnew_context_layer_shaper4r4r5rOs*     zDeiTSelfAttention.forward)N) rPrQrRrr"r$rUrtuplerOrXr4r4r2r5rws  rwcs>eZdZdZedfdd ZejejejdddZZ S)DeiTSelfOutputz The residual connection is defined in DeiTLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rxcs.tt|j|j|_t|j|_dSN) r!r"rrr%denser-r.r/rr2r4r5r"s zDeiTSelfOutput.__init__r input_tensorrcCs||}||}|Srrr/r1rrr4r4r5rOs  zDeiTSelfOutput.forward) rPrQrRrSrr"r$rUrOrXr4r4r2r5rsrcsReZdZedfdd ZeedddZd ej e ej ej dd d Z Z S) DeiTAttentionrxcs*tt||_t||_t|_dSr)r!r"rw attentionroutputset pruned_headsrr2r4r5r" s   zDeiTAttention.__init__)headscCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr=)lenrrr{r}rrrhrirjrrr~union)r1rindexr4r4r5 prune_headss zDeiTAttention.prune_headsNrcCs |||\}}|||}|Sr)rr)r1rrZself_attn_outputrLrr4r4r5rO%s zDeiTAttention.forward)N) rPrQrRrr"rrVrr$rUrrOrXr4r4r2r5r srcs6eZdZedfdd ZejejdddZZS)DeiTIntermediaterxcsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr) r!r"rrr%intermediate_sizerr\Z hidden_actstrrintermediate_act_fnrr2r4r5r"-s   zDeiTIntermediate.__init__rrcCs||}||}|Sr)rr)r1rr4r4r5rO5s  zDeiTIntermediate.forward rPrQrRrr"r$rUrOrXr4r4r2r5r,srcs:eZdZedfdd ZejejejdddZZS) DeiTOutputrxcs.tt|j|j|_t|j|_ dSr) r!r"rrrr%rr-r.r/rr2r4r5r"=s zDeiTOutput.__init__rcCs ||}||}||}|Srrrr4r4r5rOBs  zDeiTOutput.forwardrr4r4r2r5r<srcsDeZdZdZedfdd Zd ejeejejdddZ Z S) DeiTLayerz?This corresponds to the Block class in the timm implementation.rxcsbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)Nreps)r!r"Zchunk_size_feed_forwardZ seq_len_dimrrr intermediaterrr LayerNormr%layer_norm_epslayernorm_beforelayernorm_afterrr2r4r5r"Ms    zDeiTLayer.__init__NrcCsB||}|||}||}||}||}|||}|Sr)rrrrr)r1rrZhidden_states_normZattention_outputZ layer_outputr4r4r5rOWs     zDeiTLayer.forward)N) rPrQrRrSrr"r$rUrrOrXr4r4r2r5rJs rcs>eZdZedfdd ZdejeejedddZ Z S) DeiTEncoderrxcs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr4)r).0rLrxr4r5 mz(DeiTEncoder.__init__..F) r!r"rrZ ModuleListrangenum_hidden_layerslayerZgradient_checkpointingrr2rxr5r"js  zDeiTEncoder.__init__NrcCs<t|jD]&\}}|dur"||nd}|||}q t|dS)N)last_hidden_state) enumeraterr )r1rriZ layer_moduleZlayer_head_maskr4r4r5rOps zDeiTEncoder.forward)N) rPrQrRrr"r$rUrr rOrXr4r4r2r5risrc@sbeZdZUeed<dZdZdZdgZdZ dZ dZ dZ e edZeejejejfddd d ZdS) DeiTPreTrainedModelrdeitrGTr)r attentionsN)rgrcCst|tjtjfrZtjj|jjt j d|j j d|jj |j_|jdur|jjnlt|tjr|jj|jjdnDt|tr|jj|jj|jj|jdur|jjdS)zInitialize the weightsrf)meanZstdNrI)r\rrr`initZ trunc_normal_weightdatarsr$rrrZinitializer_rangernrzZzero_rZfill_rr&r,r'r()r1rgr4r4r5 _init_weightss"        z!DeiTPreTrainedModel._init_weights)rPrQrRr__annotations__Zbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingZ_no_split_modulesZ_supports_sdpaZ_supports_flash_attnZ_supports_flex_attnZ_supports_attention_backendrrwZ_can_record_outputsrrrr`rrr4r4r4r5rxs rc sxeZdZdeeeddfdd Zeddd Zd d Ze e de e j e e je e j eeeed d dZZS) DeiTModelTFN)radd_pooling_layerrrcs\t|||_t||d|_t||_tj|j |j d|_ |rJt |nd|_ |dS)z add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer use_mask_token (`bool`, *optional*, defaults to `False`): Whether to use a mask token for masked image modeling. )rrN)r!r"rrr6rencoderrrr%r layernorm DeiTPoolerpooler post_init)r1rrrr2r4r5r"s  zDeiTModel.__init__)rcCs|jjSr)r6r*)r1r4r4r5get_input_embeddingsszDeiTModel.get_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r1Zheads_to_prunerrr4r4r5 _prune_headsszDeiTModel._prune_headsrGrHrrFrurc Ks|durtd|||jj}|jjjjj}|j|krB| |}|j|||d}|j ||d}|j } | | } |j dur| | nd} t| | dS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_values)rHrF)r)rZ pooler_output)rbZ get_head_maskrrr6r*rarrnrsrrrrr ) r1rGrHrrFruZexpected_dtypeZembedding_outputZencoder_outputssequence_output pooled_outputr4r4r5rOs"   zDeiTModel.forward)TF)NNNF)rPrQrRrrTr"r)rrrrrr$rUrWrrr rOrXr4r4r2r5rs"rcs6eZdZedfdd ZejejdddZZS)rrxcs,tt|j|j|_t|j|_ dSr) r!r"rrr%Zpooler_output_sizerrZ pooler_act activationrr2r4r5r"s zDeiTPooler.__init__rcCs(|dddf}||}||}|S)Nr)rr)r1rZfirst_token_tensorrr4r4r5rOs  zDeiTPooler.forwardrr4r4r2r5rsrad DeiT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://huggingface.co/papers/2111.09886). Note that we provide a script to pre-train this model on custom data in our [examples directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining). )Z custom_introc s\eZdZeddfdd Zeed eej eej eej e e e edddZZS) DeiTForMaskedImageModelingNrrcsXt|t|ddd|_ttj|j|jd|j ddt |j|_ | dS)NFT)rrr r)Z in_channelsZ out_channelsrY) r!r"rrrZ Sequentialr`r%Zencoder_strider[Z PixelShuffledecoderrrr2r4r5r"s   z#DeiTForMaskedImageModeling.__init__FrcKs |j|f|||d|}|j}|ddddf}|j\}} } t| d} } |ddd|| | | }||} d}|dur|jj|jj }|d||}| |jj d |jj d d }t jj|| dd }|||d |jj}t|| |j|jd S) a; bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForMaskedImageModeling.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```)rHrrFNrr9r:rr none)Z reductiongh㈵>)lossZreconstructionrr)rrr?rVrBrArrrZr0Zrepeat_interleaverKrtrrCZl1_losssumr[r rr)r1rGrHrrFruoutputsrrMZsequence_lengthr[r7r8Zreconstructed_pixel_valuesZmasked_im_lossr;rNZreconstruction_lossr4r4r5rOsB&   z"DeiTForMaskedImageModeling.forward)NNNF)rPrQrRrr"rrrr$rUrWrTrrr rOrXr4r4r2r5rs rz DeiT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. c s\eZdZeddfdd Zeed eej eej eej e e e e dddZZS) DeiTForImageClassificationNrcsRt||j|_t|dd|_|jdkr 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForImageClassification >>> import torch >>> from PIL import Image >>> import requests >>> torch.manual_seed(3) # doctest: +IGNORE_RESULT >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> # note: we are loading a DeiTForImageClassificationWithTeacher from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForImageClassification.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> # model predicts one of the 1000 ImageNet classes >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: Polaroid camera, Polaroid Land camera ```rrFNr)rlogitsrr)rrrZ loss_functionrr rr) r1rGrrrFrurrrrr4r4r5rOys&*z"DeiTForImageClassification.forward)NNNF)rPrQrRrr"rrrr$rUrTrrr rOrXr4r4r2r5rfs rzC Output type of [`DeiTForImageClassificationWithTeacher`]. c@steZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejed<dZ ee ejed<dS)+DeiTForImageClassificationWithTeacherOutputaj logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores as the average of the cls_logits and distillation logits. cls_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the classification head (i.e. the linear layer on top of the final hidden state of the class token). distillation_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the distillation head (i.e. the linear layer on top of the final hidden state of the distillation token). Nr cls_logitsdistillation_logitsrr)rPrQrRrSrrr$Z FloatTensorrrrrrrr4r4r4r5rs  ra DeiT Model transformer with image classification heads on top (a linear layer on top of the final hidden state of the [CLS] token and a linear layer on top of the final hidden state of the distillation token) e.g. for ImageNet. .. warning:: This model supports inference-only. Fine-tuning with distillation (i.e. with a teacher) is not yet supported. c sTeZdZeddfdd Zeed eej eej e e e e dddZZS) %DeiTForImageClassificationWithTeacherNrcsvt||j|_t|dd|_|jdkrst       Y( 5 $M `O 3