a h~@sdZddlZddlZddlmZmZddlZddlZddlm Z ddl m Z ddl m Z ddlmZmZdd lmZdd lmZmZdd lmZd d lmZeeZGddde jZejj ddZ!ddZ"Gddde jZ#d7ej$e%e&ej$dddZ'Gddde jZ(Gddde jZ)Gdd d e jZ*Gd!d"d"e jZ+d#d$Z,d%d&Z-Gd'd(d(e Z.Gd)d*d*e jZ/e jdd+d,d-Z0eGd.d/d/eZ1eGd0d1d1e1Z2ed2d3Gd4d5d5e1eZ3gd6Z4dS)8zPyTorch ViTDet backbone.N)OptionalUnion)nn)ACT2FN)GradientCheckpointingLayer)BackboneOutputBaseModelOutput)PreTrainedModel)auto_docstringlogging) BackboneMixin) VitDetConfigcs<eZdZdZfddZddZejejdddZZ S) VitDetEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r8|n||f}t|tjj rR|n||f}|d|d|d|d}||_ ||_||_||_ |j r|d}t td||j|_nd|_t j||||d|_dS)Nrr)Z kernel_sizeZstride)super__init__Zpretrain_image_size patch_size num_channels hidden_size isinstance collectionsabcIterable image_size num_patchesZ use_absolute_position_embeddingsr Parametertorchzerosposition_embeddingsConv2d projection)selfconfigrrrrrZ num_positions __class__f/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/vitdet/modeling_vitdet.pyr+s  zVitDetEmbeddings.__init__cCs|r|ddddf}|jd}tt|}|||krDtdtjs^||ks^||krtj j | d||d dddd||fdd d }| ddddS| d||dSdS) a Calculate absolute positional embeddings. If needed, resize embeddings and remove cls_token dimension for the original embeddings. Args: abs_pos_embeddings (`torch.Tensor`): Absolute positional embeddings with (1, num_position, num_channels). has_cls_token (`bool`): If true, has 1 embedding in abs_pos_embeddings for cls token. height (`int`): Height of input image tokens. width (`int`): Width of input image tokens. Returns: Absolute positional embeddings after processing with shape (1, height, width, num_channels) Nrz5Absolute position embeddings must be a square number.rrZbicubicF)sizemodeZ align_corners) shapeintmathsqrt ValueErrorrjit is_tracingr functional interpolatereshapepermute)r"Zabs_pos_embeddingsZ has_cls_tokenheightwidthZ num_positionr*Znew_abs_pos_embeddingsr&r&r'get_absolute_positionsAs  z'VitDetEmbeddings.get_absolute_positions) pixel_valuesreturncCs|jd}||jkr,td|jd|d||}|jdur|dddd}|||jd|jd|jd}|dddd}|S) NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .rr)rT)r,rr0r!rr6r9)r"r:r embeddingsr&r&r'forwardgs"    zVitDetEmbeddings.forward) __name__ __module__ __qualname____doc__rr9rTensorr> __classcell__r&r&r$r'r%s &rcCstdt||d}|jd|krftjj|d|jddddd|dd}|d|dd}n|}t |dddft||d}t |dddft||d}|||dt||d}|| S) a Get relative positional embeddings according to the relative positions of query and key sizes. Args: q_size (`int`): Size of query q. k_size (`int`): Size of key k. rel_pos (`torch.Tensor`): Relative position embeddings (num_embeddings, num_channels). Returns: Extracted positional embeddings according to relative positions. r)rrr(Zlinear)r*r+N?) r-maxr,rr3r4r5r6rZarangelong)q_sizek_sizeZrel_posZ max_rel_distZrel_pos_resizedZq_coordsZk_coordsZrelative_coordsr&r&r' get_rel_pos}s$$rJcCs|\}}|\}} t|||} t|| |} |j\} } }|| |||}td|| } td|| }|| |||| | dddddddddf|dddddddddf| |||| }|S)a Calculate decomposed Relative Positional Embeddings as introduced in [MViT2](https://github.com/facebookresearch/mvit/blob/19786631e330df9f3622e5402b4a419a263a2c80/mvit/models/attention.py). Args: attn (`torch.Tensor`): Attention map. queries (`torch.Tensor`): Query q in the attention layer with shape (batch_size, queries_height * queries_width, num_channels). rel_pos_h (`torch.Tensor`): Relative position embeddings (Lh, num_channels) for height axis. rel_pos_w (`torch.Tensor`): Relative position embeddings (Lw, num_channels) for width axis. q_size (`tuple[int]`): Spatial sequence size of query q with (queries_height, queries_width). k_size (`tuple[int]`): Spatial sequence size of key k with (keys_height, keys_width). Returns: attn (Tensor): attention map with added relative positional embeddings. zbhwc,hkc->bhwkzbhwc,wkc->bhwkN)rJr,r5rZeinsumview)Zattnqueries rel_pos_h rel_pos_wrHrIZqueries_heightZ queries_widthZ keys_heightZ keys_widthZrelative_heightZrelative_width batch_size_dimZr_qZrelative_weightr&r&r'!add_decomposed_relative_positionss"     rRcs,eZdZdZdfdd Zd ddZZS) VitDetAttentionz=Multi-head Attention block with relative position embeddings.Ncst|j}|j}||_||}|d|_tj||d|jd|_ t|||_ |j |_ |j rt t d|dd||_t t d|dd||_dS)z Args: config (`VitDetConfig`): Model configuration. input_size (`tuple[int]`, *optional*): Input resolution, only required in case relative position embeddings are added. grbiasr)rrN)rrrZnum_attention_heads num_headsscalerLinearZqkv_biasqkvproj use_relative_position_embeddingsrrrrMrN)r"r# input_sizerQrVZhead_dimr$r&r'rs   zVitDetAttention.__init__FcCs&|j\}}}}|||||d|jdddddd}|d||j||dd\}} } ||j| dd} |jrt | ||j |j ||f||f} | j dd} | | }| ||j||d}|ddddd}||||d}||}|r| ||j| jd| jd} || f} n|f} | S) Nrr(r)rr)rQ)r,rYr5rVr6ZunbindrWZ transposer[rRrMrNZsoftmaxrKrZ)r" hidden_stateoutput_attentionsrOr7r8rPrYrLkeysvaluesZattention_scoresZattention_probsoutputsr&r&r'r>s*,&   zVitDetAttention.forward)N)Fr?r@rArBrr>rDr&r&r$r'rSsrSF)input drop_probtrainingr;cCsd|dks |s|Sd|}|jdfd|jd}|tj||j|jd}||||}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rerr)r)dtypedevice)r,ndimrZrandrirjZfloor_div)rfrgrhZ keep_probr,Z random_tensoroutputr&r&r' drop_path s rncsPeZdZdZd eeddfdd ZejejdddZ e d d d Z Z S) VitDetDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).N)rgr;cst||_dSN)rrrg)r"rgr$r&r'r"s zVitDetDropPath.__init__) hidden_statesr;cCst||j|jSrp)rnrgrh)r"rqr&r&r'r>&szVitDetDropPath.forwardr;cCs d|jS)Nzp=)rgr"r&r&r' extra_repr)szVitDetDropPath.extra_repr)N) r?r@rArBrfloatrrrCr>strrtrDr&r&r$r'rosrocs*eZdZdZdfdd ZddZZS)VitDetLayerNormaL A LayerNorm variant, popularized by Transformers, that performs point-wise mean and variance normalization over the channel dimension for inputs that have shape (batch_size, channels, height, width). https://github.com/facebookresearch/ConvNeXt/blob/d1fa8f6fef0a165b27399986cc2bdacc92777e40/models/convnext.py#L119 ư>cs@ttt||_tt||_||_ |f|_ dSrp) rrrrrZonesweightrrUepsnormalized_shape)r"r{rzr$r&r'r4s  zVitDetLayerNorm.__init__cCsn|jddd}||djddd}||t||j}|jddddf||jddddf}|S)NrT)Zkeepdimr))meanpowrr/rzryrU)r"xusr&r&r'r>;s ,zVitDetLayerNorm.forward)rxrdr&r&r$r'rw-srwcs(eZdZdZfddZddZZS)VitDetResBottleneckBlockz The standard bottleneck residual block without the last activation layer. It contains 3 conv layers with kernels 1x1, 3x3, 1x1. csttj||ddd|_t||_t|j|_ tj||dddd|_ t||_ t|j|_ tj||ddd|_ t||_dS)ar Args: config (`VitDetConfig`): Model configuration. in_channels (`int`): Number of input channels. out_channels (`int`): Number of output channels. bottleneck_channels (`int`): Number of output channels for the 3x3 "bottleneck" conv layers. rFrTr)paddingrUN)rrrr conv1rwnorm1r hidden_actZact1conv2norm2Zact2conv3norm3)r"r# in_channels out_channelsbottleneck_channelsr$r&r'rIs     z!VitDetResBottleneckBlock.__init__cCs&|}|D] }||}q ||}|Srp)children)r"r~outlayerr&r&r'r>as   z VitDetResBottleneckBlock.forwardrdr&r&r$r'rCs rcs:eZdZeeddfdd ZejejdddZZS) VitDetMlpN) in_featureshidden_featuresr;csDtt|||_t|j|_t|||_t |j |_ dSrp) rrrrXfc1rractfc2ZDropoutZ dropout_probdrop)r"r#rrr$r&r'rks   zVitDetMlp.__init__)r~r;cCs6||}||}||}||}||}|Srp)rrrr)r"r~r&r&r'r>rs      zVitDetMlp.forward) r?r@rAr-rrrCr>rDr&r&r$r'rjsrc Cs|j\}}}}||||}||||}tj|ddd|d|f}||||}} |||||| |||}|ddddddd|||} | || ffS)a  Partition into non-overlapping windows with padding if needed. Args: hidden_state (`torch.Tensor`): Input tokens with [batch_size, height, width, num_channels]. window_size (`int`): Window size. Returns: `tuple(torch.FloatTensor)` comprising various elements: - windows: windows after partition with [batch_size * num_windows, window_size, window_size, num_channels]. - (padded_height, padded_width): padded height and width before partition rrrr)r]r()r,rr3padrKr6 contiguous) r_ window_sizerOr7r8rZ pad_heightZ pad_width padded_height padded_widthwindowsr&r&r'window_partition|s$rc Cs|\}}|\}}|jd||||}||||||||d} | dddddd} | |||d} | ddd|d|ddf} | S) aB Window unpartition into original sequences and removing padding. Args: windows (`torch.Tensor`): Input tokens with [batch_size * num_windows, window_size, window_size, num_channels]. window_size (`int`): Window size. pad_height_width (`tuple[int]`): Padded height and width (padded_height, padded_width). height_width (`tuple[int]`): Original height and width before padding. Returns: hidden_state: unpartitioned sequences with [batch_size, height, width, num_channels]. rr(rrr)r]rN)r,rKr6r) rrpad_height_widthZ height_widthrrr7r8rOr_r&r&r'window_unpartitions$rcsheZdZdZd eeeeddfdd Zd e j e e j ee e e j e j fe e j fdd d ZZS) VitDetLayerzCThis corresponds to the Block class in the original implementation.rFN)r#drop_path_rateruse_residual_blockr;c st|j}|j}t|ttfr(|n||f}|j}t|ttfrH|n||f}|d|d|d|df}tj ||j d|_ t ||dkr|n||fd|_ |dkrt|nt|_tj ||j d|_t||t||jd|_||_||_|jrt||||dd|_dS) Nrr)rz)r\re)r#rrr))r#rrr)rrrrrlisttuplerr LayerNormZlayer_norm_epsrrS attentionroZIdentityrnrrr-Z mlp_ratiomlprrrresidual) r"r#rrrrQrrr\r$r&r'rs.  zVitDetLayer.__init__)rq head_maskr`r;c Cs|dddd}|}||}|jdkrN|jd|jd}}t||j\}}|j||d}|d}|dd} |jdkrt||j|||f}|||}|||| |}|dddd}|j r| |}|f| } | S)Nrr)rr)r`) r6rrr,rrrrnrrrr) r"rqrr`Zshortcutr7r8rZself_attention_outputsrcr&r&r'r>s*      zVitDetLayer.forward)rrF)NF)r?r@rArBrrur-boolrrrCrrrr>rDr&r&r$r'rs&rc sNeZdZeddfdd Zd ejeejeeee e e fddd Z Z S) VitDetEncoderN)r#r;c st||_|j}ddtjd|j|ddD}g}t|D]4}|t |||||j vrb|j nd||j vdq@t ||_d|_dS)NcSsg|] }|qSr&)item).0r~r&r&r' z*VitDetEncoder.__init__..rcpu)rj)rrrF)rrr#num_hidden_layersrZlinspacerrangeappendrZwindow_block_indicesrZresidual_block_indicesrZ ModuleListrZgradient_checkpointing)r"r#depthrZlayersir$r&r'r s    zVitDetEncoder.__init__FT)rqrr`output_hidden_states return_dictr;c Cs|rdnd}|rdnd}t|jD]P\}} |r8||f}|durH||nd} | || |} | d}|r"|| df}q"|r||f}|stdd|||fDSt|||dS)Nr&rrcss|]}|dur|VqdSrpr&)rvr&r&r' ?rz(VitDetEncoder.forward..Zlast_hidden_staterq attentions) enumeraterrr ) r"rqrr`rrZall_hidden_statesZall_self_attentionsrZ layer_moduleZlayer_head_maskZ layer_outputsr&r&r'r>#s&     zVitDetEncoder.forward)NFFT)r?r@rArrrrCrrrrr r>rDr&r&r$r'r s rmoduler;cCs2tjj|jddd|jdur.tj|jddS)a Initialize `module.weight` using the "MSRAFill" implemented in Caffe2. Also initializes `module.bias` to 0. Source: https://detectron2.readthedocs.io/en/latest/_modules/fvcore/nn/weight_init.html. Args: module (torch.nn.Module): module to initialize. Zfan_outZrelu)r+Z nonlinearityNr)rinitZkaiming_normal_ryrUZ constant_)rr&r&r'caffe2_msra_fillGs  rc@sFeZdZUeed<dZdZdZgZe e j e j e j fddddZdS) VitDetPreTrainedModelr#Zvitdetr:TNrcCst|tjtjfr\tjj|jjt j d|j j d|jj |j_|jdurX|jjn@t|tjr|jj|jjdnt|trtjj|jjt j d|j j d|jj |j_nt|tr&|j jr&tjj|jjt j d|j j d|j_tjj|jjt j d|j j d|j_nvt|tr|j|j|jfD]}t|qB|j|jfD] }|jjd|jjq^|jjj|jjjdS)zInitialize the weightsre)r|ZstdNrE) rrrXr rZ trunc_normal_rydatatorZfloat32r#Zinitializer_rangerirUZzero_rZfill_rrrSr[rMrNrrrrrrrr)r"rrr&r&r' _init_weights]sL         z#VitDetPreTrainedModel._init_weights)r?r@rAr__annotations__Zbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingZ_no_split_modulesrrrXr rrr&r&r&r'rUs rc seZdZedfdd ZedddZeee efddd d Z e de e je e je ee ee eeeefd d d ZZS) VitDetModelr#cs2t|||_t||_t||_|dSrp)rrr#rr=rencoder post_initr"r#r$r&r'rs    zVitDetModel.__init__rrcCs|jjSrpr=r!rsr&r&r'get_input_embeddingssz VitDetModel.get_input_embeddingsN)heads_to_pruner;cCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrZ prune_heads)r"rrZheadsr&r&r' _prune_headsszVitDetModel._prune_heads)r:rr`rrr;c Cs|dur |n|jj}|dur |n|jj}|dur4|n|jj}|durLtd|||jj}||}|j|||||d}|d}|s|f|ddSt ||j |j dS)a Examples: ```python >>> from transformers import VitDetConfig, VitDetModel >>> import torch >>> config = VitDetConfig() >>> model = VitDetModel(config) >>> pixel_values = torch.randn(1, 3, 224, 224) >>> with torch.no_grad(): ... outputs = model(pixel_values) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 768, 14, 14] ```Nz You have to specify pixel_values)rr`rrrrr) r#r`ruse_return_dictr0Z get_head_maskrr=rr rqr) r"r:rr`rrembedding_outputZencoder_outputsZsequence_outputr&r&r'r>s. zVitDetModel.forward)NNNNN)r?r@rArrrrdictr-rrr rrrCrrrr r>rDr&r&r$r'rs"  rzF ViTDet backbone, to be used with frameworks like Mask R-CNN. )Z custom_introcsTeZdZfddZedddZed eje e e e e e e ddd Z Z S) VitDetBackbonecsVttt|_t|_fddtjdD|_ | dS)Ncsg|] }jqSr&)r)rrPrr&r'rrz+VitDetBackbone.__init__..r) rrZ_init_backbonerr=rrrrZ num_featuresrrr$rr'rs     zVitDetBackbone.__init__rrcCs|jjSrprrsr&r&r'rsz#VitDetBackbone.get_input_embeddingsN)r:rr`rr;c Cs|dur |n|jj}|dur |n|jj}|dur4|n|jj}||}|j|d||d}|rb|jn|d}d}t|j|D]\} } | |j vrz|| f7}qz|s|r|f|dd} n|f|dd} | St ||r|jnd|j dS)a Examples: ```python >>> from transformers import VitDetConfig, VitDetBackbone >>> import torch >>> config = VitDetConfig() >>> model = VitDetBackbone(config) >>> pixel_values = torch.randn(1, 3, 224, 224) >>> with torch.no_grad(): ... outputs = model(pixel_values) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 14, 14] ```NT)rr`rrr&r)) feature_mapsrqr) r#rrr`r=rrqzipZ stage_namesZ out_featuresrr) r"r:rr`rrrcrqrZstager_rmr&r&r'r>s4    zVitDetBackbone.forward)NNN)r?r@rArrrr rrCrrrr>rDr&r&r$r'rs r)rrr)reF)5rBcollections.abcrr.typingrrrZtorch.utils.checkpointrZ activationsrZmodeling_layersrZmodeling_outputsrr Zmodeling_utilsr utilsr r Zutils.backbone_utilsr Zconfiguration_vitdetrZ get_loggerr?loggerModulerr1Zscript_if_tracingrJrRrSrCrurrnrorwrrrrrrrrrr__all__r&r&r&r'sL       X $)?' Q;3WN