a h @s ddlZddlZddlmZddlmZmZddlZ ddl Z ddl m m Zddl mZm Z ddlmZddlmZmZmZddlmZdd lmZmZdd lmZdd lmZmZm Z dd l!m"Z"d dl#m$Z$erddl%m&Z&e rddl'm(Z(ddl)m*Z*eeddGdddeZ+dLe je je jdddZ,eeedddZ-e je je jdddZ.Gdd d e j/Z0eee1ed!d"d#Z2e je je1e jd!d$d%Z3Gd&d'd'e j/Z4Gd(d)d)e j/Z5Gd*d+d+e j/Z6dMe j/e je je jee je7e7d-d.d/Z8Gd0d1d1e j/Z9Gd2d3d3e j/Z:dNe je7e;e jd4d5d6ZGd;d<dd>eZ@Gd?d@d@e jAZBGdAdBdBe j/ZCGdCdDdDe j/ZDGdEdFdFe j/ZEeGdGdHdHeZFedIdGdJdKdKeFZGdHdKgZHdS)ON) dataclass)CallableOptional)Tensornn)ACT2FN) ModelOutputis_scipy_availablerequires_backends)GradientCheckpointingLayer)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringis_accelerate_available)check_model_inputs) EomtConfig)linear_sum_assignment) PartialState)reducea Class for outputs of [`EomtForUniversalSegmentationOutput`]. This output can be directly passed to [`~EomtImageProcessor.post_process_semantic_segmentation`] or [`~EomtImageProcessor.post_process_instance_segmentation`] or [`~EomtImageProcessor.post_process_panoptic_segmentation`] to compute final segmentation maps. Please, see [`~EomtImageProcessor] for details regarding usage. )Z custom_introc@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ ee ejed<dZee ejed<dZeeejed <dS) "EomtForUniversalSegmentationOutputa+ loss (`torch.Tensor`, *optional*): The computed loss, returned when labels are present. class_queries_logits (`torch.FloatTensor`): A tensor of shape `(batch_size, num_queries, num_labels + 1)` representing the proposed classes for each query. Note the `+ 1` is needed because we incorporate the null class. masks_queries_logits (`torch.FloatTensor`): A tensor of shape `(batch_size, num_queries, height, width)` representing the proposed masks for each query. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Last hidden states (final feature map) of the last layer. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states all layers of the model. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tuple(torch.FloatTensor)` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Self and Cross Attentions weights from transformer decoder. patch_offsets (`list[torch.Tensor]`, *optional*): list of tuples indicating the image index and start and end positions of patches for semantic segementation. Nlossclass_queries_logitsmasks_queries_logitslast_hidden_state hidden_states attentions patch_offsets)__name__ __module__ __qualname____doc__rrtorchZ FloatTensor__annotations__rrrrtuplerr listrr)r)b/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/eomt/modeling_eomt.pyr2s rF)input_featurespoint_coordinatesreturncKsL|dkrd}|d}tjjj|d|dfi|}|rH|d}|S)a( A wrapper around `torch.nn.functional.grid_sample` to support 3D point_coordinates tensors. Args: input_features (`torch.Tensor` of shape (batch_size, channels, height, width)): A tensor that contains features map on a height * width grid point_coordinates (`torch.Tensor` of shape (batch_size, num_points, 2) or (batch_size, grid_height, grid_width,: 2)): A tensor that contains [0, 1] * [0, 1] normalized point coordinates add_dim (`bool`): boolean value to keep track of added dimension Returns: point_features (`torch.Tensor` of shape (batch_size, channels, num_points) or (batch_size, channels, height_grid, width_grid): A tensor that contains features for points in `point_coordinates`. rTg@?)dimZ unsqueezer%r functionalZ grid_samplesqueeze)r+r,Zadd_dimkwargsZpoint_featuresr)r)r* sample_point]s    r4)inputslabelsr-cCsd|d}dt||j}|ddddf|ddddf}d|d|d}|S)a A pair wise version of the dice loss, see `dice_loss` for usage. Args: inputs (`torch.Tensor`): A tensor representing a mask labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: `torch.Tensor`: The computed loss between each pairs. rr.N)sigmoidflattenr%matmulTsum)r5r6 numerator denominatorrr)r)r*pair_wise_dice_loss}s ,r?c Csj|jd}tjdd}||t|}||t|}t|||j}t||d|j}||}|S)a A pair wise version of the cross entropy loss, see `sigmoid_cross_entropy_loss` for usage. Args: inputs (`torch.Tensor`): A tensor representing a mask. labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: loss (`torch.Tensor`): The computed loss between each pairs. rnoneZ reduction)shaperBCEWithLogitsLossr%Z ones_likeZ zeros_liker:r;) r5r6Zheight_and_width criterionZcross_entropy_loss_posZcross_entropy_loss_negZloss_posZloss_negrr)r)r*$pair_wise_sigmoid_cross_entropy_losss  rEcs\eZdZdZd eeeedfdd Zeej ej ej ej e e e ddd Z Z S) EomtHungarianMatcheraqThis class computes an assignment between the labels and the predictions of the network. For efficiency reasons, the labels don't include the no_object. Because of this, in general, there are more predictions than labels. In this case, we do a 1-to-1 matching of the best predictions, while the others are un-matched (and thus treated as non-objects). r/1) cost_class cost_mask cost_dice num_pointscsFt|dkr*|dkr*|dkr*td||_||_||_||_dS)aHCreates the matcher Params: cost_class (`float`, *optional*, defaults to 1.0): Relative weight of the classification error in the matching cost. cost_mask (`float`, *optional*, defaults to 1.0): This is the relative weight of the focal loss of the binary mask in the matching cost. cost_dice (`float`, *optional*, defaults to 1.0): This is the relative weight of the dice loss of the binary mask in the matching cost. num_points (`int`, *optional*, defaults to 12544): No. of points to sample on which the mask loss will be calculated. The same set of K points are uniformly sampled for all prediction and ground truth masks to construct the cost matrix for bipartite matching. rzAll costs can't be 0N)super__init__ ValueErrorrKrHrIrJ)selfrHrIrJrK __class__r)r*rMs zEomtHungarianMatcher.__init__)rr mask_labels class_labelsr-cCshg}|jd}t|D]<}||d}||} |dd||f } ||| } | dddf} | dddf} tjd|jd| jd} | | jddd} t | | dd d} | | jddd}t | |dd d} t | | }t | | }|j ||j| |j|}t|td }t|td }t|d}t|}||qd d |D}|S) ao Params: masks_queries_logits (`torch.Tensor`): A tensor of dim `batch_size, num_queries, num_labels` with the classification logits. class_queries_logits (`torch.Tensor`): A tensor of dim `batch_size, num_queries, height, width` with the predicted masks. class_labels (`torch.Tensor`): A tensor of dim `num_target_boxes` (where num_target_boxes is the number of ground-truth objects in the target) containing the class labels. mask_labels (`torch.Tensor`): A tensor of dim `num_target_boxes, height, width` containing the target masks. Returns: matched_indices (`list[tuple[Tensor]]`): A list of size batch_size, containing tuples of (index_i, index_j) where: - index_i is the indices of the selected predictions (in order) - index_j is the indices of the corresponding selected labels (in order) For each batch element, it holds: len(index_i) = len(index_j) = min(num_queries, num_target_boxes). rr7Nrr.deviceFZ align_cornersg _Bg _cSs0g|](\}}tj|tjdtj|tjdfqS)dtype)r% as_tensorint64).0ijr)r)r* sz0EomtHungarianMatcher.forward..)rBrangesoftmaxtor%randrKrUrepeatr4r2rEr?rIrHrJminimumtensormaximumZ nan_to_numrcpuappend)rOrrrRrSindices batch_sizer\Z pred_probsZ pred_maskrHZ target_maskr,Ztarget_coordinatesZpred_coordinatesrIrJZ cost_matrixZassigned_indicesZmatched_indicesr)r)r*forwards4      zEomtHungarianMatcher.forward)r/r/r/rG)r!r"r#r$floatintrMr%no_gradrr(r'rk __classcell__r)r)rPr*rFs rF)r5r6 num_masksr-cCsX|d}d||d}|d|d}d|d|d}||}|S)a4 Compute the DICE loss, similar to generalized IOU for masks as follows: $$ \mathcal{L}_{\text{dice}(x, y) = 1 - \frac{2 * x \cap y }{x \cup y + 1}} $$ In practice, since `labels` is a binary mask, (only 0s and 1s), dice can be computed as follow $$ \mathcal{L}_{\text{dice}(x, y) = 1 - \frac{2 * x * y }{x + y + 1}} $$ Args: inputs (`torch.Tensor`): A tensor representing a mask. labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). num_masks (`int`): The number of masks present in the current batch, used for normalization. Returns: `torch.Tensor`: The computed loss. rr.r7)r8r9r<)r5r6rpZprobsr=r>rr)r)r* dice_losss  rqcCs,tjdd}|||}|d|}|S)a| Args: inputs (`torch.Tensor`): A float tensor of arbitrary shape. labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: loss (`torch.Tensor`): The computed loss. r@rAr)rrCmeanr<)r5r6rprDZcross_entropy_lossrr)r)r*sigmoid_cross_entropy_loss7s  rsc sPeZdZeeeefdfdd Zeee ee dddZ ee e e e fddd Z e ee e ejeee fd d d Zej eej e eje eeej fd ddZddZddZej ej dddZej e e eej dddZd!ej ej eej eej eeeej feeej fdddZej ejej ddd ZZS)"EomtLossconfig weight_dictcstt|dg|j|_||_|j|_t|jd}|j|d<| d||j |_ |j |_ |j |_ t|j|j|j|j d|_dS)aH The Eomt Loss. The loss is computed very similar to DETR. The process happens in two steps: 1) we compute hungarian assignment between ground truth masks and the outputs of the model 2) we supervise each pair of matched ground-truth / prediction (supervise class and mask) Args: config (`EomtConfig`): The configuration for Eomt model also containing loss calculation specific parameters. weight_dict (`dict[str, float]`): A dictionary of weights to be applied to the different losses. Zscipyrr7 empty_weight)rHrJrIrKN)rLrMr num_labelsrwZno_object_weightZeos_coefr%onesregister_bufferZtrain_num_pointsrKoversample_ratioimportance_sample_ratiorF class_weight dice_weight mask_weightmatcher)rOrvrwrxrPr)r*rMLs"    zEomtLoss.__init__)sizesr-cCsB|d}|ddD](}t|D]\}}t|||||<q q|S)Nrr) enumeratemax)rOrZmaxesZsublistindexitemr)r)r* _max_by_axisos zEomtLoss._max_by_axis)tensorsr-cCs|dd|D}t|g|}|\}}}}|dj}|dj} tj||| d} tj|||ftj| d} t|| | D]\\} } }| d| j dd| j dd| j df | d|d| j dd| j df<qv| | fS)NcSsg|]}t|jqSr))r(rB)r[rer)r)r*r^yz8EomtLoss._pad_images_to_max_in_batch..rrXrUrr.F) rlenrXrUr%zerosrzboolziprBZcopy_)rOrmax_sizeZ batch_shaperj_heightwidthrXrUZpadded_tensorsZ padding_masksreZ padded_tensorZ padding_maskr)r)r*_pad_images_to_max_in_batchws   2"z$EomtLoss._pad_images_to_max_in_batch)rrSrir-cCs|}|j\}}}tj|jd}||} tddt||D} tj||f|j tj |j d} | | | <| dd} || | } d| i}|S)aCompute the losses related to the labels using cross entropy. Args: class_queries_logits (`torch.Tensor`): A tensor of shape `batch_size, num_queries, num_labels` class_labels (`list[torch.Tensor]`): List of class labels of shape `(labels)`. indices (`tuple[np.array])`: The indices computed by the Hungarian matcher. Returns: `dict[str, Tensor]`: A dict of `torch.Tensor` containing the following key: - **loss_cross_entropy** -- The loss computed using cross entropy on the predicted and ground truth labels. )weightcSsg|]\}\}}||qSr)r))r[targetrr]r)r)r*r^rz(EomtLoss.loss_labels..)Z fill_valuerXrUrr.loss_cross_entropy) rBrZCrossEntropyLossrx$_get_predictions_permutation_indicesr%catrfullryrZrU transpose)rOrrSriZ pred_logitsrj num_queriesrrDidxZtarget_classes_oZtarget_classesZpred_logits_transposedZloss_celossesr)r)r* loss_labelss    zEomtLoss.loss_labels)rrRrirpr-cs|}|}||}|\}} ||}|dddf}|dddf}tD|fddjjj} t || dd d} Wdn1s0Yt || dd d} t | | |t | | |d} ~~| S)aCompute the losses related to the masks using sigmoid_cross_entropy_loss and dice loss. Args: masks_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, height, width)`. mask_labels (`torch.Tensor`): List of mask labels of shape `(labels, height, width)`. indices (`tuple[np.array])`: The indices computed by the Hungarian matcher. num_masks (`int)`: The number of masks, used for normalization. Returns: losses (`dict[str, Tensor]`): A dict of `torch.Tensor` containing two keys: - **loss_mask** -- The loss computed using sigmoid cross entropy loss on the predicted and ground truth. masks. - **loss_dice** -- The loss computed using dice loss on the predicted on the predicted and ground truth, masks. Ncs |SN)calculate_uncertaintylogitsrOr)r*rz%EomtLoss.loss_masks..FrVr) loss_mask loss_dice) r _get_targets_permutation_indicesrr%rnsample_points_using_uncertaintyrKr|r}r4r2rsrq)rOrrRrirpZsrc_idxZtgt_idxZ pred_masksZ target_masksrr,Z point_labels point_logitsrr)rr* loss_maskss.    2  zEomtLoss.loss_maskscCs4tddt|D}tdd|D}||fS)NcSs g|]\}\}}t||qSr)r%Z full_like)r[r\srcrr)r)r*r^rzAEomtLoss._get_predictions_permutation_indices..cSsg|] \}}|qSr)r))r[rrr)r)r*r^rr%rr)rOri batch_indicesZpredictions_indicesr)r)r*rsz-EomtLoss._get_predictions_permutation_indicescCs4tddt|D}tdd|D}||fS)NcSs g|]\}\}}t||qSr)r)r[r\rtgtr)r)r*r^rz=EomtLoss._get_targets_permutation_indices..cSsg|] \}}|qSr)r))r[rrr)r)r*r^rr)rOrirZtarget_indicesr)r)r*rsz)EomtLoss._get_targets_permutation_indices)rr-cCst| }|S)a In Eomt paper, uncertainty is estimated as L1 distance between 0.0 and the logit prediction in 'logits' for the foreground class in `classes`. Args: logits (`torch.Tensor`): A tensor of shape (R, 1, ...) for class-specific or class-agnostic, where R is the total number of predicted masks in all images and C is: the number of foreground classes. The values are logits. Returns: scores (`torch.Tensor`): A tensor of shape (R, 1, ...) that contains uncertainty scores with the most uncertain locations having the highest uncertainty score. )r%abs)rOrZuncertainty_scoresr)r)r*rs zEomtLoss.calculate_uncertainty)rrKr|r}r-cCs|jd}t||}tj||d|jd}t||dd} || } t||} || } tj| dddddf| ddd} |tj|tj|jd }| |dddf7} | d d| d ddf || d}| dkrtj |tj|| d|jdgdd }|S) a This function is meant for sampling points in [0, 1] * [0, 1] coordinate space based on their uncertainty. The uncertainty is calculated for each point using the passed `uncertainty function` that takes points logit prediction as input. Args: logits (`float`): Logit predictions for P points. uncertainty_function: A function that takes logit predictions for P points and returns their uncertainties. num_points (`int`): The number of points P to sample. oversample_ratio (`int`): Oversampling parameter. importance_sample_ratio (`float`): Ratio of points that are sampled via importance sampling. Returns: point_coordinates (`torch.Tensor`): Coordinates for P sampled points. rr.rTFrVNr)kr0rr7r0) rBrmr%rbrUr4Ztopkarangelongviewr)rOrZuncertainty_functionrKr|r}Z num_boxesZnum_points_sampledr,rZpoint_uncertaintiesZnum_uncertain_pointsZnum_random_pointsrshiftr)r)r*rs"   &(z(EomtLoss.sample_points_using_uncertaintyNrrrRrSauxiliary_predictionsr-c s|||||}|j||djd}i|||||||||}|durt|D]H\} | d}| d}|||||} fdd| D} || qV|S)a This performs the loss computation. Args: masks_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, height, width)`. class_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, num_labels)`. mask_labels (`torch.Tensor`): List of mask labels of shape `(labels, height, width)`. class_labels (`list[torch.Tensor]`): List of class labels of shape `(labels)`. auxiliary_predictions (`dict[str, torch.Tensor]`, *optional*): if `use_auxiliary_loss` was set to `true` in [`EomtConfig`], then it contains the logits from the inner layers of the EomtMaskedAttentionDecoder. Returns: losses (`dict[str, Tensor]`): A dict of `torch.Tensor` containing three keys: - **loss_cross_entropy** -- The loss computed using cross entropy on the predicted and ground truth labels. - **loss_mask** -- The loss computed using sigmoid cross_entropy loss on the predicted and ground truth masks. - **loss_dice** -- The loss computed using dice loss on the predicted on the predicted and ground truth masks. if `use_auxiliary_loss` was set to `true` in [`EomtConfig`], the dictionary contains additional losses for each auxiliary predictions. rrTNrrcs i|]\}}|d|qS)rr))r[keyvaluerr)r* nrz$EomtLoss.forward..) r get_num_masksrUrrrrkitemsupdate) rOrrrRrSrrirprZ aux_outputs loss_dictr)rr*rk<s$  zEomtLoss.forward)rSrUr-cCs^tdd|D}tj|tj|d}d}trHtjikrHt|}tj}tj ||dd}|S)zk Computes the average number of target masks across the batch, for normalization purposes. cSsg|] }t|qSr))r)r[classesr)r)r*r^wrz*EomtLoss.get_num_masks..rr)min) r<r%rYrlrrZ _shared_staterZ num_processesclamp)rOrSrUrpZ world_sizer)r)r*rss zEomtLoss.get_num_masks)N)r!r"r#rdictstrrlrMr(rmrrr'rnparrayrr%rrrrrrrkrUrror)r)rPr*rtKs># $ > = 7rtcs4eZdZdZfddZejejdddZZS)EomtPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r8|n||f}t|tjj rR|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr kernel_sizeZstride)rLrM image_size patch_size num_channels hidden_size isinstance collectionsabcIterable num_patchesrConv2d projection)rOrvrrrrrrPr)r*rMs  zEomtPatchEmbeddings.__init__ pixel_valuesr-cCsH|jd}||jkr,td|jd|d||ddd}|S)NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .r.)rBrrNrr9r)rOrr embeddingsr)r)r*rks  zEomtPatchEmbeddings.forward) r!r"r#r$rMr%rrkror)r)rPr*rs rcs<eZdZdZeddfdd ZejejdddZZ S) EomtEmbeddingszM Construct the CLS token, mask token, position and patch embeddings. Nrvr-cst||_|j|_ttdd|j|_ tt d|j |j|_ t ||_|jj}t|j|_d|j |_t||j|_|jdt|ddddS)Nr position_ids)rr7F) persistent)rLrMrvrr Parameterr%Zrandnr cls_tokenrZnum_register_tokensregister_tokensrpatch_embeddingsrZDropoutZhidden_dropout_probdropoutnum_prefix_tokens Embeddingposition_embeddingsr{rexpand)rOrvrrPr)r*rMs   zEomtEmbeddings.__init__rcCs~|j\}}}}|jjjj}||j|d}|j|dd}|j|dd}|| |j }t j |||gdd}| |}|S)NrWr7rr)rBrrrrXrarrrrrr%rr)rOrrjrZ target_dtyperZ cls_tokensrr)r)r*rks  zEomtEmbeddings.forward) r!r"r#r$rrMr%rrkror)r)rPr*rsr)modulequeryrrattention_maskscalingrc Ks|t||dd|}|dur(||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)Nr7)r0rX)ptrainingrr.) r%r:rrr1r`float32rarXrr contiguous) rrrrrrrr3 attn_weights attn_outputr)r)r*eager_attention_forwards  rcsLeZdZdZfddZdejeejeejeejfdddZ Z S) EomtAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkrZtd|jd|jd|jd|_ |j |_ d|_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gF)rLrMrvr embed_dimnum_attention_heads num_headshead_dimrNscaleZattention_dropoutr is_causalrLineark_projv_projq_projout_projrOrvrPr)r*rMs$   zEomtAttention.__init__N)rrr-c Ks|j\}}}||}||}||} ||||j|jdd}||||j|jdd}| |||j|jdd} t} |j j dkrt |j j } | |||| ||j |j |jsdn|jd\} } | |||} || } | | fS)z#Input shape: Batch x Time x Channelrr.eagerr)rrr)rBrrrrrrrrrvZ_attn_implementationr rrrrreshaperr) rOrrr3rjZ seq_lengthrZquerieskeysvaluesZattention_interfacerrr)r)r*rks.        zEomtAttention.forward)N) r!r"r#r$rMr%rrr'rkror)r)rPr*rs rcs6eZdZddfdd ZejejdddZZS)EomtLayerScaleNr-cs(tt|jt|j|_dSr) rLrMrrlayerscale_valuer%rzrlambda1rrPr)r*rMs zEomtLayerScale.__init__ hidden_stater-cCs ||jSr)rrOr r)r)r*rk#szEomtLayerScale.forwardr!r"r#rMr%rrkror)r)rPr*rsr)input drop_probrr-cCsd|dks |s|Sd|}|jdfd|jd}|tj||j|jd}||||}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rrr)rr)rBndimr%rbrXrUZfloor_div)r rrZ keep_probrBZ random_tensoroutputr)r)r* drop_path's rcsPeZdZdZd eeddfdd ZejejdddZ e d d d Z Z S) EomtDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).N)rr-cst||_dSr)rLrMr)rOrrPr)r*rM>s zEomtDropPath.__init__rr-cCst||j|jSr)rrrrOrr)r)r*rkBszEomtDropPath.forwardrcCs d|jS)Nzp=)rrr)r)r* extra_reprEszEomtDropPath.extra_repr)N) r!r"r#r$rrlrMr%rrkrrror)r)rPr*r;srcs6eZdZddfdd ZejejdddZZS)EomtMLPNrcsnt|j}}t|j|j}tj||dd|_t|j t rPt |j |_ n|j |_ tj||dd|_ dS)NTbias)rLrMrrm mlp_ratiorrfc1r hidden_actrr activationfc2rOrvZ in_featuresZ out_featuresZhidden_featuresrPr)r*rMJs   zEomtMLP.__init__r cCs"||}||}||}|Sr)rrrr r)r)r*rkUs   zEomtMLP.forwardr r)r)rPr*rIs rcs6eZdZddfdd ZejejdddZZS) EomtSwiGLUFFNNrcslt|j}}t|j|j}t|ddddd}tj|d|dd|_tj||dd|_dS)Nr.rTr) rLrMrrmrrr weights_in weights_outrrPr)r*rM]s   zEomtSwiGLUFFN.__init__r cCs6||}|jddd\}}tj||}||S)Nr.r7r)r#chunkrr1Zsilur$)rOr x1Zx2Zhiddenr)r)r*rkfs zEomtSwiGLUFFN.forwardr r)r)rPr*r \s r csFeZdZdZeddfdd Zd ejeejejdddZ Z S) EomtLayerzCThis corresponds to the Block class in the original implementation.Nrcsttj|j|jd|_t||_t ||_ |j dkrFt |j nt |_tj|j|jd|_|jrvt||_n t||_t ||_dS)Nepsr)rLrMr LayerNormrlayer_norm_epsnorm1r attentionr layer_scale1Zdrop_path_raterZIdentityrnorm2Zuse_swiglu_ffnr mlpr layer_scale2rrPr)r*rMps     zEomtLayer.__init__)r head_maskr-cCsb||}|||\}}||}|||}||}||}||}|||}|Sr)r,r-r.rr/r0r1)rOrr2Zhidden_states_normZself_attention_outputrZ layer_outputr)r)r*rks     zEomtLayer.forward)N) r!r"r#r$rrMr%rrrkror)r)rPr*r'msr'cs2eZdZdfdd ZejejdddZZS) EomtLayerNorm2dư>Tcstj|||ddS)N)r)Zelementwise_affine)rLrM)rOrr)ZaffinerPr)r*rMszEomtLayerNorm2d.__init__r cCs>|dddd}t||j|j|j|j}|dddd}|S)Nrr.rr)ZpermuteFZ layer_normZnormalized_shaperrr)r r)r)r*rkszEomtLayerNorm2d.forward)r4Tr r)r)rPr*r3sr3cs6eZdZedfdd ZejejdddZZS)EomtScaleLayerrvcsVt|j}tj||ddd|_t|j|_tj ||dd|dd|_ t ||_ dS)Nr.rrrF)rpaddinggroupsr) rLrMrrConvTranspose2dconv1rrrrconv2r3 layernorm2drOrvrrPr)r*rMs   zEomtScaleLayer.__init__rcCs,||}||}||}||}|Sr)r;rr<r=rr)r)r*rks     zEomtScaleLayer.forward r!r"r#rrMr%rrkror)r)rPr*r6sr6cs6eZdZedfdd ZejejdddZZS)EomtScaleBlockr7cs6tj|_tfddt|jD|_dS)Ncsg|] }tqSr))r6r[rr7r)r*r^rz+EomtScaleBlock.__init__..)rLrMZnum_upscale_blocks num_blocksr ModuleListr_blockrrPr7r*rMs zEomtScaleBlock.__init__rcCs|jD] }||}q|Sr)rD)rOrrDr)r)r*rks  zEomtScaleBlock.forwardr?r)r)rPr*r@sr@cs6eZdZedfdd ZejejdddZZS) EomtMaskHeadr7csJt|j}t|||_t|||_t|||_t|j |_ dSr) rLrMrrrrrfc3rrrr>rPr)r*rMs  zEomtMaskHead.__init__rcCs.|||}|||}||}|Sr)rrrrFrr)r)r*rks zEomtMaskHead.forwardr?r)r)rPr*rEs rEc@sPeZdZUdZeed<dZdZdZdgZ dZ dZ e e dZejd d d d Zd S) EomtPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. rvZeomtrFr'T)rrN)rr-cCsZ|jj}t|tjtjtjfrtjj|j t dd|j durtj |j \}}|dkrhdt |nd}tj|j | |nt|tjr|j jd|j jnt|tjr|j jjddd|jdur|j j|jnpt|trt|drV|jj|jjnDt|trVtjj|jjtjd|d|jj |j_|j!jdS) N)arrr/r)rrstdr)"rvZinitializer_rangerrrrr:initZkaiming_uniform_rmathsqrtrZ_calculate_fan_in_and_fan_outZuniform_r*dataZfill_Zzero_rZnormal_Z padding_idxrhasattrrrrZ trunc_normal_rrar%rrXr)rOrrJZfan_inrboundr)r)r* _init_weightss0       z!EomtPreTrainedModel._init_weights)r!r"r#r$rr&Zbase_model_prefixmain_input_nameZsupports_gradient_checkpointingZ_no_split_modulesZ_supports_sdpaZ_supports_flash_attnr'rZ_can_record_outputsrModulerQr)r)r)r*rGs rGzV The EoMT Model with head on top for instance/semantic/panoptic segmentation. c seZdZdZedfdd ZeeeeeeefeeefdddZ eeefedd d Z e e dee eee eee eeeeed d dZddZejdddZeddZZS)EomtForUniversalSegmentationrr7cst|_j|_t|_tjjj d|_ t j j|_ tfddtjD|_t|_t|_tjjd|_jjjjf|_jjjd|_t|jd|_ |!dt"#j$|%dS)Nr(csg|] }tqSr))r'rAr7r)r*r^rz9EomtForUniversalSegmentation.__init__..r)rrrruattn_mask_probs)&rLrMrvnum_hidden_layersrrrr*rr+ layernormrrrrCr_layersr@ upscale_blockrE mask_headrryclass_predictorrr grid_sizer~rrrwrtrDr{r%rzrBZ post_initrrPr7r*rM s$     z%EomtForUniversalSegmentation.__init__rc CsN|j|||||d}|jD]*\}}|D]\} } || vr.| |9} q.q|S)NrrrRrSr)rDrwr) rOrrrRrSrrrrZloss_keyrr)r)r* get_loss_dict(s  z*EomtForUniversalSegmentation.get_loss_dict)rr-cCs t|Sr)r<r)rOrr)r)r*get_loss@sz%EomtForUniversalSegmentation.get_lossN)rrRrSr r3r-cKsxd\}}d}|durtd||} t|jD]\} } | |j|jjkr|jjdddddf | j ddd | j } t j| | fdd} | |j|jjkr|js|j| |j|jjdkr|| } || \}}||f7}||f7}t j| j d| j d| j d| j t jd}tj||jd d }||d|dd}|jj}||jj}|dk|ddd||df<|j||j| |j|jj|||j d }|dddd f d|jjdd}||d }| | |} q0|| }||\}}||f7}||f7}d}|durf|durfd}t ||D],\}}|j!||||dd}||"|7}q8t#|||||dS)ah mask_labels (`list[torch.Tensor]`, *optional*): list of mask labels of shape `(num_labels, height, width)` to be fed to a model class_labels (`list[torch.LongTensor]`, *optional*): list of target class labels of shape `(num_labels, height, width)` to be fed to a model. They identify the labels of `mask_labels`, e.g. the label of `mask_labels[i][j]` if `class_labels[i][j]`. patch_offsets (`list[torch.Tensor]`, *optional*): list of tuples indicating the image index and start and end positions of patches for semantic segementation. )r)r)Nz You have to specify pixel_valuesrr7rr)rUrXZbilinear)sizemode)probnum_query_tokensencoder_start_tokensrU.gerr])rrrrr )$rNrrrXrVrvrBrrrrBrarUr%rrrUrWpredictrzrr5Z interpolater\rr`rr_disable_attention_maskrrlZ masked_fillrr^r_r)rOrrRrSr r3Zmasks_queries_logits_per_layerZclass_queries_logits_per_layerrrrZ layer_modulerZnorm_hidden_statesrrZinterpolated_logitsrcrdZsequence_outputrrr)r)r*rkCs 2     "     z$EomtForUniversalSegmentation.forwardcCs|jjSr)rrrr)r)r*get_input_embeddingssz1EomtForUniversalSegmentation.get_input_embeddingsrcCs|ddd|jjddf}||}|dd|jj|jjdddf}|dd}|j|jddg|jR}| |}| |}t d||}||fS)Nrr.rr7zbqc, bchw -> bqhw) rvrr[rrrrrBr\rZrYr%Zeinsum)rOrZ query_tokensZ class_logitsZ prefix_tokensZ mask_logitsr)r)r*res &   z$EomtForUniversalSegmentation.predictcCsD|dkr@tj|jd||d|k}d|ddd||df|<|S)NrrrT)r%rbrB)Z attn_maskrbrcrdrUZrandom_queriesr)r)r*rfsz4EomtForUniversalSegmentation._disable_attention_mask)NNN)r!r"r#rRrrMrrrr^r_rrrr(rrrrkrgr%re staticmethodrfror)r)rPr*rTs6     grT)F)r)rF)Icollections.abcrrL dataclassesrtypingrrnumpyrr%Ztorch.nn.functionalrr1r5rZ activationsrZ file_utilsr r r Zmodeling_layersr Zmodeling_utilsr rZprocessing_utilsrutilsrrrZ utils.genericrZconfiguration_eomtrZscipy.optimizerZ acceleraterZaccelerate.utilsrrr4r?rErSrFrmrqrsrtrrrlrrrrrrrr r'r*r3r6r@rErGrT__all__r)r)r)r*s          ! j:!, > *  +C