a hrV@sddlZddlmZddlmmZddlmZmZm Z ddl m Z m Z m Z mZmZerhddlmZerzddlmZejjdd ZGd d d ejZGd d d ejZdddZdS)N)is_scipy_availableis_vision_availablerequires_backends)box_iou dice_lossgeneralized_box_iounested_tensor_from_tensor_listsigmoid_focal_losslinear_sum_assignment)center_to_corners_formatcCsddt||DS)NcSsg|]\}}||dqS))logits pred_boxes).0abrrZ/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/loss/loss_rt_detr.py +z!_set_aux_loss..)zip) outputs_class outputs_coordrrr _set_aux_loss&srcs0eZdZdZfddZeddZZS)RTDetrHungarianMatcheraThis class computes an assignment between the targets and the predictions of the network For efficiency reasons, the targets don't include the no_object. Because of this, in general, there are more predictions than targets. In this case, we do a 1-to-1 matching of the best predictions, while the others are un-matched (and thus treated as non-objects). Args: config: RTDetrConfig csztt|dg|j|_|j|_|j|_|j |_ |j |_ |j |_ |j|jkrj|jkrjdkrvnntddS)NZscipyrz#All costs of the Matcher can't be 0)super__init__rZmatcher_class_cost class_costZmatcher_bbox_cost bbox_costZmatcher_giou_cost giou_costuse_focal_lossZ matcher_alphaalphaZ matcher_gammagamma ValueError)selfconfig __class__rrr9s  (zRTDetrHungarianMatcher.__init__cCs~|djdd\}}|ddd}tdd|D}td d|D}|jrt|ddd}|dd|f}d|j||jd|d  } |jd||j|d  } | | } n(|ddd d }|dd|f } tj ||dd } t t |t | } |j| |j| |j| }|||d }d d|D}ddt||d D}dd|DS)aPerforms the matching Params: outputs: This is a dict that contains at least these entries: "logits": Tensor of dim [batch_size, num_queries, num_classes] with the classification logits "pred_boxes": Tensor of dim [batch_size, num_queries, 4] with the predicted box coordinates targets: This is a list of targets (len(targets) = batch_size), where each target is a dict containing: "class_labels": Tensor of dim [num_target_boxes] (where num_target_boxes is the number of ground-truth objects in the target) containing the class labels "boxes": Tensor of dim [num_target_boxes, 4] containing the target box coordinates Returns: A list of size batch_size, containing tuples of (index_i, index_j) where: - index_i is the indices of the selected predictions (in order) - index_j is the indices of the corresponding selected targets (in order) For each batch element, it holds: len(index_i) = len(index_j) = min(num_queries, num_target_boxes) rNrrrrcSsg|] }|dqS class_labelsrrvrrrrbrz2RTDetrHungarianMatcher.forward..cSsg|] }|dqSboxesrr,rrrrcrg:0yE>)pcSsg|]}t|dqSr.lenr,rrrryrcSsg|]\}}t||qSrr )ricrrrrzrcSs0g|](\}}tj|tjdtj|tjdfqS)dtype)torch as_tensorint64)rr4jrrrr|r)shapeflattenr8catr"Fsigmoidr#r$logZsoftmaxZcdistr rr rr!viewcpu enumeratesplit)r&outputstargetsZ batch_sizeZ num_queriesZout_bboxZ target_idsZ target_bboxZout_probZneg_cost_classZpos_cost_classrr r!Z cost_matrixsizesindicesrrrforwardHs&&" zRTDetrHungarianMatcher.forward) __name__ __module__ __qualname____doc__rr8no_gradrJ __classcell__rrr(rr.s rcseZdZdZfddZdddZdddZed d Z d d Z d dZ dddZ ddZ ddZd ddZddZeddZddZZS)! RTDetrLossah This class computes the losses for RTDetr. The process happens in two steps: 1) we compute hungarian assignment between ground truth boxes and the outputs of the model 2) we supervise each pair of matched ground-truth / prediction (supervise class and box). Args: matcher (`DetrHungarianMatcher`): Module able to compute a matching between targets and proposals. weight_dict (`Dict`): Dictionary relating each loss with its weights. These losses are configured in RTDetrConf as `weight_loss_vfl`, `weight_loss_bbox`, `weight_loss_giou` losses (`list[str]`): List of all the losses to be applied. See `get_loss` for a list of all available losses. alpha (`float`): Parameter alpha used to compute the focal loss. gamma (`float`): Parameter gamma used to compute the focal loss. eos_coef (`float`): Relative classification weight applied to the no-object category. num_classes (`int`): Number of object categories, omitting the special no-object category. cs|tt||_|j|_|j|j|jd|_ ddg|_ |j |_ t |jd}|j |d<|d||j|_|j|_dS)N)loss_vfl loss_bbox loss_giouvflr/rr0 empty_weight)rrrmatcherZ num_labels num_classesZweight_loss_vflZweight_loss_bboxZweight_loss_giou weight_dictlossesZeos_coefficientZeos_coefr8ZonesZregister_bufferZfocal_loss_alphar#Zfocal_loss_gammar$)r&r'rVr(rrrs     zRTDetrLoss.__init__TcCstd|vrtdd|vr td||}|d|}tjddt||Ddd}tt|t|\} } t| } |d} td dt||D} tj | j dd |j tj | j d } | | |<tj| |j d d dddf}tj| | jd}| |j||<|d|}t| }|j||jd ||}tj| ||dd}|d | j d |}d|iS)Nr#No predicted boxes found in outputsrz$No predicted logits found in outputscSs g|]\}\}}|d|qSr.rr_target_r4rrrrrz.RTDetrLoss.loss_labels_vfl..rdimcSs g|]\}\}}|d|qSr*rr\rrrrrrr7devicerrX.r0r6none)weight reductionrR)KeyError_get_source_permutation_idxr8r>rrrdetachdiagfullr<rXr:rbr?one_hotZ zeros_liker7toZ unsqueezer@r#powr$ binary_cross_entropy_with_logitsmeansum)r&rFrGrI num_boxesrAidx src_boxes target_boxesZiousr^ src_logitstarget_classes_originaltarget_classestargetZtarget_score_originalZ target_scoreZ pred_scorerelossrrrloss_labels_vfls0    zRTDetrLoss.loss_labels_vflc Csd|vrtd|d}||}tddt||D}tj|jdd|jtj|j d} || |<t | dd| |j } d | i} | S) zClassification loss (NLL) targets dicts must contain the key "class_labels" containing a tensor of dim [nb_target_boxes] rz#No logits were found in the outputscSs g|]\}\}}|d|qSr*rr\rrrrrz*RTDetrLoss.loss_labels..Nrrarloss_ce)rgrhr8r>rrkr<rXr:rbr?Z cross_entropy transposeZ class_weight) r&rFrGrIrrrArvrsrwrxr|rZrrr loss_labelss zRTDetrLoss.loss_labelsc Csf|d}|j}tjdd|D|d}|d|jddkd}tj| | } d| i} | S)z Compute the cardinality error, i.e. the absolute error in the number of predicted non-empty boxes. This is not really a loss, it is intended for logging purposes only. It doesn't propagate gradients. rcSsg|]}t|dqSr*r2r,rrrrrz/RTDetrLoss.loss_cardinality..)rbr0rZcardinality_error) rbr8r9Zargmaxr<rqnn functionall1_lossfloat) r&rFrGrIrrrrbZtarget_lengthsZ card_predZcard_errrZrrrloss_cardinalityszRTDetrLoss.loss_cardinalityc Csd|vrtd||}|d|}tjddt||Ddd}i}tj||dd} | ||d <d tt t |t |} | ||d <|S) a; Compute the losses related to the bounding boxes, the L1 regression loss and the GIoU loss. Targets dicts must contain the key "boxes" containing a tensor of dim [nb_target_boxes, 4]. The target boxes are expected in format (center_x, center_y, w, h), normalized by the image size. rr[cSs g|]\}\}}|d|qSr.r)rtr^r4rrrrrz)RTDetrLoss.loss_boxes..rr_rdrfrSrrT) rgrhr8r>rr?rrqrjr r) r&rFrGrIrrrsrtrurZrSrTrrr loss_boxess  zRTDetrLoss.loss_boxesc Csd|vrtd||}||}|d}||}dd|D}t|\} } | |} | |} tjj|dddf| j ddddd }|ddd f d }| d } | |j } t || |t || |d } | S) z Compute the losses related to the masks: the focal loss and the dice loss. Targets dicts must contain the key "masks" containing a tensor of dim [nb_target_boxes, h, w]. Z pred_masksz#No predicted masks found in outputscSsg|] }|dqS)masksrrrrrrrrz)RTDetrLoss.loss_masks..NZbilinearF)sizemodeZ align_cornersrr)Z loss_maskZ loss_dice)rgrh_get_target_permutation_idxr Z decomposermrrZ interpolater<r=rBr r) r&rFrGrIrr source_idx target_idxZ source_masksrZ target_masksZvalidrZrrr loss_maskss(       zRTDetrLoss.loss_masksc Cs|d}||}tddt||D}tj|jdd|jtj|jd} || |<t j | |jddddd f} t j || d d d } | d |jd|} d | iS)NrcSs g|]\}\}}|d|qSr*rr\rrrr$rz.RTDetrLoss.loss_labels_bce..rrarrc.r0g?rdrZloss_bce)rhr8r>rrkr<rXr:rbr?rlrorprq r&rFrGrIrrrArvrsrwrxryrzrrrloss_labels_bce!s  zRTDetrLoss.loss_labels_bcecCs4tddt|D}tdd|D}||fS)NcSs g|]\}\}}t||qSrr8Z full_like)rr4sourcer^rrrr1rz:RTDetrLoss._get_source_permutation_idx..cSsg|] \}}|qSrr)rrr^rrrr2rr8r>rD)r&rI batch_idxrrrrrh/sz&RTDetrLoss._get_source_permutation_idxcCs4tddt|D}tdd|D}||fS)NcSs g|]\}\}}t||qSrr)rr4r^ryrrrr7rz:RTDetrLoss._get_target_permutation_idx..cSsg|] \}}|qSrr)rr^ryrrrr8rr)r&rIrrrrrr5sz&RTDetrLoss._get_target_permutation_idxc Csd|vrtd|d}||}tddt||D}tj|jdd|jtj|j d} || |<t j | |jddd dd f} t || |j |j} | d|jd|} d | iS) NrzNo logits found in outputscSs g|]\}\}}|d|qSr*rr\rrrrBrz0RTDetrLoss.loss_labels_focal..rrarrc.r0Z loss_focal)rgrhr8r>rrkr<rXr:rbr?rlr r#r$rprqrrrrloss_labels_focal;s  zRTDetrLoss.loss_labels_focalcCsL|j|j|j|j|j|j|jd}||vr:td|d||||||S)N)labelsZ cardinalityr/rZbceZfocalrUzLoss z not supported)r~rrrrrr{r%)r&rzrFrGrIrrZloss_maprrrget_lossMs zRTDetrLoss.get_lossc Cs|d|d}}dd|D}|ddj}g}t|D]\}}|dkrtj|tj|d} | |} t||t| ks~J|||| fq:|tjdtj|dtjdtj|dfq:|S)Ndn_positive_idx dn_num_groupcSsg|]}t|dqSr*r2rrrrr^rz6RTDetrLoss.get_cdn_matched_indices..rr+ra) rbrDr8Zaranger:Ztiler3appendZzeros) Zdn_metarGrrZnum_gtsrbZdn_match_indicesr4Znum_gtZgt_idxrrrget_cdn_matched_indices[s  z"RTDetrLoss.get_cdn_matched_indicesc sdd|D}||}tdd|D}tj|gtjtt|j d}tj |dd }i}j D]4} |||||fddD|qnd |vr,t|d D]p\}||}j D]T}|d krqԈ |||||fd dDfd dD|qqd |vrd|vrHtd|d|}||dd}t|d D]x\}j D]f}|d krqi} j |||||fi| fddDfddD|qqt|S)a This performs the loss computation. Args: outputs (`dict`, *optional*): Dictionary of tensors, see the output specification of the model for the format. targets (`list[dict]`, *optional*): List of dicts, such that `len(targets) == batch_size`. The expected keys in each dict depends on the losses applied, see each loss' doc. cSsi|]\}}d|vr||qS)auxiliary_outputsrrkr-rrr }rz&RTDetrLoss.forward..css|]}t|dVqdS)r+Nr2rrrr rz%RTDetrLoss.forward..rar)mincs*i|]"}|jvr||j|qSrrYrrl_dictr&rrrrrrcs*i|]"}|jvr||j|qSrrrrrrrrcs i|]\}}|d|qS)Z_aux_rrr4rrrrdn_auxiliary_outputsdenoising_meta_valuesz}The output must have the 'denoising_meta_values` key. Please, ensure that 'outputs' includes a 'denoising_meta_values' entry.rcs*i|]"}|jvr||j|qSrrrrrrrrcs i|]\}}|d|qS)Z_dn_rrrrrrr)itemsrWrqr8r9rnextitervaluesrbclampitemrZrupdaterDr%r) r&rFrGZoutputs_without_auxrIrrrZrzrkwargsr)r4rr&rrJrsJ  "         zRTDetrLoss.forward)T)T)T)T)rKrLrMrNrr{r~r8rOrrrrrhrrr staticmethodrrJrPrrr(rrQs        rQc Ks t|} | |i} || d<|| d<|jr| durdtj|| ddd\} }tj|| ddd\}}t|ddddfdd|ddddfdd}|| d <| d t|g|g| durt|dd| dd| d <| | d <| | |}t| }|||fS) NrrZ dn_num_splitrr_r0rrrrr) rQrmZauxiliary_lossr8rErr}extendrqr)rrrbrr'rrZenc_topk_logitsZenc_topk_bboxesrr criterionZ outputs_lossZ dn_out_coordZ dn_out_classrZ loss_dictrzrrrRTDetrForObjectDetectionLosss(  :  r)NNNNN)r8Ztorch.nnrZtorch.nn.functionalrr?utilsrrrZloss_for_object_detectionrrr r r Zscipy.optimizer Ztransformers.image_transformsrZjitZunusedrModulerrQrrrrrs(     Q<