a ho @sdZddlmZmZddlZddlZddlmZddlmZddl m Z ddl m Z m Z mZdd lmZmZdd lmZdd lmZmZdd lmZmZmZmZdd lmZmZddlm Z e!e"Z#Gdddej$Z%Gdddej$Z&d1ej$ej'ej'ej'eej'e(e(dddZ)Gdddej$Z*Gdddej$Z+Gdddej$Z,Gdddej$Z-Gd d!d!ej$Z.Gd"d#d#e Z/Gd$d%d%ej$Z0Gd&d'd'ej$Z1eGd(d)d)eZ2eGd*d+d+e2Z3ed,d-Gd.d/d/e2Z4gd0Z5dS)2zPyTorch ViViT model.)CallableOptionalN)nn)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) find_pruneable_heads_and_indicesprune_linear_layer)TransformersKwargsauto_docstringlogging torch_int)can_return_tuplecheck_model_inputs) VivitConfigcs>eZdZdZedfdd Zd ejeejdddZ Z S) VivitTubeletEmbeddingsa Construct Vivit Tubelet embeddings. This module turns a batch of videos of shape (batch_size, num_frames, num_channels, height, width) into a tensor of shape (batch_size, seq_len, hidden_size) to be consumed by a Transformer encoder. The seq_len (the number of patches) equals (number of frames // tubelet_size[0]) * (height // tubelet_size[1]) * (width // tubelet_size[2]). configcs|t|j|_|j|_|j|_|j|jd|j|jd|j|jd|_|j|_t j |j |j|j|jd|_ dS)Nrr)Z kernel_sizeZstride) super__init__ num_frames image_size tubelet_size patch_size num_patches hidden_sizeZ embed_dimrConv3d num_channels projectionselfr __class__d/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/vivit/modeling_vivit.pyr0s zVivitTubeletEmbeddings.__init__F pixel_valuesinterpolate_pos_encodingreturnc Cs|j\}}}}}|sV||jks(||jkrVtd|d|d|jdd|jdd |ddddd }||}|ddd}|S) NzImage image size (*z) doesn't match model (rrz).rr)shaper ValueErrorpermuter&flatten transpose) r(r.r/ batch_sizerr%heightwidthxr+r+r,forward@s( zVivitTubeletEmbeddings.forward)F) __name__ __module__ __qualname____doc__rrtorchTensorboolr< __classcell__r+r+r)r,r%s rcsVeZdZdZedfdd ZejeeejdddZ d eje ejd d d Z Z S) VivitEmbeddingsz Vivit Embeddings. Creates embeddings from a video using VivitTubeletEmbeddings, adds CLS token and positional embeddings. rcsttttdd|j|_t||_ ttd|j j d|j|_ t |j |_|jdd|_||_dS)Nr)rrr ParameterrAZzerosr# cls_tokenrpatch_embeddingsr"position_embeddingsDropouthidden_dropout_probdropoutr r!rr'r)r+r,rXs  zVivitEmbeddings.__init__) embeddingsr9r:r0c Cs|jdd}|jjdd}tjs>||kr>||kr>|jS|jddddf}|jddddf}|jd}||jd} ||jd} t|d} |d| | |}|dddd}t j j || | fdd d }|dddd dd|}tj ||fdd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNrg?rrZbicubicF)sizemodeZ align_cornersdim)r3rIrAZjit is_tracingr!rreshaper5r functionalZ interpolateviewcat) r(rMr9r:r"Z num_positionsZclass_pos_embedZpatch_pos_embedrRZ new_heightZ new_widthZsqrt_num_positionsr+r+r,r/fs(   z(VivitEmbeddings.interpolate_pos_encodingFr-c Csr|j\}}}}}|j||d}|j|ddg} tj| |fdd}|rZ|||||}n ||j}||}|S)Nr/rrQ) r3rHrGZtilerArWr/rIrL) r(r.r/r8rr%r9r:rMZ cls_tokensr+r+r,r<s  zVivitEmbeddings.forward)F) r=r>r?r@rrrArBintr/rCr<rDr+r+r)r,rEQs(rE)modulequerykeyvalueattention_maskscalingrLc Ks|t||dd|}tjj|dtjd|j}tjj |||j d}|durX||}t||} | dd } | |fS)NrN)rRdtype)ptrainingrr) rAmatmulr7rrUZsoftmaxZfloat32torbrLrd contiguous) r[r\r]r^r_r`rLkwargsZ attn_weightsZ attn_outputr+r+r,eager_attention_forwards  ricsJeZdZedfdd ZdejeejeejejfdddZ Z S) VivitSelfAttentionrcst|j|jdkr>t|ds>td|jd|jd||_|j|_t|j|j|_|j|j|_ |j |_ |jd|_ d|_ tj|j|j |jd|_tj|j|j |jd|_tj|j|j |jd|_dS) NrZembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .gF)bias)rrr#num_attention_headshasattrr4rrYattention_head_size all_head_sizeZattention_probs_dropout_prob dropout_probr` is_causalrLinearZqkv_biasr\r]r^r'r)r+r,rs"   zVivitSelfAttention.__init__N hidden_states head_maskr0c Cs|jd}|d|j|jf}||j|dd}||j|dd}||j|dd}t}|j j dkr~t |j j }|||||||j |j |jsdn|jd\} } | dd|jf} | | } | | fS) NrrNrreagerrZ)rrr`rLra)r3rmror]rVr7r^r\rirZ_attn_implementationr rrr`rdrqrOrprT) r(rurvr8Z new_shapeZ key_layerZ value_layerZ query_layerZattention_interfaceZ context_layerZattention_probsZnew_context_layer_shaper+r+r,r<s*     zVivitSelfAttention.forward)N) r=r>r?rrrArBrtupler<rDr+r+r)r,rjs  rjcs>eZdZdZedfdd ZejejejdddZZ S)VivitSelfOutputz The residual connection is defined in VivitLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rcs.tt|j|j|_t|j|_dSN) rrrrsr#denserJrKrLr'r)r+r,rs zVivitSelfOutput.__init__ru input_tensorr0cCs||}||}|Srzr{rLr(rur}r+r+r,r<s  zVivitSelfOutput.forward) r=r>r?r@rrrArBr<rDr+r+r)r,rysrycsReZdZedfdd ZeedddZd ej e ej ej dd d Z Z S) VivitAttentionrcs*tt||_t||_t|_dSrz)rrrj attentionryoutputset pruned_headsr'r)r+r,rs   zVivitAttention.__init__)headscCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)NrrrQ)lenrrrmrorrr\r]r^rr{rpunion)r(rindexr+r+r, prune_headss zVivitAttention.prune_headsNrtcCs |||\}}|||}|Srz)rr)r(rurvZself_attn_output_rr+r+r,r< s zVivitAttention.forward)N) r=r>r?rrrrYrrArBrr<rDr+r+r)r,rsrcs6eZdZedfdd ZejejdddZZS)VivitIntermediatercsPtt|j|j|_t|j|_ t |j t rDt |j |_n|j |_dSrz)rrrrsr#intermediate_sizer{rJrKrL isinstanceZ hidden_actstrrintermediate_act_fnr'r)r+r,r's   zVivitIntermediate.__init__rur0cCs"||}||}||}|Srz)r{rrL)r(rur+r+r,r<0s   zVivitIntermediate.forward r=r>r?rrrArBr<rDr+r+r)r,r&s rcs:eZdZedfdd ZejejejdddZZS) VivitOutputrcs.tt|j|j|_t|j|_ dSrz) rrrrsrr#r{rJrKrLr'r)r+r,r9s zVivitOutput.__init__r|cCs ||}||}||}|Srzr~rr+r+r,r<>s  zVivitOutput.forwardrr+r+r)r,r8srcsDeZdZdZedfdd Zd ejeejejdddZ Z S) VivitLayerzNThis corresponds to the EncoderBlock class in the scenic/vivit implementation.rcsbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)Nreps)rrZchunk_size_feed_forwardZ seq_len_dimrrr intermediaterrr LayerNormr#layer_norm_epslayernorm_beforelayernorm_afterr'r)r+r,rHs    zVivitLayer.__init__NrtcCsB||}|||}||}||}||}|||}|Srz)rrrrr)r(rurvZhidden_states_normZattention_outputZ layer_outputr+r+r,r<Rs     zVivitLayer.forward)N) r=r>r?r@rrrArBrr<rDr+r+r)r,rEs rcs>eZdZedfdd ZdejeejedddZ Z S) VivitEncoderrcs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr+)r).0rrr+r, gz)VivitEncoder.__init__..F) rrrrZ ModuleListrangenum_hidden_layerslayerZgradient_checkpointingr'r)rr,rds  zVivitEncoder.__init__NrtcCs<t|jD]&\}}|dur"||nd}|||}q t|dS)N)last_hidden_state) enumeraterr)r(rurviZ layer_moduleZlayer_head_maskr+r+r,r<js zVivitEncoder.forward)N) r=r>r?rrrArBrrr<rDr+r+r)r,rcsrcs6eZdZedfdd ZejejdddZZS) VivitPoolerrcs*tt|j|j|_t|_dSrz)rrrrsr#r{ZTanh activationr'r)r+r,rss zVivitPooler.__init__rcCs(|dddf}||}||}|S)Nr)r{r)r(ruZfirst_token_tensor pooled_outputr+r+r,r<xs  zVivitPooler.forwardrr+r+r)r,rrsrc@sHeZdZUeed<dZdZdZgZdZ dZ dZ dZ e edZddZdS) VivitPreTrainedModelrvivitr.T)ru attentionscCst|tjtjfr@|jjjd|jjd|j dur|j j nt|tj r|jjjd|jjd|j dur|jj|j  nJt|tj r|j j |jjdn"t|tr|jj |jj dS)zInitialize the weightsrZ)meanZstdNg?)rrrsr$weightdataZnormal_rZinitializer_rangerlZzero_Z EmbeddingZ padding_idxrZfill_rErGrI)r(r[r+r+r, _init_weightss       z"VivitPreTrainedModel._init_weightsN)r=r>r?r__annotations__Zbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingZ_no_split_modulesZ_supports_sdpaZ_supports_flash_attnZ_supports_flex_attnZ_supports_attention_backendrrjZ_can_record_outputsrr+r+r+r,rs rc sfeZdZdeedfdd ZddZddZee de e j e e j ee eed d d ZZS) VivitModelT)radd_pooling_layercsXt|||_t||_t||_tj|j |j d|_ |rFt |nd|_ |dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)rrrrErMrencoderrrr#r layernormrpooler post_init)r(rrr)r+r,rs   zVivitModel.__init__cCs|jjSrz)rMrH)r(r+r+r,get_input_embeddingsszVivitModel.get_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. Args: heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsrrrr)r(Zheads_to_prunerrr+r+r, _prune_headsszVivitModel._prune_headsNF)r.rvr/rhr0c Ksp|durtd|||jj}|j||d}|j||d}|j}||}|jdur`||nd}t ||dS)a Examples: ```python >>> import av >>> import numpy as np >>> from transformers import VivitImageProcessor, VivitModel >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitModel.from_pretrained("google/vivit-b-16x2-kinetics400") >>> # prepare video for the model >>> inputs = image_processor(list(video), return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 3137, 768] ```Nz You have to specify pixel_valuesrX)rv)rZ pooler_output) r4Z get_head_maskrrrMrrrrr ) r(r.rvr/rhZembedding_outputZencoder_outputssequence_outputrr+r+r,r<sT zVivitModel.forward)T)NNF)r=r>r?rrCrrrrrrrA FloatTensorr rr r<rDr+r+r)r,rs ra ViViT Transformer model with a video classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for Kinetics-400. Note that it's possible to fine-tune ViT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. )Z custom_introc sZeZdZedfdd Zeed eej eej eej e e e edddZZS) VivitForVideoClassificationrcsRt||j|_t|dd|_|jdkr 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> import av >>> import numpy as np >>> import torch >>> from transformers import VivitImageProcessor, VivitForVideoClassification >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=4, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitForVideoClassification.from_pretrained("google/vivit-b-16x2-kinetics400") >>> inputs = image_processor(list(video), return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... logits = outputs.logits >>> # model predicts one of the 400 Kinetics-400 classes >>> predicted_label = logits.argmax(-1).item() >>> print(model.config.id2label[predicted_label]) LABEL_116 ```)rvr/Nr)losslogitsrur)rrrZ loss_functionrr rur) r(r.rvrr/rhoutputsrrrr+r+r,r<Cs$]z#VivitForVideoClassification.forward)NNNF)r=r>r?rrrrrrArZ LongTensorrCr rr r<rDr+r+r)r,r(s r)rrr)rZ)6r@typingrrrAZtorch.utils.checkpointrZ activationsrZmodeling_layersrZmodeling_outputsrr r Zmodeling_utilsr r Zprocessing_utilsr Z pytorch_utilsrrutilsrrrrZ utils.genericrrZconfiguration_vivitrZ get_loggerr=loggerModulerrErBfloatrirjryrrrrrrrrr__all__r+r+r+r,sX      ,W 5 #|