a hj@sFddlmZmZddlZddlZddlmZmZddlm Z m Z m Z m Z m Z mZmZddlmZddlmZddlmZdd lmZdd lmZd d lmZmZeeZGd ddeZ GdddeZ!GdddeZ"Gddde Z#GdddeZ$Gddde Z%Gddde Z&Gddde Z'Gddde Z(gdZ)dS) )OptionalUnionN)InstructBlipQFormerConfigInstructBlipVisionConfig)$InstructBlipForConditionalGeneration/InstructBlipForConditionalGenerationModelOutputInstructBlipModelInstructBlipPreTrainedModelInstructBlipQFormerModelInstructBlipVisionModelTransformersKwargs)PretrainedConfig)FlashAttentionKwargs)!MODEL_FOR_CAUSAL_LM_MAPPING_NAMES)Unpack)logging)CONFIG_MAPPING AutoConfigc@s eZdZdS)InstructBlipVideoVisionConfigN__name__ __module__ __qualname__rr{/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/instructblipvideo/modular_instructblipvideo.pyr.src@s eZdZdS)InstructBlipVideoQFormerConfigNrrrrrr2srcsPeZdZdZdZddiZeeedZ d fdd Z e eee d d d Z ZS)InstructBlipVideoConfiga [`InstructBlipVideoConfig`] is the configuration class to store the configuration of a [`InstructBlipVideoForConditionalGeneration`]. It is used to instantiate a Instructblipvideo model according to the specified arguments, defining the vision model, Q-Former model and language model configs. Instantiating a configuration with the defaults will yield a similar configuration to that of the Instructblipvideo [Salesforce/instruct-blip-flan-t5](https://huggingface.co/Salesforce/instruct-blip-flan-t5) architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vision_config (`dict`, *optional*): Dictionary of configuration options used to initialize [`InstructBlipVideoVisionConfig`]. qformer_config (`dict`, *optional*): Dictionary of configuration options used to initialize [`InstructBlipVideoQFormerConfig`]. text_config (`dict`, *optional*): Dictionary of configuration options used to initialize any [`PretrainedConfig`]. num_query_tokens (`int`, *optional*, defaults to 32): The number of query tokens passed through the Transformer. video_token_index (`int`, *optional*): Token index of special video token. kwargs (*optional*): Dictionary of keyword arguments. Example: ```python >>> from transformers import ( ... InstructBlipVideoVisionConfig, ... InstructBlipVideoQFormerConfig, ... OPTConfig, ... InstructBlipVideoConfig, ... InstructBlipVideoForConditionalGeneration, ... ) >>> # Initializing a InstructBlipVideoConfig with Salesforce/instruct-blip-flan-t5 style configuration >>> configuration = InstructBlipVideoConfig() >>> # Initializing a InstructBlipVideoForConditionalGeneration (with random weights) from the Salesforce/instruct-blip-flan-t5 style configuration >>> model = InstructBlipVideoForConditionalGeneration(configuration) >>> # Accessing the model configuration >>> configuration = model.config >>> # We can also initialize a InstructBlipVideoConfig from a InstructBlipVideoVisionConfig, InstructBlipVideoQFormerConfig and any PretrainedConfig >>> # Initializing Instructblipvideo vision, Instructblipvideo Q-Former and language model configurations >>> vision_config = InstructBlipVideoVisionConfig() >>> qformer_config = InstructBlipVideoQFormerConfig() >>> text_config = OPTConfig() >>> config = InstructBlipVideoConfig.from_text_vision_configs(vision_config, qformer_config, text_config) ```Zinstructblipvideovideo_token_idvideo_token_index) text_configqformer_config vision_configN c stjfi||dur(i}td|dur>i}td|durTi}tdtfi||_tfi||_|dd}t |fi||_ ||_ ||_ |jj |j_|j jtv|_d|_d|_dS)NzZvision_config is None. initializing the InstructBlipVideoVisionConfig with default values.z\qformer_config is None. Initializing the InstructBlipVideoQFormerConfig with default values.zTtext_config is None. Initializing the text config with default values (`OPTConfig`). model_typeoptg?g{Gz?)super__init__loggerinforr#rr"getrr!num_query_tokensr Z hidden_sizeZencoder_hidden_sizer%ruse_decoder_only_language_modelZinitializer_factorZinitializer_range)selfr#r"r!r,r kwargsZtext_model_type __class__rrr(xs(      z InstructBlipVideoConfig.__init__r#r"r!cKs"|f|||d|S)a  Instantiate a [`InstructBlipVideoConfig`] (or a derived class) from a InstructBlipVideo vision model, Q-Former and language model configurations. Returns: [`InstructBlipVideoConfig`]: An instance of a configuration object r2)to_dict)clsr#r"r!r/rrr from_vision_qformer_text_configssz8InstructBlipVideoConfig.from_vision_qformer_text_configs)NNNr$N)rrr__doc__r%Z attribute_maprrrZ sub_configsr( classmethodrr5 __classcell__rrr0rr6s&7#rc@s eZdZdS) InstructBlipVideoPreTrainedModelNrrrrrr9sr9c@s eZdZdS)InstructBlipVideoVisionModelNrrrrrr:sr:c@s eZdZdS)InstructBlipVideoQFormerModelNrrrrrr;sr;c@s eZdZdS)4InstructBlipVideoForConditionalGenerationModelOutputNrrrrrr<sr<c@s|eZdZdejejeejeejeejeejeejeejeeeeeeeeee e e e e fdddZdS)InstructBlipVideoModelNF) pixel_valuesqformer_input_idsqformer_attention_mask input_idsattention_maskdecoder_input_idsdecoder_attention_mask inputs_embedsoutput_attentionsoutput_hidden_states return_dictinterpolate_pos_encoding use_cacher/returnc Ks>| dur | n|jj} |j\}}}}}||||||}|j|| | | | d}|d}tj|ddtj|j d}|j |jddd}tj|ddtj|j d}|durt |}|j |dd}|j |dd}tj||gdd}|j|||||| | | d}|dddd|dddf}||}|||jj|d}|dur|j|}||jjk}|durt |}n.||tj|jjtj|j dk}|d}|d||j }||j |j}|||}|jjr|jf||| | | | d|}n"|jf||||| | | | d |}t|||d S) N)r>rFrGrHrIrdtypedevicedim)rArB query_embedsencoder_hidden_statesencoder_attention_maskrFrGrHrErBrFrGrHrJ)rErBrCrDrFrGrHrJ)vision_outputsqformer_outputslanguage_model_outputs)configuse_return_dictshapereshape vision_modeltorchonessizelongrO query_tokensexpand ones_likerepeat_interleavecatqformerlanguage_projectionr,language_modelget_input_embeddingsrtensorall unsqueeze expand_astorNmasked_scatterr-r<)r.r>r?r@rArBrCrDrErFrGrHrIrJr/ batch_sizeframeschannelheightwidthrW image_embedsimage_attention_maskrcquery_attention_mask query_outputs query_outputlanguage_model_inputsspecial_image_maskoutputsrrrforwards    $           zInstructBlipVideoModel.forward) NNNNNNNNNFN)rrrr_ FloatTensorr LongTensorZTensorboolrrrtupler<rrrrrr=s8 r=c@s4eZdZdejejeejeeeedddZdejejeejeeeedddZ ejejdd d Z dejejeejeejeejeejeejeejeeeeeejeeeeee e e eefd d d ZedejeejeejeejeejeejeejdddZdS))InstructBlipVideoForConditionalGenerationNF)r>r?r@rIrHcCs@|j\}}}} } ||||| | }|j||dd} | d} tj| ddtj| jd} |j | jddd}tj|ddtj| jd}|durt |}|j |dd}|j |dd}tj ||gdd}|j |||| | dd }|dddd|dddf}||}|||jj|d}|r<|| |fS|S) a$ Encodes images into continuous embeddings that can be forwarded to the language model. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. T)r>rIrHrNrLrMrPrR)rArBrSrTrUrH)r\r]r^r_r`rarbrOrcrdrerfrgrhrirZr,)r.r>r?r@rIrHrrrsrtrurvrWrwrxrcryrzr{r|rrrget_video_features1s<   $  zr?r@rIrHrrrget_image_featuresmszInstructBlipVideoForConditionalGeneration.get_placeholder_mask)r>r?r@rArBrCrDrErFrGlabelsrHrIrJr/rKc Ksd| dur | n|jj} |j|||| dd\}}}| s:|n|}| sJ|n|}|durb||}|durtt|}||j|j }|j ||d}| ||}|jj r|j f||| | | |d|}| r|jn|d}d}| durR|jf|| |jjjd|}nL|j f||||| | | | |d |}| r6|jn|d}| rJ|jn|d }t|||||d S) a qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length)): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. Examples: ```python >>> from transformers import InstructBlipVideoProcessor, InstructBlipVideoForConditionalGeneration >>> import torch >>> from huggingface_hub import hf_hub_download >>> import av >>> import numpy as np >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> model = InstructBlipVideoForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b", device_map="auto") >>> processor = InstructBlipVideoProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample uniformly 4 frames from the videWhy is this video funny?o >>> total_frames = container.streams.video[0].frames >>> indices = np.arange(0, total_frames, total_frames / 4).astype(int) >>> clip = read_video_pyav(container, indices) >>> prompt = "What is happening in the video?" >>> inputs = processor(text=prompt, images=clip, return_tensors="pt").to(model.device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... repetition_penalty=1.5, ... length_penalty=1.0, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) "A person is eating a bowl of pasta, and they are using a fork to eat it. The person is sitting at a table, and the plate of pasta is on the table in front" ```NTr?r@rIrHrErVr)logitsr vocab_size) rErBrCrDrFrGrHrrJrR)lossrrWrXrY)rZr[rZto_tuplerkr_rerprOrNrrqr-rjrZ loss_functionr!rrr<)r.r>r?r@rArBrCrDrErFrGrrHrIrJr/r|rWrzr}r~rrrrrrstQ          z1InstructBlipVideoForConditionalGeneration.forward)r>r?r@rArBrErIrKcKs t|dr||jd} |j||||dd\} } } |dur|dur|jjg|jjd} | |jjjg}t j |gt j |j d}| | d}||}|durt |}| |j |j} |j||d }||| }||d }|jjjs||d <|jjfi||}|S) a Overrides `generate` function to be able to use the model as a conditional generator. Args: pixel_values (`torch.FloatTensor` of shape (batch_size, num_channels, height, width) or (batch_size, num_frames, num_channels, height, width)): Input images or videos to be processed. qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt for the generation. attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Embedded representation of the inputs. Should be float, not int tokens. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the positional encoding of the image embeddings. Returns: captions (list): A list of strings of length batch_size * num_captions. Z hf_device_maprTrNrMrRr)rErBrA)hasattrZ_preprocess_accelerater\rrZr r,r!Z bos_token_idr_rlrbrOrepeatrkrerprNrrqrjZis_encoder_decodergenerate)r.r>r?r@rArBrErIZgenerate_kwargsrrr|rWrzZ video_tokensZ start_tokensr}inputsr~rrrrs6"         z2InstructBlipVideoForConditionalGeneration.generate)NFF)NFF) NNNNNNNNNNFN)NNNNNF)rrrr_rrrrrrrrr rrr<rZno_gradrrrrrr0s @  r)rrrr:r9r;r=r)*typingrrr_Ztorch.utils.checkpointZ;transformers.models.instructblip.configuration_instructbliprrZ6transformers.models.instructblip.modeling_instructbliprrrr r r r Zconfiguration_utilsrZmodeling_flash_attention_utilsrZmodels.auto.modeling_autorZprocessing_utilsrutilsrautorrZ get_loggerrr)rrrr9r:r;r<r=r__all__rrrrs.$      }m/