a h 0@sddlmZmZddlZddlmZddlmZmZmZm Z m Z ddl m Z ddl mZddlmZmZd d lmZeeZGd d d e ZGd ddeZGdddejZGddde ZGdddeZGdddeZgdZdS))OptionalUnionN)nn)LlavaCausalLMOutputWithPastLlavaForConditionalGeneration LlavaModelLlavaModelOutputWithPastLlavaPreTrainedModel)ACT2FN)Cache)auto_docstringlogging)VipLlavaConfigc@s eZdZdS)VipLlavaModelOutputWithPastN__name__ __module__ __qualname__rri/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/vipllava/modular_vipllava.pyr&src@s eZdZdS)VipLlavaCausalLMOutputWithPastNrrrrrr*srcs*eZdZedfdd ZddZZS)VipLlavaMultiModalProjector)configcstt|jtrdnt|j}tj||jj |j d|_ tj ||jj |j j dd|_t|j|_tj |j j |j j dd|_dS)Nr)epsT)Zbias)super__init__ isinstancevision_feature_layersintlenrZ LayerNormZ vision_configZ hidden_sizeZprojector_layernorm_epsprojector_layernormZLinear text_configlinear_1r Zprojector_hidden_actactlinear_2)selfrZnum_feature_layers __class__rrr/s   z$VipLlavaMultiModalProjector.__init__cCs,||}||}||}||}|S)N)r"r$r%r&)r' hidden_statesrrrforward>s     z#VipLlavaMultiModalProjector.forward)rrrrrr+ __classcell__rrr(rr.src@s eZdZdS)VipLlavaPreTrainedModelNrrrrrr-Fsr-c@seZdZdejeeeeefdddZ e d ej ejeej eej ee eejeeeeefeeeeeeeeeej eeefd ddZdS) VipLlavaModelN pixel_valuesrcsv|dur |n|jj}|j|ddt|trHj|ddddf}n fdd|D}tj|dd}||}|S) aW Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_layers (`Union[int, list[int]]`): The vision feature layer, or the list of indexes of the layers to select the vision feature. Returns: image_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`). NT)output_hidden_statesrcs&g|]}j|ddddfqS)Nr)r*).0indexZ image_outputsrr ez4VipLlavaModel.get_image_features..)dim) rrZ vision_towerrr r*torchcatZmulti_modal_projector)r'r0rimage_featuresrr4rget_image_featuresKs  z VipLlavaModel.get_image_features) input_idsr0attention_mask position_idspast_key_values inputs_embedsr use_cacheoutput_attentionsr1 return_dictcache_positionreturnc  Ks| dur | n|jj} | dur | n|jj} | dur4| n|jj} |durH|n|jj}|du|duArhtd|dur|||}|dur|j||d}||j |j }|j |||d}| ||}|j f|||||| | d| d | }t|j|j|j|j|dur|ndd}| r|S|S)z vision_feature_layers (`Union[int, list[int]]`, *optional*): The vision feature layer, or the list of indexes of the layers to select the vision feature. Nz:You must specify exactly one of input_ids or inputs_embedsr/)rAr;T) r>r?r@rArBrCr1rDrE)last_hidden_stater@r* attentionsimage_hidden_states)rrCr1use_return_dictr ValueErrorZget_input_embeddingsr<toZdeviceZdtypeZget_placeholder_maskZmasked_scatterZlanguage_modelrrGr@r*rHZto_tuple)r'r=r0r>r?r@rArrBrCr1rDrE lm_kwargsr;Zspecial_image_maskoutputsoutputrrrr+jsP    zVipLlavaModel.forward)N) NNNNNNNNNNNN)rrrr9 FloatTensorrrr listr<r LongTensorTensorr booltuplerr+rrrrr.Js@  r.c@seZdZd ejeeeeefdddZ d ej ejeej eej ee eejeeeeefeej ee ee ee ee eej eeej feeefdddZdS) VipLlavaForConditionalGenerationNr/cCs|jj||dS)Nr/)modelr<)r'r0rrrrr<sz3VipLlavaForConditionalGeneration.get_image_featuresr)r=r0r>r?r@rArlabelsrBrCr1rDrElogits_to_keeprFcKs| dur | n|jj} | dur | n|jj} | dur4| n|jj} |durH|n|jj}|jf||||||| || | d| d |}|d}t|trt| dn|}| |dd|ddf}d}|dur|j |||jj j d}t |||j|j|j|jdS)a vision_feature_layers (`Union[int, list[int]]`, *optional*): The vision feature layer, or the list of indexes of the layers to select the vision feature. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> import torch >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, VipLlavaForConditionalGeneration >>> model = VipLlavaForConditionalGeneration.from_pretrained("llava-hf/vip-llava-7b-hf", device_map="auto", dtype=torch.float16) >>> processor = AutoProcessor.from_pretrained("llava-hf/vip-llava-7b-hf") >>> prompt = "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: \n{}###Assistant:" >>> question = "Can you please describe this image?" >>> prompt = prompt.format(question) >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/compel-neg.png" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(text=text, images=image, return_tensors="pt").to(0, torch.float16) >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=20) >>> processor.decode(generate_ids[0][len(inputs["input_ids"][0]):], skip_special_tokens=True) The image features a brown and white cat sitting on a green surface, with a red ball in its ```NT) r=r0r>r?r@rArBrrCr1rDrEr)logitsrX vocab_size)lossrZr@r*rHrI)rrCr1rJrrWrr sliceZlm_headZ loss_functionr#r[rr@r*rHrI)r'r=r0r>r?r@rArrXrBrCr1rDrErYrMrNr*Z slice_indicesrZr\rrrr+sH4 z(VipLlavaForConditionalGeneration.forward)N)NNNNNNNNNNNNNr)rrrr9rPrrr rQr<rRrSr rTrUrr+rrrrrVsF   rV)r.rVr-)typingrrr9rZ(transformers.models.llava.modeling_llavarrrrr Z activationsr Z cache_utilsr utilsr rZconfiguration_vipllavarZ get_loggerrloggerrrModulerr-r.rV__all__rrrrs     ff