a hz"@sdZddlZddlmZddlmZddlmZddlm Z m Z m Z ddl m Z mZmZdd lmZd d lmZeeZGd d d e ddZGddde ZdgZdS)zq Processor class for InstructBLIP. Largely copy of Blip2Processor with addition of a tokenizer for the Q-Former. N)Union) BatchFeature) ImageInput)ProcessingKwargsProcessorMixinUnpack) AddedTokenPreTokenizedInput TextInput)logging) AutoTokenizerc @s*eZdZdddddddddd idZdS)InstructBlipProcessorKwargsTFr) add_special_tokenspaddingZstrideZreturn_overflowing_tokensZreturn_special_tokens_maskZreturn_offsets_mappingZreturn_token_type_idsZ return_lengthverbose) text_kwargs images_kwargsN)__name__ __module__ __qualname__ _defaultsrrt/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/instructblip/processing_instructblip.pyr!s rF)totalcseZdZdZgdZdZdZdZdfdd Zde e e e e e e e feeedd d Zed d Zfd dZefddZZS)InstructBlipProcessora Constructs an InstructBLIP processor which wraps a BLIP image processor and a LLaMa/T5 tokenizer into a single processor. [`InstructBlipProcessor`] offers all the functionalities of [`BlipImageProcessor`] and [`AutoTokenizer`]. See the docstring of [`~BlipProcessor.__call__`] and [`~BlipProcessor.decode`] for more information. Args: image_processor (`BlipImageProcessor`): An instance of [`BlipImageProcessor`]. The image processor is a required input. tokenizer (`AutoTokenizer`): An instance of ['PreTrainedTokenizer`]. The tokenizer is a required input. qformer_tokenizer (`AutoTokenizer`): An instance of ['PreTrainedTokenizer`]. The Q-Former tokenizer is a required input. num_query_tokens (`int`, *optional*):" Number of tokens used by the Qformer as queries, should be same as in model's config. )image_processor tokenizerqformer_tokenizer)ZBlipImageProcessorZBlipImageProcessorFastrNc sPt|ds.tdddd|_|j|jgddn|j|_||_t|||dS)N image_tokenzFT) normalizedZspecial)Zspecial_tokens)hasattrr r Z add_tokensnum_query_tokenssuper__init__)selfrrrr#kwargs __class__rrr%Js  zInstructBlipProcessor.__init__)imagestextr'returnc  s|dur|durtd|jtfd|jji|}|ddd}i}|durvt|trb|g}n t|tst|dtstd|j |fi|d} | d|d <| d |d <|d d dur|dd |j 8<|j|fi|d} |durl|j j |j } d |dd<d |dd<d |dd<|j| fi|d| D]"fdd| D| <qH|| |dur|j|fi|d} || t||d}|S)a This method uses [`BlipImageProcessor.__call__`] method to prepare image(s) for the model, and [`BertTokenizerFast.__call__`] to prepare text for the model. Please refer to the docstring of the above two methods for more information. Args: images (`ImageInput`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. Both channels-first and channels-last formats are supported. text (`TextInput`, `PreTokenizedInput`, `list[TextInput]`, `list[PreTokenizedInput]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). Nz,You have to specify at least images or text.Ztokenizer_init_kwargsrreturn_tensorsrzAInvalid input text. Please provide a string, or a list of stringsZ input_idsqformer_input_idsZattention_maskqformer_attention_mask max_lengthFrrZ truncationcsg|]}|qSrr).0sampleZimage_text_encodingkrr z2InstructBlipProcessor.__call__..r)Z tensor_type) ValueErrorZ _merge_kwargsrrZ init_kwargspop isinstancestrlistrgetr#r contentupdaterr) r&r*r+ZaudioZvideosr'Z output_kwargsr-encodingZqformer_text_encodingZ text_encodingZ image_tokensZimage_encodingrr3r__call__TsH           zInstructBlipProcessor.__call__cCs$|jj}|jj}ddg}|||S)Nr.r/)rmodel_input_namesr)r&Ztokenizer_input_namesZimage_processor_input_namesZqformer_input_namesrrrrAsz'InstructBlipProcessor.model_input_namesc stj|rtd|dtj|ddtj|d}|j|d|jv}|r^|j dt j|fi|}|r|jdg7_|S)NzProvided path (z#) should be a directory, not a fileT)exist_okr) ospathisfiler7makedirsjoinrsave_pretrained attributesremover$)r&Zsave_directoryr'Zqformer_tokenizer_pathZqformer_presentoutputsr(rrrHs    z%InstructBlipProcessor.save_pretrainedc s>tj|fi|}t|tr&|d}tj|dd}||_|S)Nrr)Z subfolder)r$from_pretrainedr9tuplerr)clsZpretrained_model_name_or_pathr' processorrr(rrrLs  z%InstructBlipProcessor.from_pretrained)N)NNNN)rrr__doc__rIZimage_processor_classZtokenizer_classZqformer_tokenizer_classr%rrr r r;rrrr@propertyrArH classmethodrL __classcell__rrr(rr2s*  C  r)rPrCtypingrZimage_processing_utilsrZ image_utilsrZprocessing_utilsrrrZtokenization_utils_baser r r utilsr autorZ get_loggerrloggerrr__all__rrrrs