a h,@sbdZddlZddlmZmZddlmZddlmZddl m Z gd Z Gd d d eZ d gZ dS) z Processor class for EVOLLA. N)OptionalUnion) BatchFeature)ProcessorMixin) AutoTokenizer)aa_seqfoldseekZmsacseZdZdZddgZdgZdZdZdZdfd d Z d d d Z d!e d ddZ d"e eeeefe eeeeeefe e e e dddZddZddZddZddZfddZefddZZS)#EvollaProcessoran Constructs a EVOLLA processor which wraps a LLama tokenizer and SaProt tokenizer (EsmTokenizer) into a single processor. [`EvollaProcessor`] offers all the functionalities of [`EsmTokenizer`] and [`LlamaTokenizerFast`]. See the docstring of [`~EvollaProcessor.__call__`] and [`~EvollaProcessor.decode`] for more information. Args: protein_tokenizer (`EsmTokenizer`): An instance of [`EsmTokenizer`]. The protein tokenizer is a required input. tokenizer (`LlamaTokenizerFast`, *optional*): An instance of [`LlamaTokenizerFast`]. The tokenizer is a required input. protein_max_length (`int`, *optional*, defaults to 1024): The maximum length of the sequence to be generated. text_max_length (`int`, *optional*, defaults to 512): The maximum length of the text to be generated. protein_tokenizer tokenizerZsequence_max_lengthrNc sF|durtd|dur tdt||d|j_||_||_dS)Nz+You need to specify an `protein_tokenizer`.z"You need to specify a `tokenizer`.z<|reserved_special_token_0|>) ValueErrorsuper__init__r Z pad_tokenprotein_max_lengthtext_max_length)selfr r rrkwargs __class__h/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/evolla/processing_evolla.pyr;szEvollaProcessor.__init__c Cs`g}|D]<}|d}|d}dddt||D}||q|jj|dd|dd}|S) Nr r cSs g|]\}}||qSr)upperlower).0sfrrr Lz4EvollaProcessor.process_proteins..ptT)return_tensors truncation max_lengthpadding)getjoinzipappendr Zbatch_encode_plus) rproteinsrZ sa_sequencesZproteinr r Z sa_sequence sa_tokensrrrprocess_proteinsGs    z EvollaProcessor.process_proteins)rcCsDg}|D] }|jj|ddd}||q|j|dddd|d}|S)NFT)tokenizeZadd_generation_promptr#Zlongest)Zadd_special_tokensr$r'r%r&)r Zapply_chat_templater+)rZtextsrZpromptsmessagespromptZ prompt_inputsrrr process_textTs" zEvollaProcessor.process_text)r, messages_listrrc Ks|dus|durtd|dur$|n|j}|dur6|n|j}t|trL|g}t|ttfrrt|dttfsr|g}t|ttfrtdd|Dstdt|ttfrtdd|Dstdd t d |t|ttfrd|D]x}t|ttfstd t |d td d|Ds,tdt dd|DsRt dd|Drtd|qntdt |d | ||}| ||}t|d|d|d|dddS)avThis method takes batched or non-batched proteins and messages_list and converts them into format that can be used by the model. Args: proteins (`Union[List[dict], dict]`): A list of dictionaries or a single dictionary containing the following keys: - `"aa_seq"` (`str`) -- The amino acid sequence of the protein. - `"foldseek"` (`str`) -- The foldseek string of the protein. messages_list (`Union[List[List[dict]], List[dict]]`): A list of lists of dictionaries or a list of dictionaries containing the following keys: - `"role"` (`str`) -- The role of the message. - `"content"` (`str`) -- The content of the message. protein_max_length (`int`, *optional*, defaults to 1024): The maximum length of the sequence to be generated. text_max_length (`int`, *optional*, defaults to 512): The maximum length of the text. Return: a dict with following keys: - `protein_input_ids` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The input IDs for the protein sequence. - `protein_attention_mask` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The attention mask for the protein sequence. - `text_input_ids` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The input IDs for the text sequence. - `text_attention_mask` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The attention mask for the text sequence. Nz3You need to specify `messages_list` and `proteins`.rcss|]}t|tVqdSN isinstancedictrprrr r"z+EvollaProcessor.__call__..zUThe proteins should be a list of dictionaries, but not all elements are dictionaries.css$|]}tdd|DVqdS)css|]}|tvVqdSr4)PROTEIN_VALID_KEYS)rkrrrr:r"z5EvollaProcessor.__call__...N)allkeysr8rrrr:sz2There should be a list of dictionaries with keys: z, z for each protein.But got: z;Each messages in messages_list should be a list instead of .css|]}t|tVqdSr4r5rmrrrr:r"zfEach message in messages_list should be a list of dictionaries, but not all elements are dictionaries.css|]}t|dkVqdS)rN)lenr>r@rrrr:r"css"|]}t|ddhkVqdS)ZrolecontentN)setr>r@rrrr:szlEach message in messages_list should be a list of dictionaries with two keys: 'role' and 'content'.But got: zFThe messages_list should be a list of lists of dictionaries, but it's input_idsattention_mask)Zprotein_input_idsZprotein_attention_maskrErF)data)rrrr6r7listtupler=r)r;typeanyr.r2r) rr,r3rrrr0r-Z text_tokensrrr__call__ls`!     zEvollaProcessor.__call__cOs|jj|i|Sr4)r batch_decoderargsrrrrrMszEvollaProcessor.batch_decodecOs|jj|i|Sr4)r decoderNrrrrPszEvollaProcessor.decodecOs|jj|i|Sr4)r rMrNrrrprotein_batch_decodesz$EvollaProcessor.protein_batch_decodecOs|jj|i|Sr4)r rPrNrrrprotein_decodeszEvollaProcessor.protein_decodec s|jtj||jd|jv}|r2|jdnd}|rN|durN|jdt j|fi|}|r||dur||j |d|S)Nr ) r save_pretrainedospathr)protein_tokenizer_dir_name attributesindexremoverinsert)rZsave_directoryrZprotein_tokenizer_presentZprotein_tokenizer_indexoutputsrrrrSs    zEvollaProcessor.save_pretrainedc s@tj|fi|}t|tr&|d}tj||jd}||_|S)Nr)Z subfolder)rfrom_pretrainedr6rIrrVr )clsZpretrained_model_name_or_pathr processorr rrrr\s zEvollaProcessor.from_pretrained)Nrr)r)r)NNNN)__name__ __module__ __qualname____doc__rWZ valid_kwargsZprotein_tokenizer_classZtokenizer_classrVrr.intr2rrrHr7rLrMrPrQrRrS classmethodr\ __classcell__rrrrr s:   Y r )rbrTtypingrrZfeature_extraction_utilsrZprocessing_utilsrautorr;r __all__rrrrs   X