a h*Y@sdZddlmZddlmZmZddlZddlmZmZddl m Z ddl m Z dd l mZmZmZd d lmZd d lmZeeZeeddGdddeZeeddGdddeZeeddGdddeZeGddde ZGdddeZGdddeZGdddeZGdddeZGd d!d!eZ ed"dGd#d$d$eZ!ed%dGd&d'd'eZ"ed(dGd)d*d*e Z#gd+Z$dS),z5PyTorch DPR model for Open Domain Question Answering.) dataclass)OptionalUnionN)Tensornn)BaseModelOutputWithPooling)PreTrainedModel) ModelOutputauto_docstringlogging) BertModel) DPRConfigz6 Class for outputs of [`DPRQuestionEncoder`]. )Z custom_introc@sPeZdZUdZejed<dZee ejdfed<dZ ee ejdfed<dS)DPRContextEncoderOutputa pooler_output (`torch.FloatTensor` of shape `(batch_size, embeddings_size)`): The DPR encoder outputs the *pooler_output* that corresponds to the context representation. Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer. This output is to be used to embed contexts for nearest neighbors queries with questions embeddings. pooler_outputN. hidden_states attentions __name__ __module__ __qualname____doc__torch FloatTensor__annotations__rrtuplerrr`/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/dpr/modeling_dpr.pyr*s  rc@sPeZdZUdZejed<dZee ejdfed<dZ ee ejdfed<dS)DPRQuestionEncoderOutputa pooler_output (`torch.FloatTensor` of shape `(batch_size, embeddings_size)`): The DPR encoder outputs the *pooler_output* that corresponds to the question representation. Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer. This output is to be used to embed questions for nearest neighbors queries with context embeddings. rN.rrrrrrrr =s  r c@steZdZUdZejed<dZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dS) DPRReaderOutputa start_logits (`torch.FloatTensor` of shape `(n_passages, sequence_length)`): Logits of the start index of the span for each passage. end_logits (`torch.FloatTensor` of shape `(n_passages, sequence_length)`): Logits of the end index of the span for each passage. relevance_logits (`torch.FloatTensor` of shape `(n_passages, )`): Outputs of the QA classifier of the DPRReader that corresponds to the scores of each passage to answer the question, compared to all the other passages. start_logitsN end_logitsrelevance_logits.rr) rrrrrrrr#rr$rrrrrrrr!Ps  r!c@seZdZdZddZdS)DPRPreTrainedModelTcCst|tjr:|jjjd|jjd|jdur|jj nft|tj rz|jjjd|jjd|j dur|jj|j  n&t|tj r|jj |jj ddS)zInitialize the weightsg)meanZstdNg?) isinstancerLinearweightdataZnormal_configZinitializer_rangeZbiasZzero_Z EmbeddingZ padding_idxZ LayerNormZfill_)selfmodulerrr _init_weightsls      z DPRPreTrainedModel._init_weightsN)rrrZ_supports_sdpar.rrrrr%hsr%c sreZdZdZedfdd Zdeeeeeeeeeee e e edffdd d Z e ed d d ZZS) DPREncoder bert_modelr+csdt|t|dd|_|jjjdkr0td|j|_|jdkrXt |jjj|j|_ | dS)NF)Zadd_pooling_layerrz!Encoder hidden_size can't be zero) super__init__rr0r+ hidden_size ValueErrorprojection_dimrr( encode_proj post_initr,r+ __class__rrr3s  zDPREncoder.__init__NF. input_idsattention_masktoken_type_ids inputs_embedsoutput_attentionsoutput_hidden_states return_dictreturnc Csv|j|||||||d}|d} | dddddf} |jdkrJ|| } |sb| | f|ddSt| | |j|jdS)Nr=r>r?r@rArBrCrr )Zlast_hidden_staterrr)r0r6r7rrr) r,r=r>r?r@rArBrCoutputssequence_outputZ pooled_outputrrrforwards*    zDPREncoder.forward)rDcCs|jdkr|jjS|jjjS)Nr)r6r7Z out_featuresr0r+r4)r,rrrembeddings_sizes zDPREncoder.embeddings_size)NNNFFF)rrrbase_model_prefixrr3rrboolrrrrHpropertyintrI __classcell__rrr:rr/}s( #r/c sVeZdZdZedfdd Zd eeeeeeee e e edffdd d Z Z S) DPRSpanPredictorencoderr1csFt|t||_t|jjd|_t|jjd|_| dS)Nr r) r2r3r/rPrr(rI qa_outputs qa_classifierr8r9r:rrr3s   zDPRSpanPredictor.__init__NF.r=r>r@rArBrCrDcCs|dur|n|dd\}}|j||||||d} | d} || } | jddd\} } | d} | d} || dddddf}| ||} | ||} ||}|s| | |f| ddSt| | || j | j dS)Nr )r>r@rArBrCrr)dim)r"r#r$rr) sizerPrQsplitZsqueeze contiguousrRviewr!rr)r,r=r>r@rArBrCZ n_passagesZsequence_lengthrFrGZlogitsr"r#r$rrrrHs6 $    zDPRSpanPredictor.forward)NFFF)rrrrJrr3rrrKrr!rrHrNrrr:rrOs rOc@s"eZdZUdZeed<dZdZdS)DPRPretrainedContextEncoder An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r+N ctx_encoderrrrrrrZload_tf_weightsrJrrrrrZs rZc@s"eZdZUdZeed<dZdZdS)DPRPretrainedQuestionEncoderr[r+Nquestion_encoderr]rrrrr^s r^c@s"eZdZUdZeed<dZdZdS)DPRPretrainedReaderr[r+Nspan_predictorr]rrrrr` s r`zf The bare DPRContextEncoder transformer outputting pooler outputs as context representations. cspeZdZedfdd Zed eeeeeeeeeeeeeee e e edffdddZ Z S) DPRContextEncoderr1cs(t|||_t||_|dSN)r2r3r+r/r\r8r9r:rrr3s  zDPRContextEncoder.__init__N.r<c Cs |dur |n|jj}|dur |n|jj}|dur4|n|jj}|durV|durVtdn4|durh|}n"|dur|dd}ntd|dur|jn|j} |dur|durtj|| dn ||jj k}|durtj |tj | d}|j |||||||d} |s | ddSt | j| j| jd S) aS input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. To match pretraining, DPR input sequence should be formatted with [CLS] and [SEP] tokens as follows: (a) For sequence pairs (for a pair title+text for example): ``` tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP] token_type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1 ``` (b) For single sequences (for a question for example): ``` tokens: [CLS] the dog is hairy . [SEP] token_type_ids: 0 0 0 0 0 0 0 ``` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Examples: ```python >>> from transformers import DPRContextEncoder, DPRContextEncoderTokenizer >>> tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base") >>> model = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base") >>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors="pt")["input_ids"] >>> embeddings = model(input_ids).pooler_output ```NDYou cannot specify both input_ids and inputs_embeds at the same timerT5You have to specify either input_ids or inputs_embedsdeviceZdtypergrErrrr)r+rArBuse_return_dictr5rVrgrones pad_token_idzeroslongr\rrrr r,r=r>r?r@rArBrC input_shapergrFrrrrH&sB1      zDPRContextEncoder.forward)NNNNNNN)rrrrr3r rrrKrrrrHrNrrr:rrbs&rbzh The bare DPRQuestionEncoder transformer outputting pooler outputs as question representations. cspeZdZedfdd Zed eeeeeeeeeeeeeee e e edffdddZ Z S) DPRQuestionEncoderr1cs(t|||_t||_|dSrc)r2r3r+r/r_r8r9r:rrr3s  zDPRQuestionEncoder.__init__N.r<c Cs,|dur |n|jj}|dur |n|jj}|dur4|n|jj}|durV|durVtdn@|durt||||}n"|dur|dd}ntd|dur|jn|j} |dur|durtj || dn ||jj k}|durtj |tj | d}|j |||||||d} |s| ddSt| j| j| jd S) aj input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. To match pretraining, DPR input sequence should be formatted with [CLS] and [SEP] tokens as follows: (a) For sequence pairs (for a pair title+text for example): ``` tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP] token_type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1 ``` (b) For single sequences (for a question for example): ``` tokens: [CLS] the dog is hairy . [SEP] token_type_ids: 0 0 0 0 0 0 0 ``` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Examples: ```python >>> from transformers import DPRQuestionEncoder, DPRQuestionEncoderTokenizer >>> tokenizer = DPRQuestionEncoderTokenizer.from_pretrained("facebook/dpr-question_encoder-single-nq-base") >>> model = DPRQuestionEncoder.from_pretrained("facebook/dpr-question_encoder-single-nq-base") >>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors="pt")["input_ids"] >>> embeddings = model(input_ids).pooler_output ``` NrdrTrerfrhrErri)r+rArBrjr5%warn_if_padding_and_no_attention_maskrVrgrrkrlrmrnr_r rrrrorrrrHsD1       zDPRQuestionEncoder.forward)NNNNNNN)rrrrr3r rrrKrr rrHrNrrr:rrqs&rqzE The bare DPRReader transformer outputting span predictions. c sjeZdZedfdd Zed eeeeeeeeeeeee e e edffdddZ Z S) DPRReaderr1cs(t|||_t||_|dSrc)r2r3r+rOrar8r9r:rrr3s  zDPRReader.__init__N.rSc Cs|dur |n|jj}|dur |n|jj}|dur4|n|jj}|durV|durVtdn@|durt||||}n"|dur|dd}ntd|dur|jn|j}|durtj ||d}|j ||||||dS)a input_ids (`tuple[torch.LongTensor]` of shapes `(n_passages, sequence_length)`): Indices of input sequence tokens in the vocabulary. It has to be a sequence triplet with 1) the question and 2) the passages titles and 3) the passages texts To match pretraining, DPR `input_ids` sequence should be formatted with [CLS] and [SEP] with the format: `[CLS] [SEP] [SEP] ` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`DPRReaderTokenizer`]. See this class documentation for more details. [What are input IDs?](../glossary#input-ids) inputs_embeds (`torch.FloatTensor` of shape `(n_passages, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Examples: ```python >>> from transformers import DPRReader, DPRReaderTokenizer >>> tokenizer = DPRReaderTokenizer.from_pretrained("facebook/dpr-reader-single-nq-base") >>> model = DPRReader.from_pretrained("facebook/dpr-reader-single-nq-base") >>> encoded_inputs = tokenizer( ... questions=["What is love ?"], ... titles=["Haddaway"], ... texts=["'What Is Love' is a song recorded by the artist Haddaway"], ... return_tensors="pt", ... ) >>> outputs = model(**encoded_inputs) >>> start_logits = outputs.start_logits >>> end_logits = outputs.end_logits >>> relevance_logits = outputs.relevance_logits ``` NrdrTrerf)r@rArBrC) r+rArBrjr5rrrVrgrrkra) r,r=r>r@rArBrCrprgrrrrHs.0   zDPRReader.forward)NNNNNN)rrrrr3r rrrKrr!rrHrNrrr:rrss"rs)rbrZr%r^r`rqrs)%r dataclassesrtypingrrrrrZmodeling_outputsrZmodeling_utilsr utilsr r r Zbert.modeling_bertrZconfiguration_dprrZ get_loggerrloggerrr r!r%r/rOrZr^r`rbrqrs__all__rrrrsZ        8>  deW