a h@sdZddlZddlmZddlmZmZddlZddlZddlm Z ddl m Z m Z m Z ddlmZmZdd lmZdd lmZmZdd lmZdd lmZdd lmZmZmZddlmZe e!Z"eeddGdddeZ#eeddGdddeZ$eeddGdddeZ%eeddGdddeZ&eeddGdddeZ'eeddGd d!d!eZ(eed"dGd#d$d$eZ)eed%dGd&d'd'eZ*eed(dGd)d*d*eZ+eed+dGd,d-d-eZ,Gd.d/d/e j-Z.Gd0d1d1e j-Z/Gd2d3d3e j-Z0Gd4d5d5e j-Z1Gd6d7d7e j-Z2Gd8d9d9e j-Z3Gd:d;d;e j-Z4Gdd?d?e j-Z6Gd@dAdAe j-Z7GdBdCdCe j-Z8GdDdEdEe j-Z9eGdFdGdGeZ:edHdGdIdJdJe:Z;dKdLZedRdGdSdTdTe:Z?edUdGdVdWdWe:Z@edXdGdYdZdZe:ZAed[dGd\d]d]e:ZBed^dGd_d`d`e:ZCeGdadbdbe:ZDeGdcdddde:ZEgdeZFdS)fzPyTorch LUKE model.N) dataclass)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FNgelu)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel)apply_chunking_to_forward) ModelOutputauto_docstringlogging) LukeConfigz3 Base class for outputs of the LUKE model. )Z custom_introc@s>eZdZUdZdZeejed<dZ ee ejdfed<dS)BaseLukeModelOutputWithPoolingax pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. entity_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, entity_length, hidden_size)`): Sequence of entity hidden-states at the output of the last layer of the model. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. Nentity_last_hidden_state.entity_hidden_states __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rtupler"r"b/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/luke/modeling_luke.pyr&s  rzV Base class for model's outputs, with potential hidden states and attentions. c@s>eZdZUdZdZeejed<dZ ee ejdfed<dS)BaseLukeModelOutputa entity_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, entity_length, hidden_size)`): Sequence of entity hidden-states at the output of the last layer of the model. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. Nr.rrr"r"r"r#r$=s  r$c@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eeejed<dZeeejd fed <dZeeejd fed <dS) LukeMaskedLMOutputa: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): The sum of masked language modeling (MLM) loss and entity prediction loss. mlm_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Masked language modeling (MLM) loss. mep_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Masked entity prediction (MEP) loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). entity_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the entity prediction head (scores for each entity vocabulary token before SoftMax). entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. Nlossmlm_lossmep_losslogits entity_logits hidden_states.r attentions)rrrrr&rrrr r'r(r)r*r+r!rr,r"r"r"r#r%Qs r%z2 Outputs of entity classification models. c@seZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dZ ee ejdfed<dS) EntityClassificationOutput loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification scores (before SoftMax). entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. Nr&r).r+rr,rrrrr&rrrr r)r+r!rr,r"r"r"r#r-ss  r-z7 Outputs of entity pair classification models. c@seZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dZ ee ejdfed<dS) EntityPairClassificationOutputr.Nr&r).r+rr,r/r"r"r"r#r0s  r0z7 Outputs of entity span classification models. c@seZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dZ ee ejdfed<dS) EntitySpanClassificationOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, entity_length, config.num_labels)`): Classification scores (before SoftMax). entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. Nr&r).r+rr,r/r"r"r"r#r1s  r1z4 Outputs of sentence classification models. c@seZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dZ ee ejdfed<dS) LukeSequenceClassifierOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. Nr&r).r+rr,r/r"r"r"r#r2s  r2z@ Base class for outputs of token classification models. c@seZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dZ ee ejdfed<dS) LukeTokenClassifierOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_labels)`): Classification scores (before SoftMax). entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. Nr&r).r+rr,r/r"r"r"r#r3s  r3z/ Outputs of question answering models. c@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dZee ejdfed <dS) LukeQuestionAnsweringModelOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. Nr& start_logits end_logits.r+rr,)rrrrr&rrrr r5r6r+r!rr,r"r"r"r#r4s  r4z, Outputs of multiple choice models. c@seZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dZ ee ejdfed<dS) LukeMultipleChoiceModelOutputa loss (`torch.FloatTensor` of shape *(1,)*, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, num_choices)`): *num_choices* is the second dimension of the input tensors. (see *input_ids* above). Classification scores (before SoftMax). entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. Nr&r).r+rr,r/r"r"r"r#r7s  r7cs2eZdZdZfddZd ddZddZZS) LukeEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_|j|_tj|j|j|jd|_ dS)N padding_idxeps)super__init__r Embedding vocab_size hidden_sizeZ pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutr:selfconfig __class__r"r#r>(s  zLukeEmbeddings.__init__Nc Cs|dur0|dur&t||j|j}n ||}|durB|}n|dd}|durptj|tj|j jd}|dur| |}| |}| |}|||}| |}||}|S)Ndtypedevice)"create_position_ids_from_input_idsr:torT&create_position_ids_from_inputs_embedssizerzeroslong position_idsrBrDrFrGrK) rM input_idstoken_type_idsr[ inputs_embeds input_shaperDrF embeddingsr"r"r#forward9s"        zLukeEmbeddings.forwardcCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor NrQrrRr)rXrZaranger:rZrT unsqueezeexpand)rMr^r_Zsequence_lengthr[r"r"r#rWZs z5LukeEmbeddings.create_position_ids_from_inputs_embeds)NNNN)rrrrr>rarW __classcell__r"r"rOr#r8#s  !r8cs@eZdZedfdd ZdejejeejdddZZ S) LukeEntityEmbeddingsrNcst||_tj|j|jdd|_|j|jkrHtj |j|jdd|_ t|j |j|_ t|j |j|_tj|j|jd|_t|j|_dS)Nrr9Fbiasr;)r=r>rNrr?entity_vocab_sizeentity_emb_sizeentity_embeddingsrALinearentity_embedding_denserCrDrErFrGrHrIrJrKrLrOr"r#r>ms  zLukeEntityEmbeddings.__init__N) entity_idsr[r]c Cs|durt|}||}|jj|jjkr6||}||jdd}|dk | d}||}tj |dd}||j ddjdd}| |}|||}| |}||}|S)Nr)minrQdimgHz>)rZ zeros_likerkrNrjrArmrDclamptype_asrbsumrFrGrK) rMrnr[r]rkrDZposition_embedding_maskrFr`r"r"r#ra{s       zLukeEntityEmbeddings.forward)N) rrrrr>r LongTensorrrardr"r"rOr#relsrecs.eZdZfddZddZd ddZZS) LukeSelfAttentioncst|j|jdkr>t|ds>td|jd|jd|j|_t|j|j|_|j|j|_|j |_ t |j|j|_ t |j|j|_ t |j|j|_|j rt |j|j|_t |j|j|_t |j|j|_t |j|_dS)NrZembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .)r=r>rAnum_attention_headshasattr ValueErrorintattention_head_size all_head_sizeuse_entity_aware_attentionrrlquerykeyvalue w2e_query e2w_query e2e_queryrIZattention_probs_dropout_probrKrLrOr"r#r>s&  zLukeSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)NrQrrr )rXryr}viewpermute)rMxZ new_x_shaper"r"r#transpose_for_scoress z&LukeSelfAttention.transpose_for_scoresNFc Cs|d}|dur|}ntj||gdd}|||}|||} |jr|dur|||} |||} || |} || |} |ddddd|ddf}|ddddd|ddf}|dddd|dddf}|dddd|dddf}t | | dd}t | | dd}t | | dd}t | | dd}tj||gdd}tj||gdd}tj||gdd}n$|||}t || dd}|t |j}|dur||}tjj|dd}||}|dur||}t || }|dddd}|dd|jf}|j|}|ddd|ddf}|durd}n|dd|dddf}|r|||f}n||f}|S)NrrqrQrpr rr)rXrcatrrrrrrrrmatmulZ transposemathsqrtr}r functionalZsoftmaxrKr contiguousr~r) rMword_hidden_statesrattention_mask head_maskoutput_attentions word_sizeconcat_hidden_statesZ key_layerZ value_layerZw2w_query_layerZw2e_query_layerZe2w_query_layerZe2e_query_layerZ w2w_key_layerZ e2w_key_layerZ w2e_key_layerZ e2e_key_layerZw2w_attention_scoresZw2e_attention_scoresZe2w_attention_scoresZe2e_attention_scoresZword_attention_scoresZentity_attention_scoresZattention_scoresZ query_layerZattention_probsZ context_layerZnew_context_layer_shapeZoutput_word_hidden_statesZoutput_entity_hidden_statesoutputsr"r"r#rasV            zLukeSelfAttention.forward)NNF)rrrr>rrardr"r"rOr#rws   rwcs4eZdZfddZejejejdddZZS)LukeSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr;) r=r>rrlrAdenserGrHrIrJrKrLrOr"r#r> s zLukeSelfOutput.__init__r+ input_tensorreturncCs&||}||}|||}|SNrrKrGrMr+rr"r"r#ras  zLukeSelfOutput.forwardrrrr>rZTensorrardr"r"rOr#r s rcs.eZdZfddZddZd ddZZS) LukeAttentioncs*tt||_t||_t|_dSr)r=r>rwrMroutputsetZ pruned_headsrLrOr"r#r>s   zLukeAttention.__init__cCs tddSNz4LUKE does not support the pruning of attention headsNotImplementedError)rMZheadsr"r"r# prune_headsszLukeAttention.prune_headsNFcCs|d}||||||}|dur2|d}|} n(tj|dddd}tj||gdd} ||| } | ddd|ddf} |durd} n| dd|dddf} | | f|dd} | S)Nrrrrq)rXrMrrr)rMrrrrrrZ self_outputsZconcat_self_outputsrattention_outputZword_attention_outputZentity_attention_outputrr"r"r#ra!s(  zLukeAttention.forward)NNF)rrrr>rrardr"r"rOr#rs  rcs0eZdZfddZejejdddZZS)LukeIntermediatecsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr) r=r>rrlrAintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrLrOr"r#r>Hs   zLukeIntermediate.__init__r+rcCs||}||}|Sr)rrrMr+r"r"r#raPs  zLukeIntermediate.forwardrr"r"rOr#rGs rcs4eZdZfddZejejejdddZZS) LukeOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r=r>rrlrrArrGrHrIrJrKrLrOr"r#r>Xs zLukeOutput.__init__rcCs&||}||}|||}|Srrrr"r"r#ra^s  zLukeOutput.forwardrr"r"rOr#rWs rcs.eZdZfddZd ddZddZZS) LukeLayercs:t|j|_d|_t||_t||_t||_ dSNr) r=r>chunk_size_feed_forward seq_len_dimr attentionr intermediaterrrLrOr"r#r>fs    zLukeLayer.__init__NFc Cs|d}|j|||||d}|dur0|d}ntj|dddd}|dd} t|j|j|j|} | ddd|ddf} |durd} n| dd|dddf} | | f| } | S)Nr)rrrrq)rXrrrrfeed_forward_chunkrr) rMrrrrrrZself_attention_outputsZconcat_attention_outputr layer_outputZword_layer_outputZentity_layer_outputr"r"r#rans*    zLukeLayer.forwardcCs||}|||}|Sr)rr)rMrZintermediate_outputrr"r"r#rs  zLukeLayer.feed_forward_chunk)NNF)rrrr>rarrdr"r"rOr#res   %rcs&eZdZfddZdddZZS) LukeEncodercs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr")r).0_rfr"r# z(LukeEncoder.__init__..F) r=r>rNrZ ModuleListrangenum_hidden_layerslayerZgradient_checkpointingrLrOrfr#r>s  zLukeEncoder.__init__NFTcCs|rdnd}|rdnd} |r dnd} t|jD]n\} } |rN||f}| |f} |dur^|| nd} | |||| |}|d}|dur|d}|r.| |df} q.|r||f}| |f} |stdd||| || fDSt||| || dS)Nr"rrrcss|]}|dur|VqdSrr"rvr"r"r# sz&LukeEncoder.forward..)last_hidden_stater+r,rr) enumeraterr!r$)rMrrrrroutput_hidden_states return_dictZall_word_hidden_statesZall_entity_hidden_statesZall_self_attentionsiZ layer_moduleZlayer_head_maskZ layer_outputsr"r"r#rasN        zLukeEncoder.forward)NNFFTrrrr>rardr"r"rOr#rs rcs0eZdZfddZejejdddZZS) LukePoolercs*tt|j|j|_t|_dSr)r=r>rrlrArZTanh activationrLrOr"r#r>s zLukePooler.__init__rcCs(|dddf}||}||}|S)Nr)rr)rMr+Zfirst_token_tensor pooled_outputr"r"r#ras  zLukePooler.forwardrr"r"rOr#rs rcs$eZdZfddZddZZS)EntityPredictionHeadTransformcsVtt|j|j|_t|jt r6t |j|_ n|j|_ tj |j|j d|_ dSr)r=r>rrlrArjrrrrr transform_act_fnrGrHrLrOr"r#r>s   z&EntityPredictionHeadTransform.__init__cCs"||}||}||}|Sr)rrrGrr"r"r#ras   z%EntityPredictionHeadTransform.forwardrr"r"rOr#rs rcs$eZdZfddZddZZS)EntityPredictionHeadcsHt||_t||_tj|j|jdd|_ t t |j|_ dS)NFrg)r=r>rNr transformrrlrjridecoder ParameterrrYrhrLrOr"r#r>s   zEntityPredictionHead.__init__cCs||}|||j}|Sr)rrrhrr"r"r#ras zEntityPredictionHead.forwardrr"r"rOr#rs rc@s6eZdZUeed<dZdZddgZej dddZ d S) LukePreTrainedModelrNlukeTrre)modulecCst|tjr:|jjjd|jjd|jdur|jj n~t|tj r|j dkr^|jj n|jjjd|jjd|j dur|jj|j  n&t|tj r|jj |jjddS)zInitialize the weightsg)meanZstdNr?)rrrlweightdataZnormal_rNZinitializer_rangerhZzero_r?Z embedding_dimr:rGZfill_)rMrr"r"r# _init_weightss       z!LukePreTrainedModel._init_weightsN) rrrrr Zbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesrModulerr"r"r"r#r s rzt The bare LUKE model transformer outputting raw hidden-states for both word tokens and entities without any cseZdZdeedfdd ZddZddZd d Zd d Z d dZ e de e je e je e je e je e je e je e je e je e je e je ee ee eeeefdddZe je e jdddZZS) LukeModelT)rNadd_pooling_layercsNt|||_t||_t||_t||_|rrNr8r`rerkrencoderrpooler post_init)rMrNrrOr"r#r>,s    zLukeModel.__init__cCs|jjSrr`rBrMr"r"r#get_input_embeddings=szLukeModel.get_input_embeddingscCs ||j_dSrrrMrr"r"r#set_input_embeddings@szLukeModel.set_input_embeddingscCs|jjSrrkrr"r"r#get_entity_embeddingsCszLukeModel.get_entity_embeddingscCs ||j_dSrrrr"r"r#set_entity_embeddingsFszLukeModel.set_entity_embeddingscCs tddSrr)rMZheads_to_pruner"r"r# _prune_headsIszLukeModel._prune_headsN)r\rr]r[rnentity_attention_maskentity_token_type_idsentity_position_idsrr^rrrrc Cs| dur | n|jj} | dur | n|jj} | dur4| n|jj} |durV| durVtdn@|durt||||}n"| dur| dd}ntd|\}}|dur|jn| j}|durtj ||f|d}|durtj |tj |d}|dur6|d}|durtj ||f|d}|dur6tj ||ftj |d}| | |jj } |j|||| d}|||}|durtd}n||||}|j|||| | | | d }|d }|jdur||nd}| s||f|ddSt|||j|j|j|jd S) uz entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. Examples: ```python >>> from transformers import AutoTokenizer, LukeModel >>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-base") >>> model = LukeModel.from_pretrained("studio-ousia/luke-base") # Compute the contextualized entity representation corresponding to the entity mention "Beyoncé" >>> text = "Beyoncé lives in Los Angeles." >>> entity_spans = [(0, 7)] # character-based entity span corresponding to "Beyoncé" >>> encoding = tokenizer(text, entity_spans=entity_spans, add_prefix_space=True, return_tensors="pt") >>> outputs = model(**encoding) >>> word_last_hidden_state = outputs.last_hidden_state >>> entity_last_hidden_state = outputs.entity_last_hidden_state # Input Wikipedia entities to obtain enriched contextualized representations of word tokens >>> text = "Beyoncé lives in Los Angeles." >>> entities = [ ... "Beyoncé", ... "Los Angeles", ... ] # Wikipedia entity titles corresponding to the entity mentions "Beyoncé" and "Los Angeles" >>> entity_spans = [ ... (0, 7), ... (17, 28), ... ] # character-based entity spans corresponding to "Beyoncé" and "Los Angeles" >>> encoding = tokenizer( ... text, entities=entities, entity_spans=entity_spans, add_prefix_space=True, return_tensors="pt" ... ) >>> outputs = model(**encoding) >>> word_last_hidden_state = outputs.last_hidden_state >>> entity_last_hidden_state = outputs.entity_last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerQz5You have to specify either input_ids or inputs_embeds)rTrRr)r\r[r]r^)rrrrrr)r pooler_outputr+r,rr)rNrruse_return_dictr{Z%warn_if_padding_and_no_attention_maskrXrTrZonesrYrZZ get_head_maskrr`get_extended_attention_maskrkrrrr+r,rr)rMr\rr]r[rnrrrrr^rrrr_Z batch_sizeZ seq_lengthrTZentity_seq_lengthZword_embedding_outputextended_attention_maskZentity_embedding_outputZencoder_outputssequence_outputrr"r"r#raLspI          zLukeModel.forward)word_attention_maskrcCs|}|durtj||gdd}|dkrH|dddddddf}n8|dkrn|ddddddf}ntd|jd|j|jd}d |t|jj}|S) ac Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: word_attention_mask (`torch.LongTensor`): Attention mask for word tokens with ones indicating tokens to attend to, zeros for tokens to ignore. entity_attention_mask (`torch.LongTensor`, *optional*): Attention mask for entity tokens with ones indicating tokens to attend to, zeros for tokens to ignore. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. NrQrqr rz&Wrong shape for attention_mask (shape ))rSr) rrrrr{shaperVrSZfinforo)rMrrrrr"r"r#rs  z%LukeModel.get_extended_attention_mask)T) NNNNNNNNNNNNN)rrrrboolr>rrrrrrrrrvrrr!rrarrdr"r"rOr#r&sN  rcCs2||}tj|dd||}||S)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rrq)ner|rZcumsumrtrZ)r\r:maskZincremental_indicesr"r"r#rUs rUcs0eZdZdZfddZddZddZZS) LukeLMHeadz*Roberta Head for masked language modeling.csdtt|j|j|_tj|j|jd|_t|j|j |_ t t |j |_|j|j _dSr)r=r>rrlrArrGrH layer_normr@rrrrYrhrLrOr"r#r>s  zLukeLMHead.__init__cKs*||}t|}||}||}|Sr)rr rr)rMfeatureskwargsrr"r"r#ra$s    zLukeLMHead.forwardcCs*|jjjjdkr|j|j_n |jj|_dS)Nmeta)rrhrTtyperr"r"r# _tie_weights.s zLukeLMHead._tie_weights)rrrrr>rarrdr"r"rOr#rs  rz The LUKE model with a language modeling head and entity prediction head on top for masked language modeling and masked entity prediction. cseZdZgdZfddZfddZddZdd Zede e j e e j e e j e e j e e j e e j e e j e e j e e j e e j e e j e e j e e e e e e eeefd d d ZZS)LukeForMaskedLM)zlm_head.decoder.weightzlm_head.decoder.biasz!entity_predictions.decoder.weightcs@t|t||_t||_t||_t |_ | dSr) r=r>rrrlm_headrentity_predictionsrrloss_fnrrLrOr"r#r>@s      zLukeForMaskedLM.__init__cs$t||jj|jjjdSr)r= tie_weightsZ_tie_or_clone_weightsrrrrkrrOr"r#rMs zLukeForMaskedLM.tie_weightscCs|jjSrrrrr"r"r#get_output_embeddingsQsz%LukeForMaskedLM.get_output_embeddingscCs ||j_dSrr)rMZnew_embeddingsr"r"r#set_output_embeddingsTsz%LukeForMaskedLM.set_output_embeddingsN)r\rr]r[rnrrrlabels entity_labelsrr^rrrrcCs0|dur |n|jj}|j||||||||| | | |dd }d}d}||j}| dur| |j} ||d|jj | d}|dur|}d}d}|j dur| |j }| dur||d|jj | d}|dur|}n||}|st dd||||||j|j|jfDSt||||||j|j|jdS)aC entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` entity_labels (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` NT r\rr]r[rnrrrrr^rrrrQcss|]}|dur|VqdSrr"rr"r"r#rs z*LukeForMaskedLM.forward..)r&r'r(r)r*r+rr,)rNrrrrrVrTrrr@rrrir!r+rr,r%)rMr\rr]r[rnrrrr r rr^rrrrr&r'r)r(r*r"r"r#raWsn1    zLukeForMaskedLM.forward)NNNNNNNNNNNNNNN)rrrZ_tied_weights_keysr>rrrrrrrvrrrr!r%rardr"r"rOr#r7sN  rz The LUKE model with a classification head on top (a linear layer on top of the hidden state of the first entity token) for entity classification tasks, such as Open Entity. cseZdZfddZedeejeejeejeejeejeejeejeejeejeejeejee ee ee e e e fdddZ ZS)LukeForEntityClassificationcsJt|t||_|j|_t|j|_t |j |j|_ | dSr r=r>rr num_labelsrrIrJrKrlrA classifierrrLrOr"r#r>s   z$LukeForEntityClassification.__init__Nr\rr]r[rnrrrrr^r rrrrcCs|dur |n|jj}|j||||||||| | | | dd }|jdddddf}||}||}d}| dur| |j} | jdkrt j || }n t j | d| d|}|stdd|||j|j|jfDSt|||j|j|jd S) u entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. labels (`torch.LongTensor` of shape `(batch_size,)` or `(batch_size, num_labels)`, *optional*): Labels for computing the classification loss. If the shape is `(batch_size,)`, the cross entropy loss is used for the single-label classification. In this case, labels should contain the indices that should be in `[0, ..., config.num_labels - 1]`. If the shape is `(batch_size, num_labels)`, the binary cross entropy loss is used for the multi-label classification. In this case, labels should only contain `[0, 1]`, where 0 and 1 indicate false and true, respectively. Examples: ```python >>> from transformers import AutoTokenizer, LukeForEntityClassification >>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-open-entity") >>> model = LukeForEntityClassification.from_pretrained("studio-ousia/luke-large-finetuned-open-entity") >>> text = "Beyoncé lives in Los Angeles." >>> entity_spans = [(0, 7)] # character-based entity span corresponding to "Beyoncé" >>> inputs = tokenizer(text, entity_spans=entity_spans, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: person ```NTr rrrQcss|]}|dur|VqdSrr"rr"r"r#r?sz6LukeForEntityClassification.forward..r&r)r+rr,)rNrrrrKrrVrTndimrr cross_entropy binary_cross_entropy_with_logitsrrtr!r+rr,r-rMr\rr]r[rnrrrrr^r rrrrfeature_vectorr)r&r"r"r#rasH>     z#LukeForEntityClassification.forward)NNNNNNNNNNNNNN)rrrr>rrrrvrrrr!r-rardr"r"rOr#r sB  r z The LUKE model with a classification head on top (a linear layer on top of the hidden states of the two entity tokens) for entity pair classification tasks, such as TACRED. cseZdZfddZedeejeejeejeejeejeejeejeejeejeejeejee ee ee e e e fdddZ ZS)LukeForEntityPairClassificationcsPt|t||_|j|_t|j|_t |j d|jd|_ | dS)NrFr rLrOr"r#r>Us   z(LukeForEntityPairClassification.__init__NrcCs|dur |n|jj}|j||||||||| | | | dd }tj|jdddddf|jdddddfgdd}||}||}d}| dur| |j } | j dkrt j || }n t j |d| d|}|stdd |||j|j|jfDSt|||j|j|jd S) u entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. labels (`torch.LongTensor` of shape `(batch_size,)` or `(batch_size, num_labels)`, *optional*): Labels for computing the classification loss. If the shape is `(batch_size,)`, the cross entropy loss is used for the single-label classification. In this case, labels should contain the indices that should be in `[0, ..., config.num_labels - 1]`. If the shape is `(batch_size, num_labels)`, the binary cross entropy loss is used for the multi-label classification. In this case, labels should only contain `[0, 1]`, where 0 and 1 indicate false and true, respectively. Examples: ```python >>> from transformers import AutoTokenizer, LukeForEntityPairClassification >>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-tacred") >>> model = LukeForEntityPairClassification.from_pretrained("studio-ousia/luke-large-finetuned-tacred") >>> text = "Beyoncé lives in Los Angeles." >>> entity_spans = [ ... (0, 7), ... (17, 28), ... ] # character-based entity spans corresponding to "Beyoncé" and "Los Angeles" >>> inputs = tokenizer(text, entity_spans=entity_spans, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: per:cities_of_residence ```NTr rrrqrQcss|]}|dur|VqdSrr"rr"r"r#rsz:LukeForEntityPairClassification.forward..r)rNrrrrrrKrrVrTrrrrrrrtr!r+rr,r0rr"r"r#raasLA0     z'LukeForEntityPairClassification.forward)NNNNNNNNNNNNNN)rrrr>rrrrvrrrr!r0rardr"r"rOr#rNsB  rz The LUKE model with a span classification head on top (a linear layer on top of the hidden states output) for tasks such as named entity recognition. cseZdZfddZedeejeejeejeejeejeejeejeejeejeejeejeejeejee ee ee e e e fdddZ ZS)LukeForEntitySpanClassificationcsNt|t||_|j|_t|j|_t |j d|j|_ | dS)Nr r rLrOr"r#r>s   z(LukeForEntitySpanClassification.__init__N)r\rr]r[rnrrrentity_start_positionsentity_end_positionsrr^r rrrrcCs|dur |n|jj}|j||||||||| | ||dd }|jd}| ddd|} | j|jjkrt| |jj} t |jd| }| ddd|} | j|jjkr| |jj} t |jd| }t j |||j gdd}| |}||}d}| durT| |j} | jdkr4tj|d|j| d}n tj|d| d|}|s|tdd |||j|j|jfDSt|||j|j|jd S) u entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. entity_start_positions (`torch.LongTensor`): The start positions of entities in the word token sequence. entity_end_positions (`torch.LongTensor`): The end positions of entities in the word token sequence. labels (`torch.LongTensor` of shape `(batch_size, entity_length)` or `(batch_size, entity_length, num_labels)`, *optional*): Labels for computing the classification loss. If the shape is `(batch_size, entity_length)`, the cross entropy loss is used for the single-label classification. In this case, labels should contain the indices that should be in `[0, ..., config.num_labels - 1]`. If the shape is `(batch_size, entity_length, num_labels)`, the binary cross entropy loss is used for the multi-label classification. In this case, labels should only contain `[0, 1]`, where 0 and 1 indicate false and true, respectively. Examples: ```python >>> from transformers import AutoTokenizer, LukeForEntitySpanClassification >>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-conll-2003") >>> model = LukeForEntitySpanClassification.from_pretrained("studio-ousia/luke-large-finetuned-conll-2003") >>> text = "Beyoncé lives in Los Angeles" # List all possible entity spans in the text >>> word_start_positions = [0, 8, 14, 17, 21] # character-based start positions of word tokens >>> word_end_positions = [7, 13, 16, 20, 28] # character-based end positions of word tokens >>> entity_spans = [] >>> for i, start_pos in enumerate(word_start_positions): ... for end_pos in word_end_positions[i:]: ... entity_spans.append((start_pos, end_pos)) >>> inputs = tokenizer(text, entity_spans=entity_spans, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> predicted_class_indices = logits.argmax(-1).squeeze().tolist() >>> for span, predicted_class_idx in zip(entity_spans, predicted_class_indices): ... if predicted_class_idx != 0: ... print(text[span[0] : span[1]], model.config.id2label[predicted_class_idx]) Beyoncé PER Los Angeles LOC ```NTr rQrprrqcss|]}|dur|VqdSrr"rr"r"r#resz:LukeForEntitySpanClassification.forward..r)rNrrrrXrbrcrTrVrZgatherrrrKrrrrrrrrrtr!r+rr,r1)rMr\rr]r[rnrrrrrrr^r rrrrrAZ start_statesZ end_statesrr)r&r"r"r#rasZO        z'LukeForEntitySpanClassification.forward)NNNNNNNNNNNNNNNN)rrrr>rrrrvrrrr!r1rardr"r"rOr#rsJ  rz The LUKE Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cseZdZfddZedeejeejeejeejeejeejeejeejeejeejeejee ee ee e e e fdddZ ZS)LukeForSequenceClassificationcsZt||j|_t||_t|jdur2|jn|j|_ t |j |j|_ | dSrr=r>rrrrrIclassifier_dropoutrJrKrlrArrrLrOr"r#r>{s  z&LukeForSequenceClassification.__init__NrcCs|dur |n|jj}|j||||||||| | | | dd }|j}||}||}d}| durJ| |j} |jjdur|j dkrd|j_n4|j dkr| j t j ks| j t j krd|j_nd|j_|jjdkrt}|j dkr||| }n ||| }nN|jjdkr,t}||d|j | d}n|jjdkrJt}||| }|srtd d |||j|j|jfDSt|||j|j|jd S) a entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTr rZ regressionZsingle_label_classificationZmulti_label_classificationrQcss|]}|dur|VqdSrr"rr"r"r#rsz8LukeForSequenceClassification.forward..r)rNrrrrKrrVrTZ problem_typerrSrrZr|rsqueezerrrr!r+rr,r2)rMr\rr]r[rnrrrrr^r rrrrrr)r&loss_fctr"r"r#rasd+       "     z%LukeForSequenceClassification.forward)NNNNNNNNNNNNNN)rrrr>rrrrvrrrr!r2rardr"r"rOr#rtsB  rz The LUKE Model with a token classification head on top (a linear layer on top of the hidden-states output). To solve Named-Entity Recognition (NER) task using LUKE, `LukeForEntitySpanClassification` is more suitable than this class. cseZdZfddZedeejeejeejeejeejeejeejeejeejeejeejee ee ee e e e fdddZ ZS)LukeForTokenClassificationcs^t||j|_t|dd|_t|jdur6|jn|j|_ t |j |j|_ | dSNF)rrrLrOr"r#r>s z#LukeForTokenClassification.__init__NrcCs|dur |n|jj}|j||||||||| | | | dd }|j}||}||}d}| dur| |j} t}|| d|j | d}|st dd|||j |j |jfDSt|||j |j |jdS)aM entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) NTr rQcss|]}|dur|VqdSrr"rr"r"r#rQsz5LukeForTokenClassification.forward..r)rNrrrrKrrVrTrrrr!r+rr,r3)rMr\rr]r[rnrrrrr^r rrrrrr)r&rr"r"r#rasF+   z"LukeForTokenClassification.forward)NNNNNNNNNNNNNN)rrrr>rrrrvrrrr!r3rardr"r"rOr#r sB  r cseZdZfddZedeejeejeejeejeejeejeejeejeejeejeejeejee ee ee e e e fdddZ ZS)LukeForQuestionAnsweringcs@t||j|_t|dd|_t|j|j|_| dSr!) r=r>rrrrrlrA qa_outputsrrLrOr"r#r>bs  z!LukeForQuestionAnswering.__init__N)r\rr]r[rnrrrrr^start_positions end_positionsrrrrcCsJ|dur |n|jj}|j||||||||| | | |dd }|j}||}|jddd\}}|d}|d}d}| dur| durt| dkr| d} t| dkr| d} |d}| d|| d|t |d}||| }||| }||d }|s0t d d ||||j |j |jfDSt||||j |j |jd S) a  entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. NTr rrQrqr)Z ignore_indexrcss|]}|dur|VqdSrr"rr"r"r#rsz3LukeForQuestionAnswering.forward..)r&r5r6r+rr,)rNrrrr#splitrlenrXZclamp_rr!r+rr,r4)rMr\rr]r[rnrrrrr^r$r%rrrrrr)r5r6Z total_lossZ ignored_indexrZ start_lossZend_lossr"r"r#ramsh(             z LukeForQuestionAnswering.forward)NNNNNNNNNNNNNNN)rrrr>rrrrvrrrr!r4rardr"r"rOr#r"`sF  r"cseZdZfddZedeejeejeejeejeejeejeejeejeejeejeejee ee ee e e e fdddZ ZS)LukeForMultipleChoicecsPt|t||_t|jdur*|jn|j|_t |j d|_ | dSr) r=r>rrrrIrrJrKrlrArrrLrOr"r#r>s  zLukeForMultipleChoice.__init__NrcCs|dur |n|jj}|dur&|jdn| jd}|durJ|d|dnd}|durh|d|dnd}|dur|d|dnd}|dur|d|dnd}| dur| d| d| dnd} |dur|d|dnd}|dur|d|dnd}|dur(|d|dnd}|durP|d|d|dnd}|j||||||||| | | | dd }|j}||}||}|d|}d}| dur| |j } t }||| }|st dd|||j |j|jfDSt|||j |j|jd S) a^ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) NrrQrpTr css|]}|dur|VqdSrr"rr"r"r#rdsz0LukeForMultipleChoice.forward..r)rNrrrrXrrrKrrVrTrr!r+rr,r7)rMr\rr]r[rnrrrrr^r rrrZ num_choicesrrr)Zreshaped_logitsr&rr"r"r#ras~C       zLukeForMultipleChoice.forward)NNNNNNNNNNNNNN)rrrr>rrrrvrrrr!r7rardr"r"rOr#r(sB  r() r rrr(r"rr rrr)Grr dataclassesrtypingrrrZtorch.utils.checkpointrZtorch.nnrrrZ activationsr r Zmodeling_layersr Zmodeling_outputsr rZmodeling_utilsrZ pytorch_utilsrutilsrrrZconfiguration_lukerZ get_loggerrloggerrr$r%r-r0r1r2r3r4r7rr8rerwrrrrrrrrrrrrUrrr rrrr r"r(__all__r"r"r"r#s       I+r04E]|xgv"