a h@sdZddlZddlmZddlmZmZmZddlZddlm Z ddl m Z m Z m Z ddlmZmZdd lmZdd lmZdd lmZdd lmZmZmZd dlmZeeZdKddZ ddZ!Gddde j"Z#Gddde j"Z$Gddde j"Z%Gddde j"Z&Gddde j"Z'Gddde j"Z(Gdd d e j"Z)eGd!d"d"eZ*eed#d$Gd%d&d&eZ+eed'd$Gd(d)d)eZ,eed*d$Gd+d,d,eZ-eed-d$Gd.d/d/eZ.eed0d$Gd1d2d2eZ/eed3d$Gd4d5d5eZ0eed6d$Gd7d8d8eZ1eGd9d:d:e*Z2ed;d$Gdd$Gd?d@d@e*Z4eGdAdBdBe*Z5eGdCdDdDe*Z6edEd$GdFdGdGe*Z7eGdHdIdIe*Z8gdJZ9dS)Lz PyTorch XLNet model. N) dataclass)CallableOptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FNget_activation)GenerationMixin)PreTrainedModel)apply_chunking_to_forward) ModelOutputauto_docstringlogging) XLNetConfigc Cs.i}t|drt|dr$|jj|d<t|drRd|vrR|jjj|d<|jjj|d<t|dr|jdurd |jd |vr|jj|d |jd <|jj|d |jd <|j}| |j j|j d t |j D]\}}d |d}| |d|jjj|d|jjj|d|jj|d|jj|d|jj|d|jj|d|jj|d|jjj|d|jjj|d|jjj|d|jjj|d|jjj|d|jjji q|jrg}g}g} g} |j D]>}||jj||jj| |jj| |jjqn |jg}|jg}|jg} |jg} | ||| | d|S)z A map of modules from TF to PyTorch. I use a map to keep the PyTorch model as identical to the original PyTorch model as possible. transformerlm_losszmodel/lm_loss/biassequence_summaryz%model/sequnece_summary/summary/kernelz#model/sequnece_summary/summary/bias logits_projNzmodel/regression_z /logit/kernelz /logit/bias)z-model/transformer/word_embedding/lookup_tablez#model/transformer/mask_emb/mask_embzmodel/transformer/layer_/zrel_attn/LayerNorm/gammazrel_attn/LayerNorm/betazrel_attn/o/kernelzrel_attn/q/kernelzrel_attn/k/kernelzrel_attn/r/kernelzrel_attn/v/kernelzff/LayerNorm/gammazff/LayerNorm/betazff/layer_1/kernelzff/layer_1/biaszff/layer_2/kernelzff/layer_2/bias)zmodel/transformer/r_r_biaszmodel/transformer/r_w_biaszmodel/transformer/r_s_biaszmodel/transformer/seg_embed)hasattrrbiasrsummaryweightZfinetuning_taskrrupdateword_embeddingmask_emb enumeratelayerrel_attn layer_normoqkrvfflayer_1layer_2Zuntie_rappendr_r_biasr_w_biasr_s_bias seg_embed) modelconfig tf_weights tf_to_pt_mapibZ layer_strZr_r_listZr_w_listZr_s_listZseg_embed_listr8d/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.pybuild_tf_xlnet_to_pytorch_map'sz          r:c Cszddl}ddl}Wnty2tdYn0|j|}i}|D]4\}}td|d||j||} | ||<qHt |||} | D]\}} td|||vrt|dq||} d|vrd |vsd |vsd |vrtd | | } t | t rt| | jdksFJd t| d| jddt| D]\} } | | df}z,| j|jksJd| jd|jdWn>ty}z$|j| j|jf7_WYd}~n d}~00td|d| t|| _qNnz,| j| jks Jd| jd| jdWn>ty`}z$|j| j| jf7_WYd}~n d}~00td|t| | _||d||dd||ddqtdd||S)z&Load tf checkpoints in a pytorch modelrNzLoading a TensorFlow models in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.zLoading TF weight z with shape z Importing z( not in tf pre-trained weights, skippingZkernelr*rZlogitZ TransposingzPointer length z and array length z mismatched.zPointer shape z and array shape zInitialize PyTorch weight z for layer z/Adamz/Adam_1z%Weights not copied to PyTorch model: z, )numpyZ tensorflow ImportErrorloggererrortrainZlist_variablesinfoZ load_variabler:items transpose isinstancelistlenshaper!AssertionErrorargstorchZ from_numpydatapopjoinkeys)r2r3Ztf_pathnptfZ init_varsr4namerFarrayr5Zpointerr6Zp_iZarr_ier8r8r9load_tf_weights_in_xlnetzsj      $      rScs^eZdZfddZddZedddZeddd Zdd d ZdddZ dddZ Z S)XLNetRelativeAttentioncsnt|j|jdkr2td|jd|j|j|_|j|_|j|_d|jd|_tt |j|j|j|_ tt |j|j|j|_ tt |j|j|j|_ tt |j|j|j|_tt |j|j|j|_tt |j|j|_tt |j|j|_tt |j|j|_tt d|j|j|_tj|j|jd|_t|j|_dS)NrzThe hidden size (z6) is not a multiple of the number of attention heads (r?eps)super__init__d_modelZn_head ValueErrorZd_headscaler ParameterrI FloatTensorr&r'r)r%r(r.r0r/r1 LayerNormlayer_norm_epsr$Dropoutdropoutselfr3 __class__r8r9rZs,  zXLNetRelativeAttention.__init__cCstdSNNotImplementedError)reZheadsr8r8r9 prune_headssz"XLNetRelativeAttention.prune_headsc Cs||j}||d|d|d|d}|dddf}||d|dd|d|d}t|dtj||jtjd}|S)zbnijr )rvNrzibnd,snd->ibnszijbs,ibns->bnijz ijbn->bnijꌠ9Y>)Fdimzbnij,jbnd->ibndz bnij->ijbn)rIeinsumr/r.rxrFr0r1r]rofloat16r functionalsoftmaxrc)reZq_headk_head_hv_head_hk_head_rseg_mat attn_mask head_maskoutput_attentionsacZbdZefZ attn_score attn_probattn_vecr8r8r9 rel_attn_cores(  z$XLNetRelativeAttention.rel_attn_coreTcCs4td||j}||}|r&||}||}|S)zPost-attention processing.z ibnd,hnd->ibh)rIr}r%rcr$)rehrZresidualZattn_outoutputr8r8r9post_attention-s   z%XLNetRelativeAttention.post_attentionc  Cs|durJ|dur2|dkr2tj||gdd} n|} td| |j} td| |j} td||j}td||j}|j|| | |||| | d}| r|\}}| ||}td||j}|durtd||}|j|| | |||| | d}| r|\}}td||}n(|j|| | |||| | d}| r.|\}}| ||}| r||f}n|durv|dkrvtj||gdd} n|} td||j}td| |j} td| |j} td| |jj |j}|j|| | |||| | d}| r|\}}| ||}d}||f}| r||f}|S)Nrrr{z ibh,hnd->ibnd)rrrrzmbnd,mlb->lbndzlbnd,mlb->mbnd) r|rIcatr}r'r)r(r&rrtypero)rerg attn_mask_h attn_mask_gr(rmemstarget_mappingrrrrrrZq_head_hZ attn_vec_hZ attn_prob_houtput_hZq_head_gZ attn_vec_gZ attn_prob_goutput_grroutputsr8r8r9forward9s           zXLNetRelativeAttention.forward)rl)rl)NNNF)T)NNNF) __name__ __module__ __qualname__rZrk staticmethodrwrxrrr __classcell__r8r8rfr9rTs"    4 rTcs$eZdZfddZddZZS)XLNetFeedForwardcsvttj|j|jd|_t|j|j|_ t|j|j|_ t |j |_ t |jtrjt|j|_n|j|_dS)NrW)rYrZrr`r[rar$LinearZd_innerr+r,rbrcrCZ ff_activationstrr activation_functionrdrfr8r9rZs  zXLNetFeedForward.__init__cCsH|}||}||}||}||}||}|||}|Srh)r+rrcr,r$)reinprr8r8r9rs     zXLNetFeedForward.forward)rrrrZrrr8r8rfr9rs rcs.eZdZfddZd ddZddZZS) XLNetLayercs>tt||_t||_t|j|_|j |_ d|_ dSNr) rYrZrTr#rr*rrbrcchunk_size_feed_forward seq_len_dimrdrfr8r9rZs    zXLNetLayer.__init__NFc Csv|j||||||||| | d } | dd\}}|durJt|j|j|j|}t|j|j|j|}||f| dd} | S)N)rrrrrV)r#rff_chunkrr) rerrrrr(rrrrrrr8r8r9rs(  zXLNetLayer.forwardcCs||}|Srh)r*)reZoutput_xr8r8r9rs zXLNetLayer.ff_chunk)NNNF)rrrrZrrrr8r8rfr9rs  $rcsDeZdZdZedfdd Zd ejeejejdddZ Z S) XLNetPoolerStartLogitsz Compute SQuAD start logits from sequence hidden states. Args: config ([`XLNetConfig`]): The config used by the model, will be used to grab the `hidden_size` of the model. r3cstt|jd|_dSr)rYrZrr hidden_sizedenserdrfr8r9rZs zXLNetPoolerStartLogits.__init__N) hidden_statesp_maskreturncCsR||d}|durN|jtjkr:|d|d|}n|d|d|}|S)a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`): The final hidden states of the model. p_mask (`torch.FloatTensor` of shape `(batch_size, seq_len)`, *optional*): Mask for tokens at invalid position, such as query and special symbols (PAD, SEP, CLS). 1.0 means token should be masked. Returns: `torch.FloatTensor`: The start logits for SQuAD. rlNrryrz)rsqueezerorIr~)rerrrur8r8r9rs  zXLNetPoolerStartLogits.forward)N) rrr__doc__rrZrIr_rrrr8r8rfr9r s rcsTeZdZdZedfdd Zd ejeejeej eejejdddZ Z S) XLNetPoolerEndLogitsz Compute SQuAD end logits from sequence hidden states. Args: config ([`XLNetConfig`]): The config used by the model, will be used to grab the `hidden_size` of the model and the `layer_norm_eps` to use. rcsRtt|jd|j|_t|_tj|j|j d|_t|jd|_ dS)NrVrWr) rYrZrrrdense_0Tanh activationr`radense_1rdrfr8r9rZ<s   zXLNetPoolerEndLogits.__init__N)r start_statesstart_positionsrrcCs|dus|dusJd|durh|jdd\}}|ddddfdd|}|d|}|d|d}|tj||gdd}||}||}|| d}|dur|j tj kr|d|d|}n|d|d|}|S) a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`): The final hidden states of the model. start_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`, *optional*): The hidden states of the first tokens for the labeled span. start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): The position of the first token for the labeled span. p_mask (`torch.FloatTensor` of shape `(batch_size, seq_len)`, *optional*): Mask for tokens at invalid position, such as query and special symbols (PAD, SEP, CLS). 1.0 means token should be masked. One of `start_states` or `start_positions` should be not `None`. If both are set, `start_positions` overrides `start_states`. Returns: `torch.FloatTensor`: The end logits for SQuAD. N7One of start_states, start_positions should be not Nonerlr{rryrz) rFexpandgatherrrIrrr`rrror~)rerrrrslenhszrur8r8r9rCs"    zXLNetPoolerEndLogits.forward)NNN rrrrrrZrIr_r LongTensorrrr8r8rfr9r2s  rcsTeZdZdZedfdd Zd ejeejeej eej ejdddZ Z S) XLNetPoolerAnswerClassz Compute SQuAD 2.0 answer class from classification and start tokens hidden states. Args: config ([`XLNetConfig`]): The config used by the model, will be used to grab the `hidden_size` of the model. rcsBtt|jd|j|_t|_tj|jddd|_dS)NrVrFr) rYrZrrrrrrrrdrfr8r9rZs  zXLNetPoolerAnswerClass.__init__N)rrr cls_indexrcCs|jd}|dus"|dus"Jd|durX|ddddfdd|}|d|d}|dur|ddddfdd|}|d|d}n|dddddf}|tj||gdd}||}||d}|S)a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`): The final hidden states of the model. start_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`, *optional*): The hidden states of the first tokens for the labeled span. start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): The position of the first token for the labeled span. cls_index (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Position of the CLS token for each sentence in the batch. If `None`, takes the last token. One of `start_states` or `start_positions` should be not `None`. If both are set, `start_positions` overrides `start_states`. Returns: `torch.FloatTensor`: The SQuAD 2.0 answer class. rlNrrr{) rFrrrrrIrrr)rerrrrrZcls_token_staterur8r8r9rs  zXLNetPoolerAnswerClass.forward)NNNrr8r8rfr9rxs rcsDeZdZdZedfdd Zd ejeej ejdddZ Z S) XLNetSequenceSummarya Compute a single vector summary of a sequence hidden states. Args: config ([`XLNetConfig`]): The config used by the model. Relevant arguments in the config class of the model are (refer to the actual config class of your model for the default values it uses): - **summary_type** (`str`) -- The method to use to make this summary. Accepted values are: - `"last"` -- Take the last token hidden state (like XLNet) - `"first"` -- Take the first token hidden state (like Bert) - `"mean"` -- Take the mean of all tokens hidden states - `"cls_index"` -- Supply a Tensor of classification token position (GPT/GPT-2) - `"attn"` -- Not implemented now, use multi-head attention - **summary_use_proj** (`bool`) -- Add a projection after the vector extraction. - **summary_proj_to_labels** (`bool`) -- If `True`, the projection outputs to `config.num_labels` classes (otherwise to `config.hidden_size`). - **summary_activation** (`Optional[str]`) -- Set to `"tanh"` to add a tanh activation to the output, another string or `None` will add no activation. - **summary_first_dropout** (`float`) -- Optional dropout probability before the projection and activation. - **summary_last_dropout** (`float`)-- Optional dropout probability after the projection and activation. rcstt|dd|_|jdkr&tt|_t|drx|j rxt|drb|j rb|j dkrb|j }n|j }t |j ||_t|dd}|rt|nt|_t|_t|dr|jdkrt|j|_t|_t|d r|jdkrt|j|_dS) N summary_typelastattnsummary_use_projsummary_proj_to_labelsrZsummary_activationsummary_first_dropoutsummary_last_dropout)rYrZgetattrrrjrZIdentityrrrr num_labelsrrr r first_dropoutrrb last_dropoutr)rer3 num_classesZactivation_stringrfr8r9rZs$      zXLNetSequenceSummary.__init__N)rrrcCs|jdkr|dddf}n|jdkr8|dddf}n|jdkrP|jdd}n|jd kr|durtj|d ddddf|jd dtjd }n2|dd}|d |d| df}| d | d }n|jdkrt | |}||}||}||}|S)ak Compute a single vector summary of a sequence hidden states. Args: hidden_states (`torch.FloatTensor` of shape `[batch_size, seq_len, hidden_size]`): The hidden states of the last layer. cls_index (`torch.LongTensor` of shape `[batch_size]` or `[batch_size, ...]` where ... are optional leading dimensions of `hidden_states`, *optional*): Used if `summary_type == "cls_index"` and takes the last token of the sequence as classification token. Returns: `torch.FloatTensor`: The summary of the sequence hidden states. rNrlfirstrmeanrr{r.rro)rlr)rrrIZ full_likerFrs unsqueezerr|sizerrrjrrrr)rerrrr8r8r9rs.     "     zXLNetSequenceSummary.forward)Nrr8r8rfr9rs rc@s&eZdZUeed<eZdZddZdS)XLNetPreTrainedModelr3rc Cst|tjr:|jjjd|jjd|jdur8|jj nt|tj rz|jjjd|jjd|j durx|jj|j  nt|tj r|jj |jj dnpt|tr|j|j|j|j|j|j|j|j|jf D]}|jjd|jjdqn"t|tr|jjjd|jjddS)zInitialize the weights.g)rZstdN?)rCrrrrJZnormal_r3Zinitializer_rangerZzero_ EmbeddingZ padding_idxr`Zfill_rTr&r'r)r%r(r.r0r/r1 XLNetModelr )remoduleparamr8r8r9 _init_weights#s2         z"XLNetPreTrainedModel._init_weightsN) rrrr__annotations__rSZload_tf_weightsZbase_model_prefixrr8r8r8r9rs rz( Output type of [`XLNetModel`]. )Z custom_introc@sfeZdZUdZejed<dZee ejed<dZ ee ejdfed<dZ ee ejdfed<dS)XLNetModelOutputa last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_predict, hidden_size)`): Sequence of hidden-states at the last layer of the model. `num_predict` corresponds to `target_mapping.shape[1]`. If `target_mapping` is `None`, then `num_predict` corresponds to `sequence_length`. mems (`list[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states. Can be used (see `mems` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. last_hidden_stateNr.r attentions) rrrrrIr_rrrrDrtuplerr8r8r8r9rCs  rz. Output type of [`XLNetLMHeadModel`]. c@seZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejdfed<dZee ejdfed<dS) XLNetLMHeadModelOutputaB loss (`torch.FloatTensor` of shape *(1,)*, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, num_predict, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). `num_predict` corresponds to `target_mapping.shape[1]`. If `target_mapping` is `None`, then `num_predict` corresponds to `sequence_length`. mems (`list[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states. Can be used (see `mems` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. Nlosslogitsr.rrrrrrrrrIr_rrrrDrrrr8r8r8r9r\s rz< Output type of [`XLNetForSequenceClassification`]. c@seZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejdfed<dZee ejdfed<dS) $XLNetForSequenceClassificationOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `label` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). mems (`list[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states. Can be used (see `mems` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. Nrrr.rrrr8r8r8r9rxs  rz? Output type of [`XLNetForTokenClassificationOutput`]. c@seZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejdfed<dZee ejdfed<dS) !XLNetForTokenClassificationOutputaO loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_labels)`): Classification scores (before SoftMax). mems (`list[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states. Can be used (see `mems` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. Nrrr.rrrr8r8r8r9rs  rz4 Output type of [`XLNetForMultipleChoice`]. c@seZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejdfed<dZee ejdfed<dS) XLNetForMultipleChoiceOutputa loss (`torch.FloatTensor` of shape *(1,)*, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, num_choices)`): *num_choices* is the second dimension of the input tensors. (see *input_ids* above). Classification scores (before SoftMax). mems (`list[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states. Can be used (see `mems` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. Nrrr.rrrr8r8r8r9rs  rz= Output type of [`XLNetForQuestionAnsweringSimple`]. c@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejed<dZ eeejdfed<dZeeejdfed <dS) %XLNetForQuestionAnsweringSimpleOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. start_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length,)`): Span-start scores (before SoftMax). end_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length,)`): Span-end scores (before SoftMax). mems (`list[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states. Can be used (see `mems` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. Nr start_logits end_logitsr.rr)rrrrrrrIr_rrrrrDrrrr8r8r8r9rs  rz7 Output type of [`XLNetForQuestionAnswering`]. c@seZdZUdZdZeejed<dZ eejed<dZ eej ed<dZ eejed<dZ eej ed<dZeejed<dZeeejed <dZeeejd fed <dZeeejd fed <dS) XLNetForQuestionAnsweringOutputax loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned if both `start_positions` and `end_positions` are provided): Classification loss as the sum of start token, end token (and is_impossible if provided) classification losses. start_top_log_probs (`torch.FloatTensor` of shape `(batch_size, config.start_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the top config.start_n_top start token possibilities (beam-search). start_top_index (`torch.LongTensor` of shape `(batch_size, config.start_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Indices for the top config.start_n_top start token possibilities (beam-search). end_top_log_probs (`torch.FloatTensor` of shape `(batch_size, config.start_n_top * config.end_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the top `config.start_n_top * config.end_n_top` end token possibilities (beam-search). end_top_index (`torch.LongTensor` of shape `(batch_size, config.start_n_top * config.end_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Indices for the top `config.start_n_top * config.end_n_top` end token possibilities (beam-search). cls_logits (`torch.FloatTensor` of shape `(batch_size,)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the `is_impossible` label of the answers. mems (`list[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states. Can be used (see `mems` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. Nrstart_top_log_probsstart_top_indexend_top_log_probs end_top_index cls_logitsr.rr)rrrrrrrIr_rrrrrrrrrDrrrr8r8r8r9rs rcseZdZfddZddZddZddZd d Zd d Ze dddZ dddZ e de eje eje eje eje eje eje eje eje eje ee ee ee eeeefdddZZS)rcstj|_j|_j|_j|_j|_j|_j|_j |_ t j j|_ t tddj|_t fddtj D|_t j|_|dS)Nrcsg|] }tqSr8)r).0_rr8r9 z'XLNetModel.__init__..)rYrZmem_len reuse_lenr[ same_length attn_typebi_data clamp_lenn_layerrr vocab_sizerr^rIr_r Z ModuleListranger"rbrc post_initrdrfrr9rZ s  zXLNetModel.__init__cCs|jSrhrrer8r8r9get_input_embeddingsszXLNetModel.get_input_embeddingscCs ||_dSrhrreZnew_embeddingsr8r8r9set_input_embeddings!szXLNetModel.set_input_embeddingscCstdSrhri)reZheads_to_pruner8r8r9 _prune_heads$szXLNetModel._prune_headscCsvtj|||f|jd}|jrd|ddd|fd}||d|ddd|f|7<n||d|S)aD Creates causal attention mask. Float mask where 1.0 indicates masked, 0.0 indicates not-masked. Args: qlen: Sequence length mlen: Mask length :: same_length=False: same_length=True: < qlen > < qlen > ^ [0 0 0 0 0 1 1 1 1] [0 0 0 0 0 1 1 1 1] [0 0 0 0 0 0 1 1 1] [1 0 0 0 0 0 1 1 1] qlen [0 0 0 0 0 0 0 1 1] [1 1 0 0 0 0 0 1 1] [0 0 0 0 0 0 0 0 1] [1 1 1 0 0 0 0 0 1] v [0 0 0 0 0 0 0 0 0] [1 1 1 1 0 0 0 0 0] )rnNrlr)rIZonesrnrZtrilZtriu_)reqlenmlenmaskZmask_lor8r8r9 create_mask'szXLNetModel.create_maskcCs||jdur"|jdkr"|d|j}|jdus6|jdkridrlr{)rIr}rsincosr)Zpos_seqinv_freqbszZ sinusoid_inppos_embr8r8r9positional_embeddingXs zXLNetModel.positional_embeddingc Cs~tjd|jdtjd}dtd||j}|jdkrH|| }}n(|jdkr^|d}}ntd |jd |jr6tj||d tjd}tj| | d tjd} |j dkr| |j |j }| |j |j } |dur | |||d } | | ||d } n| ||} | | |} tj | | gdd} nDtj||d tjd}|j dkrl| |j |j }| |||} | S)Nrg@rri'biunirlzUnknown `attn_type` .grrVr{) rIrrr[Zint64floatpowrr\rrclamprr) rerrvrZfreq_seqrbegendZ fwd_pos_seqZ bwd_pos_seqZ fwd_pos_embZ bwd_pos_embrr8r8r9relative_positional_encodingcs0        z'XLNetModel.relative_positional_encoding) input_idsattention_maskr perm_maskrtoken_type_ids input_maskr inputs_embedsuse_memsroutput_hidden_states return_dictrc( Ks:| dur | n|jj} | dur | n|jj} | dur4| n|jj} d|vrXtdt|d} |jrt| durj| n|jj} n| dur| n|jj } |dur| durt dnj|dur| dd }|j d|j d}}n:| dur| dd } | j d| j d}}nt d|dur&| dd nd}|durD| dd nd}|durb| dd nd}|dur|ddd nd}|dur|ddd nd}|dur|ddur|dj dnd}||}|j}|j}|jd kr|||}|ddddddf}n"|jd kr(d}nt d |j|dusT|dusTJd |durp|durpd |}|dur|dur|d|}n<|dur|dur|d}n|dur|dur|}nd}|durX|dkrt|j d||g|}tj||gdd}|dur8|dddddddf}n ||dddddddf7}|durp|dk|}|durt|| }|dkrtjt||g||gdd}||ddddddfdk|}nd}| dur| }n ||}||}|dur4|j|j d|d}||}nd}|dur|dkrvtj||gtj|d}tj||gdd}n|}|dddf|dddfk}tjj|dd|}nd}|j |||d} | |j} || } |durf|!dkr*|"d"d"d"d}||j#dddd}n$|!dkrN|"d"d"d}|jt$|%jd}n dg|j#}d}!|durdgt&|j'}| rgnd}"| rgnd}#t(|j'D]\}$}%| r|!|)|||$f}!| r|#*|dur||fn||%||||| |||$|||$| d }&|&dd\}}| r|"*|&dq| rj|#*|durd||fn|||dur||n|}'|'ddd }'| sd}!| r|durt+dd|#D}#nt+dd|#D}#| r |durt+dd|"D}"nt+dd|"D}"| s*t+dd|'|!|#|"fDSt,|'|!|#|"dS) mems (`list[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states (see `mems` output below) . Can be used to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. `use_mems` has to be set to `True` to make use of `mems`. perm_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length, sequence_length)`, *optional*): Mask to indicate the attention pattern for each input token with values selected in `[0, 1]`: - if `perm_mask[k, i, j] = 0`, i attend to j in batch k; - if `perm_mask[k, i, j] = 1`, i does not attend to j in batch k. If not set, each token attends to all the others (full bidirectional attention). Only used during pretraining (to define factorization order) or for sequential decoding (generation). target_mapping (`torch.FloatTensor` of shape `(batch_size, num_predict, sequence_length)`, *optional*): Mask to indicate the output tokens to use. If `target_mapping[k, i, j] = 1`, the i-th predict in batch k is on the j-th token. Only used during pretraining for partial prediction or for sequential decoding (generation). input_mask (`torch.FloatTensor` of shape `batch_size, sequence_length`, *optional*): Mask to avoid performing attention on padding token indices. Negative of `attention_mask`, i.e. with 0 for real tokens and 1 for padding which is kept for compatibility with the original code base. Mask values selected in `[0, 1]`: - 1 for tokens that are **masked**, - 0 for tokens that are **not masked**. You can only uses one of `input_mask` and `attention_mask`. use_mems (`bool`, *optional*): Whether to use memory states to speed up sequential decoding. If set to `True`, the model will use the hidden states from previous forward passes to compute attention, which can significantly improve performance for sequential decoding tasks. NZ use_cachezgThe `use_cache` argument is deprecated and will be removed in a future version, use `use_mems` instead.zDYou cannot specify both input_ids and inputs_embeds at the same timerrz5You have to specify either input_ids or inputs_embedsrVrrzUnsupported attention type: z8You can only use one of input_mask (uses 1 for padding) rr{rlrorn)r)rrr8)rrr(rrrrrcss*|]"}|D]}|dddVq qdSrrrVNpermute contiguous)rhsrr8r8r9 vrz%XLNetModel.forward..css |]}|dddVqdSr'r()rr+r8r8r9r,xrcss |]}tdd|DVqdS)css"|]}|ddddVqdSrVr rrNr()rZ att_streamr8r8r9r,~rz/XLNetModel.forward...N)rrtr8r8r9r,}scss"|]}|ddddVqdSr-r(r.r8r8r9r,rcss|]}|dur|VqdSrhr8)rr)r8r8r9r,r)rrrr)-r3rr#use_return_dictwarningswarn FutureWarningZtrainingZuse_mems_trainZ use_mems_evalr\rBr*rFr)rornrr rIzerostoreyerrcr rrsrrZone_hotrr|rrnext parametersrEr"r!r r-rr)(rerrrrrrr rr!r"rr#r$kwargsrrrrvZ dtype_floatrnrZ data_maskZ mems_maskZ non_tgt_maskZ word_emb_krZ word_emb_qrZmem_padZcat_idsrrZnew_memsrrr6Z layer_modulerrr8r8r9rs4    *           "(       $          zXLNetModel.forward)N)N) NNNNNNNNNNNNN)rrrrZrrrr r rrrrrrITensorboolrrrrrr8r8rfr9rsN  & rzt XLNet Model with a language modeling head on top (linear layer with weights tied to the input embeddings). cseZdZdgZfddZddZddZdd d Zede e j e e j e e j e e j e e j e e j e e j e e j e e j e e j e e e e e e e e e eefd d d Zeee j e j ee j dddZZS)XLNetLMHeadModelzlm_loss.weightcsHt||j|_|j|_t||_tj|j|j dd|_ | dS)NTr) rYrZrrrrrrr[rrrrdrfr8r9rZs   zXLNetLMHeadModel.__init__cCs|jSrhrrr8r8r9get_output_embeddingssz&XLNetLMHeadModel.get_output_embeddingscCs ||_dSrhr=rr8r8r9set_output_embeddingssz&XLNetLMHeadModel.set_output_embeddingsNc  s|jd}tj|dftj|jd}d|rPtj|dd df|gdd}ntj||gdd}|jd}tj|||ftj|jd}d|dddddf<tj|d|ftj|jd} d| ddddf<||| |d} |rtfd d |D| d <| S) Nrrr&rVr{rrl)rrrr"c3s*|]"}|d ddddfVqdSrhr8rZ layer_pastoffsetr8r9r,rzAXLNetLMHeadModel.prepare_inputs_for_generation..r)rFrIr4rsrnrrr) rerZpast_key_valuesr"r9Zeffective_batch_sizeZ dummy_tokenZsequence_lengthrrinputsr8rAr9prepare_inputs_for_generations. & z.XLNetLMHeadModel.prepare_inputs_for_generationrrrrrrr rr!labelsr"rr#r$rcKs|dur |n|jj}|j|f|||||||| | | | |d |}||d}d}| dur~t}||d|d| d}|s|f|dd}|dur|f|S|St|||j|j |j dS)a mems (`list[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states (see `mems` output below) . Can be used to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. `use_mems` has to be set to `True` to make use of `mems`. perm_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length, sequence_length)`, *optional*): Mask to indicate the attention pattern for each input token with values selected in `[0, 1]`: - if `perm_mask[k, i, j] = 0`, i attend to j in batch k; - if `perm_mask[k, i, j] = 1`, i does not attend to j in batch k. If not set, each token attends to all the others (full bidirectional attention). Only used during pretraining (to define factorization order) or for sequential decoding (generation). target_mapping (`torch.FloatTensor` of shape `(batch_size, num_predict, sequence_length)`, *optional*): Mask to indicate the output tokens to use. If `target_mapping[k, i, j] = 1`, the i-th predict in batch k is on the j-th token. Only used during pretraining for partial prediction or for sequential decoding (generation). input_mask (`torch.FloatTensor` of shape `batch_size, sequence_length`, *optional*): Mask to avoid performing attention on padding token indices. Negative of `attention_mask`, i.e. with 0 for real tokens and 1 for padding which is kept for compatibility with the original code base. Mask values selected in `[0, 1]`: - 1 for tokens that are **masked**, - 0 for tokens that are **not masked**. You can only uses one of `input_mask` and `attention_mask`. labels (`torch.LongTensor` of shape `(batch_size, num_predict)`, *optional*): Labels for masked language modeling. `num_predict` corresponds to `target_mapping.shape[1]`. If `target_mapping` is `None`, then `num_predict` corresponds to `sequence_length`. The labels should correspond to the masked input words that should be predicted and depends on `target_mapping`. Note in order to perform standard auto-regressive language modeling a ** token has to be added to the `input_ids` (see the `prepare_inputs_for_generation` function and examples below) Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored, the loss is only computed for labels in `[0, ..., config.vocab_size]` use_mems (`bool`, *optional*): Whether to use memory states to speed up sequential decoding. If set to `True`, the model will use the hidden states from previous forward passes to compute attention, which can significantly improve performance for sequential decoding tasks. Examples: ```python >>> from transformers import AutoTokenizer, XLNetLMHeadModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("xlnet/xlnet-large-cased") >>> model = XLNetLMHeadModel.from_pretrained("xlnet/xlnet-large-cased") >>> # We show how to setup inputs to predict a next token using a bi-directional context. >>> input_ids = torch.tensor( ... tokenizer.encode("Hello, my dog is very ", add_special_tokens=False) ... ).unsqueeze( ... 0 ... ) # We will predict the masked token >>> perm_mask = torch.zeros((1, input_ids.shape[1], input_ids.shape[1]), dtype=torch.float) >>> perm_mask[:, :, -1] = 1.0 # Previous tokens don't see last token >>> target_mapping = torch.zeros( ... (1, 1, input_ids.shape[1]), dtype=torch.float ... ) # Shape [1, 1, seq_length] => let's predict one token >>> target_mapping[ ... 0, 0, -1 ... ] = 1.0 # Our first (and only) prediction will be the last token of the sequence (the masked token) >>> outputs = model(input_ids, perm_mask=perm_mask, target_mapping=target_mapping) >>> next_token_logits = outputs[ ... 0 ... ] # Output has shape [target_mapping.size(0), target_mapping.size(1), config.vocab_size] >>> # The same way can the XLNetLMHeadModel be used to be trained by standard auto-regressive language modeling. >>> input_ids = torch.tensor( ... tokenizer.encode("Hello, my dog is very ", add_special_tokens=False) ... ).unsqueeze( ... 0 ... ) # We will predict the masked token >>> labels = torch.tensor(tokenizer.encode("cute", add_special_tokens=False)).unsqueeze(0) >>> assert labels.shape[0] == 1, "only one word will be predicted" >>> perm_mask = torch.zeros((1, input_ids.shape[1], input_ids.shape[1]), dtype=torch.float) >>> perm_mask[ ... :, :, -1 ... ] = 1.0 # Previous tokens don't see last token as is done in standard auto-regressive lm training >>> target_mapping = torch.zeros( ... (1, 1, input_ids.shape[1]), dtype=torch.float ... ) # Shape [1, 1, seq_length] => let's predict one token >>> target_mapping[ ... 0, 0, -1 ... ] = 1.0 # Our first (and only) prediction will be the last token of the sequence (the masked token) >>> outputs = model(input_ids, perm_mask=perm_mask, target_mapping=target_mapping, labels=labels) >>> loss = outputs.loss >>> next_token_logits = ( ... outputs.logits ... ) # Logits have shape [target_mapping.size(0), target_mapping.size(1), config.vocab_size] ```N rrrrrr rr!r"rr#r$rrlrrrrrr) r3r0rrrviewrrrrr)rerrrrrrr rr!rFr"rr#r$r9transformer_outputsrrloss_fctrr8r8r9rsDuzXLNetLMHeadModel.forward)rbeam_idxrcsfdd|DS)z This function is used to re-order the `mems` cache if [`~PreTrainedModel.beam_search`] or [`~PreTrainedModel.beam_sample`] is called. This is required to match `mems` with the correct beam_idx at every generation step. cs g|]}|d|jqS)r)rqr5rnr@rLr8r9rsrz3XLNetLMHeadModel._reorder_cache..r8)rrLr8rMr9_reorder_cachelszXLNetLMHeadModel._reorder_cache)NN)NNNNNNNNNNNNNN)rrrZ_tied_weights_keysrZr>r?rDrrrIr:r;rrrrrrDrNrr8r8rfr9r<sP  , r<z XLNet Model with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cseZdZfddZedeejeejeejeejeejeejeejeejeejeejeeeeeeeee e e fdddZ Z S)XLNetForSequenceClassificationcsLt||j|_||_t||_t||_t |j |j|_ | dSrh) rYrZrr3rrrrrrr[rrrdrfr8r9rZ}s   z'XLNetForSequenceClassification.__init__NrEcKs|dur |n|jj}|j|f|||||||| | | | |d |}|d}||}||}d}| durH|jjdur|jdkrd|j_n4|jdkr| jtj ks| jtj krd|j_nd|j_|jjdkrt }|jdkr|| | }n ||| }nN|jjdkr*t }||d|j| d}n|jjdkrHt}||| }|sx|f|dd}|durt|f|S|St|||j|j|jd S) a. mems (`list[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states (see `mems` output below) . Can be used to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. `use_mems` has to be set to `True` to make use of `mems`. perm_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length, sequence_length)`, *optional*): Mask to indicate the attention pattern for each input token with values selected in `[0, 1]`: - if `perm_mask[k, i, j] = 0`, i attend to j in batch k; - if `perm_mask[k, i, j] = 1`, i does not attend to j in batch k. If not set, each token attends to all the others (full bidirectional attention). Only used during pretraining (to define factorization order) or for sequential decoding (generation). target_mapping (`torch.FloatTensor` of shape `(batch_size, num_predict, sequence_length)`, *optional*): Mask to indicate the output tokens to use. If `target_mapping[k, i, j] = 1`, the i-th predict in batch k is on the j-th token. Only used during pretraining for partial prediction or for sequential decoding (generation). input_mask (`torch.FloatTensor` of shape `batch_size, sequence_length`, *optional*): Mask to avoid performing attention on padding token indices. Negative of `attention_mask`, i.e. with 0 for real tokens and 1 for padding which is kept for compatibility with the original code base. Mask values selected in `[0, 1]`: - 1 for tokens that are **masked**, - 0 for tokens that are **not masked**. You can only uses one of `input_mask` and `attention_mask`. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). use_mems (`bool`, *optional*): Whether to use memory states to speed up sequential decoding. If set to `True`, the model will use the hidden states from previous forward passes to compute attention, which can significantly improve performance for sequential decoding tasks. NrGrrZ regressionZsingle_label_classificationZmulti_label_classificationrlrH)r3r0rrrZ problem_typerrorIrsintr rrrIrrrrr)rerrrrrrr rr!rFr"rr#r$r9rJrrrrKr8r8r9rsf9      "     z&XLNetForSequenceClassification.forward)NNNNNNNNNNNNNN)rrrrZrrrIr:r;rrrrrr8r8rfr9rOvsB  rOcseZdZfddZedeejeejeejeejeejeejeejeejeejeejeeeeeeeee e e fdddZ Z S)XLNetForTokenClassificationcs<t||j|_t||_t|j|j|_| dSrh) rYrZrrrrrr classifierrrdrfr8r9rZs   z$XLNetForTokenClassification.__init__NrEcKs|dur |n|jj}|j||||||||| | | | |d }|d}||}d}| durvt}||d|j| d}|s|f|dd}|dur|f|S|St|||j|j |j dS)a mems (`list[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states (see `mems` output below) . Can be used to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. `use_mems` has to be set to `True` to make use of `mems`. perm_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length, sequence_length)`, *optional*): Mask to indicate the attention pattern for each input token with values selected in `[0, 1]`: - if `perm_mask[k, i, j] = 0`, i attend to j in batch k; - if `perm_mask[k, i, j] = 1`, i does not attend to j in batch k. If not set, each token attends to all the others (full bidirectional attention). Only used during pretraining (to define factorization order) or for sequential decoding (generation). target_mapping (`torch.FloatTensor` of shape `(batch_size, num_predict, sequence_length)`, *optional*): Mask to indicate the output tokens to use. If `target_mapping[k, i, j] = 1`, the i-th predict in batch k is on the j-th token. Only used during pretraining for partial prediction or for sequential decoding (generation). input_mask (`torch.FloatTensor` of shape `batch_size, sequence_length`, *optional*): Mask to avoid performing attention on padding token indices. Negative of `attention_mask`, i.e. with 0 for real tokens and 1 for padding which is kept for compatibility with the original code base. Mask values selected in `[0, 1]`: - 1 for tokens that are **masked**, - 0 for tokens that are **not masked**. You can only uses one of `input_mask` and `attention_mask`. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices]` where *num_choices* is the size of the second dimension of the input tensors. (see *input_ids* above) use_mems (`bool`, *optional*): Whether to use memory states to speed up sequential decoding. If set to `True`, the model will use the hidden states from previous forward passes to compute attention, which can significantly improve performance for sequential decoding tasks.emory states to speed up sequential decoding. If set to `True`, the model will use the hidden states from previous forward passes to compute attention, which can significantly improve performance for sequential decoding tasks. NrGrrlrrH) r3r0rrRrrIrrrrr)rerrrrrrr rr!rFr"rr#r$r9rsequence_outputrrrKrr8r8r9r s@: z#XLNetForTokenClassification.forward)NNNNNNNNNNNNNN)rrrrZrrrIr:r;rrrrrr8r8rfr9rQsB  rQcseZdZfddZedeejeejeejeejeejeejeejeejeejeejeeeeeeeee e e fdddZ Z S)XLNetForMultipleChoicecs<t|t||_t||_t|jd|_ | dSr) rYrZrrrrrrr[rrrdrfr8r9rZms    zXLNetForMultipleChoice.__init__N)rrr rrrrrr!rFr"rr#r$rcKs|dur |n|jj}|dur&|jdn| jd}|durJ|d|dnd}|durh|d|dnd}|dur|d|dnd}|dur|d|dnd}| dur| d| d| dnd}|j|f||||||||| | | |d |}|d}||}||}|d|}d}| durFt}||| d}|sv|f|dd}|durr|f|S|St |||j |j |j dS)a& input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) input_mask (`torch.FloatTensor` of shape `batch_size, num_choices, sequence_length`, *optional*): Mask to avoid performing attention on padding token indices. Negative of `attention_mask`, i.e. with 0 for real tokens and 1 for padding which is kept for compatibility with the original code base. Mask values selected in `[0, 1]`: - 1 for tokens that are **masked**, - 0 for tokens that are **not masked**. You can only uses one of `input_mask` and `attention_mask`. mems (`list[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states (see `mems` output below) . Can be used to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. `use_mems` has to be set to `True` to make use of `mems`. perm_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length, sequence_length)`, *optional*): Mask to indicate the attention pattern for each input token with values selected in `[0, 1]`: - if `perm_mask[k, i, j] = 0`, i attend to j in batch k; - if `perm_mask[k, i, j] = 1`, i does not attend to j in batch k. If not set, each token attends to all the others (full bidirectional attention). Only used during pretraining (to define factorization order) or for sequential decoding (generation). target_mapping (`torch.FloatTensor` of shape `(batch_size, num_predict, sequence_length)`, *optional*): Mask to indicate the output tokens to use. If `target_mapping[k, i, j] = 1`, the i-th predict in batch k is on the j-th token. Only used during pretraining for partial prediction or for sequential decoding (generation). inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., use_mems (`bool`, *optional*): Whether to use memory states to speed up sequential decoding. If set to `True`, the model will use the hidden states from previous forward passes to compute attention, which can significantly improve performance for sequential decoding tasks. Nrrlr) rr rrrrrr!r"rr#r$rrH) r3r0rFrIrrrrrrrrr)rerrr rrrrrr!rFr"rr#r$r9Z num_choicesZflat_input_idsZflat_token_type_idsZflat_attention_maskZflat_input_maskZflat_inputs_embedsrJrrZreshaped_logitsrrKr8r8r9rws\J    zXLNetForMultipleChoice.forward)NNNNNNNNNNNNNN)rrrrZrrrIr:r;rrrrrr8r8rfr9rTksB  rTz XLNet Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cseZdZfddZedeejeejeejeejeejeejeejeejeejeejeejeeeeeeeee e e fdddZ Z S)XLNetForQuestionAnsweringSimplecs<t||j|_t||_t|j|j|_| dSrh) rYrZrrrrrr qa_outputsrrdrfr8r9rZs   z(XLNetForQuestionAnsweringSimple.__init__N)rrrrrrr rr!r end_positionsr"rr#r$rcKsd|dur |n|jj}|j|f|||||||| | | ||d |}|d}||}|jddd\}}|d}|d}d}| dur| durt| dkr| d} t| dkr| d} |d}| d|} | d|} t |d}||| }||| }||d}|sJ||f|dd}|durF|f|S|St ||||j |j |jd S) r%NrGrrrlr{)Z ignore_indexrV)rrrrrr)r3r0rrVsplitrr*rErrrrrrr)rerrrrrrr rr!rrWr"rr#r$r9rrSrrr total_lossZ ignored_indexrK start_lossend_lossrr8r8r9r s`6          z'XLNetForQuestionAnsweringSimple.forward)NNNNNNNNNNNNNNN)rrrrZrrrIr:r;rrrrrr8r8rfr9rUsF  rUcseZdZfddZedeejeejeejeejeejeejeejeejeejeejeejeejeejeejeeeeeeeee e e fdddZ Z S)XLNetForQuestionAnsweringcsPt||j|_|j|_t||_t||_t||_ t ||_ | dSrh) rYrZ start_n_top end_n_toprrrrrrr answer_classrrdrfr8r9rZzs     z"XLNetForQuestionAnswering.__init__N)rrrrrrr rr!rrW is_impossiblerrr"rr#r$rc-Ks|dur |n|jj}|j|f|||||||| ||||d |}|d}|j||d}|dd}| durP| durP| | | | fD]"}|dur|dkr|dq|j|| |d}t}||| }||| }||d}| dur| dur|j|| | d }t }||| } || d 7}|s6|f|ddSt ||j |j |jd SnR|\}!}"}#t jj|dd }$tj|$|jdd \}%}&|&ddd|#}'t|d |'}(|(dd|"dd}(|d|(})|dur|dnd}|j|)|(|d}t jj|dd }*tj|*|jdd \}+},|+d|j|j}+|,d|j|j},td||$}(|j||(| d}|s|%|&|+|,|f}||ddSt |%|&|+|,||j |j |jdSdS)a mems (`list[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states (see `mems` output below) . Can be used to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. `use_mems` has to be set to `True` to make use of `mems`. perm_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length, sequence_length)`, *optional*): Mask to indicate the attention pattern for each input token with values selected in `[0, 1]`: - if `perm_mask[k, i, j] = 0`, i attend to j in batch k; - if `perm_mask[k, i, j] = 1`, i does not attend to j in batch k. If not set, each token attends to all the others (full bidirectional attention). Only used during pretraining (to define factorization order) or for sequential decoding (generation). target_mapping (`torch.FloatTensor` of shape `(batch_size, num_predict, sequence_length)`, *optional*): Mask to indicate the output tokens to use. If `target_mapping[k, i, j] = 1`, the i-th predict in batch k is on the j-th token. Only used during pretraining for partial prediction or for sequential decoding (generation). input_mask (`torch.FloatTensor` of shape `batch_size, sequence_length`, *optional*): Mask to avoid performing attention on padding token indices. Negative of `attention_mask`, i.e. with 0 for real tokens and 1 for padding which is kept for compatibility with the original code base. Mask values selected in `[0, 1]`: - 1 for tokens that are **masked**, - 0 for tokens that are **not masked**. You can only uses one of `input_mask` and `attention_mask`. is_impossible (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels whether a question has an answer or no answer (SQuAD 2.0) cls_index (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the classification token to use as input for computing plausibility of the answer. p_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Optional mask of tokens which can't be in answers (e.g. [CLS], [PAD], ...). 1.0 means token should be masked. 0.0 mean token is not masked. use_mems (`bool`, *optional*): Whether to use memory states to speed up sequential decoding. If set to `True`, the model will use the hidden states from previous forward passes to compute attention, which can significantly improve performance for sequential decoding tasks. Example: ```python >>> from transformers import AutoTokenizer, XLNetForQuestionAnswering >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("xlnet/xlnet-base-cased") >>> model = XLNetForQuestionAnswering.from_pretrained("xlnet/xlnet-base-cased") >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze( ... 0 ... ) # Batch size 1 >>> start_positions = torch.tensor([1]) >>> end_positions = torch.tensor([3]) >>> outputs = model(input_ids, start_positions=start_positions, end_positions=end_positions) >>> loss = outputs.loss ```NrGr)rrrl)rrrV)rrrU)rrrrr{r)rrz blh,bl->bh)rr)rrrrrrrr)r3r0rrr|Zsqueeze_rrr_rrrrrrrrrrIZtopkr]rrrZ expand_asr^rIr})-rerrrrrrr rr!rrWr`rrr"rr#r$r9rJrrrrurrKrZr[rYrZ loss_fct_clsZcls_lossrrrZstart_log_probsrrZstart_top_index_exprZhidden_states_expandedZ end_log_probsrrr8r8r9rsS           z!XLNetForQuestionAnswering.forward)NNNNNNNNNNNNNNNNNN)rrrrZrrrIr:r;rrrrrr8r8rfr9r\xsR  r\) rTr\rUrOrQr<rrrS)N):rr1 dataclassesrtypingrrrrIrZtorch.nnrrr Z activationsr r Z generationr Zmodeling_utilsrZ pytorch_utilsrutilsrrrZconfiguration_xlnetrZ get_loggerrr=r:rSModulerTrrrrrrrrrrrrrrrr<rOrQrTrUr\__all__r8r8r8r9s        SC3'FBc%!gm zJ