a h @sdZddlmZmZmZddlZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZmZdd lmZdd lmZdd lmZdd lmZdd lmZmZmZmZddlmZm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,e'rddl-m.Z.ddl/m0Z0e(1e2Z3Gdddej4Z5d/ej6ej7ej7ej7eej7e8e8dddZ9Gdddej6Z:GdddeZ;e%Gd d!d!e ZGd&d'd'er-r-r+r.rUhs*#  rUcseZdZd eeedfdd Zedddddej eej eej ee ej ee ee eej eej e ee ejee ejejffd d d ZZS)OPTDecoderLayerNrVcst|j|_t||d|_|j|_|j|_t|j |_ t j |j|j d|_t j|j|j|jd|_t j|j|j|jd|_t j |j|j d|_dS)NrVZelementwise_affinerY)r(r)r[r\rU self_attndo_layer_norm_beforerEr Zactivation_function activation_fnr LayerNormlayer_norm_elementwise_affineself_attn_layer_normrcZffn_dimr^fc1fc2final_layer_norm)r*rWrXr+r-r.r)s  zOPTDecoderLayer.__init__rhrirjrkF) rnr/rorirp use_cacher1rqrQrrc  Ks|} |jr||}|jf|||||||d| \}} tjj||j|jd}| |}|jsf||}|j} |d| d}|} |jr| |}| |}| |}| |}tjj||j|jd}| || }|js| |}|f} |r| | f7} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`, *optional*): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence.. )rnrir1r/rorprqrIrF)rrrrrNrErKshaperzrvrrrrrw)r*rnr/rorirprr1rqrQZresidualZself_attn_weightsZhidden_states_shapeoutputsr-r-r.r7sD         zOPTDecoderLayer.forward)N)NNNFFNN)r8r9r:rrr<r)rr4r|r}r~r=rr FloatTensorr7r>r-r-r+r.rs*  rc@s@eZdZUeed<dZdZdgZdZdZ dZ dZ dZ ddZ dS)OPTPreTrainedModelrWmodelTrcCs|jj}t|tjr>|jjjd|d|jdur|jj nbt|tj rz|jjjd|d|j dur|jj|j  n&t|tj r|jj d|jj dS)Nr?)meanstdru)rWZinit_std isinstancerrcweightdataZnormal_rZZzero_ Embedding padding_idxrZfill_)r*r@rr-r-r. _init_weightsAs     z OPTPreTrainedModel._init_weightsN)r8r9r:r__annotations__Zbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_supports_attention_backendZ_supports_flash_attnZ_supports_sdpaZ_supports_flex_attnZ_can_compile_fullgraphrr-r-r-r.r4s rcseZdZdZedfdd Zdeejdfejeje e ddd Z e eje e ejeje d d d Zedeejeejeejee eejee ee ee ee eejeejeeeeefd ddZZS) OPTDecoderz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`OPTDecoderLayer`] Args: config: OPTConfig rWcstj|_j|_j|_j|_j|_t jj |j|_ t jj|_j jkr~t jjj dd|_nd|_j jkrt jj jdd|_nd|_jrЈjst jjjd|_nd|_t fddtjD|_d|_|dS)NFrYrcsg|]}t|dqS))rX)r).0irr-r. wz'OPTDecoder.__init__..)r(r)rE layerdrop pad_token_idrZmax_position_embeddingsZmax_target_positions vocab_sizerrword_embed_proj_dim embed_tokensr"r[embed_positionsrc project_out project_inrZ_remove_final_layer_normrrrZ ModuleListrangeZnum_hidden_layerslayersgradient_checkpointing post_initr*rWr+rr.r)Xs,      zOPTDecoder.__init__Fr )r/ input_tensorrqrirpc CsB|jjdkr(|dur$|dkr$|SdS|jjdkrLt|tjrHt|}|S|dur\|nd}|durn|jnd}|jjdkr|s|st j ||||j drdS|j }|j d} |r|} n"t|tjr|j d n || d} |j|| | |||j dd } |jjdkr>|dur>|jjd vr>|s>t|j} t | | } | S) NZflash_attention_2r?Zflex_attentionrFrt) inputs_embedsr0Z is_trainingrrF)sequence_length target_lengthrHrq batch_size)cudaZxpuZnpu)rWryanyrr4r|r!get_seq_lengthZis_compileablerZ_ignore_causal_mask_sdparKrHrZget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positiondevicetypefinfominZ_unmask_unattended) r*r/rrqrirppast_seen_tokensZusing_compilable_cacherHrr causal_mask min_dtyper-r-r._update_causal_mask~sZ           zOPTDecoder._update_causal_mask)r/rrrHrqrc KsF|dur|dkr|}n&t|j}tj||f|||jd}|dkrVtj|dd}|tj||jd|ddk9}|ddddddf |ddd}|durB| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N)Z fill_valuerHrr)ZdiagonalrrFr)r3r4rrfullrZtriuarangerzexpandclonerrOZ masked_fill) r/rrrHrqrrQrrZ mask_lengthZ padding_maskr-r-r.rs*  $  6  z@OPTDecoder._prepare_4d_causal_attention_mask_with_cache_positionN input_idsr/ head_maskrirrrpoutput_hidden_states return_dictr1rqrQrrc  Ks|dur |n|jj}|dur |n|jj}|dur4|n|jj}| durH| n|jj} |du|duArhtd|jr|jr|rt dd}|dur| d|j d}|dur| |}|r|durt |jd}|dur|nd} | durtj| | |j d|jd } |dur4| |j d}tj|j d||jd }|||| ||}| durtj|dd } | |d} | dd| df} |j|| | d }|jdur||}|||j}|rd nd}|rd nd}t|gd gD]V\}}|dur|dt|jkrtd|dt|jd|ddqt|jD]\}}|r\||f7}|jr~tg}||jkr~qD||f|| |dur||nd|||| d| }|d}|rD||df7}qD|j dur| |}|j!dur|!|}|r ||f7}t"||||dS)a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(num_hidden_layers, num_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. for padding use -1. [What are position IDs?](../glossary#position-ids) cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FrFrrrrr2)r1r-rzThe `z` should be specified for z layers, but it is for .)r/r1rorirprrqlast_hidden_staterirn attentions)#rWrprruse_return_dictrbrrKr_r`rwrrr rr4rrZonesrr5r6rrrOziprvlenr enumerateZrandrrrr)r*rr/rrirrrprrr1rqrQrZ seq_lengthrZ pos_embedsrnZall_hidden_statesZall_self_attnsZ attn_maskZ mask_nameidxZ decoder_layerZdropout_probabilityZ layer_outputsr-r-r.r7sH                     zOPTDecoder.forward)F) NNNNNNNNNNN)r8r9r:r;rr)rr4r|r r~r staticmethodr<rHrrrr=rrrr}rr7r>r-r-r+r.rPsZ,  D6 rcseZdZedfdd ZddZddZeed e e j e e j e e j e e ee jefe e je ee ee ee ee e j e e j eee eefd d d ZZS) OPTModelrcs"t|t||_|dSN)r(r)rdecoderrrr+r-r.r)s  zOPTModel.__init__cCs|jjSrrrr*r-r-r.get_input_embeddingsszOPTModel.get_input_embeddingscCs ||j_dSrrr*rCr-r-r.set_input_embeddingsszOPTModel.set_input_embeddingsNrc Ks|dur |n|jj}|dur |n|jj}|dur4|n|jj}| durH| n|jj} |jf||| ||||||d| d | } t| j| j| j | j dS)NT rr/r1rrirrrprrrqr) rWrprrrrrrrirnr)r*rr/rrirrrprrr1rqrQZdecoder_outputsr-r-r.r7s4 zOPTModel.forward) NNNNNNNNNNN)r8r9r:rr)rrrrrr4r=r|rlistrr r~rrr}rr7r>r-r-r+r.rs> rcseZdZdgZfddZddZddZdd Zd d Ze e de e j e e je e je eee jefe e je e j e ee ee ee ee e j e e jeeeeefd ddZZS)OPTForCausalLMzlm_head.weightcs8t|t||_tj|j|jdd|_| dSNFrY) r(r)rrrrcrrlm_headrrr+r-r.r)s  zOPTForCausalLM.__init__cCs |jjjSrrrrrr-r-r.rsz#OPTForCausalLM.get_input_embeddingscCs||jj_dSrrrr-r-r.rsz#OPTForCausalLM.set_input_embeddingscCs ||j_dSrrr)r*rr-r-r. set_decoderszOPTForCausalLM.set_decodercCs|jjSrrrr-r-r. get_decoderszOPTForCausalLM.get_decoderN)rr/rrirlabelsrrprrr1rqrQrrc Ks|dur |n|jj}| dur | n|jj} | dur4| n|jj} |jjf||| |||||| d| d | }||d}d}|dur||j }|j ||fd|jj i| }t |||j |j|jdS)an labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, OPTForCausalLM >>> model = OPTForCausalLM.from_pretrained("facebook/opt-350m") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious. I'm just a little bit of a weirdo." ```NTrrrlosslogitsrirnr)rWrprrrrrrPrOrZ loss_functionrrrirnr)r*rr/rrirrrrprrr1rqrQrrrr-r-r.r7 sL)  zOPTForCausalLM.forward) NNNNNNNNNNNN)r8r9r:Z_tied_weights_keysr)rrrrrrrr4r=r|rrrr r~rrr}rr7r>r-r-r+r.rsH  ra The OPT Model transformer with a sequence classification head on top (linear layer). [`OPTForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). )Z custom_introcseZdZedfdd Zed eejeej eej ee e ej e feej eejee ee ee ee eeje eefd ddZdd Zd d ZZS) OPTForSequenceClassificationrcs@t||j|_t||_tj|j|jdd|_| dSr) r(r) num_labelsrrrrcrscorerrr+r-r.r)ns   z%OPTForSequenceClassification.__init__N) rr/rrirrrrprrr1rrc  Cs\| dur | n|jj} |j|||| ||||| | d } | d} || }|dur`|jdd\}}n|jdd\}}|jjdur|dkrtd|jjdurd}nb|dur||jjk|jt j }t j |jd|jt j d}|| d}nd}t |jjd |t j ||jd |f}d}|dur|jjdur|jdkrLd |j_n:|jdkr~|jt jkst|jt jkr~d |j_nd |j_|jjd krt}|jdkr|||}n |||}nN|jjd krt}||d|j|d}n|jjd krt}|||}| sD|f| dd}|dur@|f|S|St||| j| j| jdS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N rir/r1rrrrprrrr&rz=Cannot handle batch sizes > 1 if no padding token is defined.rF)rrHz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`rZ regressionZsingle_label_classificationZmulti_label_classificationr)rWrrrrrrbrOrr4Zint32rZargmaxr_r`r,r8Z problem_typerrHr6r<rsqueezerrwrrrirnr)r*rr/rrirrrrprrr1transformer_outputsrnrrrZlast_non_pad_tokenZ non_pad_maskZ token_indicesZ pooled_logitsrloss_fctoutputr-r-r.r7wsx       (    z$OPTForSequenceClassification.forwardcCs |jjjSrrrr-r-r.rsz1OPTForSequenceClassification.get_input_embeddingscCs||jj_dSrrrr-r-r.rsz1OPTForSequenceClassification.set_input_embeddings) NNNNNNNNNNN)r8r9r:rr)rrr4r=rrrr r~r}rr7rrr>r-r-r+r.r_s:  ^rcseZdZedfdd Zed eejeej eej ee e ej e feej eejeejee ee ee ee eeje eefd ddZdd Zd d ZZS) OPTForQuestionAnsweringrcs2t|t||_t|jd|_|dSr%) r(r)rrrrcr qa_outputsrrr+r-r.r)s  z OPTForQuestionAnswering.__init__N) rr/rrirstart_positions end_positionsrrprrr1rrc  Csb| dur | n|jj} |j|||| |||| | | d } | d}||}|jddd\}}|d}|d}d}|dur|durt|dkr|d}t|dkr|d}|d}| d| |j }| d| |j }t |d}|||}|||}||d}| sL||f| dd}|durH|f|S|St |||| j| jd S) a Example: ```python >>> from transformers import AutoTokenizer, OPTForQuestionAnswering >>> import torch >>> torch.manual_seed(4) # doctest: +IGNORE_RESULT >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") >>> # note: we are loading a OPTForQuestionAnswering from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> model = OPTForQuestionAnswering.from_pretrained("facebook/opt-350m") >>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet" >>> inputs = tokenizer(question, text, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> answer_offset = len(tokenizer(question)[0]) >>> predict_answer_tokens = inputs.input_ids[ ... 0, answer_offset + answer_start_index : answer_offset + answer_end_index + 1 ... ] >>> predicted = tokenizer.decode(predict_answer_tokens) >>> predicted ' a nice puppet' ```NrrrrFr2)Z ignore_indexr&)r start_logits end_logitsrnr)rWrrrsplitrrPrrvclamprOrrrrnr)r*rr/rrirrrrrprrr1rrnrrrZ total_lossZ ignored_indexrZ start_lossZend_lossrr-r-r.r7sR0         zOPTForQuestionAnswering.forwardcCs |jjjSrrrr-r-r.rIsz,OPTForQuestionAnswering.get_input_embeddingscCs||jj_dSrrrr-r-r.rLsz,OPTForQuestionAnswering.set_input_embeddings) NNNNNNNNNNNN)r8r9r:rr)rrr4r=rrrr r~r}rr7rrr>r-r-r+r.rs> ar)rrrrr)r?)Cr;typingrrrr4Ztorch.utils.checkpointrZtorch.nnrrrZ activationsr Z cache_utilsr r Z generationr Zmodeling_attn_mask_utilsrZmodeling_flash_attention_utilsrZmodeling_layersrZmodeling_outputsrrrrZmodeling_utilsrrZprocessing_utilsrutilsrrrrrZutils.deprecationrZconfiguration_optrZ!torch.nn.attention.flex_attentionr Zintegrations.flex_attentionr!Z get_loggerr8r_rr"Moduler|floatrTrUrrrrrrr__all__r-r-r-r.sb            $ ffe=npr