a h}@sfdZddlmZmZddlZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZdd lmZmZmZdd lmZdd lmZmZmZdd lmZmZd dlmZee Z!ddZ"ddZ#d'ddZ$Gdddej%Z&ddZ'Gdddej%Z(eGdddeZ)eGddde)Z*edd Gd!d"d"e)eZ+ed#d Gd$d%d%e)Z,gd&Z-dS)(zPyTorch CTRL model.)OptionalUnionN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss) DynamicCache)GenerationMixin)BaseModelOutputWithPastCausalLMOutputWithPastSequenceClassifierOutput)PreTrainedModel)Conv1D find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging) CTRLConfigcCs$dtdd|d|}||S)Nri')torchpow)posi d_model_sizeZ angle_ratesrb/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/ctrl/modeling_ctrl.py angle_defn(srcCsttj|tjd|dtj|tjd|d|}t|dddddf}t|dddddf}tj||gdd}|S)Ndtyperrrdim) rrarangeZint64to unsqueezesincoscat)positionrr Z angle_radsZsinesZcosines pos_encodingrrrpositional_encoding-sr,c Cst||dddd}|jd}|t|}|durn|d|d} } ||| | | d| fd7}|dur~||}tj|dd} |dur| |} t| |} | | fS) Nrrrrr!gr")rmatmulpermuteshapenpsqrtsizeZsoftmax) qkvmaskattention_mask head_maskZ matmul_qkZdkZscaled_attention_logitsndnsZattention_weightsoutputrrrscaled_dot_product_attention<s   r=cs8eZdZd fdd ZddZddZd d d ZZS) MultiHeadAttentionNcspt||_||_||_t||j|_t|||_ t|||_ t|||_ t|||_ t |_dSN)super__init__ num_headsr layer_idxintdepthrLinearWqWkWvdenseset pruned_heads)selfrrBrC __class__rrrAWs zMultiHeadAttention.__init__cCs|j|j}t|dkrdSt||j||j\}}t|j||_t|j||_t|j||_t|j |dd|_ |jt||_||j|_|j ||_dS)Nrrr") rrBlenrrLrrGrHrIrJunion)rMheadsZattention_head_sizeindexrrr prune_headsfs   zMultiHeadAttention.prune_headscCs"||d|j|j}|gdS)Nr!rrrr)reshaperBrEr/)rMx batch_sizerrrsplit_into_headswsz#MultiHeadAttention.split_into_headsFc Cs|jd} ||}||}||}||| }||| }||| }|durn||||jd| i\}}t||||||} | dgd} | d}| | d|j }| |} | |fS)Nrcache_positionrUrr!) r0rGrHrIrYupdaterCr=r/rVrrJ)rMr6r5r4r7 layer_pastr8r9 use_cacheoutput_attentionsrZrXr<Zscaled_attentionZattnZoriginal_size_attentionrrrforward{s        zMultiHeadAttention.forward)N)NNNFFN)__name__ __module__ __qualname__rArTrYr_ __classcell__rrrNrr>Vs r>cCs"tt||tt||Sr?)rZ SequentialrFZReLU)rdffrrrpoint_wise_feed_forward_networksrecs(eZdZdfdd Zd ddZZS) EncoderLayer皙?Ncsbtt|||d|_t|||_tj|dd|_tj|dd|_ t ||_ t ||_ dS)NrCgư>eps) r@rAr>multi_head_attentionreffnr LayerNorm layernorm1 layernorm2Dropoutdropout1dropout2)rMrrBrdZraterCrNrrrAs   zEncoderLayer.__init__Fc  Cs~||} |j| | | |||||||d } | d} || } || } || } || }||}| |} | f| dd}|S)Nr\r8r9r]r^rZrr)rnrkrqrorlrr)rMrWr7r\r8r9r]r^rZnormedZ attn_outputsZ attn_outputZout1Zout2Z ffn_outputoutputsrrrr_s,      zEncoderLayer.forward)rgN)NNNFFN)r`rarbrAr_rcrrrNrrfsrfc@s"eZdZUeed<dZddZdS)CTRLPreTrainedModelconfig transformercCst|tjtfr>|jjjd|jjd|j dur|j j nft|tj r~|jjjd|jjd|j dur|jj|j  n&t|tj r|j j |jjddS)zInitialize the weights.g)meanZstdN?) isinstancerrFrweightdataZnormal_rwZinitializer_rangebiasZzero_ EmbeddingZ padding_idxrmZfill_)rMmodulerrr _init_weightss     z!CTRLPreTrainedModel._init_weightsN)r`rarbr__annotations__Zbase_model_prefixrrrrrrvs rvcseZdZfddZddZddZddZed ee j ee e e j ee j ee j ee j ee j ee j ee ee ee ee ee jee e jefd d d ZZS) CTRLModelcstj|_j|_tj|jtj |_ t j j|_t j|_t fddtjD|_t jjjd|_|dS)Nc s&g|]}tjjjj|dqS)rh)rfn_embdZn_headrdZ resid_pdrop).0rrwrr sz&CTRLModel.__init__..ri)r@rArrn_layerZ num_layersr,Z n_positionsrfloatr+rr vocab_sizewrpZ embd_pdropdropoutZ ModuleListrangehrmZlayer_norm_epsilon layernorm post_initrMrwrNrrrAs  zCTRLModel.__init__cCs|jSr?r)rMrrrget_input_embeddingsszCTRLModel.get_input_embeddingscCs ||_dSr?r)rMZnew_embeddingsrrrset_input_embeddingsszCTRLModel.set_input_embeddingscCs(|D]\}}|j|j|qdS)zv Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsrrkrT)rMZheads_to_prunelayerrRrrr _prune_headsszCTRLModel._prune_headsN) input_idspast_key_valuesr8token_type_ids position_idsr9 inputs_embedsr]r^output_hidden_states return_dictrZreturnc  KsT| dur | n|jj} |dur |n|jj}| dur4| n|jj} | durH| n|jj} |durj|durjtdnd|dur||||}|d|d}|j d}n,|dur|dd}|j d}ntd|dur|j n|j }|r|durt |jd}|r t |t r tdt |}|dur2|nd}|durftj||d|tj|d}|d}|dur|dkrtd ||d}|d d }|j|jd }d |t|jj}|||jj}|dur|d|d}||}|t|j9}nd}|dur ||}|d}t t!||||d |}|t|j9}|j"||_"|j"|ddf}|||}|#|}| rdnd}| rdnd}t$|j%D]R\}}| r||f}||||||||| | d}|d}| r||d f7}q|&|}| r$||f}| sDt dd||||fDSt'||||dS)aE input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) Example: ```python >>> from transformers import AutoTokenizer, CTRLModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 5, 1280] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer!rz5You have to specify either input_ids or inputs_embedsrzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `DynamicCache` instead, e.g. `past_key_values=DynamicCache.from_legacy_cache(past_key_values)`.)r devicez$batch_size has to be defined and > 0rrrrzrrscss|]}|dur|VqdSr?r)rr6rrr sz$CTRLModel.forward..)Zlast_hidden_stater hidden_states attentions)(rwr^r]ruse_return_dict ValueErrorZ%warn_if_padding_and_no_attention_maskr3viewr0rr r{tuplelogger warning_onceZfrom_legacy_cacheget_seq_lengthrr$longr&r%r ZfinfominZ get_head_maskrrr1r2rZtriuZonesr+r enumeraterrr )rMrrr8rrr9rr]r^rrrZkwargsZ input_shaperXr past_lengthZtoken_type_embedsZseq_lenr7Z pos_embedsrZall_hidden_statesZall_attentionsrrrurrrr_s1                "       zCTRLModel.forward) NNNNNNNNNNNN)r`rarbrArrrrrr LongTensorr FloatTensorboolTensorrr r_rcrrrNrrs@ rz The CTRL Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). )Z custom_introcseZdZdgZfddZed eejee e ej eej eejeejeej eej eejee ee ee ee eej e e ej efdddZd dd ZZS) CTRLLMHeadModelzlm_head.weightcs8t|t||_tj|j|jdd|_| dS)NTr~) r@rArrxrrFrrlm_headrrrNrrrAs  zCTRLLMHeadModel.__init__N)rrr8rrr9rlabelsr]r^rrrZrcKs| dur | n|jj} |j|||||||| | | | | d }|d}||}d}|durp|j||fd|jji|}| s|f|dd}|dur|f|S|St|||j|j|j dS)a  input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Example: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLLMHeadModel >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLLMHeadModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Wikipedia The llama is", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> sequence_ids = model.generate(inputs["input_ids"]) >>> sequences = tokenizer.batch_decode(sequence_ids) >>> sequences ['Wikipedia The llama is a member of the family Bovidae. It is native to the Andes of Peru,'] >>> outputs = model(**inputs, labels=inputs["input_ids"]) >>> round(outputs.loss.item(), 2) 9.21 >>> list(outputs.logits.shape) [1, 5, 246534] ```N) rr8rrr9rr]r^rrrZrrr)losslogitsrrr) rwrrxrZ loss_functionrr rrr)rMrrr8rrr9rrr]r^rrrZrtransformer_outputsrZ lm_logitsrr<rrrr_sJ< zCTRLLMHeadModel.forwardcKsR|durF|}|jd|kr$|}n|jdd}|dd|df}|||dS)Nr)rrr])rr0)rMrrr]rrZremove_prefix_lengthrrrprepare_inputs_for_generation-sz-CTRLLMHeadModel.prepare_inputs_for_generation) NNNNNNNNNNNNN)NN)r`rarbZ_tied_weights_keysrArrrrrrrrrr r_rrcrrrNrrsB era The CTRL Model transformer with a sequence classification head on top (linear layer). [`CTRLForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). cseZdZfddZedeejeeeej eej eejeejeej eej eejee ee ee ee e eej e fd ddZZS)CTRLForSequenceClassificationcs@t||j|_t||_tj|j|jdd|_| dS)NFr) r@rA num_labelsrrxrrFr classifierrrrNrrrALs   z&CTRLForSequenceClassification.__init__N) rrr8rrr9rrr]r^rrrc  CsZ| dur | n|jj} |j|||||||| | | | d } | d}||}|durb|jdd\}}n|jdd\}}|jjdur|dkrtd|jjdurd}nb|dur||jjk|jt j }t j |jd|jt j d}|| d}nd}t |jjd |t j ||jd |f}d}|dur|jjdur|jdkrNd |j_n:|jdkr|jt jksv|jt jkrd |j_nd |j_|jjd krt}|jdkr|||}n |||}nN|jjd krt}||d|j|d}n|jjd krt}|||}| sF|f| dd}|durB|f|S|St||| j| jdS)a2 input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example of single-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> import torch >>> torch.manual_seed(42) # doctest: +IGNORE_RESULT >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> labels = torch.tensor(1) >>> loss = model(**inputs, labels=labels).loss >>> round(loss.item(), 2) 0.93 ``` Example of multi-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained( ... "Salesforce/ctrl", problem_type="multi_label_classification" ... ) >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> num_labels = len(model.config.id2label) >>> labels = torch.nn.functional.one_hot(torch.tensor([predicted_class_id]), num_classes=num_labels).to( ... torch.float ... ) >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() # doctest: +IGNORE_RESULT ```N) rr8rrr9rr]r^rrrrrz=Cannot handle batch sizes > 1 if no padding token is defined.r!)rr z will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`)rZ regressionZsingle_label_classificationZmulti_label_classification)rrrr)rwrrxrr0Z pad_token_idrr%rrZint32r$ZargmaxrrrOr`Z problem_typerr rrDrZsqueezerrrr rr)rMrrr8rrr9rrr]r^rrrrrrXZsequence_lengthZlast_non_pad_tokenZ non_pad_maskZ token_indicesZ pooled_logitsrZloss_fctr<rrrr_Usxh      (    z%CTRLForSequenceClassification.forward) NNNNNNNNNNNN)r`rarbrArrrrrrrrrr r_rcrrrNrr@s: r)rrrrv)NN).__doc__typingrrnumpyr1rrZtorch.nnrrrZ cache_utilsr Z generationr Zmodeling_outputsr r r Zmodeling_utilsrZ pytorch_utilsrrrutilsrrZconfiguration_ctrlrZ get_loggerr`rrr,r=Moduler>rerfrvrrr__all__rrrrsF       G2M ?