a hn?@sndZddlmZmZmZddlZddlZddlmZddlm Z ddl m Z ddl m Z dd lmZmZdd lmZdd lmZdd lmZdd lmZmZddlmZddlmZmZmZm Z m!Z!m"Z"ddl#m$Z$e%e&Z'Gdddej(Z)Gddde!Z*ddZ+d&ddZ,GdddeZ-GdddeZ.Gdd d e Z/Gd!d"d"e Z0Gd#d$d$eZ1gd%Z2dS)'zPyTorch Cohere model.)CallableOptionalUnionN)nn)Cache)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)dynamic_rope_update)ALL_ATTENTION_FUNCTIONS)Unpack)TransformersKwargslogging)deprecate_kwarg)LlamaAttentionLlamaForCausalLMLlamaMLP LlamaModelLlamaRotaryEmbeddingeager_attention_forward) CohereConfigcs&eZdZdfdd ZddZZS) CohereLayerNormNh㈵>Fcs&ttt||_||_dS)zcThe hidden size can be a tuple or an int. The tuple is used for QKNorm to normalize across head_dimN)super__init__r ParametertorchZonesweightvariance_epsilon)self hidden_sizeepsbias __class__e/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/cohere/modular_cohere.pyr7s zCohereLayerNorm.__init__cCsl|j}|tj}|jddd}||djddd}||t||j}|jtj|}||S)NT)Zkeepdimr) dtypetor Zfloat32meanpowZrsqrtr"r!)r# hidden_statesZ input_dtyper.Zvariancer)r)r*forward=s zCohereLayerNorm.forward)NrF)__name__ __module__ __qualname__rr1 __classcell__r)r)r'r*r6src@s eZdZeeddZdS)CohereRotaryEmbeddingc Cs|jddddf|jddd}|dddddf}t|jjtrd|jjdkrd|jjnd}tj |ddT|| dd}tj |ddd }| |j }||j } Wdn1s0Y|j|jd | j|jd fS) Nrr+rZmpscpuF) device_typeZenabledrdimr,)Zinv_freqfloatexpandshape isinstanceZdevicetypestrr Zautocast transposeZrepeat_interleavecosZattention_scalingsinr-r,) r#x position_idsZinv_freq_expandedZposition_ids_expandedr8ZfreqsZembrCrDr)r)r*r1Hs(&,zCohereRotaryEmbedding.forwardN)r2r3r4r Zno_gradr r1r)r)r)r*r6Gsr6cCsB|ddddf}|ddddf}tj| |gddd}|S)N.rrr+r9)r stackflatten)rEx1Zx2Zrot_xr)r)r* rotate_halfXsrKc Csj|j}|}|}||}||}||t||}||t||}|j|d|j|dfS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. r;)r,r<Z unsqueezerKr-) qkrCrDrFZ unsqueeze_dimr,Zq_embedZk_embedr)r)r*apply_rotary_pos_emb`s  rNcseZdZfddZZS) CohereMLPcsRt|tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_dS)NF)r&) rrrZLinearr$Zintermediate_sizeZ gate_projZup_projZ down_projr#configr'r)r*rs zCohereMLP.__init__)r2r3r4rr5r)r)r'r*rO~srOc seZdZdZd eeedfdd Zedddd de j e e j e j fee j ee ee j eee e j ee j fd d d ZZS)CohereAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNrQ layer_idxcsPt|||j|_|jrLt|j|jf|jd|_t|j|jf|jd|_ dS)Nr$r%) rr use_qk_normrZnum_attention_headshead_dimlayer_norm_epsq_normZnum_key_value_headsk_normr#rQrTr'r)r*rszCohereAttention.__init__past_key_valuepast_key_values4.58new_nameversion)r0position_embeddingsattention_maskr]cache_positionkwargsreturncKsL|jdd}g|d|jR}|||} |||} |||} |jrl|| } || } | dd} | dd} | dd} |\} } t | | | | \} } |dur| | |d}| | | |j |\} } t }|jjdkrt|jj}||| | | |f|js dn|j|jd|\}}|jg|dR}||}||fS)Nr+rr)rDrCrdeagerg)Zdropoutscaling)r>rWZq_projviewZk_projZv_projrVrYrZrBrNupdaterTrrQZ_attn_implementationr ZtrainingZattention_dropoutrhZreshape contiguousZo_proj)r#r0rbrcr]rdreZ input_shapeZ hidden_shapeZ query_statesZ key_statesZ value_statesrCrDZ cache_kwargsZattention_interfaceZ attn_outputZ attn_weightsr)r)r*r1sD           zCohereAttention.forward)N)NN)r2r3r4__doc__rrintrrr Tensortupler LongTensorrrr1r5r)r)r'r*rRs rRcseZdZeedfdd Zeddddd eje eje ej e e e e e ej e e ejejfeee eje e ejejffd d d ZZS)CohereDecoderLayerrScs@t|j|_t||d|_t||_t|j|jd|_ dS)NrSrU) rrr$rR self_attnrOmlprrXinput_layernormr[r'r)r*rs   zCohereDecoderLayer.__init__r\r]r^r_NF) r0rcrFr] use_cacherdrbrerfc KsL|} ||}|jf|||||||d|\} } ||} | | | }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. past_key_values (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. )r0rcrFr]rurdrb)rtrrrs) r#r0rcrFr]rurdrbreZresidualZhidden_states_attention_Zhidden_states_mlpr)r)r*r1s    zCohereDecoderLayer.forward)NNNFNN)r2r3r4rrmrrr rnrrprboolrorr FloatTensorr1r5r)r)r'r*rqs& rqcs"eZdZedfdd ZZS) CohereModelrQcsNttfddtjD|_td|_t j j d|_ dS)Ncsg|]}t|qSr))rq).0rTrzr)r* z(CohereModel.__init__..rzrU) rrrZ ModuleListrangeZnum_hidden_layersZlayersr6Z rotary_embrr$rXZnormrPr'rzr*rs   zCohereModel.__init__)r2r3r4rrr5r)r)r'r*rysrycseZdZfddZdeejeejeejeee e ej feej eejee ee ee eejee ejfeeed ddZZS) CohereForCausalLMcs*t|t||_|j|_|j|_dS)N)rrrymodel logit_scaleZtie_word_embeddingsrPr'r)r*rs  zCohereForCausalLM.__init__Nr) input_idsrcrFr] inputs_embedslabelsruoutput_attentionsoutput_hidden_statesrdlogits_to_keeprerfc  Ks|dur |n|jj}| dur | n|jj} |jf|||||||| | d | } | j}t| trht| dn| }||dd|ddf}||j }d}|dur|j f|||jj d| }t ||| j | j| jdS)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >> from transformers import AutoTokenizer, CohereForCausalLM >> model = CohereForCausalLM.from_pretrained("CohereForAI/c4ai-command-r-v01") >> tokenizer = AutoTokenizer.from_pretrained("CohereForAI/c4ai-command-r-v01") >> prompt = "Hey, are you conscious? Can you talk to me?" >> inputs = tokenizer(prompt, return_tensors="pt") >> # Generate >> generate_ids = model.generate(inputs.input_ids, max_length=30) >> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrcrFr]rrurrrd)logitsr vocab_size)lossrr]r0 attentions)rQrrrZlast_hidden_stater?rmsliceZlm_headrZ loss_functionrr r]r0r)r#rrcrFr]rrrurrrdrreoutputsr0Z slice_indicesrrr)r)r*r1s<%   zCohereForCausalLM.forward) NNNNNNNNNNr)r2r3r4rrr rprnrrlistrxrwrmrrr r1r5r)r)r'r*rs6  r)rryZCoherePreTrainedModel)Nr)3rltypingrrrr Ztorch.utils.checkpointrZ cache_utilsrZmodeling_flash_attention_utilsrZmodeling_layersr Zmodeling_outputsr r Zmodeling_rope_utilsr Zmodeling_utilsr Zprocessing_utilsrutilsrrZutils.deprecationrZllama.modeling_llamarrrrrrZconfiguration_cohererZ get_loggerr2loggerModulerr6rKrNrOrRrqryr__all__r)r)r)r*s4            D: R