a h\ @sddlmZmZmZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZmZddlmZdd lmZdd lmZmZdd lmZmZdd lmZmZdd lmZddl m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)Gdddej*Z+Gdddej*Z,ej-e.ej-dddZ/d/ej*ej-ej-ej-eej-e0e0ee!dddZ1ddZ2d0d d!Z3Gd"d#d#ej*Z4Gd$d%d%ej*Z5Gd&d'd'eZ6e"Gd(d)d)eZ7e"Gd*d+d+e7Z8e"Gd,d-d-e7e Z9gd.Z:dS)1)CallableOptionalUnionN)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_mask!create_sliding_window_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)deprecate_kwarg)check_model_inputs) Cohere2ConfigcsDeZdZUejed<dedfdd Zee ddZ Z S) Cohere2RotaryEmbeddinginv_freqNconfigcstt|dr:t|jtr:|jd|jd|_nd|_|j|_ |j|_ ||_ t |j|_ | |j |\}|_|jd|dd|j|_dS)N rope_scaling rope_typetypedefaultrF) persistent)super__init__hasattr isinstancer dictgetr!Zmax_position_embeddingsZmax_seq_len_cachedZoriginal_max_seq_lenrrZ rope_init_fnattention_scalingZregister_bufferrZoriginal_inv_freq)selfrdevicer __class__h/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/cohere2/modeling_cohere2.pyr&.s  zCohere2RotaryEmbedding.__init__c Cs|jddddf|jddd}|dddddf}t|jjtrd|jjdkrd|jjnd}tj |ddT|| dd}tj |ddd }| |j }||j } Wdn1s0Y|j|jd | j|jd fS) NrrZmpscpuF) device_typeZenableddimdtype)rfloatexpandshaper(r-r"strtorchZautocast transposeZrepeat_interleavecosr+sintor9) r,x position_idsZinv_freq_expandedZposition_ids_expandedr4ZfreqsZembr@rAr0r0r1forward?s(&,zCohere2RotaryEmbedding.forward)N) __name__ __module__ __qualname__r>Tensor__annotations__rr&Zno_gradrrE __classcell__r0r0r.r1r+s  rcs&eZdZdfdd ZddZZS) Cohere2LayerNormNh㈵>Fcs&ttt||_||_dS)zcThe hidden size can be a tuple or an int. The tuple is used for QKNorm to normalize across head_dimN)r%r&nn Parameterr>Zonesweightvariance_epsilon)r, hidden_sizeepsbiasr.r0r1r&Ps zCohere2LayerNorm.__init__cCsl|j}|tj}|jddd}||djddd}||t||j}|jtj|}||S)Nr2T)Zkeepdimr5) r9rBr>float32meanpowZrsqrtrQrP)r, hidden_statesZ input_dtyperVZvariancer0r0r1rEVs zCohere2LayerNorm.forward)NrMFrFrGrHr&rErKr0r0r.r1rLOsrL)rXn_repreturncCs^|j\}}}}|dkr|S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r<r;reshape)rXrZbatchnum_key_value_headsslenhead_dimr0r0r1 repeat_kv`s 0ra)modulequerykeyvalueattention_maskscalingdropoutkwargsc Kst||j}t||j} t||dd|} |durf|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)Nr5rr2)r7r9)ptrainingr)ranum_key_value_groupsr>matmulr?r<rNZ functionalZsoftmaxrUrBr9rirm contiguous) rcrdrerfrgrhrirj key_states value_states attn_weightsZ causal_mask attn_outputr0r0r1eager_attention_forwardls  & rucCsB|ddddf}|ddddf}tj| |gddd}|S)N.r5rr2r6rk)r>stackflatten)rCx1Zx2Zrot_xr0r0r1 rotate_halfsryc Csj|j}|}|}||}||}||t||}||t||}|j|d|j|dfS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. r8)r9r: unsqueezeryrB) qkr@rArDZ unsqueeze_dimr9Zq_embedZk_embedr0r0r1apply_rotary_pos_embs  r}cseZdZdZd eeedfdd Zedddd de j e e j e j fee j ee ee j eee e j ee j ee e j fd d d ZZS)Cohere2Attentionz=Multi-headed attention from 'Attention Is All You Need' paperNr layer_idxcst||_||_t|d|j|j|_|j|j|_ |jd|_ |j |_ d|_ |j |dkrh|jnd|_tj|j|j|j|jd|_tj|j|j|j|jd|_tj|j|j|j|jd|_tj|j|j|j|jd|_dS)Nr`gTsliding_attentionrT)r%r&rrgetattrrRZnum_attention_headsr`r^rnrhattention_dropoutZ is_causal layer_typessliding_windowrNLinearZattention_biasq_projk_projv_projo_projr,rrr.r0r1r&s*  zCohere2Attention.__init__past_key_valuepast_key_values4.58new_nameversion)rXposition_embeddingsrgrcache_positionrjr[cKs2|jdd}g|d|jR}|||dd} |||dd} |||dd} |\} } |jdurt| | | | \} } |dur| | |d}| | | |j |\} } t }|j j dkrt|j j }||| | | |f|jsdn|j|j|jd|\}}|jg|dR}||}||fS)Nr2rr5)rAr@reagerrb)rirhr)r<r`rviewr?rrrr}updaterrurZ_attn_implementationrrmrrhr\rpr)r,rXrrgrrrjZ input_shapeZ hidden_shapeZ query_statesrqrrr@rAZ cache_kwargsZattention_interfacertrsr0r0r1rEs<        zCohere2Attention.forward)N)NN)rFrGrH__doc__rrintr&rr>rItupler LongTensorrr rErKr0r0r.r1r~s r~cs$eZdZfddZddZZS) Cohere2MLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFr) r%r&rrRZintermediate_sizerNr gate_projup_proj down_projrZ hidden_actact_fnr,rr.r0r1r&s zCohere2MLP.__init__cCs$||||||}|S)N)rrrr)r,rCrr0r0r1rEs zCohere2MLP.forwardrYr0r0r.r1rs rcseZdZeedfdd Zeddddd eje ejejfe eje e e e e ej eee eje e ejejffd d d ZZS)Cohere2DecoderLayerrcsLt|j|_t||d|_t||_t|j|jd|_ |j ||_ dS)NrrRrS) r%r&rRr~ self_attnrmlprLlayer_norm_epsinput_layernormrattention_typerr.r0r1r&s   zCohere2DecoderLayer.__init__rrrrNF)rXrrgr use_cacherrjr[c KsJ|}||}|jf||||||d|\} } ||} || | }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. past_key_values (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. )rXrrgrrr)rrr) r,rXrrgrrrrjZresidualZhidden_states_attention_Zhidden_states_mlpr0r0r1rEs   zCohere2DecoderLayer.forward)NNFN)rFrGrHrrr&rr>rIrrrboolrrr FloatTensorrErKr0r0r.r1rs  rc@sHeZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZdS)Cohere2PreTrainedModelrmodelTrr)rX attentionsN)rFrGrHrrJZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_skip_keys_device_placementZ_supports_flash_attnZ_supports_sdpaZ_supports_flex_attnZ_can_compile_fullgraphZ_supports_attention_backendrr~Z_can_record_outputsr0r0r0r1r=s rc steZdZedfdd Zeedeej eej eej ee eej ee eej eeed ddZZS) Cohere2Modelrcstj|_j|_tjj|j|_t fddt j D|_ t jjd|_td|_d|_|dS)Ncsg|]}t|qSr0)r).0rrr0r1 Yz)Cohere2Model.__init__..rrF)r%r&Z pad_token_idZ padding_idx vocab_sizerNZ EmbeddingrR embed_tokensZ ModuleListrangeZnum_hidden_layerslayersrLrnormr rotary_embZgradient_checkpointing post_initrr.rr1r&Rs  zCohere2Model.__init__N) input_idsrgrDr inputs_embedsrrrjr[c Ks&|du|duArtd|dur*||}|rH|durH|jsHt|jd}|dur|dur`|nd} tj| | |jd|j d}|dur| d}t |} t s|j|||||d} t fi| tfi| d} |} || |} |jD]&}|| f| | |j|||d|} q|| } t| |d S) Nz:You must specify exactly one of input_ids or inputs_embedsrrr)r-)rZ input_embedsrgrrrD)Zfull_attentionr)rrgrrr)last_hidden_stater) ValueErrorrrmrrZget_seq_lengthr>Zaranger<r-rzr(r)r r rrrrr)r,rrgrDrrrrrjZpast_seen_tokensZcausal_mask_mappingZ mask_kwargsrXrZ decoder_layerr0r0r1rEbsV         zCohere2Model.forward)NNNNNNN)rFrGrHrr&rrrr>rrIrrrrrrrErKr0r0r.r1rPs*rcseZdZdgZddiZddgdgfiZfddZeed e e j e e j e e j e e eee jfe e je e j e ee ee ee e j e ee j feeed d d ZZS)Cohere2ForCausalLMzlm_head.weightlm_headZ colwise_reprXlogitscsPt|t||_|j|_tj|j|jdd|_|j |_ |j |_ | dSr) r%r&rrrrNrrRr logit_scaleZtie_word_embeddingsrrr.r0r1r&s  zCohere2ForCausalLM.__init__Nr) rrgrDrrlabelsroutput_attentionsoutput_hidden_statesrlogits_to_keeprjr[c  Ks|dur |n|jj}| dur | n|jj} |jf|||||||| | d | } | j}t| trht| dn| }||dd|ddf}||j }d}|dur|j f|||jj d| }t ||| j | j| jdS)a~ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >> from transformers import AutoTokenizer, Cohere2ForCausalLM >> model = Cohere2ForCausalLM.from_pretrained("Cohere2ForAI/c4ai-command-r-v01") >> tokenizer = AutoTokenizer.from_pretrained("Cohere2ForAI/c4ai-command-r-v01") >> prompt = "Hey, are you conscious? Can you talk to me?" >> inputs = tokenizer(prompt, return_tensors="pt") >> # Generate >> generate_ids = model.generate(inputs.input_ids, max_length=30) >> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrgrDrrrrrr)rrr)lossrrrXr)rrrrrr(rslicerrZ loss_functionrrrrXr)r,rrgrDrrrrrrrrrjoutputsrXZ slice_indicesrrr0r0r1rEs<'   zCohere2ForCausalLM.forward) NNNNNNNNNNr)rFrGrHZ_tied_weights_keysZ_tp_planZ_pp_planr&rrrr>rrIrrlistrrrrrrrErKr0r0r.r1rs@  r)rrr)rb)Nr);typingrrrr>Ztorch.nnrNZ activationsrZ cache_utilsrrZ generationr Z masking_utilsr r Zmodeling_flash_attention_utilsr Zmodeling_layersr Zmodeling_outputsrrZmodeling_rope_utilsrrZmodeling_utilsrrZprocessing_utilsrutilsrrrZutils.deprecationrZ utils.genericrZconfiguration_cohere2rModulerrLrIrrar:ruryr}r~rrrrr__all__r0r0r0r1sT         $  I8R]