a h2_ @sddlmZmZmZddlZddlmZddlmZddlm Z m Z ddl m Z ddl mZdd lmZdd lmZdd lmZmZdd lmZmZdd lmZmZddlmZddlmZm Z m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'Gdddej(Z)Gdddej(Z*Gdddej(Z+ej,e-ej,dddZ.d0ej(ej,ej,ej,eej,e/e/eeddd Z0d!d"Z1d1d#d$Z2Gd%d&d&ej(Z3Gd'd(d(eZ4e Gd)d*d*eZ5e Gd+d,d,e5Z6e Gd-d.d.e5e Z7gd/Z8dS)2)CallableOptionalUnionN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)deprecate_kwarg)check_model_inputs) CohereConfigcs&eZdZdfdd ZddZZS) CohereLayerNormNh㈵>Fcs&ttt||_||_dS)zcThe hidden size can be a tuple or an int. The tuple is used for QKNorm to normalize across head_dimN)super__init__r ParametertorchZonesweightvariance_epsilon)self hidden_sizeepsbias __class__f/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/cohere/modeling_cohere.pyr4s zCohereLayerNorm.__init__cCsl|j}|tj}|jddd}||djddd}||t||j}|jtj|}||S)NT)Zkeepdim) dtypetor!float32meanpowZrsqrtr#r")r$ hidden_statesZ input_dtyper1Zvariancer*r*r+forward:s zCohereLayerNorm.forward)NrF__name__ __module__ __qualname__rr4 __classcell__r*r*r(r+r3srcsDeZdZUejed<dedfdd Zee ddZ Z S) CohereRotaryEmbeddinginv_freqNconfigcstt|dr:t|jtr:|jd|jd|_nd|_|j|_ |j|_ ||_ t |j|_ | |j |\}|_|jd|dd|j|_dS)N rope_scaling rope_typetypedefaultr;F) persistent)rrhasattr isinstancer>dictgetr?Zmax_position_embeddingsZmax_seq_len_cachedZoriginal_max_seq_lenr=rZ rope_init_fnattention_scalingZregister_bufferr;Zoriginal_inv_freq)r$r=devicer;r(r*r+rGs  zCohereRotaryEmbedding.__init__c Cs|jddddf|jddd}|dddddf}t|jjtrd|jjdkrd|jjnd}tj |ddT|| dd}tj |ddd }| |j }||j } Wdn1s0Y|j|jd | j|jd fS) Nrr,rZmpscpuF) device_typeZenabledr-dimr.)r;floatexpandshaperDrHr@strr!Zautocast transposeZrepeat_interleavecosrGsinr/r.) r$x position_idsZinv_freq_expandedZposition_ids_expandedrJZfreqsZembrSrTr*r*r+r4Xs(&,zCohereRotaryEmbedding.forward)N) r6r7r8r!Tensor__annotations__rrZno_gradrr4r9r*r*r(r+r:Ds  r:cs$eZdZfddZddZZS) CohereMLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFr') rrr=r%Zintermediate_sizerLinear gate_projup_proj down_projrZ hidden_actact_fnr$r=r(r*r+ris zCohereMLP.__init__cCs$||||||}|S)N)r_r`r]r^)r$rUr_r*r*r+r4ss zCohereMLP.forwardr5r*r*r(r+rYhs rY)r3n_repreturncCs^|j\}}}}|dkr|S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rPrOreshape)r3rbbatchnum_key_value_headsslenhead_dimr*r*r+ repeat_kvxs 0ri)modulequerykeyvalueattention_maskscalingdropoutkwargsc Kst||j}t||j} t||dd|} |durf|ddddddd|jdf} | | } tjj| dtj d |j } tjj | ||j d} t| | } | dd} | | fS)Nr-rr,)rLr.)ptrainingr)rinum_key_value_groupsr!matmulrRrPrZ functionalZsoftmaxr0r/r.rqru contiguous) rkrlrmrnrorprqrr key_states value_states attn_weights causal_mask attn_outputr*r*r+eager_attention_forwards  & r~cCsB|ddddf}|ddddf}tj| |gddd}|S)N.r-rr,rKrs)r!stackflatten)rUx1Zx2Zrot_xr*r*r+ rotate_halfsrc Csj|j}|}|}||}||}||t||}||t||}|j|d|j|dfS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. rM)r.rN unsqueezerr/) qkrSrTrVZ unsqueeze_dimr.Zq_embedZk_embedr*r*r+apply_rotary_pos_embs  rc seZdZdZd eeedfdd Zedddd de j e e j e j fee j ee ee j eee e j ee j fd d d ZZS)CohereAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNr= layer_idxcst||_||_t|d|j|j|_|j|j|_ |jd|_ |j |_ d|_ t j|j|j|j|jd|_t j|j|j|j|jd|_t j|j|j|j|jd|_t j|j|j|j|jd|_|j|_|jr t|j|jf|jd|_t|j|jf|jd|_dS)NrhgTr[r%r&)rrr=rgetattrr%Znum_attention_headsrhrfrvrpattention_dropoutZ is_causalrr\Zattention_biasq_projk_projv_projo_proj use_qk_normrlayer_norm_epsq_normk_normr$r=rr(r*r+rs8  zCohereAttention.__init__past_key_valuepast_key_values4.58new_nameversion)r3position_embeddingsrorcache_positionrrrccKsL|jdd}g|d|jR}|||} |||} |||} |jrl|| } || } | dd} | dd} | dd} |\} } t | | | | \} } |dur| | |d}| | | |j |\} } t }|jjdkrt|jj}||| | | |f|js dn|j|jd|\}}|jg|dR}||}||fS)Nr,rr-)rTrSreagerrj)rqrp)rPrhrviewrrrrrrRrupdaterr~r=Z_attn_implementationrrurrprdrxr)r$r3rrorrrrZ input_shapeZ hidden_shapeZ query_statesryrzrSrTZ cache_kwargsZattention_interfacer}r{r*r*r+r4sD           zCohereAttention.forward)N)NN)r6r7r8__doc__rrintrrr!rWtupler LongTensorrr r4r9r*r*r(r+rs rcseZdZeedfdd Zeddddd eje eje ej e e e e e ej e e ejejfeee eje e ejejffd d d ZZS)CohereDecoderLayerrcs@t|j|_t||d|_t||_t|j|jd|_ dS)Nrr) rrr%r self_attnrYmlprrinput_layernormrr(r*r+rs   zCohereDecoderLayer.__init__rrrrNF) r3rorVr use_cacherrrrrcc KsL|} ||}|jf|||||||d|\} } ||} | | | }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. past_key_values (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. )r3rorVrrrr)rrr) r$r3rorVrrrrrrZresidualZhidden_states_attention_Zhidden_states_mlpr*r*r+r4$s    zCohereDecoderLayer.forward)NNNFNN)r6r7r8rrrrr!rWrrrboolrrr FloatTensorr4r9r*r*r(r+rs& rc@sHeZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZdS)CoherePreTrainedModelr=modelTrr)r3 attentionsN)r6r7r8rrXZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_skip_keys_device_placementZ_supports_flash_attnZ_supports_sdpaZ_supports_flex_attnZ_can_compile_fullgraphZ_supports_attention_backendrrZ_can_record_outputsr*r*r*r+rVs rc steZdZedfdd Zeedeej eej eej ee eej eej ee eeed ddZZS) CohereModelr<cstj|_j|_tjj|j|_t fddt j D|_ t jjd|_td|_d|_|dS)Ncsg|]}t|qSr*)r).0rr<r*r+ rz(CohereModel.__init__..rr<F)rrZ pad_token_idZ padding_idx vocab_sizerZ Embeddingr% embed_tokensZ ModuleListrangenum_hidden_layerslayersrrnormr: rotary_embZgradient_checkpointing post_initrar(r<r+rks  zCohereModel.__init__N) input_idsrorVr inputs_embedsrrrrrcc Ks|du|duArtd|dur*||}|rB|durBt|jd}|durz|durZ|nd} tj| | |jd|jd}|dur| d}t |j|||||d} |} | | |} |j d|jj D] } | | f| |||| d|} q|| } t| |dS) Nz:You must specify exactly one of input_ids or inputs_embedsr<rr)rH)r=Z input_embedsrorrrV)rorVrrr)last_hidden_stater) ValueErrorrr r=Zget_seq_lengthr!ZarangerPrHrr rrrrr)r$rrorVrrrrrrZpast_seen_tokensr|r3rZ decoder_layerr*r*r+r4{sP        zCohereModel.forward)NNNNNNN)r6r7r8rrrrrr!rrWrrrrrrr4r9r*r*r(r+ris*rcseZdZdgZddiZddgdgfiZfddZeed e e j e e j e e j e e eee jfe e je e j e ee ee ee e j e ee j feeed d d ZZS)CohereForCausalLMzlm_head.weightlm_headZ colwise_repr3logitscsPt|t||_|j|_tj|j|jdd|_|j |_ |j |_ | dSrZ) rrrrrrr\r%r logit_scaleZtie_word_embeddingsrrar(r*r+rs  zCohereForCausalLM.__init__Nr) rrorVrrlabelsroutput_attentionsoutput_hidden_statesrlogits_to_keeprrrcc  Ks|dur |n|jj}| dur | n|jj} |jf|||||||| | d | } | j}t| trht| dn| }||dd|ddf}||j }d}|dur|j f|||jj d| }t ||| j | j| jdS)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >> from transformers import AutoTokenizer, CohereForCausalLM >> model = CohereForCausalLM.from_pretrained("CohereForAI/c4ai-command-r-v01") >> tokenizer = AutoTokenizer.from_pretrained("CohereForAI/c4ai-command-r-v01") >> prompt = "Hey, are you conscious? Can you talk to me?" >> inputs = tokenizer(prompt, return_tensors="pt") >> # Generate >> generate_ids = model.generate(inputs.input_ids, max_length=30) >> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrorVrrrrrr)rrr)lossrrr3r)r=rrrrrDrslicerrZ loss_functionrrrr3r)r$rrorVrrrrrrrrrroutputsr3Z slice_indicesrrr*r*r+r4s<'   zCohereForCausalLM.forward) NNNNNNNNNNr)r6r7r8Z_tied_weights_keysZ_tp_planZ_pp_planrrrrr!rrWrrlistrrrrrrr4r9r*r*r(r+rs@  r)rrr)rj)Nr)9typingrrrr!rZ activationsrZ cache_utilsrr Z generationr Z masking_utilsr Zmodeling_flash_attention_utilsr Zmodeling_layersr Zmodeling_outputsrrZmodeling_rope_utilsrrZmodeling_utilsrrZprocessing_utilsrutilsrrrZutils.deprecationrZ utils.genericrZconfiguration_cohererModulerr:rYrWrrirNr~rrrrrrr__all__r*r*r*r+sT          $  X:N]