a hI @s dZddlmZmZmZddlZddlZddlmZddlm Z ddl m Z m Z ddl mZdd lmZdd lmZdd lmZmZmZdd lmZmZdd lmZmZddlmZmZddl m!Z!ddl"m#Z#m$Z$m%Z%m&Z&ddl'm(Z(ddl)m*Z*e%rddl+m,Z,ddl-m.Z.e&/e0Z1Gdddej2Z3ddZ4d1ddZ5Gdddej2Z6d2ej2ej7ej7ej7eej7e8e8dd d!Z9Gd"d#d#ej2Z:Gd$d%d%eZ;e#Gd&d'd'eZGd,d-d-ee<Z?Gd.d/d/ee<Z@gd0ZAdS)3zPyTorch Persimmon model.)CallableOptionalUnionN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs) GenericForSequenceClassificationGenericForTokenClassificationGradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringcan_return_tupleis_torch_flex_attn_availablelogging)deprecate_kwarg)PersimmonConfig) BlockMask)make_flex_block_causal_maskcsDeZdZUejed<dedfdd Zee ddZ Z S) PersimmonRotaryEmbeddinginv_freqNconfigcstt|dr:t|jtr:|jd|jd|_nd|_|j|_ |j|_ ||_ t |j|_ | |j |\}|_|jd|dd|j|_dS)N rope_scaling rope_typetypedefaultr!F) persistent)super__init__hasattr isinstancer$dictgetr%Zmax_position_embeddingsZmax_seq_len_cachedZoriginal_max_seq_lenr#rZ rope_init_fnattention_scalingZregister_bufferr!Zoriginal_inv_freq)selfr#devicer! __class__l/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/persimmon/modeling_persimmon.pyr*?s  z!PersimmonRotaryEmbedding.__init__c Cs|jddddf|jddd|j}|dddddf}t|jjtrl|jjdkrl|jjnd}t j |ddV|| dd}t j ||fdd }| |j}||j} Wdn1s0Y|j|jd | j|jd fS) NrrZmpscpuF) device_typeZenableddim)dtype)r!floatexpandshapetor1r,r&strtorchZautocast transposecatcosr/sinr<) r0x position_idsZinv_freq_expandedZposition_ids_expandedr8ZfreqsZembrErFr4r4r5forwardPs0&,z PersimmonRotaryEmbedding.forward)N) __name__ __module__ __qualname__rBTensor__annotations__rr*Zno_gradrrI __classcell__r4r4r2r5r <s  r cCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr6r9r:)r?rBrD)rGx1Zx2r4r4r5 rotate_halfasrQcCsD||}||}||t||}||t||}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezerQ)qkrErFrHZ unsqueeze_dimZq_embedZk_embedr4r4r5apply_rotary_pos_embis   rUcs$eZdZfddZddZZS) PersimmonMLPcs>tt|j|j|_t|j|j|_t|j |_ dSN) r)r*rLinear hidden_sizeZintermediate_size dense_h_to_4h dense_4h_to_hrZ hidden_actactr0r#r2r4r5r*s zPersimmonMLP.__init__cCs"||}||}||}|SrW)rZr\r[)r0 hidden_statesr4r4r5rIs   zPersimmonMLP.forward)rJrKrLr*rIrOr4r4r2r5rVs rV)modulequerykeyvalueattention_maskscalingdropoutc Kst||dd|}|durN|ddddddd|jdf} || }tjj|dtjd|j }tjj |||j d}t||} | dd } | |fS)Nr9rr6)r;r<)ptrainingr) rBmatmulrCr?rZ functionalZsoftmaxZfloat32r@r<rfri contiguous) r`rarbrcrdrerfkwargs attn_weights causal_mask attn_outputr4r4r5eager_attention_forwards & rpcseZdZdZdeeedfdd Zej e ej ej ej fdddZ e d d d d dej eej eej eeeeeej ee ej ej feee ej eej ee ej fd ddZZS)PersimmonAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNr# layer_idxcsHt||_||_|dur4td|jjd|j|_|j |_ |j|j |_ |j |_ t |j |j|_d|_|j |j |jkrtd|jd|j dtj|jd|jdd|_tj|j |j |jdd|_|j|_|j d |_|jr(tj|j|j |jdd |_tj|j|j |jdd |_t|j|_t|jd |_dS) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz?hidden_size must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).rbiasg)epsZelementwise_affiner") r)r*r#rslogger warning_oncer3rJrYZnum_attention_heads num_headshead_dimZ rope_thetaintZpartial_rotary_factor rotary_ndimsZ is_causal ValueErrorrrXquery_key_valuedense qk_layernormre LayerNormlayer_norm_eps q_layernorm k_layernormDropoutattention_dropoutr rotary_embr0r#rsr2r4r5r*s@   zPersimmonAttention.__init__) fused_qkvreturncCsV|j\}}}||||jd|j}|ddddf|ddddf|ddddffS)a Split the last dimension into (num_heads, head_dim) without making any copies, results share same memory storage as `fused_qkv` Args: fused_qkv (`torch.tensor`): [batch_size, seq_length, num_heads * 3 * head_dim] Returns: query: [batch_size, seq_length, num_heads, head_dim] key: [batch_size, seq_length, num_heads, head_dim] value: [batch_size, seq_length, num_heads, head_dim] r.rNrr9)r?viewryrz)r0r batch_sizeZ seq_lengthZthree_times_hidden_sizer4r4r5 _split_headss zPersimmonAttention._split_headspast_key_valuepast_key_values4.58new_nameversionF r^rdrHroutput_attentions use_cachecache_positionposition_embeddingsrlrc Ks|\} } } ||} || \}}}|jrB||}||}|dd}|dd}|dd}|\}}|dd|jf|d|jdf}}|dd|jf|d|jdf}}t||||\}}t j ||fdd}t j ||fdd}|dur |||j|d}| |||j |\}}t }|jjdkr>t|jj}||||||f|jsXdn|jj|jd | \}}|| | d}||}|sd}||fS) Nrr9.r6r:)rFrEZpartial_rotation_sizereagerr_)rfre)sizer~rrrrrCr|rUrBrDupdatersrpr#_attn_implementationrrirrereshaper)r0r^rdrHrrrrrrlZbszZq_len_rZ query_statesZ key_statesZ value_statesrErFZ query_rotZ query_passZkey_rotZkey_passZ cache_kwargsZattention_interfacerormr4r4r5rIs\           zPersimmonAttention.forward)N)NNNFFNN)rJrKrL__doc__rrr{r*rBrMtuplerr LongTensorrboolrr rIrOr4r4r2r5rqs.&" rqcseZdZeedfdd Zeddddd eje eje ej e e eje e e e e ej e e ejejfe ee eje e ejejffd d d ZZS)PersimmonDecoderLayerrrcsdt|j|_t||d|_t||_tj|j|j d|_ tj|j|j d|_ t |j |_dS)Nrrrv)r)r*rYrq self_attnrVmlprrrinput_layernormpost_attention_layernormrZhidden_dropoutrfrr2r4r5r*6s  zPersimmonDecoderLayer.__init__rrrrNFrc Ks~|} ||}|jf||||||||d| \}} | |}|} ||}||}||}|| }|f} |rz| | f7} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_values (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. )r^rdrHrrrrr)rrrrrf) r0r^rdrHrrrrrrlZresidualZself_attn_weightsoutputsr4r4r5rI?s0%       zPersimmonDecoderLayer.forward)NNNFFNN)rJrKrLrr{r*rrBrMrrrrrr FloatTensorrIrOr4r4r2r5r5s*  rc@s@eZdZUeed<dZdZdgZdZdZ dZ dZ dZ ddZ dS) PersimmonPreTrainedModelr#modelTrrcCs|jj}t|tjr>|jjjd|d|jdur|jj nbt|tj rz|jjjd|d|j dur|jj|j  n&t|tj r|jj d|jj dS)Nr_)meanstdg?)r#Zinitializer_ranger,rrXweightdataZnormal_ruZzero_ Embedding padding_idxrZfill_)r0r`rr4r4r5 _init_weightss     z&PersimmonPreTrainedModel._init_weightsN)rJrKrLrrNZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_skip_keys_device_placementZ_can_compile_fullgraphZ_supports_sdpaZ_supports_flash_attnZ_supports_attention_backendrr4r4r4r5rs rcseZdZdZedfdd Zeedee j ee j ee j ee ee j eeeeeeee j eeed ddZdee j d fe j e j e ed d d Zee j eee je j edddZZS)PersimmonModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`PersimmonDecoderLayer`] Args: config: PersimmonConfig r"cstj|_j|_tjj|j|_t fddt j D|_ tj jjd|_td|_d|_|dS)Ncsg|]}t|qSr4)r).0rsr"r4r5 z+PersimmonModel.__init__..rr"F)r)r*Z pad_token_idr vocab_sizerrrY embed_tokensZ ModuleListrangeZnum_hidden_layerslayersrrfinal_layernormr rgradient_checkpointing post_initr]r2r"r5r*s  zPersimmonModel.__init__N) input_idsrdrHr inputs_embedsrroutput_hidden_statesrrlrc  Ks|dur |n|jj}|dur |n|jj}|dur4|n|jj}|du|duArTtd|jrr|jrr|rrtdd}t |t dt fstd|r|durt |jd}|dur| |}| dur|dur|nd} tj| | |jd|jd} |dur| d}|||| ||} |} || |}|r.d nd}|r|dur>|jjd vr>|s>t|j} t | | } | S) NZflash_attention_2r_Zflex_attentionrFZsdpa)rZpast_key_values_lengthZ is_trainingrr6)sequence_length target_lengthr<rr)cudaZxpuZnpu)r#ranyr,rBrMrrZis_compileabler Z_ignore_causal_mask_sdparir<r?Zget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionr1r&finfominZ_unmask_unattended) r0rdrrrrrZusing_compilable_cacher<rrrn min_dtyper4r4r5rsZ           z"PersimmonModel._update_causal_mask)rdrrr<rrc KsF|dur|dkr|}n&t|j}tj||f|||jd}|dkrVtj|dd}|tj||jd|ddk9}|ddddddf |ddd}|durB| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N)Z fill_valuer<r1r)Zdiagonalrr6r)r;rBrrfullr1Ztriurrr>cloner?r@Z masked_fill) rdrrr<rrrlrnrZ mask_lengthZ padding_maskr4r4r5ras*  $  6  zDPersimmonModel._prepare_4d_causal_attention_mask_with_cache_position) NNNNNNNNN)F)rJrKrLrrr*rrrrBrrMrrrrr rrIrr staticmethodr{r<rrOr4r4r2r5rsTe  DrcseZdZdgZfddZeed eej eej eej ee eej eej ee ee ee eej eeej fed ddZZS) PersimmonForCausalLMzlm_head.weightcs@t|t||_|j|_tj|j|jdd|_| dS)NFrt) r)r*rrrrrXrYlm_headrr]r2r4r5r*s   zPersimmonForCausalLM.__init__Nr) rrdrHrrlabelsrrrrlogits_to_keeprc  Ks|dur |n|jj}| dur | n|jj} |jf|||||||| | d | } | j}t| trht| dn| }||dd|ddf}d}|dur|j ||fd|jj i| }t ||| j | j | jdS)uk labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, PersimmonForCausalLM >>> model = PersimmonForCausalLM.from_pretrained("adept/persimmon-8b-base") >>> tokenizer = AutoTokenizer.from_pretrained("adept/persimmon-8b-base") >>> prompt = "human: Hey, what should I eat for dinner?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] 'human: Hey, what should I eat for dinner?\n\ncat: 🐱\n\nhuman: 😐\n\n' ```N) rrdrHrrrrrrr)losslogitsrr^r)r#rrrrr,r{slicerZ loss_functionrrrr^r)r0rrdrHrrrrrrrrrlrr^Z slice_indicesrrr4r4r5rIsH(  zPersimmonForCausalLM.forward) NNNNNNNNNNr)rJrKrLZ_tied_weights_keysr*rrrrBrrMrrrrr{rrIrOr4r4r2r5rs:  rc@s eZdZdS)"PersimmonForSequenceClassificationNrJrKrLr4r4r4r5rrrc@s eZdZdS)PersimmonForTokenClassificationNrr4r4r4r5rrr)rrrrr)Nr)r_)BrtypingrrrrBZtorch.utils.checkpointrZ activationsrZ cache_utilsrr Z generationr Zmodeling_attn_mask_utilsr Zmodeling_flash_attention_utilsr Zmodeling_layersr rrZmodeling_outputsrrZmodeling_rope_utilsrrZmodeling_utilsrrZprocessing_utilsrutilsrrrrZutils.deprecationrZconfiguration_persimmonrZ!torch.nn.attention.flex_attentionrZintegrations.flex_attentionrZ get_loggerrJrwModuler rQrUrVrMr=rprqrrrrrr__all__r4r4r4r5s^           %   Qx_