a h@s"dZddlmZmZddlZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZdd lmZd d lmZd dlmZmZmZmZmZmZmZddlmZee Z!GdddeZ"GdddeZ#GdddeZ$GdddeZ%GdddeZ&GdddeZ'gdZ(dS)zPyTorch BitNet model.)CallableOptionalN)Cache)FlashAttentionKwargs)CausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONS)Unpack)logging)deprecate_kwarg)GemmaMLP)LlamaAttentionLlamaDecoderLayerLlamaForCausalLM LlamaModel LlamaRMSNormapply_rotary_pos_embeager_attention_forward) BitNetConfigc@s eZdZdS) BitNetRMSNormN__name__ __module__ __qualname__rre/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/bitnet/modular_bitnet.pyr+srcs*eZdZedfdd ZddZZS) BitNetMLP)configcs"t|t|j|jd|_dSN)eps)super__init__rZintermediate_size rms_norm_eps ffn_sub_norm)selfr __class__rrr#0s zBitNetMLP.__init__c Cs*|||||||}|S)N) down_projr%Zact_fnZ gate_projZup_proj)r&xr)rrrforward4s&zBitNetMLP.forward)rrrrr#r+ __classcell__rrr'rr/src seZdZeedfdd Zeddddd eje ejejfe eje e e ej e ee eje ejfd d d ZZS) BitNetAttention)r layer_idxcs$t||t|j|jd|_dSr )r"r#rZ hidden_sizer$ attn_sub_norm)r&rr.r'rrr#:szBitNetAttention.__init__Zpast_key_valuepast_key_valuesz4.58)new_nameversionN) hidden_statesposition_embeddingsattention_maskr0cache_positionkwargsreturncKs.|jdd}g|d|jR}|||dd} |||dd} |||dd} |\} } t| | | | \} } |dur| | |d}|| | |j |\} } t }|j j dkrt |j j }||| | | |f|jsdn|j|jd|\}}|jg|dR}||}||}||fS)Nrr )sincosr6eagerg)Zdropoutscaling)shapeZhead_dimZq_projviewZ transposeZk_projZv_projrupdater.rrZ_attn_implementationrZtrainingZattention_dropoutr=Zreshape contiguousr/Zo_proj)r&r3r4r5r0r6r7Z input_shapeZ hidden_shapeZ query_statesZ key_statesZ value_statesr;r:Z cache_kwargsZattention_interfaceZ attn_outputZ attn_weightsrrrr+>s:       zBitNetAttention.forward)NN)rrrrintr#r torchZTensortuplerrZ LongTensorr rr+r,rrr'rr-9s r-c@s eZdZdS)BitNetDecoderLayerNrrrrrrEmsrEc@s eZdZdS) BitNetModelNrrrrrrFqsrFcs0eZdZdgZdZdZedfdd ZZS)BitNetForCausalLMzlm_head.weightN)r8c stjfi|S)a$ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, transformers., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, transformers., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, BitNetForCausalLM >>> model = BitNetForCausalLM.from_pretrained("microsoft/bitnet-b1.58-2B-4T") >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/bitnet-b1.58-2B-4T") >>> prompt = f'<|begin_of_text|>User: Hey, are you conscious? Can you talk to me?<|eot_id|>Assistant: ' >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=100) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "User: Hey, are you conscious? Can you talk to me?Assistant: No, I'm not conscious. I'm an artificial intelligence designed to assist with information and tasks. How can I help you today?" ```)r"r+)r&Z super_kwargsr'rrr+zszBitNetForCausalLM.forward) rrrZ_tied_weights_keysZ_tp_planZ_pp_planrr+r,rrr'rrGus rG)rGrFZBitNetPreTrainedModel))__doc__typingrrrCZ cache_utilsrZmodeling_flash_attention_utilsrZmodeling_outputsrZmodeling_utilsrZprocessing_utilsr utilsr Zutils.deprecationr Zgemma.modeling_gemmar Zllama.modeling_llamarrrrrrrZconfiguration_bitnetrZ get_loggerrloggerrrr-rErFrG__all__rrrrs(        $   4"