a h@s&ddlmZmZddlZddlmZddlmZddlm Z ddl m Z ddl m Z dd lmZmZdd lmZd d lmZmZmZmZd d lmZddlmZddlmZeeZ dZ!GdddeZ"Gddde Z#GdddeZ$GdddeZ%GdddeZ&GdddeZ'gdZ(dS))OptionalUnionN)Cache)FlashAttentionKwargs)GradientCheckpointingLayer)CausalLMOutputWithPast)Unpack)TransformersKwargslogging)deprecate_kwarg) GlmAttentionGlmForCausalLMGlmForSequenceClassificationGlmForTokenClassification)Phi3MLP) Glm4Config) Glm4RMSNormzTHUDM/GLM-4-9B-0414c@s eZdZdS)Glm4MLPN__name__ __module__ __qualname__rra/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/models/glm4/modular_glm4.pyr&srcseZdZeedfdd Zeddddd eje eje ej e e e e e ej e e ejejfeee eje e ejejffd d d ZZS)Glm4DecoderLayerconfig layer_idxcsvt|j|_t||d|_t||_t|j|jd|_ t|j|jd|_ t|j|jd|_ t|j|jd|_ dS)Nr)eps) super__init__Z hidden_size Glm4Attention self_attnrmlprZ rms_norm_epsinput_layernormpost_attention_layernormpost_self_attn_layernormpost_mlp_layernorm)selfrr  __class__rrr#+s  zGlm4DecoderLayer.__init__Zpast_key_valuepast_key_valuesz4.58)new_nameversionNF) hidden_statesattention_mask position_idsr. use_cachecache_positionposition_embeddingskwargsreturnc Ksr|} ||}|jf|||||||d|\}} ||}| |}|} ||}||}||}| |}|S)N)r1r2r3r.r4r5r6)r'r%r)r(r&r*) r+r1r2r3r.r4r5r6r7Zresidual_rrrforward6s*      zGlm4DecoderLayer.forward)NNNFNN)rrrrintr#r torchZTensorrZ LongTensorrbooltupler rZ FloatTensorr: __classcell__rrr,rr*s& rc@s eZdZdS)r$Nrrrrrr$[sr$cs0eZdZeeeeefdfdd ZZ S)Glm4ForCausalLM) super_kwargsr8c stjfi|S)ah labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Glm4ForCausalLM >>> model = Glm4ForCausalLM.from_pretrained("THUDM/GLM-4-9B-0414") >>> tokenizer = AutoTokenizer.from_pretrained("THUDM/GLM-4-9B-0414") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)r"r:)r+rAr,rrr:`szGlm4ForCausalLM.forward) rrrr r rr>rr:r?rrr,rr@_s r@c@s eZdZdS)Glm4ForSequenceClassificationNrrrrrrB}srBc@s eZdZdS)Glm4ForTokenClassificationNrrrrrrCsrC)ZGlm4PreTrainedModelZ Glm4Modelr@rBrC))typingrrr<Z cache_utilsrZmodeling_flash_attention_utilsrZmodeling_layersrZmodeling_outputsrZprocessing_utilsr utilsr r Zutils.deprecationr Zglm.modeling_glmrrrrZphi3.modeling_phi3rZconfiguration_glm4rZ modeling_glm4rZ get_loggerrloggerZ_CHECKPOINT_FOR_DOCrrr$r@rBrC__all__rrrrs*          1