a h|@s&ddlZddlZddlZddlZddlZddlZddlZddlZddlm Z m Z ddl m Z ddl mZmZddlmZddlmZddlZddlmZmZddlmZmZmZdd lmZdd lmZm Z dd l!m"Z"m#Z#e$d krddl%Z%e"rdd l&m'Z'ddl(m)Z)ddl*m+Z+e#rDddl,Z,ddlm-Z-mZm.Z.mZe/ej0ej1Z2e/ej0ej3ej1dZ4ddiddiddiddiddiddiddidZ5dZ6dd7e58dZ9Gdd d Z:eGd!d"d"Z;e d#d$d%Zd(kr"e;Z?d)e?_@d*e?_@e=e?ZAeABdS)+N)ArgumentParser Namespace) AsyncIterator) dataclassfield)Thread)Optional)AsyncInferenceClientChatCompletionStreamOutput) AutoTokenizerGenerationConfigPreTrainedTokenizer)BaseTransformersCLICommand)ServeArguments ServeCommand)is_rich_availableis_torch_availableWindows)Console)Live)Markdown)AutoModelForCausalLMr BitsAndBytesConfigr z .!\"#$%&'()*+,\-/:<=>?@[]^_`{|}~textz5There is a Llama in my lawn, how can I get rid of it?zyWrite a Python function that integrates any Python function f(x) numerically over an arbitrary interval [x_start, x_end].z4How many helicopters can a human eat in one sitting?z4Count to 10 but skip every number ending with an 'e'zWhy aren't birds real?z2Why is it important to eat socks after meditating?z$Which number is larger, 9.9 or 9.11?)llamacode helicopternumbersZbirdssocksZnumbers2a **TRANSFORMERS CHAT INTERFACE** Chat interface to try out a model. Besides chatting with the model, here are some basic commands: - **!help**: shows all available commands (set generation settings, save chat, etc.) - **!status**: shows the current status of the model and generation settings - **!clear**: clears the current conversation and starts a new one - **!exit**: closes the interface am **TRANSFORMERS CHAT INTERFACE HELP** Full command list: - **!help**: shows this help message - **!clear**: clears the current conversation and starts a new one - **!status**: shows the current status of the model and generation settings - **!example {NAME}**: loads example named `{NAME}` from the config and uses it as the user input. Available example names: `z`, `a%` - **!set {ARG_1}={VALUE_1} {ARG_2}={VALUE_2}** ...: changes the system prompt or generation settings (multiple settings are separated by a space). Accepts the same flags and format as the `generate_flags` CLI argument. If you're a new user, check this basic flag guide: https://huggingface.co/docs/transformers/llm_tutorial#common-options - **!save {SAVE_NAME} (optional)**: saves the current chat and settings to file by default to `./chat_history/{MODEL_NAME}/chat_{DATETIME}.yaml` or `{SAVE_NAME}` if provided - **!exit**: closes the interface c@seZdZdeeeedddZeeeee fdddZ edd d Z d d Z ed ddZ eedddZdedddZeeedddZdS) RichInterfaceN model_name user_namecCs8t|_|durd|_n||_|dur.d|_n||_dS)N assistantuser)r_consoler!r")selfr!r"r'V/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/commands/chat.py__init__sszRichInterface.__init__)streamreturnc s|jd|jdt|jdd}d}|IdH2z3dHW}|jdjj}|sTq4tdd|}||7}g}| D].}| || d r| d qv| d qvt d |d d }|j|ddq46Wdn1s0Y|j|S)Nz [bold blue]:)consolerefresh_per_secondrz <(/*)(\w*)>z\<\1\2\>z``` z z github-dark)Z code_themeT)refresh)r%printr!rchoicesdeltacontentresub splitlinesappend startswithrjoinstripupdate) r&r*livertokenoutputslineslinemarkdownr'r'r( stream_output~s&      0 zRichInterface.stream_outputr+cCs$|jd|jd}|j|S)z!Gets user input from the console. [bold red]: )r%inputr"r2)r&rGr'r'r(rGs zRichInterface.inputcCs|jdS)zClears the console.N)r%clearr&r'r'r(rHszRichInterface.clear)rcCs(|jd|jd||jdS)z%Prints a user message to the console.rFz>:[/ bold red] N)r%r2r")r&rr'r'r(print_user_messagesz RichInterface.print_user_messagercolorcCs&|jd|d||jdS)z,Prints text in a given color to the console.z[bold ]Nr%r2)r&rrLr'r'r( print_colorszRichInterface.print_colorFminimalcCs&|jt|rtnt|jdS)z'Prints the help message to the console.N)r%r2rHELP_STRING_MINIMAL HELP_STRING)r&rQr'r'r( print_helpszRichInterface.print_helpr!generation_config model_kwargscCsJ|jd|d|r*|jd||jd||jdS)zFPrints the status of the model and generation settings to the console.z[bold blue]Model: r0z[bold blue]Model kwargs: z [bold blue]NrN)r&r!rVrWr'r'r( print_statuss zRichInterface.print_status)NN)F)__name__ __module__ __qualname__rstrr)rr tupleintrDrGrHrJrOboolrTr dictrXr'r'r'r(rrs .rc@seZdZUdZedddidZeeed<edddidZ eeed<eddd idZ eeed <ed dd idZ eed <edddidZ eeed<edddidZ eed<edddidZeeed<edddidZeed<edddidZeed<eddgdddZeeed<eddgdddZeeed <eddd!idZeed"<eddd#idZeeed$<eddd%idZeed&<eddd'idZeed(<ed)d*d+d)gddZeed,<eddd-idZeed.<ed/dd0idZeed1<ed2dd3idZeed4<d5d6ZdS)7 ChatArgumentsz Arguments for the chat CLI. See the metadata arg for each argument's description -- the medatata will be printed with `transformers chat --help` Nhelpz_Name of the pre-trained model. The positional argument will take precedence if both are passed.)defaultmetadatamodel_name_or_pathzKUsername to display in chat interface. Defaults to the current user's name.r$zSystem prompt. system_promptz./chat_history/zFolder to save chat history. save_folderz"Path to a yaml file with examples. examples_pathFz7Whether to show runtime warnings in the chat interface.verbosezPath to a local generation config file or to a HuggingFace repo containing a `generation_config.json` file. Other generation settings passed as CLI arguments will be applied on top of this generation config.rVmainzLSpecific model version to use (can be a branch name, tag name or commit id).model_revisionautozDevice to use for inference.devicezA`torch_dtype` is deprecated! Please use `dtype` argument instead.)rlZbfloat16Zfloat16Zfloat32)rbr3 torch_dtypezOverride the default `torch.dtype` and load the model under this dtype. If `'auto'` is passed, the dtype will be automatically derived from the model's weights.dtypez2Whether to trust remote code when loading a model.trust_remote_codezWhich attention implementation to use; you can run --attn_implementation=flash_attention_2, in which case you must install this manually by running `pip install flash-attn --no-build-isolation`.attn_implementationzIWhether to use 8 bit precision for the base model - works only with LoRA. load_in_8bitzIWhether to use 4 bit precision for the base model - works only with LoRA. load_in_4bitZnf4zQuantization type.Zfp4bnb_4bit_quant_typez#Whether to use nested quantization.use_bnb_nested_quant localhostz%Interface the server will listen to..host@zPort the server will listen to.portcCs |jdur|jdkr|j|_dS)z(Only used for BC `torch_dtype` argument.Nrl)rnrorIr'r'r( __post_init__!szChatArguments.__post_init__)rYrZr[__doc__rrerr\__annotations__r$rfrgrhrir_rVrkrmrnrorprqrrrsrtrurwryr^rzr'r'r'r(rast  raargscCst|S)z; Factory function used to chat with a local model. ) ChatCommandr}r'r'r(chat_command_factory(src @sLeZdZeedddZddZeedddZed(e e eed d d Z ed)e ee e d ddZe ee dddZe eeee fdddZeeee ee eeee efdddZee e ddddZe edefddd Zee ee ee eeffee e e ee e ee fd!d"d#Zd$d%Zd&d'Zd S)*r)parsercCsTtf}|jd|d}|d}|jdtddd|jdtdd d d |jtd dS) z Register this command to argparse so it's available for the transformer-cli Args: parser: Root parser to register command-specific arguments chat)dataclass_typeszPositional argumentsmodel_name_or_path_or_addressNz7Name of the pre-trained model or address to connect to.)typercrbgenerate_flagsaFlags to pass to `generate`, using a space as a separator between flags. Accepts booleans, numbers, and lists of integers, more advanced parameterization should be set through --generation-config. Example: `transformers chat max_new_tokens=100 do_sample=False eos_token_id=[1,2]`. If you're a new user, check this basic flag guide: https://huggingface.co/docs/transformers/llm_tutorial#common-options*)rrcrbnargs)func)ra add_parseradd_argument_group add_argumentr\ set_defaultsr)rrZ chat_parsergroupr'r'r(register_subcommand0s"  zChatCommand.register_subcommandcCs|jdur|j}|ds.|ds.|drzd|_|jdksH|jdkrPtd|jdd\|_|_|jdurtd nd |_|j|_tst s|jrt d n$tst d nt s|jrt d ||_ dS)NhttphttpsrvFrxuuLooks like you’ve set both a server address and a custom host/port. Please pick just one way to specify the server.:z\When connecting to a server, please specify a model name with the --model_name_or_path flag.TzYou need to install rich to use the chat interface. Additionally, you have not specified a remote endpoint and are therefore spawning a backend. Torch is required for this: (`pip install rich torch`)zHYou need to install rich to use the chat interface. (`pip install rich`)zYou have not specified a remote endpoint and are therefore spawning a backend. Torch is required for this: (`pip install rich torch`)) rr: spawn_backendrwry ValueErrorrsplitrerr ImportErrorr~)r&r~namer'r'r(r)Qs4    zChatCommand.__init__rEcCs(tdkrtSttjSdS)z)Returns the username of the current user.rN)platformsystemosgetloginpwdgetpwuidgetuidpw_namer'r'r'r( get_usernamexs zChatCommand.get_usernameN)r~filenamer+cCsi}t||d<||d<|j}|durPtd}|jd|d}tj||}tjtj |ddt |d  }t j ||d d Wdn1s0Ytj |S) z!Saves the chat history to a file.settingsZ chat_historyNz%Y-%m-%d_%H-%M-%Sz/chat_.jsonT)exist_okwr,)indent)varsrgtimestrftimerrpathr;makedirsdirnameopenjsondumpabspath)rr~rZ output_dictfolderZtime_strfr'r'r( save_chats   .zChatCommand.save_chat)rfr+cCs|durg}n d|dg}|S)zClears the chat history.NrZroler5r')rfrr'r'r(clear_chat_historys zChatCommand.clear_chat_history)rr+cst|dkriSdd|D}dd|D}dd|D}ttdddfd d|D}d d d |D}d |d}|dd}|dd}|dd}|dd}|dd}|dd}zt|}Wntjyt dYn0|S)zUParses the generate flags from the user input into a dictionary of `generate` kwargs.rcSs.i|]&}d|ddd|ddqS)"=rr)split).0flagr'r'r( z4ChatCommand.parse_generate_flags..cSs*i|]"\}}||dvr"|n|qS))truefalse)lowerrkvr'r'r(rscSs"i|]\}}||dkrdn|qS)Nonenullr'rr'r'r(rr)sr+cSs(|dr|dd}|dddS)N-r.r/)r:replaceisdigit)rr'r'r( is_numbers  z3ChatCommand.parse_generate_flags..is_numbercs*i|]"\}}||s"d|dn|qS)rr'rrr'r(rrz, cSsg|]\}}|d|qS)z: r'rr'r'r( rz4ChatCommand.parse_generate_flags..{}z"null"rz"true"rz"false"rz"[[z]"rMrrzFailed to convert `generate_flags` into a valid JSON object. `generate_flags` = {generate_flags} Converted JSON string = {generate_flags_string}) lenitemsr\r_r;rrloadsJSONDecodeErrorr)r&rZgenerate_flags_as_dictZgenerate_flags_stringZprocessed_generate_flagsr'rr(parse_generate_flagss0         z ChatCommand.parse_generate_flags)r~model_generation_configr+cCs|jdurLd|jvr>tj|j}tj|j}t||}qlt|j}n t|}|j fiddd| |j }|j fi|}||fS)zj Returns a GenerationConfig object holding the generation parameters for the CLI command. NrT)Z do_sampleZmax_new_tokens) rVrrrbasenamer from_pretrainedcopydeepcopyr=rr)r&r~rrrrVZparsed_generate_flagsrWr'r'r(get_generation_parameterizations    z+ChatCommand.get_generation_parameterization) tokenizerrV eos_tokens eos_token_idsr+cCs||jdur|j}n|j}g}|dur:|||d|dur\|dd|dDt|dkrt||j||fS)z:Retrieves the pad token ID and all possible EOS token IDs.N,cSsg|] }t|qSr')r^)rZtoken_idr'r'r(rrz0ChatCommand.parse_eos_tokens..r) pad_token_idZ eos_token_idextendZconvert_tokens_to_idsrrr9)rrVrrrZall_eos_token_idsr'r'r(parse_eos_tokenss   zChatCommand.parse_eos_tokensr) model_argsr+cCs<|jr"td|j|j|j|jd}n|jr4tdd}nd}|S)NT)rsZbnb_4bit_compute_dtypertZbnb_4bit_use_double_quantZbnb_4bit_quant_storage)rr)rsrrortrurr)rquantization_configr'r'r(get_quantization_config sz#ChatCommand.get_quantization_configr)r~r+cCstj|j|j|jd}|jdvr&|jn tt|j}||}|j|j |d|d}t j|jfd|ji|}t|dddur| |j }||fS)N)revisionrp)rlNrl)rrqroZ device_maprrpZ hf_device_map) r rZmodel_name_or_path_positionalrkrprogetattrtorchrrqrtorm)r&r~rrorrWmodelr'r'r(load_model_and_tokenizer s.  z$ChatCommand.load_model_and_tokenizer) user_inputr~ interfaceexamplesrVrWrr+cCsd}|dkr$||j}|n|dkr8|n|drt|dkr|} t| dkrp| d} nd} |||| } |jd| d d d nT|d r|d d } | } | D](} d| vr|jd| ddd qq| | } |j fi| }|j fi|n|drt|dkr|d}||vr|g}| ||d| d||ddn(d|dt|d}|j|dd n@|dkr|j|j||dn"d}|jd|ddd |||||fS)z Handles all user commands except for `!exit`. May update the chat history (e.g. reset it) or the generation config (e.g. set a new flag). Tz!clearz!helpz!saverNzChat saved in !greenrKz!setr,rz(Invalid flag format, missing `=` after `z;`. Please use the format `arg_1=value_1 arg_2=value_2 ...`.red!examplerr$rzExample z* not found in list of available examples: rz!statusrUF'z/' is not a valid command. Showing help message.)rrfrHrTr:rrrrOr<rr=rJr9listkeysrXre)r&rr~rrrVrWr valid_commandZ split_inputrZnew_generate_flagsrZparsed_new_generate_flagsZnew_model_kwargsZ example_nameZ example_errorr'r'r(handle_non_exit_user_commands;s\           z)ChatCommand.handle_non_exit_user_commandscCst|dS)N)asynciorun _inner_runrIr'r'r(rszChatCommand.runc s|jrnt|jj|jj|jj|jj|jj|jj|jj |jj |jj |jj dd }t |}t|jd}d|_||jjd|jj}|jj dkrdn|jj }t|d|jj }|j}|jdurt}n4t|j} t| }Wdn1s0Y|jdur|} n|j} t|j} ||| \} } t|j| d }| |!|j"}|j#dd z z|$}|%d r|d krWW|&IdHqn |j'||||| | |d \}}} } |r|%dsWW|&IdHq`n|(d|d|j)|d| *|dd}|+|IdH}|(d|dWn*t,y\YW|&IdHqYn0W|&IdHn|&IdH0q`dS)Nerror) rmrorprqrrrsrtrurwryZ log_level)targetT@rvzhttp://localhostrr rPrz!exit)rr~rrrVrWrrr$r)rVr)r*Z extra_bodyr#)-rrr~rmrorprqrrrsrtrurwryrrrdaemonstartrerkr rhDEFAULT_EXAMPLESryamlZ safe_loadr$rr rrrrHrrfrTrGr:closerr9Zchat_completionZto_json_stringrDKeyboardInterrupt)r&Z serve_argsZ serve_commandthreadrrwclientr~rrr$rrVrWrrrrr*Z model_outputr'r'r(rs    (       !   zChatCommand._inner_run)N)N)rYrZr[ staticmethodrrr)r\rrarrrr`rrr r]rr r^rrr rrrrrr'r'r'r(r/sD ' 7  Tr__main__z meta-llama/Llama-3.2-3b-Instructzhttp://localhost:8000)Crrrrrr6stringrargparserrcollections.abcr dataclassesrr threadingrtypingrrZhuggingface_hubr r Z transformersr r r Ztransformers.commandsrZtransformers.commands.servingrrZtransformers.utilsrrrrZ rich.consolerZ rich.liverZ rich.markdownrrrrset ascii_letters whitespaceZALLOWED_KEY_CHARSdigitsZALLOWED_VALUE_CHARSrrRr;rrSrrarrrYr~rrrr'r'r'r(sv          \Y@