a h2@s~ddlmZddlZddlmZddlmZer The API for the streamer classes is still under development and may change in the future. Parameters: tokenizer (`AutoTokenizer`): The tokenized used to decode the tokens. skip_prompt (`bool`, *optional*, defaults to `False`): Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots. decode_kwargs (`dict`, *optional*): Additional keyword arguments to pass to the tokenizer's `decode` method. Examples: ```python >>> from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer >>> tok = AutoTokenizer.from_pretrained("openai-community/gpt2") >>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2") >>> inputs = tok(["An increasing sequence: one,"], return_tensors="pt") >>> streamer = TextStreamer(tok) >>> # Despite returning the usual output, the streamer will also print the generated text to stdout. >>> _ = model.generate(**inputs, streamer=streamer, max_new_tokens=20) An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven, ``` Frbool) tokenizer skip_promptcKs(||_||_||_g|_d|_d|_dS)NrT)rr decode_kwargs token_cache print_lennext_tokens_are_prompt)r rrrr r r__init__Ks zTextStreamer.__init__cCst|jdkr&|jddkr&tdnt|jdkr<|d}|jrR|jrRd|_dS|j||jj |jfi|j }| dr||j d}g|_d|_ njt|dkr| t|dr||j d}|j t|7_ n*||j |dd}|j t|7_ ||dS) zm Receives tokens, decodes them, and prints them to stdout as soon as they form entire words. rz'TextStreamer only supports batch size 1FN  )lenshape ValueErrorrrrextendtolistrdecoderendswithr_is_chinese_charordrfindon_finalized_text)r r textprintable_textr r rrUs&   zTextStreamer.putcCs^t|jdkrB|jj|jfi|j}||jd}g|_d|_nd}d|_|j|dddS)z;Flushes any remaining cache and prints a newline to stdout.rNT) stream_end)r#rrr(rrrr-)r r.r/r r rrwszTextStreamer.endstrr.r1cCst|d|sdndddS)zNPrints the new text to stdout. If the stream is ending, also prints a newline.Tr0N)flushr)printr r.r1r r rr-szTextStreamer.on_finalized_textcCs|dkr|dks|dkr |dks|dkr0|dks|dkr@|dks|d krP|d ks|d kr`|d ks|d krp|dks|dkr|dkrdSdS)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTFr )r cpr r rr*sD  zTextStreamer._is_chinese_charN)F)F) rrrrrrrr-r*r r r rr)s ! "rcsNeZdZdZdddddfdd Zdd dd d d ZddZddZZS)TextIteratorStreamera Streamer that stores print-ready text in a queue, to be used by a downstream application as an iterator. This is useful for applications that benefit from accessing the generated text in a non-blocking way (e.g. in an interactive Gradio demo). The API for the streamer classes is still under development and may change in the future. Parameters: tokenizer (`AutoTokenizer`): The tokenized used to decode the tokens. skip_prompt (`bool`, *optional*, defaults to `False`): Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots. timeout (`float`, *optional*): The timeout for the text queue. If `None`, the queue will block indefinitely. Useful to handle exceptions in `.generate()`, when it is called in a separate thread. decode_kwargs (`dict`, *optional*): Additional keyword arguments to pass to the tokenizer's `decode` method. Examples: ```python >>> from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer >>> from threading import Thread >>> tok = AutoTokenizer.from_pretrained("openai-community/gpt2") >>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2") >>> inputs = tok(["An increasing sequence: one,"], return_tensors="pt") >>> streamer = TextIteratorStreamer(tok) >>> # Run the generation in a separate thread, so that we can fetch the generated text in a non-blocking way. >>> generation_kwargs = dict(inputs, streamer=streamer, max_new_tokens=20) >>> thread = Thread(target=model.generate, kwargs=generation_kwargs) >>> thread.start() >>> generated_text = "" >>> for new_text in streamer: ... generated_text += new_text >>> generated_text 'An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,' ``` FNrr float | Nonerrtimeoutc s.tj||fi|t|_d|_||_dSN)superrr text_queue stop_signalr;r rrr;r __class__r rrszTextIteratorStreamer.__init__r2r3cCs.|jj||jd|r*|jj|j|jddS)\Put the new text in the queue. If the stream is ending, also put a stop signal in the queue.r;N)r>rr;r?r6r r rr-sz&TextIteratorStreamer.on_finalized_textcCs|Sr<r rr r r__iter__szTextIteratorStreamer.__iter__cCs*|jj|jd}||jkr"tn|SdSNrD)r>getr;r? StopIterationr r r r__next__s zTextIteratorStreamer.__next__)FN)F) rrrrrr-rErI __classcell__r r rArr8s .r8csNeZdZdZdddddfdd Zdd dd d d ZddZddZZS)AsyncTextIteratorStreamera' Streamer that stores print-ready text in a queue, to be used by a downstream application as an async iterator. This is useful for applications that benefit from accessing the generated text asynchronously (e.g. in an interactive Gradio demo). The API for the streamer classes is still under development and may change in the future. Parameters: tokenizer (`AutoTokenizer`): The tokenized used to decode the tokens. skip_prompt (`bool`, *optional*, defaults to `False`): Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots. timeout (`float`, *optional*): The timeout for the text queue. If `None`, the queue will block indefinitely. Useful to handle exceptions in `.generate()`, when it is called in a separate thread. decode_kwargs (`dict`, *optional*): Additional keyword arguments to pass to the tokenizer's `decode` method. Raises: TimeoutError: If token generation time exceeds timeout value. Examples: ```python >>> from transformers import AutoModelForCausalLM, AutoTokenizer, AsyncTextIteratorStreamer >>> from threading import Thread >>> import asyncio >>> tok = AutoTokenizer.from_pretrained("openai-community/gpt2") >>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2") >>> inputs = tok(["An increasing sequence: one,"], return_tensors="pt") >>> # Run the generation in a separate thread, so that we can fetch the generated text in a non-blocking way. >>> async def main(): ... # Important: AsyncTextIteratorStreamer must be initialized inside a coroutine! ... streamer = AsyncTextIteratorStreamer(tok) ... generation_kwargs = dict(inputs, streamer=streamer, max_new_tokens=20) ... thread = Thread(target=model.generate, kwargs=generation_kwargs) ... thread.start() ... generated_text = "" ... async for new_text in streamer: ... generated_text += new_text >>> print(generated_text) >>> asyncio.run(main()) An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven, ``` FNrrr9r:c sFtj||fi|t|_d|_||_t|_t td|_ dS)Nr;) r=rasynciorr>r?r;Zget_running_looploophasattrhas_asyncio_timeoutr@rAr rrs   z"AsyncTextIteratorStreamer.__init__r2r3cCs.|j|jj||r*|j|jj|jdS)rCN)rMZcall_soon_threadsafer> put_nowaitr?r6r r rr-(sz+AsyncTextIteratorStreamer.on_finalized_textcCs|Sr<r rr r r __aiter__.sz#AsyncTextIteratorStreamer.__aiter__c szx|jrZt|j4IdH&|jIdH}WdIdHqv1IdHsN0Yntj|j|jdIdH}WntjytYn0||jkrtn|SdSrF) rOrLr;r>rGwait_for TimeoutErrorr?StopAsyncIterationr r r r __anext__1s<   z#AsyncTextIteratorStreamer.__anext__)FN)F) rrrrrr-rQrUrJr r rArrKs 5 rK) __future__rrLqueuertypingrZ models.autorrrr8rKr r r rs    yG