a h"@sddlZddlmZmZddlmZddlmZddlZddl m Z ddl m Z e dZee jeeejeeefd d d ZGd d d eZeGdddZeGdddZdS)N) dataclassfield)Enum)Optional)logging)tracedZContinuousBatchingLoggerreturncCstjrPtd}tjtjtj|j}tj|}tj |}nVtj j rtj j rtd}tj }|tj }d}ntd}d}d}d}||||fS)Ncudampsrcpu)torchr Z is_availabledeviceZ empty_cacheZ synchronizeZget_device_properties total_memoryZmemory_reservedZmemory_allocatedbackendsr Zis_builtZdriver_allocated_memoryZrecommended_max_memory)rrZreserved_memoryZallocated_memoryro/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/generation/continuous_batching/classes.pyget_device_and_memory_breakdowns"        rc@s,eZdZdZdZdZdZdZdZdZ dZ d S) RequestStatusz5Status of a generation request through its lifecycle.pendingZ prefillingZprefilling_splitZsplit_pending_remainderZdecodingfinishedfailedN) __name__ __module__ __qualname____doc__PENDINGZ PREFILLINGZPREFILLING_SPLITZSPLIT_PENDING_REMAINDERDECODINGFINISHEDZFAILEDrrrrr6src@seZdZUdZeed<eedZee ed<eedZ ee ed<eedZ ee ed<dZ eeed<ejZeed <eejdZe ed <ee dZee ed <dS) GenerationOutputayTracks the output of a generation request. Attributes: request_id (str): The ID of the generation request. prompt_ids (list[int]): The IDs of the prompt tokens. generated_tokens (list[int]): The generated tokens. logprobs (list[float]): The log probabilities of the generated tokens. error (Optional[str]): Any error message associated with the request. When None, the request was successful. status (RequestStatus): The status of the request. created_time (float): The time the request was created. next_token (Optional[int]): The next token to be generated. request_iddefault_factory prompt_idsgenerated_tokenslogprobsNerrorstatus created_time next_token)rrrrstr__annotations__rlistr$intr%r&floatr'rrrr(timer)r*rrrrr Bs  r c@sleZdZUdZeed<dZeee ed<dZ eee ed<e edZ ee ed<e edZ ee ed<e edZee ed <d Ze ed <ejZeed <d Ze ed<dZe ed<e ejdZeed<dZeeed<dZeeed<dZeeefed<eedddZejedddZddZe dddZ e ddd Z!e"e e#d!d"d#Z$d$d%Z%d&d'Z&dS)( RequestStateaTracks the state of a generation request through its lifecycle. Attributes: request_id (str): The ID of the generation request. full_prompt_ids (list[int] | None): The tokens IDs of the full prompt. prompt_ids (list[int] | None): The tokens IDs currently being processed. remaining_prompt_ids (list[int]): The tokens IDs remaining to be processed (for split requests). static_outputs (list[int]): The generated tokens. allocated_blocks (list[int]): The identifiers of the allocated blocks to the request. position_offset (int): The current position in the sequence for position_ids. status (RequestStatus): The status of the request: can be one of PENDING, PREFILLING, PREFILLING_SPLIT, SPLIT_PENDING_REMAINDER, DECODING, FINISHED, FAILED max_new_tokens (int): The maximum number of new tokens to generate. eos_token_id (int): The ID of the end-of-sequence token. created_time (float): The time the request was created. error (Optional[str]): Any error message associated with the request. When None, has had no error yet. next_token (Optional[str]): The next token to be generated. r!Nfull_prompt_idsr$r"remaining_prompt_idsstatic_outputsallocated_blocksrposition_offset_statusmax_new_tokens eos_token_idr)r'r*)r:r:lifespanr cCs|jS)N)r7selfrrrr(szRequestState.status)valuecCsL|jtjkrtdf|_n&|tjkrB|jdtf|_|||_dS)Nr:r)r7rrr0r<rlog_end_of_request)r>r?rrrr(s   c Cs`t|j}|}|jd|j}|jd|j}td|jd|d|d|d| dS)NrzRequest z finished: prefill_len = z decode_len = z start_time = z end_time = )lenr2 generated_lenr<r)loggerinfor!)r>Z prefill_lenZ decode_len start_timeZend_timerrrr@s "zRequestState.log_end_of_requestcCs|jS)zCGet the current length of the sequence (prompt + generated tokens).)r6r=rrr current_lenszRequestState.current_lencCs t|jS)z*Get the number of tokens generated so far.)rBr4r=rrrrCszRequestState.generated_len)token_idr cCs`|jtjkrdS||jko"|jdk}||jk}|r:|rH|j|g|sP|r\tj|_dSdS)zUpdate the request with a newly generated token and check for completion. Args: token_id: The token ID to add to the output sequence Returns: bool: True if the request is now complete, False otherwise Fr:T) r(rrr;rCr9r4extendr)r>rHZis_eosZ is_max_lenrrrupdate_with_tokens zRequestState.update_with_tokenc Cs~d|jd|jd|dt|jdt|jd|jdt|jd|jd |j g }d d |d S) Nz request_id=zstatus=z out_tokens=z query_length=zremaining_tokens=z kv_length=zfull_prompt_length=zallocated_blocks=zgenerated_tokens=zRequestState( z, z )) r!r7rCrBr$r3r6r2r5r4join)r>msgrrr__repr__s       zRequestState.__repr__c Cs"t|j|j|j|jg|j|jdS)z7Convert the request state to a GenerationOutput object.)r!r$r(r%r&r'r*)r r!r2r(r4r'r*r=rrrto_generation_outputsz!RequestState.to_generation_output)'rrrrr+r,r2rr-r.r$rr3r4r5r6rrr7r9r;r0r)r/r'r*r<tuplepropertyr(setterr@rGrCrboolrJrMrNrrrrr1[s4     r1)r0 dataclassesrrenumrtypingrrZ utils.loggingrZ utils.metricsr getLoggerrDsetLevelINFO staticmethodrOrr.rrr r1rrrrs