a ht= @sddlZddlZddlZddlmZddlmZmZmZm Z ddl Z GdddeZ z(ddl m Z ddlmZmZmZdZWneyd ZYn0dd d Zddd dd eeeeee eeegefffd ddZeeZeGdddZdS)N)Enum)AnyCallableOptionalUnionc@s,eZdZdZdZdZdZdZdZdZ dZ d S) RequestStatusz5Status of a generation request through its lifecycle.pendingZ prefillingZprefilling_splitZsplit_pending_remainderZdecodingfinishedfailedN) __name__ __module__ __qualname____doc__ZPENDING PREFILLINGPREFILLING_SPLITZSPLIT_PENDING_REMAINDERDECODINGFINISHEDZFAILEDrrV/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/utils/metrics.pyr sr)metrics)Status StatusCode get_tracerTFcsts ddSfdd}|S)a Decorator that attaches a tracer to a class. This decorator should be applied to classes that need OpenTelemetry tracing. It adds a tracer attribute to the class instance that can be used by the traced decorator. Args: tracer_name_template: Optional template string for the tracer name. If provided, it should contain {module} which will be replaced with the class's full module path and {class_name} for the class name. If None, a default naming scheme will be used where: - If the module already starts with "transformers.", it will use that directly - Otherwise, it will prepend "transformers." to the module name Returns: Class decorator function cSs|S)Nr)clsrrr2zattach_tracer..cs*jtfdd}|_S)Ncsr|g|Ri|j}j}durV|drD|d|}qdd|d|}nj||d}t||_dS)N transformers..)module class_name)r r startswithformatrtracer)selfargskwargs module_namerZ tracer_name)r original_inittracer_name_templaterrinit_with_tracer7s z:attach_tracer..decorator..init_with_tracer)__init__ functoolswraps)rr)r()rr'r decorator4s z attach_tracer..decorator)_has_opentelemetry)r(r.rr-r attach_tracers r0) span_name standaloneadditional_attributes)r3cs$fdd}|dur|S||S)a Decorator to trace function calls with OpenTelemetry. Can be used as @traced or @traced(span_name="custom_name") Args: func: The function to trace span_name: Optional custom name for the span (defaults to function name) standalone: If True, creates a parentless span additional_attributes: Optional list of additional attributes to set on the span. Each item is a tuple of (instance_attribute_name, span_attribute_key, value_or_transform_function) where: - instance_attribute_name: Name of the attribute to get from the class instance - span_attribute_key: Key to use when setting the attribute on the span - value_or_transform_function: Either a raw value to use directly, or a function to transform the attribute value before setting it on the span Returns: Decorated function with tracing cs0tsSddl}|fdd}|S)Nrc s\|r tdr jdur |dnd}|du}|rBt|drB|j}ntdjdj}pbj}rn|jn|j}||}|dj|dj|d||r t |D]T\}} t | t t t tfs| dur|d |t | q|d |t t| q|rv|D]Z\} } t | t t t tfs@| durX|d | t | n|d | t t| qr|r҈D]J} | \} }}t|| rt|| }t|r||}n|}|||qz"|i|}|WWdSty6}z(|ttj||WYd}~n d}~00Wdn1sN0YdS) N__self__rr"rrz function.namezfunction.modulezfunction.is_methodzargs.zkwargs.)hasattrr4r"rr r Z start_spanZstart_as_current_spanZ set_attribute enumerate isinstancestrintfloatbooltypeitemsgetattrcallable ExceptionZ set_statusrrERRORZrecord_exception)r$r%instanceZ is_methodr"nameZspan_fnspaniargkeyvalueZ attr_configZinstance_attribute_nameZspan_attribute_keyZvalue_or_transform_functionZattribute_valueZtransformed_valueresulte)r3funcr1r2rrwrapperpsJ$          z*traced..decorator..wrapper)r/r+r,)rKr+rLr3r1r2)rKrr.js 0ztraced..decoratorNr)rKr1r2r3r.rrMrtracedNs9rNc@seZdZdZedddZddZeee ddd d Z ee dd d d Z eddddZ eeeddddZeee ddddZdS)ContinuousBatchProcessorMetricsz0Metrics collection for ContinuousBatchProcessor.)max_batch_tokenscCs||_|dS)zInitialize metrics for continuous batch processor. Args: max_batch_tokens: Maximum number of tokens in a batch N)rP_setup_metrics)r#rPrrrr*sz(ContinuousBatchProcessorMetrics.__init__cCststddStd|_gd}|jjddd|d|_|jjd d d d |_ |jjd dd d |_ gd}|jjddd|d|_ |jjdddd |_ |jj dddd |_|jj dddd |_gd}|jjddd|d|_|jjddd d |_|jjd!d"d d |_dS)#zIInitialize OpenTelemetry metrics and tracing if the library is available.zIOpenTelemetry is not installed. Metrics and tracing will not be recorded.Nz2transformers.generation.continuous_batch_processor) 2Kdi,i'Zttft_millisecondsz#Time to first token in millisecondsms)rC descriptionunitZ#explicit_bucket_boundaries_advisoryZactive_requests_countz3Number of active requests currently being processedrequests)rCr_r`Zwaiting_requests_countz*Number of requests waiting to be processed) rTrVrYrZr[r\r]i Ni0ui`Zrequest_latency_millisecondsz9End-to-end latency for completed requests in millisecondsZdecode_prefill_ratioz3Ratio of decode tokens to prefill tokens in a batchratioZprefill_tokens_processedz"Number of prefill tokens processedtokensZdecode_tokens_processedz!Number of decode tokens processed) rR(rT<FPZ_brVZbatch_fill_percentagez5Percentage of max_batch_tokens utilized in each batchpercentZkv_cache_free_memory_bytesz/Free memory of the PagedAttentionCache in bytesbytesZkv_cache_memory_bytesz0Memory usage of the PagedAttentionCache in bytes)r/loggerinforZ get_meterZmeterZcreate_histogramttft_histogramZ create_gaugeactive_requests_gaugewaiting_requests_gaugerequest_latency_histogramdecode_prefill_ratio_gaugeZcreate_counterprefill_tokens_counterdecode_tokens_counterbatch_fill_percentage_histogramkv_cache_free_memory_gaugekv_cache_memory_gauge)r#Z ttft_bucketsZlatency_bucketsZbatch_fill_bucketsrrrrQsx  z.ContinuousBatchProcessorMetrics._setup_metricsN) created_time request_idreturnc Cs|tsdSt|d}z*|j|td|d|ddWn4tyv}ztd|WYd}~n d}~00dS)zRecord Time to First Token (TTFT). Args: created_time: The time the request was created request_id: The ID of the request N@@zRecorded TTFT for request : .2fr^zFailed to record TTFT metric: )r/timersrecordrqdebugr@warning)r#r}r~Zttft_msrJrrrrecord_ttft_metric s z2ContinuousBatchProcessorMetrics.record_ttft_metric)requests_in_batchrc Cs(tr|s dSd}d}|D]:}|jtjkr2|d7}q|jtjtjfvr|t|j7}q||}z|dkrr|j ||dkr|j ||dkr||}|j |||j d}|j|td|d|d|dd |d |j d Wn6ty"}ztd |WYd}~n d}~00dS) zRecord metrics about the batch composition including decode/prefill ratio and batch fill percentage. Args: requests_in_batch: List of request states in the current batch NrgY@zBatch metrics: z decode tokens, z prefill tokens, batch fill: rz% (/)z Failed to record batch metrics: )r/statusrrrrlenZ prompt_idsrxaddryrwsetrPrzrrqrr@r) r#rZ decode_tokensZprefill_tokensstateZtotal_batch_tokensrcZfill_percentagerJrrrrecord_batch_metricss>      z4ContinuousBatchProcessorMetrics.record_batch_metrics)rc CstsdSz|jt|j}t|j}|jtjtjfvr:dnd}|||j |j |j d|}|t|j|j |j |j d|}|j ||j |td|ddd|d|jd ||jd d d Wn6ty}ztd |WYd}~n d}~00dS)a&Record memory usage of the PagedAttentionCache without GPU synchronization. This calculates the theoretical memory usage based on cache configuration and the number of blocks currently in use. Args: cache: The PagedAttentionCache object to measure NzKV Cache memory: irzMB, Used blocks: rz (rVz.1fz%)z*Failed to record KV cache memory metrics: )r/Z num_blocksrZ _free_blocksZ key_cacheZdtypetorchZfloat16Zbfloat16 block_sizeZnum_key_value_headsZhead_dimr|rr{rqrr@r)r#cacheZnum_used_blocksZ num_layersZbytes_per_parameterZ memory_bytesZfree_memory_bytesrJrrrrecord_kv_cache_memory_metricsFs^     z>ContinuousBatchProcessorMetrics.record_kv_cache_memory_metrics)active_requestswaiting_requestsrc CsvtsdSz4|j||j|td|d|dWn4typ}ztd|WYd}~n d}~00dS)zRecord metrics about active and waiting requests. Args: active_requests: Number of active requests waiting_requests: Number of waiting requests NzQueue metrics: z active requests, z waiting requestsz Failed to record queue metrics: )r/rtrrurqrr@r)r#rrrJrrrrecord_queue_metrics|s  z4ContinuousBatchProcessorMetrics.record_queue_metricsc Cs|tsdSt|d}z*|j|td|d|ddWn4tyv}ztd|WYd}~n d}~00dS)zRecord metrics about a completed request. Args: created_time: The time the request was created request_id: The ID of the request Nrz Recorded request completion for rrr^z,Failed to record request completion metric: )r/rrvrrqrr@r)r#r}r~Z latency_msrJrrrrecord_request_completions z9ContinuousBatchProcessorMetrics.record_request_completion)r r r rr9r*rQrNr:r8rlistrrrrrrrrrOs Q(5rO)N)N)r+loggingrenumrtypingrrrrrrZ opentelemetryrZopentelemetry.tracerrrr/ ImportErrorr0rtupler8rN getLoggerr rqrOrrrrs0      0& Z