a hG@sddlmZddlmZmZddlmZmZddlZddl m Z ddl m Z ddl mZmZd d lmZmZmZeGd d d ZGd ddZdS))deque)floorsqrt)OptionalUnionN)PretrainedConfig)GenerationConfig) attach_tracertraced) RequestStateget_device_and_memory_breakdownloggerc @seZdZejdddfeeejeje e e e e e eje ffe e ddddZee e ee dddZee dd d d Ze d d dZe ee d ddZeeee ee dddZeejeje eejejfdddZdS)PagedAttentionCachedN)configgeneration_configdevicedtype num_requestslayer_device_maptp_sizereturncCs||_||_t|dd}|dur$|n|j|_t|dd} | durD| n |j|j|_|j|_t|dd|_|dur|dkr|j|dkrt d|jd |d t |j|j|j|j|j|j d } | j t|d dt|d dt|dd|jd\} } | |_ | |_td|j d|jd|j| |j|jf|_g|_g|_t|jD]t} |durV|| n|}tj|j|j|d}tj|j|j|d}tj|tj||j||j|q@tt| |_i|_dS)aInitialize a paged attention cache for efficient memory usage. Args: config: Model configuration generation_config: Generation configuration containing cache parameters device: Device for the cache tensors dtype: Data type for the cache tensors layer_device_map: Optional mapping of layer indices to devices initial_prompt_shapes: Optional sample prompts to help calculate optimal cache size num_key_value_headsNhead_dim block_size r rzNumber of key value heads z+ must be divisible by tensor parallel size .rr num_heads num_layers hidden_size vocab_size num_blocksmax_batch_tokensZ max_memory?)r$r%max_memory_percent cache_dtypez7PagedAttentionCache initialized with self.num_blocks = z and self.max_batch_tokens =  )rr)rrgetattrZnum_attention_headsrr"rZnum_hidden_layersr ValueErrorPagedAttentionMemoryHandlerr#%infer_num_blocks_and_max_batch_tokensr$r%rwarningZ cache_shape key_cache value_cacherangetorchZzerosZ_dynamoZmark_static_addressappendr _free_blocks _block_tables)selfrrrrrrrZkv_headsrZmemory_handlerr$r%idxZ layer_deviceZnew_layer_key_cacheZnew_layer_value_cacher8m/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/generation/continuous_batching/cache.py__init__sV         zPagedAttentionCache.__init__)n_blocks request_idrcCs\t|j|krdSg}t|D]}||jq||jvrHg|j|<|j|||S)z*Allocates n_blocks for a given request_id.F)lenr4r1r3popleftr5extend)r6r;r< allocated_r8r8r9allocate_blocksns   z#PagedAttentionCache.allocate_blocks)r<rcCs8||jvr$|j|}|j|ntd|dS)z.Frees all blocks associated with a request_id.z6Attempted to free blocks for non-existent request_id: N)r5popr4r?rinfo)r6r<Zblocks_to_freer8r8r9 free_blocks}s  zPagedAttentionCache.free_blocks)rcCs t|jS)z,Returns the number of free blocks available.)r=r4)r6r8r8r9get_num_free_blockssz'PagedAttentionCache.get_num_free_blockscCs|j|gS)z&Returns the block table for a request.)r5get)r6r<r8r8r9get_block_tablesz#PagedAttentionCache.get_block_table)statelogical_indicesrc Cs|j}|j|}|s$td||j}g}|D]X}||}||} |t|krltd|d|d|||} | || } || q2|S)a Maps logical sequence indices to physical cache indices using the block table, using PyTorch. Args: request_id: The request ID. logical_indices: A list of logical indices. Returns: A list of physical indices. Raises: ValueError: If no block table is found for the request ID. IndexError: If a logical index maps to a block index that is out of bounds. z!No block table found for request zLogical index z maps to block index z$ which is out of bounds for request )r<r5rGr+rr= IndexErrorr3) r6rIrJr<Z block_tablerZphysical_indicesr7Z block_idxZ block_offsetZphysical_block_numZphysical_indexr8r8r9_get_physical_indicess&    z)PagedAttentionCache._get_physical_indices) key_states value_states layer_idxrc Ks|j|j}|j||j||j}|j||j||j} |d|dd|ddf<|d| dd|ddf<|ddd|ddf| ddd|ddffS)Nr)r$rr/viewrrr0) r6rMrNrOZ read_indexZ write_indexkwargsZ total_slotsZ k_cache_flatZ v_cache_flatr8r8r9updates zPagedAttentionCache.update)__name__ __module__ __qualname__r2float16rr rrintrdictrstrr:r listrBrErFrHr rLZTensortuplerRr8r8r8r9rs8 Q'rc@seZdZejZdZejZdZ dZ e e e e e e ddddZ e dee d d d Zddd ejfee ee eejee e fd ddZd ejdfeejeee e fdddZd ejfe eeje dddZd ejfe eeje dddZddejfee ee ejee e e fdddZdS)r,iN)rrr r!r"r#rcCs(||_||_||_||_||_||_dSNr)r6rrr r!r"r#r8r8r9r:s z$PagedAttentionMemoryHandler.__init__?)r'rcCs,t\}}}}|t||}t||}|Sr^)rmaxrW)r'rAtotalreservedr@available_memoryr8r8r9get_available_memorys z0PagedAttentionMemoryHandler.get_available_memoryr&)r$r%r'r(rcCs|dur"|dur"|||\}}n>|durB|durB||||}n|dur`|dur`||||}||}|j|||d}t||krtd|d|||fS)a The memory footprint depends on the cache size C and the max batch tokens M in the following way: Mem = Mem(cache) + Mem(activation) + Mem(static_tensors) where: Mem(cache) = 2 * num_heads * head_dim * num_layers * cache_dtype.itemsize * C Mem(activation) = M * (hidden_size + vocab_size) * activation_dtype.itemsize Mem(static_tensors) ~= 8M * input_dtype.itemsize + M * C * activation_dtype.itemsize Depending on if C or M is given, we use different methods to infer the values (C = num_blocks * block_size) and since block_size is fixed, num_blocks is the true variable to find. N)r%r$r(zMemory footprint z is more than available memory )'compute_num_blocks_and_max_batch_tokenscompute_max_batch_tokenscompute_num_blocksrdcompute_memory_footprintsum MemoryError)r6r$r%r'r(rcZmemory_footprintr8r8r9r-s"   zAPagedAttentionMemoryHandler.infer_num_blocks_and_max_batch_tokensg{Gz?)r'r(mrcCsz||}td|||jj|j|j}d|j|j|j |j}d||j j}td|td|td|||jj}|||} | } | dd|| } | dkrt d | | t | d|} | dkrt d | t | |j} | |jkrs   .