a h2@sddlZddlZddlmZmZddlmZGdddeZGdddeZ Gd d d e Z Gd d d e Z Gd ddeZ GdddeZ dS)N)DatasetIterableDataset) ModelOutputc@s$eZdZddZddZddZdS)PipelineDatasetcCs||_||_||_dSNdatasetprocessparams)selfr r r r [/var/www/html/assistant/venv/lib/python3.9/site-packages/transformers/pipelines/pt_utils.py__init__ szPipelineDataset.__init__cCs t|jSrlenr r r r r__len__szPipelineDataset.__len__cCs"|j|}|j|fi|j}|Srr)r iitem processedr r r __getitem__s zPipelineDataset.__getitem__N)__name__ __module__ __qualname__rrrr r r rrsrc@s6eZdZd ddZddZddZdd Zd d ZdS) PipelineIteratorNcCs4||_||_||_|dkrd}||_d|_d|_dS)a Roughly equivalent to ``` for item in loader: yield infer(item, **params) ``` Arguments: loader (`torch.utils.data.DataLoader` or `Iterable`): The iterator that will be used to apply `infer` on. infer (any function): The function to apply of each element of `loader`. params (`dict`): The parameters passed to `infer` along with every item loader_batch_size (`int`, *optional*): If specified, the items of `loader` are supposed to come as batch, and are loader_batched here making it roughly behave as ``` for items in loader: for i in loader_batch_size: item = items[i] yield infer(item, **params) ```N)loaderinferr loader_batch_size_loader_batch_index_loader_batch_datar rrr rr r rrszPipelineIterator.__init__cCs t|jSr)rrrr r rr?szPipelineIterator.__len__cCst|j|_|Sriterriteratorrr r r__iter__Bs zPipelineIterator.__iter__cstjtjr$jjd}ni}jD]d\}}t|tr|}t|dtjrzt fdd|D||<q2t|dt j r2t fdd|D||<q2|dvrt|t rt|dtjrt fdd|D||<q2t|dt j r2t fdd|D||<q2|dkr q2|d ur4d ||<q2t|jtjr^|jd||<q2t|jt j rt |jd||<q2|j||<q2j |}jd 7_|S) ze Return item located at `loader_batch_index` within the current `loader_batch_data`. rc3s|]}|jdVqdSrNr unsqueeze.0elrr r Uz5PipelineIterator.loader_batch_item..c3s |]}t|jdVqdSr'np expand_dimsr r*rr rr-Wr.>Z hidden_statesZ attentionsc3s|]}|jdVqdSr'r(r*rr rr-\r.c3s |]}t|jdVqdSr'r/r*rr rr-^r.Zpast_key_valuesNr) isinstancer!torchTensorr r)itemsrZto_tupletupler0Zndarrayr1 __class__)r resultZloader_batchedkelementr rrloader_batch_itemFs:     z"PipelineIterator.loader_batch_itemcCs|jdur|j|jkr|St|j}|j|fi|j}|jdurt|tj rX|}n,t|t rl|d}nt | d}||}t|t rt |}n |jd}d|kr|jkrnn||_t|t r|dn||_d|_|S|SdS)Nr)r rr;nextr%rr r2r3r4r6listkeysrshaper!)r rr first_tensorkeyobserved_batch_sizer r r__next__ws(        zPipelineIterator.__next__)N)rrrrrr&r;rCr r r rrs  '1rcs.eZdZdfdd ZddZddZZS) PipelineChunkIteratorNcst|||dS)aI Roughly equivalent to ``` for iterator in loader: for item in iterator: yield infer(item, **params) ``` Arguments: loader (`torch.utils.data.DataLoader` or `Iterable`): The iterator that will be used to apply `infer` on. infer (any function): The function to apply of each element of `loader`. params (`dict`): The parameters passed to `infer` along with every item N)superrr"r7r rrszPipelineChunkIterator.__init__cCst|j|_d|_|Sr)r$rr% subiteratorrr r rr&s zPipelineChunkIterator.__iter__cCsr|jdur&|jt|jfi|j|_zt|j}Wn8tyl|jt|jfi|j|_t|j}Yn0|Sr)rGrr<r%r StopIteration)r rr r rrCs  zPipelineChunkIterator.__next__)N)rrrrr&rC __classcell__r r rFrrDsrDc@s eZdZdZddZddZdS)PipelinePackIteratora Roughly equivalent to ``` packed = [] for item in loader: packed.append(item) if item["is_last"]: yield packed packed = [] ``` but it also handles cases where `item` are batched (meaning it's a dict of Tensor with first dimension > 1. In that case it does ``` packed = [] for batch in loader: # item is batched for item in batch: packed.append(item) if item["is_last"]: yield packed packed = [] ``` Arguments: loader (`torch.utils.data.DataLoader` or `Iterable`): The iterator that will be used to apply `infer` on. infer (any function): The function to apply of each element of `loader`. params (`dict`): The parameters passed to `infer` along with every item loader_batch_size (`int`, *optional*): If specified, the items of `loader` are supposed to come as batch, and are loader_batched here making it roughly behave as ``` for items in loader: for i in loader_batch_size: item = items[i] yield infer(item, **params) ```cCst|j|_|Srr#rr r rr&s zPipelinePackIterator.__iter__cCsDd}g}|jdurP|j|jkrP|j|jkrP|}|d}|||r|Sq|s@|jt|jfi|j}|jdur&t |t j r|}nt | d}||}t |t rt|}n |jd}d|kr|jkrnn||_||_d|_|j|jkr>|}|d}|||r|SqqP|}|d}||qP|S)NFis_lastr)r rr;popappendrr<r%r r2r3r4r=r>rr?r!)r rKZ accumulatorrrr@rArBr r rrCsB             zPipelinePackIterator.__next__N)rrr__doc__r&rCr r r rrJs-rJc@s,eZdZeedddZddZddZdS) KeyDatasetr rAcCs||_||_dSrrP)r r rAr r rr.szKeyDataset.__init__cCs t|jSrrrr r rr2szKeyDataset.__len__cCs|j||jSrrPr rr r rr5szKeyDataset.__getitem__Nrrrrstrrrrr r r rrO-srOc@s.eZdZeeedddZddZddZdS) KeyPairDatasetr key1key2cCs||_||_||_dSrrU)r r rVrWr r rr:szKeyPairDataset.__init__cCs t|jSrrrr r rr?szKeyPairDataset.__len__cCs"|j||j|j||jdS)N)textZ text_pairrUrQr r rrBszKeyPairDataset.__getitem__NrRr r r rrT9srT)numpyr0r3Ztorch.utils.datarrZ utils.genericrrrrDrJrOrTr r r rs -d