community[patch]: fixed multithreading returning List[List[Documents]] instead of List[Documents] (#20230)

Description: When multithreading is set to True and using the
DirectoryLoader, there was a bug that caused the return type to be a
double nested list. This resulted in other places upstream not being
able to utilize the from_documents method as it was no longer a
`List[Documents]` it was a `List[List[Documents]]`. The change made was
to just loop through the `future.result()` and yield every item.
Issue: #20093
Dependencies: N/A
Twitter handle: N/A
pull/20233/head
Chip Davis 6 months ago committed by GitHub
parent 230376f183
commit 806d4ae48f
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194

@ -174,7 +174,8 @@ class DirectoryLoader(BaseLoader):
) )
) )
for future in concurrent.futures.as_completed(futures): for future in concurrent.futures.as_completed(futures):
yield future.result() for item in future.result():
yield item
else: else:
for i in items: for i in items:
yield from self._lazy_load_file(i, p, pbar) yield from self._lazy_load_file(i, p, pbar)

Loading…
Cancel
Save