blob42/langchain
1
0
Fork 0
forked from Archives/langchain
Code Issues Pull Requests Packages Projects Releases Wiki Activity

bugfix: return whole document when loading with ConfluenceLoader.load by label (#3980)

Browse Source 
Method confluence.get_all_pages_by_label, returns only metadata about
documents with a certain label (such as pageId, titles, ...). To return
all documents with a certain label we need to extract all page ids given
a certain label and get pages content by these ids.

---------

Co-authored-by: Andrea Biondo <a.biondo@reply.it>
This commit is contained in:
obbiondo 2023-05-04 06:52:05 +02:00 committed by GitHub
parent 5f30cc8713
commit d18e788ee3
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 5 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
langchain/document_loaders/confluence.py
View File

@ -219,9 +219,11 @@ class ConfluenceLoader(BaseLoader):
max_pages=max_pages,
expand="body.storage.value",
)
docs += self.process_pages(
pages, include_restricted_content, include_attachments, include_comments
)
ids_by_label = [page["id"] for page in pages]
if page_ids:
page_ids = list(set(page_ids + ids_by_label))
else:
page_ids = list(set(ids_by_label))
if cql:
pages = self.paginate_request(