Archives/langchain
2
0
Fork 1
mirror of https://github.com/hwchase17/langchain synced 2024-11-06 03:20:49 +00:00
Code Issues Packages Projects Releases Wiki Activity

Support recursive sitemaps in SitemapLoader (#3146)

Browse Source 
A (very) simple addition to support multiple sitemap urls.

---------

Co-authored-by: Johann-Peter Hartmann <johann-peter.hartmann@mayflower.de>
This commit is contained in:
Johann-Peter Hartmann 2023-04-22 17:48:04 +02:00 committed by GitHub
parent 215dcc2d26
commit 7e79f8c136
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 7 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
langchain/document_loaders/sitemap.py
View File

@ -61,6 +61,13 @@ class SitemapLoader(WebBaseLoader):
} }
) )
for sitemap in soup.find_all("sitemap"):
loc = sitemap.find("loc")
if not loc:
continue
soup_child = self.scrape_all([loc.text], "xml")[0]
els.extend(self.parse_sitemap(soup_child))
return els return els
def load(self) -> List[Document]: def load(self) -> List[Document]: