Archives/langchain
2
0
Fork 1
mirror of https://github.com/hwchase17/langchain synced 2024-11-02 09:40:22 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: load all namespaces (#13549)

Browse Source 
- **Description:** This change allows for the `MWDumpLoader` to load all
namespaces including custom by default instead of only loading the
[default
namespaces](https://www.mediawiki.org/wiki/Help:Namespaces#Localisation).
  - **Tag maintainer:** @hwchase17
This commit is contained in:
Andrew Teeter 2023-11-19 20:35:17 -05:00 committed by GitHub
parent 47451764a7
commit 325bdac673
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
libs/langchain/langchain/document_loaders/mediawikidump.py
View File

@ -55,7 +55,7 @@ class MWDumpLoader(BaseLoader):
self.file_path = file_path if isinstance(file_path, str) else str(file_path)
self.encoding = encoding
# Namespaces range from -2 to 15, inclusive.
self.namespaces = namespaces or list(range(-2, 16))
self.namespaces = namespaces
self.skip_redirects = skip_redirects
self.stop_on_error = stop_on_error
@ -76,7 +76,7 @@ class MWDumpLoader(BaseLoader):
for page in dump.pages:
if self.skip_redirects and page.redirect:
continue
if page.namespace not in self.namespaces:
if self.namespaces and page.namespace not in self.namespaces:
continue
try:
for revision in page: