forked from Archives/langchain
enhancement: add elements mode to `UnstructuredURLLoader` (#3456)
### Summary Updates the `UnstructuredURLLoader` to include a "elements" mode that retains additional metadata from `unstructured`. This makes `UnstructuredURLLoader` consistent with other unstructured loaders, which also support "elements" mode. Patched mode into the existing `UnstructuredURLLoader` class instead of inheriting from `UnstructuredBaseLoader` because it significantly simplified the implementation. ### Testing This should still work and show the url in the source for the metadata ```python from langchain.document_loaders import UnstructuredURLLoader urls = ["https://www.understandingwar.org/sites/default/files/Russian%20Offensive%20Campaign%20Assessment%2C%20April%2011%2C%202023.pdf"] loader = UnstructuredURLLoader(urls=urls, headers={"Accept": "application/json"}, strategy="fast") docs = loader.load() print(docs[0].page_content[:1000]) docs[0].metadata ``` This should now work and show additional metadata from `unstructured`. This should still work and show the url in the source for the metadata ```python from langchain.document_loaders import UnstructuredURLLoader urls = ["https://www.understandingwar.org/sites/default/files/Russian%20Offensive%20Campaign%20Assessment%2C%20April%2011%2C%202023.pdf"] loader = UnstructuredURLLoader(urls=urls, headers={"Accept": "application/json"}, strategy="fast", mode="elements") docs = loader.load() print(docs[0].page_content[:1000]) docs[0].metadata ```fix_agent_callbacks
parent
a3e3f26090
commit
8e10ac422e
Loading…
Reference in New Issue