Lance Martin 7a00f17033 Web research retriever (#8102) ... Given a user question, this will - * Use LLM to generate a set of queries. * Query for each. * The URLs from search results are stored in self.urls. * A check is performed for any new URLs that haven't been processed yet (not in self.url_database). * Only these new URLs are loaded, transformed, and added to the vectorstore. * The vectorstore is queried for relevant documents based on the questions generated by the LLM. * Only unique documents are returned as the final result. This code will avoid reprocessing of URLs across multiple runs of similar queries, which should improve the performance of the retriever. It also keeps track of all URLs that have been processed, which could be useful for debugging or understanding the retriever's behavior. --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>		2023-07-25 19:58:00 -07:00
..
document_transformers	mv module integrations docs (#8101)	2023-07-23 23:23:16 -07:00
retrievers	Web research retriever (#8102)	2023-07-25 19:58:00 -07:00