mirror of
https://github.com/hwchase17/langchain
synced 2024-11-02 09:40:22 +00:00
7be68228da
The `_amake_session()` method does not allow modifying the `self.session_factory` with anything other than `async_sessionmaker`. This prohibits advanced uses of `index()`. In a RAG architecture, it is necessary to import document chunks. To keep track of the links between chunks and documents, we can use the `index()` API. This API proposes to use an SQL-type record manager. In a classic use case, using `SQLRecordManager` and a vector database, it is impossible to guarantee the consistency of the import. Indeed, if a crash occurs during the import (problem with the network, ...) there is an inconsistency between the SQL database and the vector database. With the [PR](https://github.com/langchain-ai/langchain-postgres/pull/32) we are proposing for `langchain-postgres`, it is now possible to guarantee the consistency of the import of chunks into a vector database. It's possible only if the outer session is built with the connection. ```python def main(): db_url = "postgresql+psycopg://postgres:password_postgres@localhost:5432/" engine = create_engine(db_url, echo=True) embeddings = FakeEmbeddings() pgvector:VectorStore = PGVector( embeddings=embeddings, connection=engine, ) record_manager = SQLRecordManager( namespace="namespace", engine=engine, ) record_manager.create_schema() with engine.connect() as connection: session_maker = scoped_session(sessionmaker(bind=connection)) # NOTE: Update session_factories record_manager.session_factory = session_maker pgvector.session_maker = session_maker with connection.begin(): loader = CSVLoader( "data/faq/faq.csv", source_column="source", autodetect_encoding=True, ) result = index( source_id_key="source", docs_source=loader.load()[:1], cleanup="incremental", vector_store=pgvector, record_manager=record_manager, ) print(result) ``` The same thing is possible asynchronously, but a bug in `sql_record_manager.py` in `_amake_session()` must first be fixed. ```python async def _amake_session(self) -> AsyncGenerator[AsyncSession, None]: """Create a session and close it after use.""" # FIXME: REMOVE if not isinstance(self.session_factory, async_sessionmaker):~~ if not isinstance(self.engine, AsyncEngine): raise AssertionError("This method is not supported for sync engines.") async with self.session_factory() as session: yield session ``` Then, it is possible to do the same thing asynchronously: ```python async def main(): db_url = "postgresql+psycopg://postgres:password_postgres@localhost:5432/" engine = create_async_engine(db_url, echo=True) embeddings = FakeEmbeddings() pgvector:VectorStore = PGVector( embeddings=embeddings, connection=engine, ) record_manager = SQLRecordManager( namespace="namespace", engine=engine, async_mode=True, ) await record_manager.acreate_schema() async with engine.connect() as connection: session_maker = async_scoped_session( async_sessionmaker(bind=connection), scopefunc=current_task) record_manager.session_factory = session_maker pgvector.session_maker = session_maker async with connection.begin(): loader = CSVLoader( "data/faq/faq.csv", source_column="source", autodetect_encoding=True, ) result = await aindex( source_id_key="source", docs_source=loader.load()[:1], cleanup="incremental", vector_store=pgvector, record_manager=record_manager, ) print(result) asyncio.run(main()) ``` --------- Signed-off-by: Rahul Tripathi <rauhl.psit.ec@gmail.com> Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com> Co-authored-by: Sean <sean@upstage.ai> Co-authored-by: JuHyung-Son <sonju0427@gmail.com> Co-authored-by: Erick Friis <erick@langchain.dev> Co-authored-by: YISH <mokeyish@hotmail.com> Co-authored-by: Bagatur <baskaryan@gmail.com> Co-authored-by: Jason_Chen <820542443@qq.com> Co-authored-by: Joan Fontanals <joan.fontanals.martinez@jina.ai> Co-authored-by: Pavlo Paliychuk <pavlo.paliychuk.ca@gmail.com> Co-authored-by: fzowl <160063452+fzowl@users.noreply.github.com> Co-authored-by: samanhappy <samanhappy@gmail.com> Co-authored-by: Lei Zhang <zhanglei@apache.org> Co-authored-by: Tomaz Bratanic <bratanic.tomaz@gmail.com> Co-authored-by: merdan <48309329+merdan-9@users.noreply.github.com> Co-authored-by: ccurme <chester.curme@gmail.com> Co-authored-by: Andres Algaba <andresalgaba@gmail.com> Co-authored-by: davidefantiniIntel <115252273+davidefantiniIntel@users.noreply.github.com> Co-authored-by: Jingpan Xiong <71321890+klaus-xiong@users.noreply.github.com> Co-authored-by: kaka <kaka@zbyte-inc.cloud> Co-authored-by: jingsi <jingsi@leadincloud.com> Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com> Co-authored-by: Rahul Triptahi <rahul.psit.ec@gmail.com> Co-authored-by: Rahul Tripathi <rauhl.psit.ec@gmail.com> Co-authored-by: Shengsheng Huang <shannie.huang@gmail.com> Co-authored-by: Michael Schock <mjschock@users.noreply.github.com> Co-authored-by: Anish Chakraborty <anish749@users.noreply.github.com> Co-authored-by: am-kinetica <85610855+am-kinetica@users.noreply.github.com> Co-authored-by: Dristy Srivastava <58721149+dristysrivastava@users.noreply.github.com> Co-authored-by: Matt <matthew.gotteiner@microsoft.com> Co-authored-by: William FH <13333726+hinthornw@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| langchain_community | ||
| scripts | ||
| tests | ||
| Makefile | ||
| poetry.lock | ||
| pyproject.toml | ||
| README.md | ||
🦜️🧑🤝🧑 LangChain Community
Quick Install
pip install langchain-community
What is it?
LangChain Community contains third-party integrations that implement the base interfaces defined in LangChain Core, making them ready-to-use in any LangChain application.
For full documentation see the API reference.
📕 Releases & Versioning
langchain-community is currently on version 0.0.x
All changes will be accompanied by a patch version increase.
💁 Contributing
As an open-source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infrastructure, or better documentation.
For detailed information on how to contribute, see the Contributing Guide.