mirror of https://github.com/hwchase17/langchain
Docs: Add lcel to combine_docs chains (#12310)
Bagatur
11 months ago
committed by
GitHub
parent
bc6f6e968e
commit
f3449ccd20
@ -0,0 +1,251 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2accd7d9-20b6-47f9-9cec-923809cc36c7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Map reduce\n",
|
||||
"\n",
|
||||
"The map reduce documents chain first applies an LLM chain to each document individually (the Map step), treating the chain output as a new document. It then passes all the new documents to a separate combine documents chain to get a single output (the Reduce step). It can optionally first compress, or collapse, the mapped documents to make sure that they fit in the combine documents chain (which will often pass them to an LLM). This compression step is performed recursively if necessary.\n",
|
||||
"\n",
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "343fc972-40be-44f8-8ed3-305322661a00",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Recreating with LCEL\n",
|
||||
"\n",
|
||||
"With [LangChain Expression Language](/docs/expression_language) we can recreate the `MapReduceDocumentsChain` functionality, with the additional benefit of getting all the built-in LCEL features (batch, async, etc.) and with much more ability to customize specific parts of the chain."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "7bc161bc-4054-457a-9d04-7245093acd16",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from functools import partial\n",
|
||||
"\n",
|
||||
"from langchain.callbacks.manager import CallbackManagerForChainRun\n",
|
||||
"from langchain.chains.combine_documents.reduce import _collapse_docs, _split_list_of_docs\n",
|
||||
"from langchain.chat_models import ChatAnthropic\n",
|
||||
"from langchain.prompts import PromptTemplate\n",
|
||||
"from langchain.schema import StrOutputParser\n",
|
||||
"from langchain.schema.prompt_template import format_document\n",
|
||||
"from langchain.schema.runnable import RunnableParallel, RunnablePassthrough"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "a93ba908-5b81-4e91-a598-ee6fa05eac01",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"llm = ChatAnthropic()\n",
|
||||
"\n",
|
||||
"# Prompt and method for converting Document -> str.\n",
|
||||
"document_prompt = PromptTemplate.from_template(\"{page_content}\")\n",
|
||||
"partial_format_document = partial(format_document, prompt=document_prompt)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "75918baa-df6b-4570-91eb-1acd1c87e09b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# The chain we'll apply to each individual document.\n",
|
||||
"# Returns a summary of the document.\n",
|
||||
"map_chain = (\n",
|
||||
" {\"context\": partial_format_document} \n",
|
||||
" | PromptTemplate.from_template(\"Summarize this content:\\n\\n{context}\") \n",
|
||||
" | llm \n",
|
||||
" | StrOutputParser()\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# A wrapper chain to keep the original Document metadata\n",
|
||||
"map_as_doc_chain = (\n",
|
||||
" RunnableParallel({\"doc\": RunnablePassthrough(), \"content\": map_chain}) \n",
|
||||
" | (lambda x: Document(page_content=x[\"content\"], metadata=x[\"doc\"].metadata))\n",
|
||||
").with_config(run_name=\"Summarize (return doc)\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "720cb117-8a3e-4595-9e2c-dbbd7a3777b5",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# The chain we'll repeatedly apply to collapse subsets of the documents \n",
|
||||
"# into a consolidate document until the total token size of our \n",
|
||||
"# documents is below some max size.\n",
|
||||
"def format_docs(docs):\n",
|
||||
" return \"\\n\\n\".join(partial_format_document(doc) for doc in docs)\n",
|
||||
" \n",
|
||||
"collapse_chain = (\n",
|
||||
" {\"context\": format_docs}\n",
|
||||
" | PromptTemplate.from_template(\"Collapse this content:\\n\\n{context}\")\n",
|
||||
" | llm\n",
|
||||
" | StrOutputParser()\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"def get_num_tokens(docs):\n",
|
||||
" return llm.get_num_tokens(format_docs(docs))\n",
|
||||
"\n",
|
||||
"def collapse(\n",
|
||||
" docs, \n",
|
||||
" config,\n",
|
||||
" token_max=4000,\n",
|
||||
"):\n",
|
||||
" collapse_ct = 1\n",
|
||||
" while get_num_tokens(docs) > token_max:\n",
|
||||
" config[\"run_name\"] = f\"Collapse {collapse_ct}\"\n",
|
||||
" invoke = partial(collapse_chain.invoke, config=config)\n",
|
||||
" split_docs = _split_list_of_docs(docs, get_num_tokens, token_max)\n",
|
||||
" docs = [_collapse_docs(_docs, invoke) for _docs in split_docs]\n",
|
||||
" collapse_ct += 1\n",
|
||||
" return docs"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"id": "fe5c5597-3ea3-443e-ad9f-e2f055cf092f",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# The chain we'll use to combine our individual document summaries\n",
|
||||
"# (or summaries over subset of documents if we had to collapse the map results)\n",
|
||||
"# into a final summary.\n",
|
||||
"\n",
|
||||
"reduce_chain = (\n",
|
||||
" {\"context\": format_docs}\n",
|
||||
" | PromptTemplate.from_template(\"Combine these summaries:\\n\\n{context}\")\n",
|
||||
" | llm\n",
|
||||
" | StrOutputParser()\n",
|
||||
").with_config(run_name=\"Reduce\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "fd1148ce-f693-42b5-91e4-304983e26be6",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# The final full chain\n",
|
||||
"map_reduce = (map_as_doc_chain.map() | collapse | reduce_chain).with_config(run_name=\"Map reduce\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5a10615c-f3ab-4603-bf0c-e6aea73c5450",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Example run"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "28a8f74c-2441-431a-b352-541d0ad1e75b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from langchain.schema import Document\n",
|
||||
"\n",
|
||||
"text = \"\"\"Nuclear power in space is the use of nuclear power in outer space, typically either small fission systems or radioactive decay for electricity or heat. Another use is for scientific observation, as in a Mössbauer spectrometer. The most common type is a radioisotope thermoelectric generator, which has been used on many space probes and on crewed lunar missions. Small fission reactors for Earth observation satellites, such as the TOPAZ nuclear reactor, have also been flown.[1] A radioisotope heater unit is powered by radioactive decay and can keep components from becoming too cold to function, potentially over a span of decades.[2]\n",
|
||||
"\n",
|
||||
"The United States tested the SNAP-10A nuclear reactor in space for 43 days in 1965,[3] with the next test of a nuclear reactor power system intended for space use occurring on 13 September 2012 with the Demonstration Using Flattop Fission (DUFF) test of the Kilopower reactor.[4]\n",
|
||||
"\n",
|
||||
"After a ground-based test of the experimental 1965 Romashka reactor, which used uranium and direct thermoelectric conversion to electricity,[5] the USSR sent about 40 nuclear-electric satellites into space, mostly powered by the BES-5 reactor. The more powerful TOPAZ-II reactor produced 10 kilowatts of electricity.[3]\n",
|
||||
"\n",
|
||||
"Examples of concepts that use nuclear power for space propulsion systems include the nuclear electric rocket (nuclear powered ion thruster(s)), the radioisotope rocket, and radioisotope electric propulsion (REP).[6] One of the more explored concepts is the nuclear thermal rocket, which was ground tested in the NERVA program. Nuclear pulse propulsion was the subject of Project Orion.[7]\n",
|
||||
"\n",
|
||||
"Regulation and hazard prevention[edit]\n",
|
||||
"After the ban of nuclear weapons in space by the Outer Space Treaty in 1967, nuclear power has been discussed at least since 1972 as a sensitive issue by states.[8] Particularly its potential hazards to Earth's environment and thus also humans has prompted states to adopt in the U.N. General Assembly the Principles Relevant to the Use of Nuclear Power Sources in Outer Space (1992), particularly introducing safety principles for launches and to manage their traffic.[8]\n",
|
||||
"\n",
|
||||
"Benefits\n",
|
||||
"\n",
|
||||
"Both the Viking 1 and Viking 2 landers used RTGs for power on the surface of Mars. (Viking launch vehicle pictured)\n",
|
||||
"While solar power is much more commonly used, nuclear power can offer advantages in some areas. Solar cells, although efficient, can only supply energy to spacecraft in orbits where the solar flux is sufficiently high, such as low Earth orbit and interplanetary destinations close enough to the Sun. Unlike solar cells, nuclear power systems function independently of sunlight, which is necessary for deep space exploration. Nuclear-based systems can have less mass than solar cells of equivalent power, allowing more compact spacecraft that are easier to orient and direct in space. In the case of crewed spaceflight, nuclear power concepts that can power both life support and propulsion systems may reduce both cost and flight time.[9]\n",
|
||||
"\n",
|
||||
"Selected applications and/or technologies for space include:\n",
|
||||
"\n",
|
||||
"Radioisotope thermoelectric generator\n",
|
||||
"Radioisotope heater unit\n",
|
||||
"Radioisotope piezoelectric generator\n",
|
||||
"Radioisotope rocket\n",
|
||||
"Nuclear thermal rocket\n",
|
||||
"Nuclear pulse propulsion\n",
|
||||
"Nuclear electric rocket\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"docs = [\n",
|
||||
" Document(\n",
|
||||
" page_content=split, \n",
|
||||
" metadata={\"source\": \"https://en.wikipedia.org/wiki/Nuclear_power_in_space\"}\n",
|
||||
" ) \n",
|
||||
" for split in text.split(\"\\n\\n\")\n",
|
||||
"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"id": "846fe4ce-7016-4bc7-a8e0-7e914675d568",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" Here is a summary that combines the key points about nuclear power in space:\n",
|
||||
"\n",
|
||||
"Nuclear power is used in space for electricity, heat, and scientific observation. The most common type is a radioisotope thermoelectric generator, which has powered space probes and lunar missions using the heat from radioactive decay. Small nuclear fission reactors have also been used to generate electricity for Earth observation satellites like the TOPAZ reactor. In addition, radioisotope heater units use radioactive decay to provide reliable heat that can keep components functioning properly over decades in the harsh space environment. Overall, nuclear power has proven useful for providing long-lasting power for space applications where solar power is not practical. Technologies like radioisotope decay heat and small fission reactors allow probes, satellites, and missions to operate far from the Sun and for extended periods by generating electricity and heat without reliance on solar energy.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(map_reduce.invoke(docs[0:1], config={\"max_concurrency\": 5}))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "a8b77e13-6db4-4096-a1a4-d0abe2979b6b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "poetry-venv",
|
||||
"language": "python",
|
||||
"name": "poetry-venv"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.1"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@ -1,5 +0,0 @@
|
||||
# Map reduce
|
||||
|
||||
The map reduce documents chain first applies an LLM chain to each document individually (the Map step), treating the chain output as a new document. It then passes all the new documents to a separate combine documents chain to get a single output (the Reduce step). It can optionally first compress, or collapse, the mapped documents to make sure that they fit in the combine documents chain (which will often pass them to an LLM). This compression step is performed recursively if necessary.
|
||||
|
||||
|
||||
@ -0,0 +1,183 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3bbf49ee-f3f1-40c1-b48a-828e4166bfe0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Map re-rank\n",
|
||||
"\n",
|
||||
"The map re-rank documents chain runs an initial prompt on each document, that not only tries to complete a task but also gives a score for how certain it is in its answer. The highest scoring response is returned.\n",
|
||||
"\n",
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d4cfac68-f2c4-49bf-9aad-d3e07eb9ee53",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Recreating with LCEL\n",
|
||||
"\n",
|
||||
"With [LangChain Expression Language](/docs/expression_language) we can recreate the `MapRerankDocumentsChain` functionality, with the additional benefit of getting all the built-in LCEL features (batch, async, etc.) and with much more ability to customize specific parts of the chain."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "27c523f9-f9f1-4ad5-8bdb-38f8faa9c6e3",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from langchain.chat_models import ChatOpenAI\n",
|
||||
"from langchain.output_parsers.openai_functions import PydanticOutputFunctionsParser\n",
|
||||
"from langchain.prompts import PromptTemplate\n",
|
||||
"from langchain.pydantic_v1 import BaseModel, Field\n",
|
||||
"from langchain.schema.prompt_template import format_document\n",
|
||||
"from langchain.utils.openai_functions import convert_pydantic_to_openai_function"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "ea687400-9410-445c-9adc-f8c8b9f66327",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Chain to apply to each individual document. Chain\n",
|
||||
"# provides an answer to the question based on the document\n",
|
||||
"# and scores it's confidence in the answer.\n",
|
||||
"\n",
|
||||
"map_prompt = PromptTemplate.from_template(\n",
|
||||
" \"Answer the user question using the context.\"\n",
|
||||
" \"\\n\\nContext:\\n\\n{context}\\n\\nQuestion: {question}\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"class AnswerAndScore(BaseModel):\n",
|
||||
" \"\"\"Return the answer to the question and a relevance score.\"\"\"\n",
|
||||
" answer: str = Field(description=\"The answer to the question, which is based ONLY on the provided context.\")\n",
|
||||
" score: float = Field(decsription=\"A 0.0-1.0 relevance score, where 1.0 indicates the provided context answers the question completely and 0.0 indicates the provided context does not answer the question at all.\")\n",
|
||||
"\n",
|
||||
"function = convert_pydantic_to_openai_function(AnswerAndScore)\n",
|
||||
"map_chain = (\n",
|
||||
" map_prompt \n",
|
||||
" | ChatOpenAI().bind(temperature=0, functions=[function], function_call={\"name\": \"AnswerAndScore\"})\n",
|
||||
" | PydanticOutputFunctionsParser(pydantic_schema=AnswerAndScore)\n",
|
||||
").with_config(run_name=\"Map\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "ace2b1d5-a9ea-4a70-8d39-2826a5445aa7",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Final chain, which after answer and scoring based on\n",
|
||||
"# each doc return the answer with the highest score.\n",
|
||||
"\n",
|
||||
"def top_answer(scored_answers):\n",
|
||||
" return max(scored_answers, key=lambda x: x.score).answer\n",
|
||||
"\n",
|
||||
"document_prompt = PromptTemplate.from_template(\"{page_content}\")\n",
|
||||
"map_rerank_chain = (\n",
|
||||
" (lambda x: [{\"context\": format_document(doc, document_prompt), \"question\": x[\"question\"]} for doc in x[\"docs\"]])\n",
|
||||
" | map_chain.map()\n",
|
||||
" | top_answer\n",
|
||||
").with_config(run_name=\"Map rerank\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "62b863c9-2316-42ad-9581-ebc688889855",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Example run"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "7b46c373-64c9-4b69-b64f-9bc6e52ae91c",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from langchain.schema import Document\n",
|
||||
"\n",
|
||||
"text = \"\"\"Nuclear power in space is the use of nuclear power in outer space, typically either small fission systems or radioactive decay for electricity or heat. Another use is for scientific observation, as in a Mössbauer spectrometer. The most common type is a radioisotope thermoelectric generator, which has been used on many space probes and on crewed lunar missions. Small fission reactors for Earth observation satellites, such as the TOPAZ nuclear reactor, have also been flown.[1] A radioisotope heater unit is powered by radioactive decay and can keep components from becoming too cold to function, potentially over a span of decades.[2]\n",
|
||||
"\n",
|
||||
"The United States tested the SNAP-10A nuclear reactor in space for 43 days in 1965,[3] with the next test of a nuclear reactor power system intended for space use occurring on 13 September 2012 with the Demonstration Using Flattop Fission (DUFF) test of the Kilopower reactor.[4]\n",
|
||||
"\n",
|
||||
"After a ground-based test of the experimental 1965 Romashka reactor, which used uranium and direct thermoelectric conversion to electricity,[5] the USSR sent about 40 nuclear-electric satellites into space, mostly powered by the BES-5 reactor. The more powerful TOPAZ-II reactor produced 10 kilowatts of electricity.[3]\n",
|
||||
"\n",
|
||||
"Examples of concepts that use nuclear power for space propulsion systems include the nuclear electric rocket (nuclear powered ion thruster(s)), the radioisotope rocket, and radioisotope electric propulsion (REP).[6] One of the more explored concepts is the nuclear thermal rocket, which was ground tested in the NERVA program. Nuclear pulse propulsion was the subject of Project Orion.[7]\n",
|
||||
"\n",
|
||||
"Regulation and hazard prevention[edit]\n",
|
||||
"After the ban of nuclear weapons in space by the Outer Space Treaty in 1967, nuclear power has been discussed at least since 1972 as a sensitive issue by states.[8] Particularly its potential hazards to Earth's environment and thus also humans has prompted states to adopt in the U.N. General Assembly the Principles Relevant to the Use of Nuclear Power Sources in Outer Space (1992), particularly introducing safety principles for launches and to manage their traffic.[8]\n",
|
||||
"\n",
|
||||
"Benefits\n",
|
||||
"\n",
|
||||
"Both the Viking 1 and Viking 2 landers used RTGs for power on the surface of Mars. (Viking launch vehicle pictured)\n",
|
||||
"While solar power is much more commonly used, nuclear power can offer advantages in some areas. Solar cells, although efficient, can only supply energy to spacecraft in orbits where the solar flux is sufficiently high, such as low Earth orbit and interplanetary destinations close enough to the Sun. Unlike solar cells, nuclear power systems function independently of sunlight, which is necessary for deep space exploration. Nuclear-based systems can have less mass than solar cells of equivalent power, allowing more compact spacecraft that are easier to orient and direct in space. In the case of crewed spaceflight, nuclear power concepts that can power both life support and propulsion systems may reduce both cost and flight time.[9]\n",
|
||||
"\n",
|
||||
"Selected applications and/or technologies for space include:\n",
|
||||
"\n",
|
||||
"Radioisotope thermoelectric generator\n",
|
||||
"Radioisotope heater unit\n",
|
||||
"Radioisotope piezoelectric generator\n",
|
||||
"Radioisotope rocket\n",
|
||||
"Nuclear thermal rocket\n",
|
||||
"Nuclear pulse propulsion\n",
|
||||
"Nuclear electric rocket\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"docs = [\n",
|
||||
" Document(\n",
|
||||
" page_content=split, \n",
|
||||
" metadata={\"source\": \"https://en.wikipedia.org/wiki/Nuclear_power_in_space\"}\n",
|
||||
" ) \n",
|
||||
" for split in text.split(\"\\n\\n\")\n",
|
||||
"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "d1998c41-1ebb-4d55-9c28-d2f3feb12657",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"The Viking missions were powered by radioisotope thermoelectric generators (RTGs). These generators used the heat produced by the natural decay of plutonium-238 to generate electricity.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(map_rerank_chain.invoke({\"docs\": docs, \"question\": \"How were the vikings powered\"}))"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "poetry-venv",
|
||||
"language": "python",
|
||||
"name": "poetry-venv"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.1"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@ -1,5 +0,0 @@
|
||||
# Map re-rank
|
||||
|
||||
The map re-rank documents chain runs an initial prompt on each document, that not only tries to complete a task but also gives a score for how certain it is in its answer. The highest scoring response is returned.
|
||||
|
||||
|
||||
@ -0,0 +1,235 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "raw",
|
||||
"id": "15868084-aec1-4e58-8524-32cbb12aa272",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"sidebar_position: 1\n",
|
||||
"title: Refine\n",
|
||||
"---"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2d592270-ebbc-4310-abd8-86ef8d70fe81",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Refine\n",
|
||||
"\n",
|
||||
"The Refine documents chain constructs a response by looping over the input documents and iteratively updating its answer. For each document, it passes all non-document inputs, the current document, and the latest intermediate answer to an LLM chain to get a new answer.\n",
|
||||
"\n",
|
||||
"Since the Refine chain only passes a single document to the LLM at a time, it is well-suited for tasks that require analyzing more documents than can fit in the model's context.\n",
|
||||
"The obvious tradeoff is that this chain will make far more LLM calls than, for example, the Stuff documents chain.\n",
|
||||
"There are also certain tasks which are difficult to accomplish iteratively. For example, the Refine chain can perform poorly when documents frequently cross-reference one another or when a task requires detailed information from many documents.\n",
|
||||
"\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "644838eb-6dbd-4d29-a557-628c1fa2d4c6",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Recreating with LCEL\n",
|
||||
"\n",
|
||||
"With [LangChain Expression Language](/docs/expression_language) we can easily recreate the `RefineDocumentsChain`, with the additional benefit of getting all the built-in LCEL features (batch, async, etc.) and with much more ability to customize specific parts of the chain."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "2560d994-e1a4-4fe0-97c7-182b6bd7798b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from functools import partial\n",
|
||||
"from operator import itemgetter\n",
|
||||
"\n",
|
||||
"from langchain.callbacks.manager import trace_as_chain_group\n",
|
||||
"from langchain.chat_models import ChatAnthropic\n",
|
||||
"from langchain.prompts import PromptTemplate\n",
|
||||
"from langchain.schema import StrOutputParser\n",
|
||||
"from langchain.schema.prompt_template import format_document"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "f8bb92d8-3961-4a75-b4ad-568a224c78b2",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Chain for generating initial summary based on the first document\n",
|
||||
"\n",
|
||||
"llm = ChatAnthropic()\n",
|
||||
"first_prompt = PromptTemplate.from_template(\"Summarize this content:\\n\\n{context}\")\n",
|
||||
"document_prompt = PromptTemplate.from_template(\"{page_content}\")\n",
|
||||
"partial_format_doc = partial(format_document, prompt=document_prompt)\n",
|
||||
"summary_chain = {\"context\": partial_format_doc} | first_prompt | llm | StrOutputParser()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "9aedc203-b842-4d1b-84f0-23946b0acc7a",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Chain for refining an existing summary based on\n",
|
||||
"# an additional document\n",
|
||||
"\n",
|
||||
"refine_prompt = PromptTemplate.from_template(\n",
|
||||
" \"Here's your first summary: {prev_response}. \"\n",
|
||||
" \"Now add to it based on the following context: {context}\"\n",
|
||||
")\n",
|
||||
"refine_chain = (\n",
|
||||
" {\n",
|
||||
" \"prev_response\": itemgetter(\"prev_response\"), \n",
|
||||
" \"context\": lambda x: partial_format_doc(x[\"doc\"])\n",
|
||||
" } | refine_prompt\n",
|
||||
" | llm \n",
|
||||
" | StrOutputParser()\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "cd118a07-1678-4274-9248-2c88656da686",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# The final refine loop, which generates an initial summary\n",
|
||||
"# then iteratively refines it based on each of the rest of the documents\n",
|
||||
"\n",
|
||||
"def refine_loop(docs):\n",
|
||||
" with trace_as_chain_group(\"refine loop\", inputs={\"input\": docs}) as manager:\n",
|
||||
" summary = summary_chain.invoke(\n",
|
||||
" docs[0], \n",
|
||||
" config={\"callbacks\": manager, \"run_name\": \"initial summary\"}\n",
|
||||
" )\n",
|
||||
" for i, doc in enumerate(docs[1:]):\n",
|
||||
" summary = refine_chain.invoke(\n",
|
||||
" {\"prev_response\": summary, \"doc\": doc}, \n",
|
||||
" config={\"callbacks\": manager, \"run_name\": f\"refine {i}\"}\n",
|
||||
" )\n",
|
||||
" manager.on_chain_end({\"output\": summary})\n",
|
||||
" return summary"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0d2ec3b0-a47a-4344-8f12-8bb644fdccae",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Example run"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "d8242337-3aa4-4377-bd4c-3bf0c01f9dd7",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from langchain.schema import Document\n",
|
||||
"\n",
|
||||
"text = \"\"\"Nuclear power in space is the use of nuclear power in outer space, typically either small fission systems or radioactive decay for electricity or heat. Another use is for scientific observation, as in a Mössbauer spectrometer. The most common type is a radioisotope thermoelectric generator, which has been used on many space probes and on crewed lunar missions. Small fission reactors for Earth observation satellites, such as the TOPAZ nuclear reactor, have also been flown.[1] A radioisotope heater unit is powered by radioactive decay and can keep components from becoming too cold to function, potentially over a span of decades.[2]\n",
|
||||
"\n",
|
||||
"The United States tested the SNAP-10A nuclear reactor in space for 43 days in 1965,[3] with the next test of a nuclear reactor power system intended for space use occurring on 13 September 2012 with the Demonstration Using Flattop Fission (DUFF) test of the Kilopower reactor.[4]\n",
|
||||
"\n",
|
||||
"After a ground-based test of the experimental 1965 Romashka reactor, which used uranium and direct thermoelectric conversion to electricity,[5] the USSR sent about 40 nuclear-electric satellites into space, mostly powered by the BES-5 reactor. The more powerful TOPAZ-II reactor produced 10 kilowatts of electricity.[3]\n",
|
||||
"\n",
|
||||
"Examples of concepts that use nuclear power for space propulsion systems include the nuclear electric rocket (nuclear powered ion thruster(s)), the radioisotope rocket, and radioisotope electric propulsion (REP).[6] One of the more explored concepts is the nuclear thermal rocket, which was ground tested in the NERVA program. Nuclear pulse propulsion was the subject of Project Orion.[7]\n",
|
||||
"\n",
|
||||
"Regulation and hazard prevention[edit]\n",
|
||||
"After the ban of nuclear weapons in space by the Outer Space Treaty in 1967, nuclear power has been discussed at least since 1972 as a sensitive issue by states.[8] Particularly its potential hazards to Earth's environment and thus also humans has prompted states to adopt in the U.N. General Assembly the Principles Relevant to the Use of Nuclear Power Sources in Outer Space (1992), particularly introducing safety principles for launches and to manage their traffic.[8]\n",
|
||||
"\n",
|
||||
"Benefits\n",
|
||||
"\n",
|
||||
"Both the Viking 1 and Viking 2 landers used RTGs for power on the surface of Mars. (Viking launch vehicle pictured)\n",
|
||||
"While solar power is much more commonly used, nuclear power can offer advantages in some areas. Solar cells, although efficient, can only supply energy to spacecraft in orbits where the solar flux is sufficiently high, such as low Earth orbit and interplanetary destinations close enough to the Sun. Unlike solar cells, nuclear power systems function independently of sunlight, which is necessary for deep space exploration. Nuclear-based systems can have less mass than solar cells of equivalent power, allowing more compact spacecraft that are easier to orient and direct in space. In the case of crewed spaceflight, nuclear power concepts that can power both life support and propulsion systems may reduce both cost and flight time.[9]\n",
|
||||
"\n",
|
||||
"Selected applications and/or technologies for space include:\n",
|
||||
"\n",
|
||||
"Radioisotope thermoelectric generator\n",
|
||||
"Radioisotope heater unit\n",
|
||||
"Radioisotope piezoelectric generator\n",
|
||||
"Radioisotope rocket\n",
|
||||
"Nuclear thermal rocket\n",
|
||||
"Nuclear pulse propulsion\n",
|
||||
"Nuclear electric rocket\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"docs = [\n",
|
||||
" Document(\n",
|
||||
" page_content=split, \n",
|
||||
" metadata={\"source\": \"https://en.wikipedia.org/wiki/Nuclear_power_in_space\"}\n",
|
||||
" ) \n",
|
||||
" for split in text.split(\"\\n\\n\")\n",
|
||||
"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "68f23401-cb6b-4500-8576-7e1c9254dfef",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" Here is the updated summary with the additional context:\n",
|
||||
"\n",
|
||||
"Here is a summary of the key points about nuclear power in space:\n",
|
||||
"\n",
|
||||
"- Nuclear power is used in space for electricity, heat, and scientific observation. The most common type is a radioisotope thermoelectric generator (RTG), which uses radioactive decay to generate electricity. RTGs have powered space probes and crewed lunar missions. \n",
|
||||
"\n",
|
||||
"- Small nuclear fission reactors have also been used to power Earth observation satellites, like the TOPAZ reactor. The United States tested the SNAP-10A nuclear reactor in space for 43 days in 1965.\n",
|
||||
"\n",
|
||||
"- After a ground-based test of the experimental 1965 Romashka reactor, which used uranium and direct thermoelectric conversion to electricity, the USSR sent about 40 nuclear-electric satellites into space, mostly powered by the BES-5 reactor. The more powerful TOPAZ-II reactor produced 10 kilowatts of electricity.\n",
|
||||
"\n",
|
||||
"- Radioisotope heater units use radioactive decay for heat. They can keep components warm enough to function over decades.\n",
|
||||
"\n",
|
||||
"- Nuclear power concepts have also been proposed and tested for space propulsion. Examples include the nuclear electric rocket (nuclear powered ion thruster(s)), the radioisotope\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(refine_loop(docs))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "1db2ea25-0f93-4e23-8793-b7b94df8be07",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "poetry-venv",
|
||||
"language": "python",
|
||||
"name": "poetry-venv"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.1"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@ -1,12 +0,0 @@
|
||||
---
|
||||
sidebar_position: 1
|
||||
---
|
||||
# Refine
|
||||
|
||||
The Refine documents chain constructs a response by looping over the input documents and iteratively updating its answer. For each document, it passes all non-document inputs, the current document, and the latest intermediate answer to an LLM chain to get a new answer.
|
||||
|
||||
Since the Refine chain only passes a single document to the LLM at a time, it is well-suited for tasks that require analyzing more documents than can fit in the model's context.
|
||||
The obvious tradeoff is that this chain will make far more LLM calls than, for example, the Stuff documents chain.
|
||||
There are also certain tasks which are difficult to accomplish iteratively. For example, the Refine chain can perform poorly when documents frequently cross-reference one another or when a task requires detailed information from many documents.
|
||||
|
||||
|
||||
@ -0,0 +1,173 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "raw",
|
||||
"id": "bafbef65-ace3-42ce-83f3-553909b48685",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"sidebar_position: 0\n",
|
||||
"title: Stuff\n",
|
||||
"---"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1c473378-ff18-45cf-b718-43f6f94af040",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The stuff documents chain (\"stuff\" as in \"to stuff\" or \"to fill\") is the most straightforward of the document chains. It takes a list of documents, inserts them all into a prompt and passes that prompt to an LLM.\n",
|
||||
"\n",
|
||||
"This chain is well-suited for applications where documents are small and only a few are passed in for most calls.\n",
|
||||
"\n",
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1f20da9a-2a7f-4fd1-9dbc-c01f164b078b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Recreating with LCEL\n",
|
||||
"\n",
|
||||
"With [LangChain Expression Language](/docs/expression_language) we can easily recreate the `StuffDocumentsChain` functionality, with the additional benefit of getting all the built-in LCEL features (batch, async, etc.) and with much more ability to customize specific parts of the chain."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "c0f33d9f-2cf2-4064-92f6-39fbd5a597e3",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from langchain.chat_models import ChatAnthropic\n",
|
||||
"from langchain.prompts import PromptTemplate\n",
|
||||
"from langchain.schema import StrOutputParser\n",
|
||||
"from langchain.schema.prompt_template import format_document"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "3e355e46-6375-4f5e-a102-fdbf72476422",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"doc_prompt = PromptTemplate.from_template(\"{page_content}\")\n",
|
||||
" \n",
|
||||
"chain = (\n",
|
||||
" {\"content\": lambda docs: \"\\n\\n\".join(format_document(doc, doc_prompt) for doc in docs)}\n",
|
||||
" | PromptTemplate.from_template(\"Summarize the following content:\\n\\n{content}\")\n",
|
||||
" | ChatAnthropic()\n",
|
||||
" | StrOutputParser()\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ece7db13-5d4b-490f-8677-387bf4eac944",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Example run\n",
|
||||
"\n",
|
||||
"Lets run this summarization chain on some sample data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"id": "a0ddd725-d97c-4e22-8509-7337d3a71dff",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from langchain.schema import Document\n",
|
||||
"\n",
|
||||
"text = \"\"\"Nuclear power in space is the use of nuclear power in outer space, typically either small fission systems or radioactive decay for electricity or heat. Another use is for scientific observation, as in a Mössbauer spectrometer. The most common type is a radioisotope thermoelectric generator, which has been used on many space probes and on crewed lunar missions. Small fission reactors for Earth observation satellites, such as the TOPAZ nuclear reactor, have also been flown.[1] A radioisotope heater unit is powered by radioactive decay and can keep components from becoming too cold to function, potentially over a span of decades.[2]\n",
|
||||
"\n",
|
||||
"The United States tested the SNAP-10A nuclear reactor in space for 43 days in 1965,[3] with the next test of a nuclear reactor power system intended for space use occurring on 13 September 2012 with the Demonstration Using Flattop Fission (DUFF) test of the Kilopower reactor.[4]\n",
|
||||
"\n",
|
||||
"After a ground-based test of the experimental 1965 Romashka reactor, which used uranium and direct thermoelectric conversion to electricity,[5] the USSR sent about 40 nuclear-electric satellites into space, mostly powered by the BES-5 reactor. The more powerful TOPAZ-II reactor produced 10 kilowatts of electricity.[3]\n",
|
||||
"\n",
|
||||
"Examples of concepts that use nuclear power for space propulsion systems include the nuclear electric rocket (nuclear powered ion thruster(s)), the radioisotope rocket, and radioisotope electric propulsion (REP).[6] One of the more explored concepts is the nuclear thermal rocket, which was ground tested in the NERVA program. Nuclear pulse propulsion was the subject of Project Orion.[7]\n",
|
||||
"\n",
|
||||
"Regulation and hazard prevention[edit]\n",
|
||||
"After the ban of nuclear weapons in space by the Outer Space Treaty in 1967, nuclear power has been discussed at least since 1972 as a sensitive issue by states.[8] Particularly its potential hazards to Earth's environment and thus also humans has prompted states to adopt in the U.N. General Assembly the Principles Relevant to the Use of Nuclear Power Sources in Outer Space (1992), particularly introducing safety principles for launches and to manage their traffic.[8]\n",
|
||||
"\n",
|
||||
"Benefits\n",
|
||||
"\n",
|
||||
"Both the Viking 1 and Viking 2 landers used RTGs for power on the surface of Mars. (Viking launch vehicle pictured)\n",
|
||||
"While solar power is much more commonly used, nuclear power can offer advantages in some areas. Solar cells, although efficient, can only supply energy to spacecraft in orbits where the solar flux is sufficiently high, such as low Earth orbit and interplanetary destinations close enough to the Sun. Unlike solar cells, nuclear power systems function independently of sunlight, which is necessary for deep space exploration. Nuclear-based systems can have less mass than solar cells of equivalent power, allowing more compact spacecraft that are easier to orient and direct in space. In the case of crewed spaceflight, nuclear power concepts that can power both life support and propulsion systems may reduce both cost and flight time.[9]\n",
|
||||
"\n",
|
||||
"Selected applications and/or technologies for space include:\n",
|
||||
"\n",
|
||||
"Radioisotope thermoelectric generator\n",
|
||||
"Radioisotope heater unit\n",
|
||||
"Radioisotope piezoelectric generator\n",
|
||||
"Radioisotope rocket\n",
|
||||
"Nuclear thermal rocket\n",
|
||||
"Nuclear pulse propulsion\n",
|
||||
"Nuclear electric rocket\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"docs = [\n",
|
||||
" Document(\n",
|
||||
" page_content=split, \n",
|
||||
" metadata={\"source\": \"https://en.wikipedia.org/wiki/Nuclear_power_in_space\"}\n",
|
||||
" ) \n",
|
||||
" for split in text.split()\n",
|
||||
"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"id": "bd18f647-bdb4-46ce-b01f-fe8afc208ffa",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" Here is a summary of the key points:\n",
|
||||
"\n",
|
||||
"- Nuclear power has been used in space for electricity, heat, and scientific observation. The most common type is a radioisotope thermoelectric generator, used on many probes and lunar missions. \n",
|
||||
"\n",
|
||||
"- Small fission reactors have been used for Earth observation satellites. Radioisotope heater units use radioactive decay to keep components warm for decades.\n",
|
||||
"\n",
|
||||
"- The US tested a nuclear reactor in space in 1965. The Soviet Union launched around 40 nuclear-powered satellites, mostly with BES-5 reactors.\n",
|
||||
"\n",
|
||||
"- Concepts for nuclear propulsion include nuclear thermal rockets, nuclear electric rockets, and nuclear pulse propulsion. The NERVA program ground tested nuclear thermal rockets.\n",
|
||||
"\n",
|
||||
"- After the 1967 Outer Space Treaty banned nuclear weapons in space, safety principles were introduced for nuclear power launch and traffic management.\n",
|
||||
"\n",
|
||||
"- Benefits of nuclear power in space include functioning independently of sunlight needed for deep space exploration, less mass than equivalent solar power, and ability to power both life support and propulsion.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(chain.invoke(docs))"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.1"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@ -1,12 +0,0 @@
|
||||
---
|
||||
sidebar_position: 0
|
||||
---
|
||||
# Stuff
|
||||
|
||||
The stuff documents chain ("stuff" as in "to stuff" or "to fill") is the most straightforward of the document chains. It takes a list of documents, inserts them all into a prompt and passes that prompt to an LLM.
|
||||
|
||||
This chain is well-suited for applications where documents are small and only a few are passed in for most calls.
|
||||
|
||||
|
||||
|
||||
|
||||
Loading…
Reference in New Issue