"LangChain provides async support for LLMs by leveraging the [asyncio](https://docs.python.org/3/library/asyncio.html) library.\n",
"\n",
"Async support is particularly useful for calling multiple LLMs concurrently, as these calls are network-bound. Currently, only `OpenAI` and `PromptLayerOpenAI` is supported, but async support for other LLMs is on the roadmap.\n",
"Async support is particularly useful for calling multiple LLMs concurrently, as these calls are network-bound. Currently, only `OpenAI` `OpenAIChat`, and `PromptLayerOpenAI` are supported, but async support for other LLMs is on the roadmap.\n",
"\n",
"You can use the `agenerate` method to call an OpenAI LLM asynchronously."
]
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 1,
"id": "5e49e96c-0f88-466d-b3d3-ea0966bdf19e",
"metadata": {
"tags": []
@ -29,64 +28,66 @@
"text": [
"\n",
"\n",
"I'm doing well. How about you?\n",
"As an AI language model, I don't have feelings like humans, but I'm functioning properly. How may I assist you?\n",
"\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"I'm an AI language model, so I don't have emotions, but I'm functioning properly. How may I assist you today?\n",
"\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"As an AI language model, I do not have emotions like humans, but I'm functioning normally. How can I assist you today?\n",
"\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"I am an AI language model, so I do not have feelings, but I am here to assist you. How may I help you today?\n",
"\n",
"I am doing quite well. How about you?\n",
"\n",
"As an AI language model, I do not have feelings or emotions but I'm always ready to assist you. How may I assist you today?\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\n",
"As an AI language model, I don't have feelings, but I'm functioning normally. How may I assist you today?\n",
"\n",
"I'm doing great, thank you! How about you?\n",
"\n",
"As an AI language model, I don't have feelings, but I'm functioning properly. Thank you. How may I assist you today?\n",
"\n",
"I'm doing well, thanks for asking. How about you?\n",
"\n",
"As an AI language model, I don't have emotions, so I don't have a specific feeling or emotion. How can I assist you today?\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\n",
"As an AI language model, I do not have feelings or emotions. However, I am functioning as intended and ready to assist you with any queries you may have. How can I be of assistance today?\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\u001b[1mConcurrent executed in 1.93 seconds.\u001b[0m\n",
"\n",
"As an AI language model, I do not have feelings, but I am functioning well. Thank you for asking. How can I assist you today?\n",
"\u001b[1mConcurrent executed in 0.92 seconds.\u001b[0m\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\n",
"As an AI language model, I don't have feelings, but I'm functioning well. How can I assist you today?\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\n",
"As an AI language model, I don't have feelings, but I'm functioning well. Thank you for asking. How may I assist you today?\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\n",
"I'm an AI language model, so I don't have feelings, but I'm functioning well. How can I assist you today?\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\n",
"As an AI language model, I don't have feelings, but I'm functioning well. Thank you for asking. How may I assist you today?\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\n",
"As an AI language model, I don't have feelings, but I am functioning well. How can I assist you today?\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"As an AI language model, I don't have feelings but I'm functioning well. How can I assist you today?\n",
"\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"As an AI language model, I do not have personal emotions. However, I am functioning well and ready to assist you with any queries or tasks you have. How may I assist you today?\n",
"\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"As an AI language model, I do not have feelings or emotions, but I'm functioning well. How can I assist you today?\n",
"\n",
"\n",
"I'm doing great, thank you. How about you?\n",
"\u001b[1mSerial executed in 10.54 seconds.\u001b[0m\n"
"I am an AI language model and do not have feelings. But I am functioning properly and ready to assist you with any task. How may I help you today?\n",
"\n",
"\n",
"As an AI language model, I do not have emotions, but I am functioning well. How can I assist you today?\n",
"\u001b[1mSerial executed in 5.00 seconds.\u001b[0m\n"
]
}
],
@ -94,10 +95,10 @@
"import time\n",
"import asyncio\n",
"\n",
"from langchain.llms import OpenAI\n",
"from langchain.llms import OpenAIChat\n",
"\n",
"def generate_serially():\n",
" llm = OpenAI(temperature=0.9)\n",
" llm = OpenAIChat(temperature=0.9)\n",
" for _ in range(10):\n",
" resp = llm.generate([\"Hello, how are you?\"])\n",
" print(resp.generations[0][0].text)\n",
@ -109,7 +110,7 @@
"\n",
"\n",
"async def generate_concurrently():\n",
" llm = OpenAI(temperature=0.9)\n",
" llm = OpenAIChat(temperature=0.9)\n",
" tasks = [async_generate(llm) for _ in range(10)]\n",
" await asyncio.gather(*tasks)\n",
"\n",
@ -125,6 +126,14 @@
"elapsed = time.perf_counter() - s\n",
"print('\\033[1m' + f\"Serial executed in {elapsed:0.2f} seconds.\" + '\\033[0m')"
"LangChain provides streaming support for LLMs. Currently, we only support streaming for the `OpenAI` LLM implementation, but streaming support for other LLM implementations is on the roadmap. To utilize streaming, use a [`CallbackHandler`](https://github.com/hwchase17/langchain/blob/master/langchain/callbacks/base.py) that implements `on_llm_new_token`. In this example, we are using [`StreamingStdOutCallbackHandler`]()."
"LangChain provides streaming support for LLMs. Currently, we only support streaming for the `OpenAI` and `OpenAIChat` LLM implementation, but streaming support for other LLM implementations is on the roadmap. To utilize streaming, use a [`CallbackHandler`](https://github.com/hwchase17/langchain/blob/master/langchain/callbacks/base.py) that implements `on_llm_new_token`. In this example, we are using [`StreamingStdOutCallbackHandler`]()."
"LLMResult(generations=[[Generation(text='\\n\\nQ: What did the fish say when it hit the wall?\\nA: Dam!', generation_info={'finish_reason': 'stop', 'logprobs': None})]], llm_output={'token_usage': {}})"
"LLMResult(generations=[[Generation(text='\\n\\nQ: What did the fish say when it hit the wall?\\nA: Dam!', generation_info={'finish_reason': None, 'logprobs': None})]], llm_output={'token_usage': {}})"