Archives/langchain
2
0
Fork 1
mirror of https://github.com/hwchase17/langchain synced 2024-11-08 07:10:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Generalize Comment on Streaming Support for LLM Implementations and add examples (#8115)

Browse Source 
The example provided demonstrates the usage of the
HuggingFaceTextGenInference implementation with streaming enabled.
This commit is contained in:
Adarsh Shirawalmath 2023-07-24 11:29:59 +05:30 committed by GitHub
parent 80b3ec5869
commit 8ea840432f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 31 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

30
docs/extras/modules/model_io/models/llms/integrations/huggingface_textgen_inference.ipynb
View File

@ -48,6 +48,36 @@
")\n",
"llm(\"What did foo say about bar?\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Streaming"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from langchain.llms import HuggingFaceTextGenInference\n",
"from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
"\n",
"\n",
"llm = HuggingFaceTextGenInference(\n",
" inference_server_url=\"http://localhost:8010/\",\n",
" max_new_tokens=512,\n",
" top_k=10,\n",
" top_p=0.95,\n",
" typical_p=0.95,\n",
" temperature=0.01,\n",
" repetition_penalty=1.03,\n",
" stream=True\n",
")\n",
"llm(\"What did foo say about bar?\", callbacks=[StreamingStdOutCallbackHandler()])"
]
}
],
"metadata": {

3
docs/snippets/modules/model_io/models/llms/how_to/streaming_llm.mdx
View File

@ -1,5 +1,4 @@
Currently, we support streaming for the `OpenAI`, `ChatOpenAI`, and `ChatAnthropic` implementations. To utilize streaming, use a [`CallbackHandler`](https://github.com/hwchase17/langchain/blob/master/langchain/callbacks/base.py) that implements `on_llm_new_token`. In this example, we are using `StreamingStdOutCallbackHandler`.
Currently, we support streaming for a broad range of LLM implementations, including but not limited to `OpenAI`, `ChatOpenAI`, `ChatAnthropic`, `Hugging Face Text Generation Inference`, and `Replicate`. This feature has been expanded to accommodate most of the models. To utilize streaming, use a [`CallbackHandler`](https://github.com/hwchase17/langchain/blob/master/langchain/callbacks/base.py) that implements `on_llm_new_token`. In this example, we are using `StreamingStdOutCallbackHandler`.
```python
from langchain.llms import OpenAI
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler