mirror of
https://github.com/hwchase17/langchain
synced 2024-10-31 15:20:26 +00:00
621da3c164
Adds [DeepSparse](https://github.com/neuralmagic/deepsparse) as an LLM backend. DeepSparse supports running various open-source sparsified models hosted on [SparseZoo](https://sparsezoo.neuralmagic.com/) for performance gains on CPUs. Twitter handles: @mgoin_ @neuralmagic --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
36 lines
1.2 KiB
Plaintext
36 lines
1.2 KiB
Plaintext
# DeepSparse
|
|
|
|
This page covers how to use the [DeepSparse](https://github.com/neuralmagic/deepsparse) inference runtime within LangChain.
|
|
It is broken into two parts: installation and setup, and then examples of DeepSparse usage.
|
|
|
|
## Installation and Setup
|
|
|
|
- Install the Python package with `pip install deepsparse`
|
|
- Choose a [SparseZoo model](https://sparsezoo.neuralmagic.com/?useCase=text_generation) or export a support model to ONNX [using Optimum](https://github.com/neuralmagic/notebooks/blob/main/notebooks/opt-text-generation-deepsparse-quickstart/OPT_Text_Generation_DeepSparse_Quickstart.ipynb)
|
|
|
|
## Wrappers
|
|
|
|
### LLM
|
|
|
|
There exists a DeepSparse LLM wrapper, which you can access with:
|
|
|
|
```python
|
|
from langchain.llms import DeepSparse
|
|
```
|
|
|
|
It provides a unified interface for all models:
|
|
|
|
```python
|
|
llm = DeepSparse(model='zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none')
|
|
|
|
print(llm('def fib():'))
|
|
```
|
|
|
|
Additional parameters can be passed using the `config` parameter:
|
|
|
|
```python
|
|
config = {'max_generated_tokens': 256}
|
|
|
|
llm = DeepSparse(model='zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none', config=config)
|
|
```
|