mirror of
https://github.com/hwchase17/langchain
synced 2024-11-18 09:25:54 +00:00
docs: Hugging Face
platform page (#13831)
`Hugging Face` is definitely a platform. It includes many integrations for many modules (LLM, Embedding, DocumentLoader, Tool) So, a doc page was added that defines Hugging Face as a platform.
This commit is contained in:
parent
8a0951d934
commit
283c2994de
136
docs/docs/integrations/platforms/huggingface.mdx
Normal file
136
docs/docs/integrations/platforms/huggingface.mdx
Normal file
@ -0,0 +1,136 @@
|
||||
# Hugging Face
|
||||
|
||||
All functionality related to the [Hugging Face Platform](https://huggingface.co/).
|
||||
|
||||
## LLMs
|
||||
|
||||
### Hugging Face Hub
|
||||
|
||||
>The [Hugging Face Hub](https://huggingface.co/docs/hub/index) is a platform
|
||||
> with over 350k models, 75k datasets, and 150k demo apps (Spaces), all open source
|
||||
> and publicly available, in an online platform where people can easily
|
||||
> collaborate and build ML together. The Hub works as a central place where anyone
|
||||
> can explore, experiment, collaborate, and build technology with Machine Learning.
|
||||
|
||||
To use, we should have the `huggingface_hub` python [package installed](https://huggingface.co/docs/huggingface_hub/installation).
|
||||
|
||||
```bash
|
||||
pip install huggingface_hub
|
||||
```
|
||||
|
||||
See a [usage example](/docs/integrations/llms/huggingface_hub).
|
||||
|
||||
```python
|
||||
from langchain.llms import HuggingFaceHub
|
||||
```
|
||||
|
||||
### Hugging Face Local Pipelines
|
||||
|
||||
Hugging Face models can be run locally through the `HuggingFacePipeline` class.
|
||||
|
||||
We need to install `transformers` python package.
|
||||
|
||||
```bash
|
||||
pip install transformers
|
||||
```
|
||||
|
||||
See a [usage example](/docs/integrations/llms/huggingface_pipelines).
|
||||
|
||||
```python
|
||||
from langchain.llms.huggingface_pipeline import HuggingFacePipeline
|
||||
```
|
||||
|
||||
### Hugging Face TextGen Inference
|
||||
|
||||
>[Text Generation Inference](https://github.com/huggingface/text-generation-inference) is
|
||||
> a Rust, Python and gRPC server for text generation inference. Used in production at
|
||||
> [HuggingFace](https://huggingface.co/) to power LLMs api-inference widgets.
|
||||
|
||||
We need to install `text_generation` python package.
|
||||
|
||||
```bash
|
||||
pip install text_generation
|
||||
```
|
||||
|
||||
See a [usage example](/docs/integrations/llms/huggingface_textgen_inference).
|
||||
|
||||
```python
|
||||
from langchain.llms import HuggingFaceTextGenInference
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Document Loaders
|
||||
|
||||
### Hugging Face dataset
|
||||
|
||||
>[Hugging Face Hub](https://huggingface.co/docs/hub/index) is home to over 75,000
|
||||
> [datasets](https://huggingface.co/docs/hub/index#datasets) in more than 100 languages
|
||||
> that can be used for a broad range of tasks across NLP, Computer Vision, and Audio.
|
||||
> They used for a diverse range of tasks such as translation, automatic speech
|
||||
> recognition, and image classification.
|
||||
|
||||
We need to install `datasets` python package.
|
||||
|
||||
```bash
|
||||
pip install datasets
|
||||
```
|
||||
|
||||
See a [usage example](/docs/integrations/document_loaders/hugging_face_dataset).
|
||||
|
||||
```python
|
||||
from langchain.document_loaders.hugging_face_dataset import HuggingFaceDatasetLoader
|
||||
```
|
||||
|
||||
|
||||
## Embedding Models
|
||||
|
||||
### Hugging Face Hub
|
||||
|
||||
>The [Hugging Face Hub](https://huggingface.co/docs/hub/index) is a platform
|
||||
> with over 350k models, 75k datasets, and 150k demo apps (Spaces), all open source
|
||||
> and publicly available, in an online platform where people can easily
|
||||
> collaborate and build ML together. The Hub works as a central place where anyone
|
||||
> can explore, experiment, collaborate, and build technology with Machine Learning.
|
||||
|
||||
We need to install the `sentence_transformers` python package.
|
||||
|
||||
```bash
|
||||
pip install sentence_transformers
|
||||
```
|
||||
|
||||
|
||||
#### HuggingFaceEmbeddings
|
||||
|
||||
See a [usage example](/docs/integrations/text_embedding/huggingfacehub).
|
||||
|
||||
```python
|
||||
from langchain.embeddings import HuggingFaceEmbeddings
|
||||
```
|
||||
#### HuggingFaceInstructEmbeddings
|
||||
|
||||
See a [usage example](/docs/integrations/text_embedding/instruct_embeddings).
|
||||
|
||||
```python
|
||||
from langchain.embeddings import HuggingFaceInstructEmbeddings
|
||||
```
|
||||
|
||||
|
||||
## Tools
|
||||
|
||||
### Hugging Face Hub Tools
|
||||
|
||||
>[Hugging Face Tools](https://huggingface.co/docs/transformers/v4.29.0/en/custom_tools)
|
||||
> support text I/O and are loaded using the `load_huggingface_tool` function.
|
||||
|
||||
We need to install several python packages.
|
||||
|
||||
```bash
|
||||
pip install transformers huggingface_hub
|
||||
```
|
||||
|
||||
See a [usage example](/docs/integrations/tools/huggingface_tools).
|
||||
|
||||
```python
|
||||
from langchain.agents import load_huggingface_tool
|
||||
```
|
@ -1,69 +0,0 @@
|
||||
# Hugging Face
|
||||
|
||||
This page covers how to use the Hugging Face ecosystem (including the [Hugging Face Hub](https://huggingface.co)) within LangChain.
|
||||
It is broken into two parts: installation and setup, and then references to specific Hugging Face wrappers.
|
||||
|
||||
## Installation and Setup
|
||||
|
||||
If you want to work with the Hugging Face Hub:
|
||||
- Install the Hub client library with `pip install huggingface_hub`
|
||||
- Create a Hugging Face account (it's free!)
|
||||
- Create an [access token](https://huggingface.co/docs/hub/security-tokens) and set it as an environment variable (`HUGGINGFACEHUB_API_TOKEN`)
|
||||
|
||||
If you want work with the Hugging Face Python libraries:
|
||||
- Install `pip install transformers` for working with models and tokenizers
|
||||
- Install `pip install datasets` for working with datasets
|
||||
|
||||
## Wrappers
|
||||
|
||||
### LLM
|
||||
|
||||
There exists two Hugging Face LLM wrappers, one for a local pipeline and one for a model hosted on Hugging Face Hub.
|
||||
Note that these wrappers only work for models that support the following tasks: [`text2text-generation`](https://huggingface.co/models?library=transformers&pipeline_tag=text2text-generation&sort=downloads), [`text-generation`](https://huggingface.co/models?library=transformers&pipeline_tag=text-classification&sort=downloads)
|
||||
|
||||
To use the local pipeline wrapper:
|
||||
```python
|
||||
from langchain.llms import HuggingFacePipeline
|
||||
```
|
||||
|
||||
To use a the wrapper for a model hosted on Hugging Face Hub:
|
||||
```python
|
||||
from langchain.llms import HuggingFaceHub
|
||||
```
|
||||
For a more detailed walkthrough of the Hugging Face Hub wrapper, see [this notebook](/docs/integrations/llms/huggingface_hub)
|
||||
|
||||
|
||||
### Embeddings
|
||||
|
||||
There exists two Hugging Face Embeddings wrappers, one for a local model and one for a model hosted on Hugging Face Hub.
|
||||
Note that these wrappers only work for [`sentence-transformers` models](https://huggingface.co/models?library=sentence-transformers&sort=downloads).
|
||||
|
||||
To use the local pipeline wrapper:
|
||||
```python
|
||||
from langchain.embeddings import HuggingFaceEmbeddings
|
||||
```
|
||||
|
||||
To use a the wrapper for a model hosted on Hugging Face Hub:
|
||||
```python
|
||||
from langchain.embeddings import HuggingFaceHubEmbeddings
|
||||
```
|
||||
For a more detailed walkthrough of this, see [this notebook](/docs/integrations/text_embedding/huggingfacehub)
|
||||
|
||||
### Tokenizer
|
||||
|
||||
There are several places you can use tokenizers available through the `transformers` package.
|
||||
By default, it is used to count tokens for all LLMs.
|
||||
|
||||
You can also use it to count tokens when splitting documents with
|
||||
```python
|
||||
from langchain.text_splitter import CharacterTextSplitter
|
||||
CharacterTextSplitter.from_huggingface_tokenizer(...)
|
||||
```
|
||||
For a more detailed walkthrough of this, see [this notebook](/docs/modules/data_connection/document_transformers/text_splitters/huggingface_length_function)
|
||||
|
||||
|
||||
### Datasets
|
||||
|
||||
The Hugging Face Hub has lots of great [datasets](https://huggingface.co/datasets) that can be used to evaluate your LLM chains.
|
||||
|
||||
For a detailed walkthrough of how to use them to do so, see [this notebook](/docs/integrations/document_loaders/hugging_face_dataset)
|
@ -304,6 +304,10 @@
|
||||
"source": "/docs/integrations/providers/amazon_api_gateway",
|
||||
"destination": "/docs/integrations/platforms/aws"
|
||||
},
|
||||
{
|
||||
"source": "/docs/integrations/providers/huggingface",
|
||||
"destination": "/docs/integrations/platforms/huggingface"
|
||||
},
|
||||
{
|
||||
"source": "/docs/integrations/providers/azure_blob_storage",
|
||||
"destination": "/docs/integrations/platforms/microsoft"
|
||||
|
Loading…
Reference in New Issue
Block a user