docs: Hugging Face platform page (#13831)

`Hugging Face` is definitely a platform. It includes many integrations
for many modules (LLM, Embedding, DocumentLoader, Tool)
So, a doc page was added that defines Hugging Face as a platform.
This commit is contained in:
Leonid Ganeline 2023-12-03 11:06:43 -08:00 committed by GitHub
parent 8a0951d934
commit 283c2994de
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 140 additions and 69 deletions

View File

@ -0,0 +1,136 @@
# Hugging Face
All functionality related to the [Hugging Face Platform](https://huggingface.co/).
## LLMs
### Hugging Face Hub
>The [Hugging Face Hub](https://huggingface.co/docs/hub/index) is a platform
> with over 350k models, 75k datasets, and 150k demo apps (Spaces), all open source
> and publicly available, in an online platform where people can easily
> collaborate and build ML together. The Hub works as a central place where anyone
> can explore, experiment, collaborate, and build technology with Machine Learning.
To use, we should have the `huggingface_hub` python [package installed](https://huggingface.co/docs/huggingface_hub/installation).
```bash
pip install huggingface_hub
```
See a [usage example](/docs/integrations/llms/huggingface_hub).
```python
from langchain.llms import HuggingFaceHub
```
### Hugging Face Local Pipelines
Hugging Face models can be run locally through the `HuggingFacePipeline` class.
We need to install `transformers` python package.
```bash
pip install transformers
```
See a [usage example](/docs/integrations/llms/huggingface_pipelines).
```python
from langchain.llms.huggingface_pipeline import HuggingFacePipeline
```
### Hugging Face TextGen Inference
>[Text Generation Inference](https://github.com/huggingface/text-generation-inference) is
> a Rust, Python and gRPC server for text generation inference. Used in production at
> [HuggingFace](https://huggingface.co/) to power LLMs api-inference widgets.
We need to install `text_generation` python package.
```bash
pip install text_generation
```
See a [usage example](/docs/integrations/llms/huggingface_textgen_inference).
```python
from langchain.llms import HuggingFaceTextGenInference
```
## Document Loaders
### Hugging Face dataset
>[Hugging Face Hub](https://huggingface.co/docs/hub/index) is home to over 75,000
> [datasets](https://huggingface.co/docs/hub/index#datasets) in more than 100 languages
> that can be used for a broad range of tasks across NLP, Computer Vision, and Audio.
> They used for a diverse range of tasks such as translation, automatic speech
> recognition, and image classification.
We need to install `datasets` python package.
```bash
pip install datasets
```
See a [usage example](/docs/integrations/document_loaders/hugging_face_dataset).
```python
from langchain.document_loaders.hugging_face_dataset import HuggingFaceDatasetLoader
```
## Embedding Models
### Hugging Face Hub
>The [Hugging Face Hub](https://huggingface.co/docs/hub/index) is a platform
> with over 350k models, 75k datasets, and 150k demo apps (Spaces), all open source
> and publicly available, in an online platform where people can easily
> collaborate and build ML together. The Hub works as a central place where anyone
> can explore, experiment, collaborate, and build technology with Machine Learning.
We need to install the `sentence_transformers` python package.
```bash
pip install sentence_transformers
```
#### HuggingFaceEmbeddings
See a [usage example](/docs/integrations/text_embedding/huggingfacehub).
```python
from langchain.embeddings import HuggingFaceEmbeddings
```
#### HuggingFaceInstructEmbeddings
See a [usage example](/docs/integrations/text_embedding/instruct_embeddings).
```python
from langchain.embeddings import HuggingFaceInstructEmbeddings
```
## Tools
### Hugging Face Hub Tools
>[Hugging Face Tools](https://huggingface.co/docs/transformers/v4.29.0/en/custom_tools)
> support text I/O and are loaded using the `load_huggingface_tool` function.
We need to install several python packages.
```bash
pip install transformers huggingface_hub
```
See a [usage example](/docs/integrations/tools/huggingface_tools).
```python
from langchain.agents import load_huggingface_tool
```

View File

@ -1,69 +0,0 @@
# Hugging Face
This page covers how to use the Hugging Face ecosystem (including the [Hugging Face Hub](https://huggingface.co)) within LangChain.
It is broken into two parts: installation and setup, and then references to specific Hugging Face wrappers.
## Installation and Setup
If you want to work with the Hugging Face Hub:
- Install the Hub client library with `pip install huggingface_hub`
- Create a Hugging Face account (it's free!)
- Create an [access token](https://huggingface.co/docs/hub/security-tokens) and set it as an environment variable (`HUGGINGFACEHUB_API_TOKEN`)
If you want work with the Hugging Face Python libraries:
- Install `pip install transformers` for working with models and tokenizers
- Install `pip install datasets` for working with datasets
## Wrappers
### LLM
There exists two Hugging Face LLM wrappers, one for a local pipeline and one for a model hosted on Hugging Face Hub.
Note that these wrappers only work for models that support the following tasks: [`text2text-generation`](https://huggingface.co/models?library=transformers&pipeline_tag=text2text-generation&sort=downloads), [`text-generation`](https://huggingface.co/models?library=transformers&pipeline_tag=text-classification&sort=downloads)
To use the local pipeline wrapper:
```python
from langchain.llms import HuggingFacePipeline
```
To use a the wrapper for a model hosted on Hugging Face Hub:
```python
from langchain.llms import HuggingFaceHub
```
For a more detailed walkthrough of the Hugging Face Hub wrapper, see [this notebook](/docs/integrations/llms/huggingface_hub)
### Embeddings
There exists two Hugging Face Embeddings wrappers, one for a local model and one for a model hosted on Hugging Face Hub.
Note that these wrappers only work for [`sentence-transformers` models](https://huggingface.co/models?library=sentence-transformers&sort=downloads).
To use the local pipeline wrapper:
```python
from langchain.embeddings import HuggingFaceEmbeddings
```
To use a the wrapper for a model hosted on Hugging Face Hub:
```python
from langchain.embeddings import HuggingFaceHubEmbeddings
```
For a more detailed walkthrough of this, see [this notebook](/docs/integrations/text_embedding/huggingfacehub)
### Tokenizer
There are several places you can use tokenizers available through the `transformers` package.
By default, it is used to count tokens for all LLMs.
You can also use it to count tokens when splitting documents with
```python
from langchain.text_splitter import CharacterTextSplitter
CharacterTextSplitter.from_huggingface_tokenizer(...)
```
For a more detailed walkthrough of this, see [this notebook](/docs/modules/data_connection/document_transformers/text_splitters/huggingface_length_function)
### Datasets
The Hugging Face Hub has lots of great [datasets](https://huggingface.co/datasets) that can be used to evaluate your LLM chains.
For a detailed walkthrough of how to use them to do so, see [this notebook](/docs/integrations/document_loaders/hugging_face_dataset)

View File

@ -304,6 +304,10 @@
"source": "/docs/integrations/providers/amazon_api_gateway",
"destination": "/docs/integrations/platforms/aws"
},
{
"source": "/docs/integrations/providers/huggingface",
"destination": "/docs/integrations/platforms/huggingface"
},
{
"source": "/docs/integrations/providers/azure_blob_storage",
"destination": "/docs/integrations/platforms/microsoft"