diff --git a/docs/integrations.rst b/docs/integrations.rst index 66128fd8..e9405697 100644 --- a/docs/integrations.rst +++ b/docs/integrations.rst @@ -20,6 +20,12 @@ Integrations by Module - `Toolkit Integrations <./modules/agents/toolkits.html>`_ +Dependencies +---------------- + +| LangChain depends on `several hungered Python packages `_. + + All Integrations ------------------------------------------- diff --git a/docs/integrations/airbyte_json.md b/docs/integrations/airbyte_json.md new file mode 100644 index 00000000..f3918ce5 --- /dev/null +++ b/docs/integrations/airbyte_json.md @@ -0,0 +1,29 @@ +Airbyte JSON + +>[Airbyte](https://github.com/airbytehq/airbyte) is a data integration platform for ELT pipelines from APIs, +> databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases. + +## Installation and Setup + +This instruction shows how to load any source from `Airbyte` into a local `JSON` file that can be read in as a document. + +**Prerequisites:** +Have `docker desktop` installed. + +**Steps:** +1. Clone Airbyte from GitHub - `git clone https://github.com/airbytehq/airbyte.git`. +2. Switch into Airbyte directory - `cd airbyte`. +3. Start Airbyte - `docker compose up`. +4. In your browser, just visit http://localhost:8000. You will be asked for a username and password. By default, that's username `airbyte` and password `password`. +5. Setup any source you wish. +6. Set destination as Local JSON, with specified destination path - lets say `/json_data`. Set up a manual sync. +7. Run the connection. +8. To see what files are created, navigate to: `file:///tmp/airbyte_local/`. + +## Document Loader + +See a [usage example](../modules/indexes/document_loaders/examples/airbyte_json.ipynb). + +```python +from langchain.document_loaders import AirbyteJSONLoader +``` diff --git a/docs/integrations/aleph_alpha.md b/docs/integrations/aleph_alpha.md new file mode 100644 index 00000000..52746049 --- /dev/null +++ b/docs/integrations/aleph_alpha.md @@ -0,0 +1,36 @@ +# Aleph Alpha + +>[Aleph Alpha](https://docs.aleph-alpha.com/) was founded in 2019 with the mission to research and build the foundational technology for an era of strong AI. The team of international scientists, engineers, and innovators researches, develops, and deploys transformative AI like large language and multimodal models and runs the fastest European commercial AI cluster. + +>[The Luminous series](https://docs.aleph-alpha.com/docs/introduction/luminous/) is a family of large language models. + +## Installation and Setup + +```bash +pip install aleph-alpha-client +``` + +You have to create a new token. Please, see [instructions](https://docs.aleph-alpha.com/docs/account/#create-a-new-token). + +```python +from getpass import getpass + +ALEPH_ALPHA_API_KEY = getpass() +``` + + +## LLM + +See a [usage example](../modules/models/llms/integrations/aleph_alpha.ipynb). + +```python +from langchain.llms import AlephAlpha +``` + +## Text Embedding Models + +See a [usage example](../modules/models/text_embedding/examples/aleph_alpha.ipynb). + +```python +from langchain.embeddings import AlephAlphaSymmetricSemanticEmbedding, AlephAlphaAsymmetricSemanticEmbedding +``` diff --git a/docs/integrations/arxiv.md b/docs/integrations/arxiv.md new file mode 100644 index 00000000..749e1feb --- /dev/null +++ b/docs/integrations/arxiv.md @@ -0,0 +1,28 @@ +# Arxiv + +>[arXiv](https://arxiv.org/) is an open-access archive for 2 million scholarly articles in the fields of physics, +> mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and +> systems science, and economics. + + +## Installation and Setup + +First, you need to install `arxiv` python package. + +```bash +pip install arxiv +``` + +Second, you need to install `PyMuPDF` python package which transforms PDF files downloaded from the `arxiv.org` site into the text format. + +```bash +pip install pymupdf +``` + +## Document Loader + +See a [usage example](../modules/indexes/document_loaders/examples/arxiv.ipynb). + +```python +from langchain.document_loaders import ArxivLoader +``` diff --git a/docs/integrations/azure_openai.md b/docs/integrations/azure_openai.md new file mode 100644 index 00000000..9e39f38a --- /dev/null +++ b/docs/integrations/azure_openai.md @@ -0,0 +1,50 @@ +# Azure OpenAI + +>[Microsoft Azure](https://en.wikipedia.org/wiki/Microsoft_Azure), often referred to as `Azure` is a cloud computing platform run by `Microsoft`, which offers access, management, and development of applications and services through global data centers. It provides a range of capabilities, including software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS). `Microsoft Azure` supports many programming languages, tools, and frameworks, including Microsoft-specific and third-party software and systems. + + +>[Azure OpenAI](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/) is an `Azure` service with powerful language models from `OpenAI` including the `GPT-3`, `Codex` and `Embeddings model` series for content generation, summarization, semantic search, and natural language to code translation. + + +## Installation and Setup + +```bash +pip install openai +pip install tiktoken +``` + + +Set the environment variables to get access to the `Azure OpenAI` service. + +```python +import os + +os.environ["OPENAI_API_TYPE"] = "azure" +os.environ["OPENAI_API_BASE"] = "https://[Amazon SageMaker](https://aws.amazon.com/sagemaker/) is a system that can build, train, and deploy machine learning (ML) models with fully managed infrastructure, tools, and workflows. + +We use `SageMaker` to host our model and expose it as the `SageMaker Endpoint`. + + +## Installation and Setup + +```bash +pip install boto3 +``` + +For instructions on how to expose model as a `SageMaker Endpoint`, please see [here](https://www.philschmid.de/custom-inference-huggingface-sagemaker). + +**Note**: In order to handle batched requests, we need to adjust the return line in the `predict_fn()` function within the custom `inference.py` script: + +Change from + +``` +return {"vectors": sentence_embeddings[0].tolist()} +``` + +to: + +``` +return {"vectors": sentence_embeddings.tolist()} +``` + + + +We have to set up following required parameters of the `SagemakerEndpoint` call: +- `endpoint_name`: The name of the endpoint from the deployed Sagemaker model. + Must be unique within an AWS Region. +- `credentials_profile_name`: The name of the profile in the ~/.aws/credentials or ~/.aws/config files, which + has either access keys or role information specified. + If not specified, the default credential profile or, if on an EC2 instance, + credentials from IMDS will be used. + See [this guide](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html). + +## LLM + +See a [usage example](../modules/models/llms/integrations/sagemaker.ipynb). + +```python +from langchain import SagemakerEndpoint +from langchain.llms.sagemaker_endpoint import LLMContentHandler +``` + +## Text Embedding Models + +See a [usage example](../modules/models/text_embedding/examples/sagemaker-endpoint.ipynb). +```python +from langchain.embeddings import SagemakerEndpointEmbeddings +from langchain.llms.sagemaker_endpoint import ContentHandlerBase +``` diff --git a/docs/modules/indexes/document_loaders/examples/arxiv.ipynb b/docs/modules/indexes/document_loaders/examples/arxiv.ipynb index 96ef81a0..07dd45d1 100644 --- a/docs/modules/indexes/document_loaders/examples/arxiv.ipynb +++ b/docs/modules/indexes/document_loaders/examples/arxiv.ipynb @@ -47,7 +47,7 @@ "tags": [] }, "source": [ - "Second, you need to install `PyMuPDF` python package which transform PDF files from the `arxiv.org` site into the text format." + "Second, you need to install `PyMuPDF` python package which transforms PDF files downloaded from the `arxiv.org` site into the text format." ] }, {