Leonid Ganeline a3598193a0

docs: ecosystem/integrations update 2 (#5282 )

# docs: ecosystem/integrations update 2

#5219 - part 1 
The second part of this update (parts are independent of each other! no
overlap):

- added diffbot.md
- updated confluence.ipynb; added confluence.md
- updated college_confidential.md
- updated openai.md
- added blackboard.md
- added bilibili.md
- added azure_blob_storage.md
- added azlyrics.md
- added aws_s3.md

## Who can review?

@hwchase17@agola11
@agola11
 @vowelparrot
 @dev2049

2023-05-29 07:19:43 -07:00

2.2 KiB

Raw Blame History

OpenAI

OpenAI is American artificial intelligence (AI) research laboratory consisting of the non-profit OpenAI Incorporated and its for-profit subsidiary corporation OpenAI Limited Partnership. OpenAI conducts AI research with the declared intention of promoting and developing a friendly AI. OpenAI systems run on an Azure-based supercomputing platform from Microsoft.

The OpenAI API is powered by a diverse set of models with different capabilities and price points.

ChatGPT is the Artificial Intelligence (AI) chatbot developed by OpenAI.

Installation and Setup

Install the Python SDK with

pip install openai

Get an OpenAI api key and set it as an environment variable (OPENAI_API_KEY)
If you want to use OpenAI's tokenizer (only available for Python 3.9+), install it

pip install tiktoken

LLM

from langchain.llms import OpenAI

If you are using a model hosted on Azure, you should use different wrapper for that:

from langchain.llms import AzureOpenAI

For a more detailed walkthrough of the Azure wrapper, see this notebook

Text Embedding Model

from langchain.embeddings import OpenAIEmbeddings

For a more detailed walkthrough of this, see this notebook

Tokenizer

There are several places you can use the tiktoken tokenizer. By default, it is used to count tokens for OpenAI LLMs.

You can also use it to count tokens when splitting documents with

from langchain.text_splitter import CharacterTextSplitter
CharacterTextSplitter.from_tiktoken_encoder(...)

For a more detailed walkthrough of this, see this notebook

Chain

See a usage example.

from langchain.chains import OpenAIModerationChain

Document Loader