langchain/docs/extras/integrations/callbacks/infino.ipynb

406 lines
183 KiB
Plaintext
Raw Normal View History

Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
{
"cells": [
{
"cell_type": "markdown",
"id": "8d10861f-a550-4443-bc63-4ce2ae13b841",
"metadata": {},
"source": [
"# Infino\n",
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
"\n",
"This example shows how one can track the following while calling OpenAI models via `LangChain` and [Infino](https://github.com/infinohq/infino):\n",
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
"\n",
"* prompt input,\n",
"* response from `ChatGPT` or any other `LangChain` model,\n",
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
"* latency,\n",
"* errors,\n",
"* number of tokens consumed"
]
},
{
"cell_type": "markdown",
"id": "64d14c88-b71c-4524-ab1b-4250a7dbb62b",
"metadata": {},
"source": [
"## Initializing"
]
},
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
{
"cell_type": "code",
"execution_count": null,
"id": "ed46c894-caa6-49b2-85d1-f275374fa308",
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
"metadata": {},
"outputs": [],
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
"source": [
"# Install necessary dependencies.\n",
"!pip install infinopy\n",
"!pip install matplotlib"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3a5a0976-9953-41d8-880c-eb3f2992e936",
"metadata": {},
"outputs": [],
"source": [
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
"# Remove the (1) import sys and sys.path.append(..) and (2) uncomment `!pip install langchain` after merging the PR for Infino/LangChain integration.\n",
"import sys\n",
Fix `make docs_build` and related scripts (#7276) **Description: a description of the change** Fixed `make docs_build` and related scripts which caused errors. There are several changes. First, I made the build of the documentation and the API Reference into two separate commands. This is because it takes less time to build. The commands for documents are `make docs_build`, `make docs_clean`, and `make docs_linkcheck`. The commands for API Reference are `make api_docs_build`, `api_docs_clean`, and `api_docs_linkcheck`. It looked like `docs/.local_build.sh` could be used to build the documentation, so I used that. Since `.local_build.sh` was also building API Rerefence internally, I removed that process. `.local_build.sh` also added some Bash options to stop in error or so. Futher more added `cd "${SCRIPT_DIR}"` at the beginning so that the script will work no matter which directory it is executed in. `docs/api_reference/api_reference.rst` is removed, because which is generated by `docs/api_reference/create_api_rst.py`, and added it to .gitignore. Finally, the description of CONTRIBUTING.md was modified. **Issue: the issue # it fixes (if applicable)** https://github.com/hwchase17/langchain/issues/6413 **Dependencies: any dependencies required for this change** `nbdoc` was missing in group docs so it was added. I installed it with the `poetry add --group docs nbdoc` command. I am concerned if any modifications are needed to poetry.lock. I would greatly appreciate it if you could pay close attention to this file during the review. **Tag maintainer** - General / Misc / if you don't know who to tag: @baskaryan If this PR needs any additional changes, I'll be happy to make them! --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-07-12 02:05:14 +00:00
"\n",
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
"sys.path.append(\"../../../../../langchain\")\n",
"\n",
"\n",
"import datetime as dt\n",
"import json\n",
"from langchain.llms import OpenAI\n",
"import matplotlib.pyplot as plt\n",
"import matplotlib.dates as md\n",
"import os\n",
"import time\n",
"import sys\n",
"\n",
"from infinopy import InfinoClient\n",
"from langchain.callbacks import InfinoCallbackHandler"
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
]
},
{
"cell_type": "markdown",
"id": "9f90210d-c805-4a0c-81e4-d5298942afc4",
"metadata": {},
"source": [
"## Start Infino server, initialize the Infino client"
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "748b9858-5145-4351-976a-ca2d54e836a6",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"497a621125800abdd19f57ce7e033349b3cf83ca8cea6a74e8e28433a42ecadd\n"
]
}
],
"source": [
"# Start server using the Infino docker image.\n",
"!docker run --rm --detach --name infino-example -p 3000:3000 infinohq/infino:latest\n",
"\n",
"# Create Infino client.\n",
"client = InfinoClient()"
]
},
{
"cell_type": "markdown",
"id": "b6b81cda-b841-43ee-8c5e-b1576555765f",
"metadata": {},
"source": [
"## Read the questions dataset"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "b659fd0c-0d8c-470e-8b6c-867a117f2a27",
"metadata": {},
"outputs": [],
"source": [
Fix `make docs_build` and related scripts (#7276) **Description: a description of the change** Fixed `make docs_build` and related scripts which caused errors. There are several changes. First, I made the build of the documentation and the API Reference into two separate commands. This is because it takes less time to build. The commands for documents are `make docs_build`, `make docs_clean`, and `make docs_linkcheck`. The commands for API Reference are `make api_docs_build`, `api_docs_clean`, and `api_docs_linkcheck`. It looked like `docs/.local_build.sh` could be used to build the documentation, so I used that. Since `.local_build.sh` was also building API Rerefence internally, I removed that process. `.local_build.sh` also added some Bash options to stop in error or so. Futher more added `cd "${SCRIPT_DIR}"` at the beginning so that the script will work no matter which directory it is executed in. `docs/api_reference/api_reference.rst` is removed, because which is generated by `docs/api_reference/create_api_rst.py`, and added it to .gitignore. Finally, the description of CONTRIBUTING.md was modified. **Issue: the issue # it fixes (if applicable)** https://github.com/hwchase17/langchain/issues/6413 **Dependencies: any dependencies required for this change** `nbdoc` was missing in group docs so it was added. I installed it with the `poetry add --group docs nbdoc` command. I am concerned if any modifications are needed to poetry.lock. I would greatly appreciate it if you could pay close attention to this file during the review. **Tag maintainer** - General / Misc / if you don't know who to tag: @baskaryan If this PR needs any additional changes, I'll be happy to make them! --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-07-12 02:05:14 +00:00
"# These are a subset of questions from Stanford's QA dataset -\n",
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
"# https://rajpurkar.github.io/SQuAD-explorer/\n",
Fix `make docs_build` and related scripts (#7276) **Description: a description of the change** Fixed `make docs_build` and related scripts which caused errors. There are several changes. First, I made the build of the documentation and the API Reference into two separate commands. This is because it takes less time to build. The commands for documents are `make docs_build`, `make docs_clean`, and `make docs_linkcheck`. The commands for API Reference are `make api_docs_build`, `api_docs_clean`, and `api_docs_linkcheck`. It looked like `docs/.local_build.sh` could be used to build the documentation, so I used that. Since `.local_build.sh` was also building API Rerefence internally, I removed that process. `.local_build.sh` also added some Bash options to stop in error or so. Futher more added `cd "${SCRIPT_DIR}"` at the beginning so that the script will work no matter which directory it is executed in. `docs/api_reference/api_reference.rst` is removed, because which is generated by `docs/api_reference/create_api_rst.py`, and added it to .gitignore. Finally, the description of CONTRIBUTING.md was modified. **Issue: the issue # it fixes (if applicable)** https://github.com/hwchase17/langchain/issues/6413 **Dependencies: any dependencies required for this change** `nbdoc` was missing in group docs so it was added. I installed it with the `poetry add --group docs nbdoc` command. I am concerned if any modifications are needed to poetry.lock. I would greatly appreciate it if you could pay close attention to this file during the review. **Tag maintainer** - General / Misc / if you don't know who to tag: @baskaryan If this PR needs any additional changes, I'll be happy to make them! --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-07-12 02:05:14 +00:00
"data = \"\"\"In what country is Normandy located?\n",
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
"When were the Normans in Normandy?\n",
"From which countries did the Norse originate?\n",
"Who was the Norse leader?\n",
"What century did the Normans first gain their separate identity?\n",
"Who gave their name to Normandy in the 1000's and 1100's\n",
"What is France a region of?\n",
"Who did King Charles III swear fealty to?\n",
"When did the Frankish identity emerge?\n",
"Who was the duke in the battle of Hastings?\n",
"Who ruled the duchy of Normandy\n",
"What religion were the Normans\n",
"What type of major impact did the Norman dynasty have on modern Europe?\n",
"Who was famed for their Christian spirit?\n",
"Who assimilted the Roman language?\n",
"Who ruled the country of Normandy?\n",
"What principality did William the conquerer found?\n",
"What is the original meaning of the word Norman?\n",
"When was the Latin version of the word Norman first recorded?\n",
Fix `make docs_build` and related scripts (#7276) **Description: a description of the change** Fixed `make docs_build` and related scripts which caused errors. There are several changes. First, I made the build of the documentation and the API Reference into two separate commands. This is because it takes less time to build. The commands for documents are `make docs_build`, `make docs_clean`, and `make docs_linkcheck`. The commands for API Reference are `make api_docs_build`, `api_docs_clean`, and `api_docs_linkcheck`. It looked like `docs/.local_build.sh` could be used to build the documentation, so I used that. Since `.local_build.sh` was also building API Rerefence internally, I removed that process. `.local_build.sh` also added some Bash options to stop in error or so. Futher more added `cd "${SCRIPT_DIR}"` at the beginning so that the script will work no matter which directory it is executed in. `docs/api_reference/api_reference.rst` is removed, because which is generated by `docs/api_reference/create_api_rst.py`, and added it to .gitignore. Finally, the description of CONTRIBUTING.md was modified. **Issue: the issue # it fixes (if applicable)** https://github.com/hwchase17/langchain/issues/6413 **Dependencies: any dependencies required for this change** `nbdoc` was missing in group docs so it was added. I installed it with the `poetry add --group docs nbdoc` command. I am concerned if any modifications are needed to poetry.lock. I would greatly appreciate it if you could pay close attention to this file during the review. **Tag maintainer** - General / Misc / if you don't know who to tag: @baskaryan If this PR needs any additional changes, I'll be happy to make them! --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-07-12 02:05:14 +00:00
"What name comes from the English words Normans/Normanz?\"\"\"\n",
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
"\n",
Fix `make docs_build` and related scripts (#7276) **Description: a description of the change** Fixed `make docs_build` and related scripts which caused errors. There are several changes. First, I made the build of the documentation and the API Reference into two separate commands. This is because it takes less time to build. The commands for documents are `make docs_build`, `make docs_clean`, and `make docs_linkcheck`. The commands for API Reference are `make api_docs_build`, `api_docs_clean`, and `api_docs_linkcheck`. It looked like `docs/.local_build.sh` could be used to build the documentation, so I used that. Since `.local_build.sh` was also building API Rerefence internally, I removed that process. `.local_build.sh` also added some Bash options to stop in error or so. Futher more added `cd "${SCRIPT_DIR}"` at the beginning so that the script will work no matter which directory it is executed in. `docs/api_reference/api_reference.rst` is removed, because which is generated by `docs/api_reference/create_api_rst.py`, and added it to .gitignore. Finally, the description of CONTRIBUTING.md was modified. **Issue: the issue # it fixes (if applicable)** https://github.com/hwchase17/langchain/issues/6413 **Dependencies: any dependencies required for this change** `nbdoc` was missing in group docs so it was added. I installed it with the `poetry add --group docs nbdoc` command. I am concerned if any modifications are needed to poetry.lock. I would greatly appreciate it if you could pay close attention to this file during the review. **Tag maintainer** - General / Misc / if you don't know who to tag: @baskaryan If this PR needs any additional changes, I'll be happy to make them! --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-07-12 02:05:14 +00:00
"questions = data.split(\"\\n\")"
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
]
},
{
"cell_type": "markdown",
"id": "dce1b820-3f1a-4b94-b848-4c6032cadc18",
"metadata": {},
"source": [
"## LangChain OpenAI Q&A; Publish metrics and logs to Infino"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "d5cebf35-2d10-48b8-ab11-c4a574c595d2",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"In what country is Normandy located?\n",
"generations=[[Generation(text='\\n\\nNormandy is located in France.', generation_info={'finish_reason': 'stop', 'logprobs': None})]] llm_output={'token_usage': {'total_tokens': 16, 'completion_tokens': 9, 'prompt_tokens': 7}, 'model_name': 'text-davinci-003'} run=RunInfo(run_id=UUID('8de21639-acec-4bd1-a12d-8124de1e20da'))\n",
"When were the Normans in Normandy?\n",
"generations=[[Generation(text='\\n\\nThe Normans first settled in Normandy in the late 9th century.', generation_info={'finish_reason': 'stop', 'logprobs': None})]] llm_output={'token_usage': {'total_tokens': 24, 'completion_tokens': 16, 'prompt_tokens': 8}, 'model_name': 'text-davinci-003'} run=RunInfo(run_id=UUID('cf81fc86-250b-4e6e-9d92-2df3bebb019a'))\n",
"From which countries did the Norse originate?\n",
"generations=[[Generation(text='\\n\\nThe Norse originated from Scandinavia, which includes modern-day Norway, Sweden, and Denmark.', generation_info={'finish_reason': 'stop', 'logprobs': None})]] llm_output={'token_usage': {'total_tokens': 29, 'completion_tokens': 21, 'prompt_tokens': 8}, 'model_name': 'text-davinci-003'} run=RunInfo(run_id=UUID('50f42f5e-b4a4-411a-a049-f92cb573a74f'))\n",
"Who was the Norse leader?\n",
"generations=[[Generation(text='\\n\\nThe most famous Norse leader was the legendary Viking king Ragnar Lodbrok. He is believed to have lived in the 9th century and is renowned for his exploits in England and France.', generation_info={'finish_reason': 'stop', 'logprobs': None})]] llm_output={'token_usage': {'total_tokens': 45, 'completion_tokens': 39, 'prompt_tokens': 6}, 'model_name': 'text-davinci-003'} run=RunInfo(run_id=UUID('e32f31cb-ddc9-4863-8e6e-cb7a281a0ada'))\n",
"What century did the Normans first gain their separate identity?\n",
"generations=[[Generation(text='\\n\\nThe Normans first gained their separate identity in the 11th century.', generation_info={'finish_reason': 'stop', 'logprobs': None})]] llm_output={'token_usage': {'total_tokens': 28, 'completion_tokens': 16, 'prompt_tokens': 12}, 'model_name': 'text-davinci-003'} run=RunInfo(run_id=UUID('da9d8f73-b3b3-4bc5-8495-da8b11462a51'))\n",
"Who gave their name to Normandy in the 1000's and 1100's\n",
"generations=[[Generation(text='\\n\\nThe Normans, a people from northern France, gave their name to Normandy in the 1000s and 1100s. The Normans were descended from Viking settlers who had come to the region in the late 800s.', generation_info={'finish_reason': 'stop', 'logprobs': None})]] llm_output={'token_usage': {'total_tokens': 58, 'completion_tokens': 45, 'prompt_tokens': 13}, 'model_name': 'text-davinci-003'} run=RunInfo(run_id=UUID('bb5829bf-b6a6-4429-adfa-414ac5be46e5'))\n",
"What is France a region of?\n",
"generations=[[Generation(text='\\n\\nFrance is a region of Europe.', generation_info={'finish_reason': 'stop', 'logprobs': None})]] llm_output={'token_usage': {'total_tokens': 16, 'completion_tokens': 9, 'prompt_tokens': 7}, 'model_name': 'text-davinci-003'} run=RunInfo(run_id=UUID('6943880b-b4e4-4c74-9ca1-8c03c10f7e9c'))\n",
"Who did King Charles III swear fealty to?\n",
"generations=[[Generation(text='\\n\\nKing Charles III swore fealty to Pope Innocent III.', generation_info={'finish_reason': 'stop', 'logprobs': None})]] llm_output={'token_usage': {'total_tokens': 23, 'completion_tokens': 13, 'prompt_tokens': 10}, 'model_name': 'text-davinci-003'} run=RunInfo(run_id=UUID('c91fd663-09e6-4d00-b746-4c7fd96f9ceb'))\n",
"When did the Frankish identity emerge?\n",
"generations=[[Generation(text='\\n\\nThe Frankish identity began to emerge in the late 5th century, when the Franks began to expand their power and influence in the region. The Franks were a Germanic tribe that had migrated to the area from the east and had established a kingdom in what is now modern-day France. The Franks were eventually able to establish a powerful kingdom that lasted until the 10th century.', generation_info={'finish_reason': 'stop', 'logprobs': None})]] llm_output={'token_usage': {'total_tokens': 86, 'completion_tokens': 78, 'prompt_tokens': 8}, 'model_name': 'text-davinci-003'} run=RunInfo(run_id=UUID('23f86775-e592-4cb8-baa3-46ebe74305b2'))\n",
"Who was the duke in the battle of Hastings?\n",
"generations=[[Generation(text='\\n\\nThe Duke of Normandy, William the Conqueror, was the leader of the Norman forces at the Battle of Hastings in 1066.', generation_info={'finish_reason': 'stop', 'logprobs': None})]] llm_output={'token_usage': {'total_tokens': 39, 'completion_tokens': 28, 'prompt_tokens': 11}, 'model_name': 'text-davinci-003'} run=RunInfo(run_id=UUID('ad5b7984-8758-4d95-a5eb-ee56e0218f6b'))\n"
]
}
],
"source": [
"# Set your key here.\n",
Fix `make docs_build` and related scripts (#7276) **Description: a description of the change** Fixed `make docs_build` and related scripts which caused errors. There are several changes. First, I made the build of the documentation and the API Reference into two separate commands. This is because it takes less time to build. The commands for documents are `make docs_build`, `make docs_clean`, and `make docs_linkcheck`. The commands for API Reference are `make api_docs_build`, `api_docs_clean`, and `api_docs_linkcheck`. It looked like `docs/.local_build.sh` could be used to build the documentation, so I used that. Since `.local_build.sh` was also building API Rerefence internally, I removed that process. `.local_build.sh` also added some Bash options to stop in error or so. Futher more added `cd "${SCRIPT_DIR}"` at the beginning so that the script will work no matter which directory it is executed in. `docs/api_reference/api_reference.rst` is removed, because which is generated by `docs/api_reference/create_api_rst.py`, and added it to .gitignore. Finally, the description of CONTRIBUTING.md was modified. **Issue: the issue # it fixes (if applicable)** https://github.com/hwchase17/langchain/issues/6413 **Dependencies: any dependencies required for this change** `nbdoc` was missing in group docs so it was added. I installed it with the `poetry add --group docs nbdoc` command. I am concerned if any modifications are needed to poetry.lock. I would greatly appreciate it if you could pay close attention to this file during the review. **Tag maintainer** - General / Misc / if you don't know who to tag: @baskaryan If this PR needs any additional changes, I'll be happy to make them! --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-07-12 02:05:14 +00:00
"# os.environ[\"OPENAI_API_KEY\"] = \"YOUR_API_KEY\"\n",
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
"\n",
"# Create callback handler. This logs latency, errors, token usage, prompts as well as prompt responses to Infino.\n",
Fix `make docs_build` and related scripts (#7276) **Description: a description of the change** Fixed `make docs_build` and related scripts which caused errors. There are several changes. First, I made the build of the documentation and the API Reference into two separate commands. This is because it takes less time to build. The commands for documents are `make docs_build`, `make docs_clean`, and `make docs_linkcheck`. The commands for API Reference are `make api_docs_build`, `api_docs_clean`, and `api_docs_linkcheck`. It looked like `docs/.local_build.sh` could be used to build the documentation, so I used that. Since `.local_build.sh` was also building API Rerefence internally, I removed that process. `.local_build.sh` also added some Bash options to stop in error or so. Futher more added `cd "${SCRIPT_DIR}"` at the beginning so that the script will work no matter which directory it is executed in. `docs/api_reference/api_reference.rst` is removed, because which is generated by `docs/api_reference/create_api_rst.py`, and added it to .gitignore. Finally, the description of CONTRIBUTING.md was modified. **Issue: the issue # it fixes (if applicable)** https://github.com/hwchase17/langchain/issues/6413 **Dependencies: any dependencies required for this change** `nbdoc` was missing in group docs so it was added. I installed it with the `poetry add --group docs nbdoc` command. I am concerned if any modifications are needed to poetry.lock. I would greatly appreciate it if you could pay close attention to this file during the review. **Tag maintainer** - General / Misc / if you don't know who to tag: @baskaryan If this PR needs any additional changes, I'll be happy to make them! --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-07-12 02:05:14 +00:00
"handler = InfinoCallbackHandler(\n",
" model_id=\"test_openai\", model_version=\"0.1\", verbose=False\n",
")\n",
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
"\n",
"# Create LLM.\n",
"llm = OpenAI(temperature=0.1)\n",
"\n",
"# Number of questions to ask the OpenAI model. We limit to a short number here to save $$ while running this demo.\n",
"num_questions = 10\n",
"\n",
"questions = questions[0:num_questions]\n",
"for question in questions:\n",
" print(question)\n",
"\n",
" # We send the question to OpenAI API, with Infino callback.\n",
" llm_result = llm.generate([question], callbacks=[handler])\n",
" print(llm_result)"
]
},
{
"cell_type": "markdown",
"id": "b68ec697-c922-4fd9-aad1-f49c6ac24e8a",
"metadata": {},
"source": [
"## Create Metric Charts\n",
"\n",
"We now use matplotlib to create graphs of latency, errors and tokens consumed."
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "f078c612-89e0-4a1d-b1a8-bf36b664a10e",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAgsAAAGiCAYAAABppIV1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABnYUlEQVR4nO3deXhTVfoH8O9Nuq/QvaUtpaVlXwpl37cCoiKIDoviCoqIC8qIM/MTdMZBdFBREBAUFFFRQUUUyr4Isu97W1poS+lK9yZNk/P7I02gLKUlaW+W7+d5eMamN8mbd26TN+ee8x5JCCFAREREdAcKuQMgIiIiy8ZigYiIiGrEYoGIiIhqxGKBiIiIasRigYiIiGrEYoGIiIhqxGKBiIiIasRigYiIiGrEYoGIiIhqxGKBiIiIasRigcjGrVixApIk4dChQyY9TllZGWbPno0dO3aYJzAishosFoioVsrKyvD222+zWCCyQywWiIiIqEYsFojsXEVFBd566y107twZ3t7ecHd3R58+fbB9+3bjMampqfD39wcAvP3225AkCZIkYfbs2cZjzp07hzFjxsDHxwcuLi6Ii4vDunXrqj2X4ZLInj17MH36dPj7+8Pd3R2jRo1CTk7OLbFt2LAB/fr1g6enJ7y8vNClSxd8++23AIBZs2bB0dHxtvebPHkyGjVqBJVKZY4UEdk9FgtEdq6oqAjLli1D//79MXfuXMyePRs5OTkYOnQojh07BgDw9/fHokWLAACjRo3CypUrsXLlSowePRoAcPr0aXTv3h1nz57FzJkzMW/ePLi7u+Ohhx7Czz//fMtzTps2DcePH8esWbMwZcoU/Pbbb3jxxRerHbNixQqMGDEC+fn5ePPNN/Hee++hY8eO2LhxIwDg8ccfR2VlJVavXl3tfhUVFfjpp5/w8MMPw8XFxdzpIrJPgohs2vLlywUAcfDgwdv+vrKyUqjV6mq3Xbt2TQQGBoqnn37aeFtOTo4AIGbNmnXLYwwaNEi0a9dOqFQq4206nU707NlTREdH3xLL4MGDhU6nM97+6quvCqVSKQoKCoQQQhQUFAhPT0/RrVs3UV5eXu25brxfjx49RLdu3ar9fu3atQKA2L59+x0yQkR1xZEFIjunVCrh5OQEANDpdMjPz0dlZSXi4uJw5MiRu94/Pz8f27Ztw6OPPori4mLk5uYiNzcXeXl5GDp0KBITE5GRkVHtPpMnT4YkScaf+/TpA61Wi0uXLgEANm/ejOLiYsycOfOW0YEb7zdx4kTs378fycnJxttWrVqFsLAw9OvXr+7JIKLbYrFARPjqq6/Qvn17uLi4wNfXF/7+/vj9999RWFh41/smJSVBCIH/+7//g7+/f7V/s2bNAgBkZ2dXu094eHi1nxs3bgwAuHbtGgAYP/zbtm1b43P/7W9/g7OzM1atWgUAKCwsxPr16zFhwoRqRQURmcZB7gCISF7ffPMNnnzySTz00EOYMWMGAgICoFQqMWfOnGrf2O9Ep9MBAF5//XUMHTr0tsc0b9682s9KpfK2xwkh6hR748aNcf/992PVqlV466238NNPP0GtVuOxxx6r0+MQUc1YLBDZuZ9++gmRkZFYu3ZttW/jhlEBgzt9U4+MjAQAODo6YvDgwWaJKSoqCgBw6tSpWwqNm02cOBEjR47EwYMHsWrVKsTGxqJNmzZmiYOI9HgZgsjOGb7l3/itfv/+/fjrr7+qHefm5gYAKCgoqHZ7QEAA+vfvjyVLliAzM/OWx7/d0sa7iY+Ph6enJ+bMmXPL8sebRx+GDx8OPz8/zJ07Fzt37uSoAlE94MgCkZ348ssvjcsOb9S/f3+sXbsWo0aNwogRI5CSkoLFixejdevWKCkpMR7n6uqK1q1bY/Xq1YiJiYGPjw/atm2Ltm3bYuHChejduzfatWuHSZMmITIyEllZWfjrr7+Qnp6O48eP1ylWLy8vfPTRR3j22WfRpUsXjB8/Ho0bN8bx48dRVlaGr776yniso6Mjxo4diwULFkCpVGLcuHH3niQiui0WC0R2wtAn4WaXL19GSUkJlixZgoSEBLRu3RrffPMNfvzxx1taOy9btgzTpk3Dq6++ioqKCsyaNQtt27ZF69atcejQIbz99ttYsWIF8vLyEBAQgNjYWLz11lv3FO8zzzyDgIAAvPfee/j3v/8NR0dHtGzZEq+++uotx06cOBELFizAoEGDEBwcfE/PR0R3Jom6zigiIrIwx48fR8eOHfH111/j8ccflzscIpvDOQtEZPWWLl0KDw8PY0dJIjIvXoYgIqv122+/4cyZM/j888/x4osvwt3dXe6QiGwSL0MQkdWKiIhAVlYWhg4dipUrV8LT01PukIhsEosFIiIiqhHnLBAREVGNWCwQERFRjax6gqNOp8OVK1fg6enJTWOIiIjqQAiB4uJihISEQKGoeezAqouFK1euICwsTO4wiIiIrFZaWhpCQ0NrPMaqiwXDzOe0tDR4eXnJHA0REZH1KCoqQlhYWK1WEVl1sWC49ODl5cVigYiI6B7U5jI+JzgSERFRjVgsEBERUY1YLBAREVGNWCwQERFRjVgsEBERUY1YLBAREVGNWCwQERFRjVgsEBERUY1YLBAREdWgRF2Jb/ZdQmG5Ru5QZGPVHRyJiIjq2xtrTuD3E5nIKlLhtfgWcocjC44sEBER3cGpjEL8fiITAHDmSpHM0ciHxQIREdEdzNt03vjfyTklMkYiLxYLREREt3EwNR/bz+fAsM/S5fwyqCu18gYlExYLRERENxFC4P2N5wAA47qGw9PZAToBXMorkzkyebBYICIiusnOCzk4mHoNzg4KvDQwGpEBHgCA5Gz7vBTBYoGIiOgGOp3ABwn6uQpP9IxAkLcLovzcAdjvvAUWC0RERDfYcOoqTl8pgoezA57vFwUAiKoaWbiYUypnaLJhsUBERFSlUqvDvM36UYVJfSLh4+4EAIjy58gCERERAVh7JAMXc0rh4+6EZ/o0M94e5V81ZyGnFEIIucKTDYsFIiIiAOpKLT7ecgEA8EL/KHg4X29yHO7rBqVCQom6EtnFarlClA2LBSIiIgDf7r+MK4UqBHu74LHuTav9ztlBiXAfNwD2uSKCxQIREdm9UnUlFmxLAgC8NCgaLo7KW46JtOMVESwWiIjI7i3fk4K80gpE+LphTOfQ2x5jWBGRbIcrIlgsEBGRXSsoq8CSXRcBANPjW8BRefuPRnteEcFigYiI7NrinRdRrKpEyyBP3N8u+I7HGVZE2GOvBRYLRERkt7KLVFixNwUAMGNoCygU0h2PNRQLGQXlKKuobJD4LAWLBSIislsLtidBpdGhc9PGGNgyoMZjG7s7GZs02dvoAosFIiKyS2n5ZfjuwGUA+lEFSbrzqIKBvc5bYLFARER26aMtF6DRCvSJ9kP3SN9a3SfSzz5XRLBYICIiu3Mhqxg/H80AAPx9aMta3y8qQD+ycJEjC0RERLZt3qbzEAIY3jYI7UK9a32/G/eIsCcsFoiIyK4cTytAwuksKCTgtfiYOt33+vLJEuh09rOhFIsFIiKyKx8k6LegHt0pFM0DPOt039DGrnBSKqCu1CGjoLw+wrNIshYLs2fPhiRJ1f61bFn7a0dERER1sTcpF38m5cJRKeHlQdF1vr+DUoEIv6oNpexo3oLD3Q+pX23atMGWLVuMPzs4yB4SERHZICEEPtikH1UY3zUcYVW7SNZVpJ8HLmSVIDmnFP1bmDNCyyX7J7ODgwOCgoLkDoOIiGzclrPZOHq5AK6OSkwd2PyeHycqwB04bV8jC7LPWUhMTERISAgiIyMxYcIEXL58+Y7HqtVqFBUVVftHRER0NzqdwP+q5io81SsCAZ4u9/xYN05ytBeyFgvdunXDihUrsHHjRixatAgpKSno06cPiouLb3v8nDlz4O3tbfwXFhbWwBETEZE1+u3EFZzPKoaXiwOe6xtl0mPZ4/JJSQhhMWs/CgoK0LRpU3z44Yd45plnbvm9Wq2GWq02/lxUVISwsDAUFhbCy8urIUMlIiIrodHqMPjDnbiUV4YZQ1tg6oB7vwQBAMUqDdrN3gQ
"text/plain": [
"<Figure size 600x400 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAiwAAAGiCAYAAADEJZ3cAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA9oUlEQVR4nO3dd3gU9f728XtTKSEBEkgMBIOIAkqPQAAFpQQEFQQLYuMgWLGA/gT1AGJBPBZQUSwHsKEcRRE5CCKgcATpRVoEC9WEJgkBUkg+zx88WV0JkLAbdpK8X9eVS3d2ZvOd+5pl78xOcZmZCQAAwMEC/D0AAACA06GwAAAAx6OwAAAAx6OwAAAAx6OwAAAAx6OwAAAAx6OwAAAAx6OwAAAAx6OwAAAAx6OwAAAAx6OwAHCEyZMny+VynfTnhx9+8PcQAfhRkL8HAAB/NWrUKNWuXfuE6eeff74fRgPAKSgsAByla9euSkhIKPT8x44dU15enkJCQk547vDhw6pYseIZj8XMlJmZqfLly5/xawDwDb4SAlBi/Pbbb3K5XHrhhRc0duxY1alTR6Ghodq4caNGjhwpl8uljRs36qabblKVKlXUtm1bScdLzVNPPeWePz4+Xo899piysrI8Xj8+Pl7du3fXnDlzlJCQoPLly+vNN9+UJM2dO1dt27ZV5cqVFRYWpgsvvFCPPfbYWc8AKKvYwwLAUdLS0rRv3z6PaS6XS5GRke7HkyZNUmZmpgYOHKjQ0FBVrVrV/dx1112nunXr6tlnn5WZSZLuuOMOvfvuu+rdu7eGDBmipUuXavTo0dq0aZM+//xzj9+VnJysPn366M4779SAAQN04YUXasOGDerevbsaNWqkUaNGKTQ0VFu3btX3339fjEkA+CsKCwBH6dix4wnTQkNDlZmZ6X68c+dObd26VdWqVTth3saNG2vKlCnux2vXrtW7776rO+64Q2+//bYk6Z577lH16tX1wgsvaMGCBbr88svd82/dulWzZ89WUlKSe9rYsWOVnZ2tr776SlFRUT5ZTwBFQ2EB4Cjjx4/XBRdc4DEtMDDQ43GvXr0KLCuSdNddd3k8njVrliRp8ODBHtOHDBmiF154Qf/97389Ckvt2rU9yookVa5cWZL0xRdfqF+/fgoI4Nt04GyjsABwlBYtWpz2oNuCziI62XPbtm1TQEDACWcZxcTEqHLlytq2bdtpX/uGG27QO++8ozvuuENDhw5Vhw4ddO2116p3796UF+As4Z0GoMQ51Vk7J3vO5XKd8WuXL19eCxcu1DfffKNbbrlF69at0w033KBOnTopNze3cIMG4BUKC4BS7dxzz1VeXp62bNniMT01NVUHDx7UueeeW6jXCQgIUIcOHfTSSy9p48aNeuaZZzR//nwtWLCgOIYN4G8oLABKtSuvvFLS8QNn/+qll16SJHXr1u20r3HgwIETpjVp0kSSTjg1GkDx4BgWAI7y1VdfafPmzSdMb9269RkdL9K4cWPddttteuutt3Tw4EG1a9dOy5Yt07vvvqsePXp4HHB7MqNGjdLChQvVrVs3nXvuudqzZ49ef/111axZ032tFwDFi8ICwFGGDx9e4PRJkyapffv2Z/Sa77zzjs477zxNnjxZn3/+uWJiYjRs2DCNGDGiUMtfffXV+u233zRx4kTt27dPUVFRateunZ588klFRESc0ZgAFI3L8q+sBAAA4FAcwwIAAByPwgIAAByPwgIAAByPwgIAAByPwgIAAByPwgIAAByP67D4QF5ennbv3q1KlSoV+n4lAABAMjMdOnRIsbGxp7w4JIXFB3bv3q24uDh/DwMAgBJrx44dqlmz5kmfp7D4QKVKlSQdDzs8PNzPowEAoORIT09XXFyc+7P0ZCgsPpD/NVB4eDiFBQCAM3C6Qyo46BYAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADgehQUAADheiSss48ePV3x8vMqVK6eWLVtq2bJlp5z/k08+Ub169VSuXDk1bNhQs2bNOum8d911l1wul8aOHevjUQMAAG+UqMIydepUDR48WCNGjNCqVavUuHFjJSUlac+ePQXOv3jxYvXp00f9+/fX6tWr1aNHD/Xo0UPr168/Yd7PP/9cP/zwg2JjY4t7NQAAQBGVqMLy0ksvacCAAerXr58aNGigCRMmqEKFCpo4cWKB848bN05dunTRI488ovr16+upp55Ss2bN9Nprr3nMt2vXLg0aNEgffvihgoODz8aqAACAIigxhSU7O1srV65Ux44d3dMCAgLUsWNHLVmypMBllixZ4jG/JCUlJXnMn5eXp1tuuUWPPPKILrrookKNJSsrS+np6R4/AACg+JSYwrJv3z7l5uYqOjraY3p0dLRSUlIKXCYlJeW0848ZM0ZBQUG6//77Cz2W0aNHKyIiwv0TFxdXhDUBAABFVWIKS3FYuXKlxo0bp8mTJ8vlchV6uWHDhiktLc39s2PHjmIcJQAAKDGFJSoqSoGBgUpNTfWYnpqaqpiYmAKXiYmJOeX8ixYt0p49e1SrVi0FBQUpKChI27Zt05AhQxQfH3/SsYSGhio8PNzjBwAAFJ8SU1hCQkLUvHlzzZs3zz0tLy9P8+bNU2JiYoHLJCYmeswvSXPnznXPf8stt2jdunVas2aN+yc2NlaPPPKI5syZU3wrAwAAiiTI3wMoisGDB+u2225TQkKCWrRoobFjx+rw4cPq16+fJOnWW29VjRo1NHr0aEnSAw88oHbt2unFF19Ut27d9PHHH2vFihV66623JEmRkZGKjIz0+B3BwcGKiYnRhRdeeHZXDgAAnFSJKiw33HCD9u7dq+HDhyslJUVNmjTR7Nmz3QfWbt++XQEBf+40at26taZMmaInnnhCjz32mOrWravp06fr4osv9tcqAACAM+AyM/P3IEq69PR0RUREKC0tjeNZAAAogsJ+hpaYY1gAAEDZRWEBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOR2EBAACOV+IKy/jx4xUfH69y5cqpZcuWWrZs2Snn/+STT1SvXj2VK1dODRs21KxZs9zP5eTk6NFHH1XDhg1VsWJFxcbG6tZbb9Xu3buLezUAAEARlKjCMnXqVA0ePFgjRozQqlWr1LhxYyUlJWnPnj0Fzr948WL16dNH/fv31+rVq9WjRw/16NFD69evlyQdOXJEq1at0j//+U+tWrVKn332mZKTk3X11VefzdUCAACn4TIz8/cgCqtly5a65JJL9Nprr0mS8vLyFBcXp0GDBmno0KEnzH/DDTfo8OHDmjlzpntaq1at1KRJE02YMKHA37F8+XK1aNFC27ZtU61atQo1rvT0dEVERCgtLU3h4eFnsGYAAJRNhf0MLTF7WLKzs7Vy5Up17NjRPS0gIEAdO3bUkiVLClxmyZIlHvNLUlJS0knnl6S0tDS5XC5VrlzZJ+MGAADeC/L3AApr3759ys3NVXR0tMf06Ohobd68ucBlUlJSCpw/JSWlwPkzMzP16KOPqk+fPqdseVlZWcrKynI/Tk9PL+xqAACAM1Bi9rAUt5ycHF1//fUyM73xxhunnHf06NGKiIhw/8TFxZ2lUQIAUDaVmMISFRWlwMBApaamekxPTU1VTExMgcv
"text/plain": [
"<Figure size 600x400 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAhQAAAGiCAYAAAC/AV8QAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABo4UlEQVR4nO3dd1iT5/oH8G9C2HsrgqIo4kQU98I60arVqkdrW1cdra1a257qr8vaYe3pcFZt6+6y51itte6qddaNe4GoKCqyd4Dk+f2BCeAEXpI34/u5Lq5zEkJ4cjfgzfPez30rhBACRERERBIo5V4AERERmT8mFERERCQZEwoiIiKSjAkFERERScaEgoiIiCRjQkFERESSMaEgIiIiyZhQEBERkWRMKIiIiEgyJhREZNEUCgVeffVVuZdBZPGYUBCZoBUrVkChUOg/HBwcEBoaildffRV37tyRe3mVkpiYiBkzZiAmJuaJjy392h/3sXv3boOvm4jKRyX3Aojo0WbOnInatWsjPz8f+/btw6JFi7Bp0yacOXMGTk5Oci+vQhITE/Hhhx8iODgYzZo1e+xjV69eXeb2qlWrsH379gfub9CgQVUvk4gqiQkFkQmLjo5GZGQkAOCll16Ct7c3vvrqK/z+++8YNmzYQ78mJycHzs7OxlxmlXv++efL3P7nn3+wffv2B+4nItPBSx5EZuSpp54CAMTHxwMARo4cCRcXF8TFxaF3795wdXXF8OHDARQnFm+88QaCgoJgb2+P+vXr44svvsD9A4Z1NQb//e9/0bBhQzg6OqJt27Y4ffo0AGDJkiWoW7cuHBwcEBUVhatXr5b5+qioKDRu3BjHjh1Du3bt4OjoiNq1a2Px4sX6x+zevRstW7YEAIwaNUp/yWLFihWVjkV5X9/DfPzxx1AqlZg/f77+vs2bN6Njx45wdnaGq6sr+vTpg7Nnz5b5Ol28b968iWeeeQYuLi7w9fXFm2++CY1GU+axv/zyC1q0aAFXV1e4ubmhSZMmmDt3bqVfL5GpY0JBZEbi4uIAAN7e3vr7ioqK0LNnT/j5+eGLL77As88+CyEE+vXrh6+//hq9evXCV199hfr16+Ott97C1KlTH3jevXv34o033sCIESMwY8YMnD9/Hk8//TQWLlyIefPm4ZVXXsFbb72FgwcPYvTo0Q98fVpaGnr37o0WLVrg888/R2BgIF5++WUsW7YMQPGliZkzZwIAxo0bh9WrV2P16tXo1KlTpeJQ0ddX2rvvvov3338fS5YswWuvvQag+BJLnz594OLigtmzZ+O9997DuXPn0KFDhwcSKI1Gg549e8Lb2xtffPEFOnfujC+//BLffvut/jHbt2/HsGHD4OnpidmzZ+Ozzz5DVFQU9u/fX6nXS2QWBBGZnOXLlwsAYseOHeLu3bsiISFB/PLLL8Lb21s4OjqKGzduCCGEGDFihAAgpk2bVubr169fLwCIjz/+uMz9gwYNEgqFQsTGxurvAyDs7e1FfHy8/r4lS5YIAKJatWoiMzNTf//06dMFgDKP7dy5swAgvvzyS/19arVaNGvWTPj5+YmCggIhhBBHjhwRAMTy5csrHI+JEyeK0r+uKvr6Jk6cKIQQ4o033hBKpVKsWLFC//msrCzh4eEhxo4dW+a5bt++Ldzd3cvcr4v3zJkzyzw2IiJCtGjRQn978uTJws3NTRQVFVX4tRKZK+5QEJmwbt26wdfXF0FBQRg6dChcXFywbt061KhRo8zjXn755TK3N23aBBsbG0yaNKnM/W+88QaEENi8eXOZ+7t27Yrg4GD97datWwMAnn32Wbi6uj5w/5UrV8p8vUqlwvjx4/W37ezsMH78eCQlJeHYsWMVfNVPVtHXJ4TAq6++irlz5+KHH37AiBEj9J/bvn070tPTMWzYMCQnJ+s/bGxs0Lp1a+zateuB7z9hwoQytzt27FgmJh4eHsjJycH27dur4uUSmQUWZRKZsIULFyI0NBQqlQr+/v6oX78+lMqyfweoVCoEBgaWue/atWsICAgokwwAJacirl27Vub+mjVrlrnt7u4OAAgKCnro/WlpaWXuDwgIeKAQNDQ0FABw9epVtGnT5vEvtIIq+vpWrVqF7OxsLFq06IFi1suXLwMoqU+5n5ubW5nbDg4O8PX1LXOfp6dnmZi88sor+PXXXxEdHY0aNWqgR48eGDJkCHr16lWBV0lkXphQEJmwVq1a6U95PIq9vf0DSUZF2djYVOh+UY7CR1PSvn17xMTEYMGCBRgyZAi8vLz0n9NqtQCK6yiqVav2wNeqVGV/TT4qJqX5+fkhJiYGW7duxebNm7F582YsX74cL774IlauXCnx1RCZJl7yILJAtWrVQmJiIrKyssrcf+HCBf3nq1JiYiJycnLK3Hfp0iUA0F9KUSgUVfb9Kvr66tati23btiExMRG9evUq83UhISEAipOAbt26PfARFRVVqTXa2dmhb9+++OabbxAXF4fx48dj1apViI2NrdTzEZk6JhREFqh3797QaDRYsGBBmfu//vprKBQKREdHV+n3KyoqwpIlS/S3CwoKsGTJEvj6+qJFixYAoL8kkp6eLvn7Veb1NW3aFJs2bcL58+fRt29f5OXlAQB69uwJNzc3fPrppygsLHzg6+7evVvh9aWkpJS5rVQq0bRpUwCAWq2u8PMRmQNe8iCyQH379kWXLl3wzjvv4OrVqwgPD8e2bdvw+++/Y8qUKfq/yqtKQEAAZs+ejatXryI0NBRr1qxBTEwMvv32W9ja2gIo3gnw8PDA4sWL4erqCmdnZ7Ru3Rq1a9c22utr06YNfv/9d/Tu3RuDBg3C+vXr4ebmhkWLFuGFF15A8+bNMXToUPj6+uL69ev4888/0b59+wcSlyd56aWXkJqaiqeeegqBgYG4du0a5s+fj2bNmrG7J1ks7lAQWSClUokNGzZgypQp2LhxI6ZMmYJz587hP//5D7766qsq/36enp7YtGkTjh49irfeegsJCQlYsGABxo4dq3+Mra0tVq5cCRsbG0yYMAHDhg3D33//XanvJ+X1PfXUU/j111+xbds2vPDCC9BqtXjuuefw119/oUaNGvjPf/6DyZMn45dffkGzZs0watSoCq/v+eefh4ODA7755hu88sorWLlyJf71r39h8+bNkutdiEyVQphbdRURmZSoqCgkJyfjzJkzci+FiGTEVJmIiIgkY0JBREREkjGhICIiIslYQ0FERESScYeCiIiIJGNCQURERJJZfGMrrVaLxMREuLq6VmnrXyIiIksnhEBWVhYCAgKe2EPF4hOKxMTEByYmEhERUfklJCQ8MNX4fhafUOjGGyckJDwwhpiIiIgeLTMzE0FBQfp/Sx/H4hMK3WUONzc3JhRERESVUJ6SARZlEhERkWRMKIiIiEgyJhREREQkGRMKIiIikowJBREREUkma0KxZ88e9O3bFwEBAVAoFFi/fn2Zz8+YMQNhYWFwdnaGp6cnunXrhkOHDsmzWCIiInokWROKnJwchIeHY+HChQ/9fGhoKBYsWIDTp09j3759CA4ORo8ePXD37l0jr5SIiIgex2SmjSoUCqxbtw7PPPPMIx+TmZkJd3d37NixA127di3X8+q+JiMjg30oiIiIKqAi/4aaTQ1FQUEBvv32W7i7uyM8PFzu5RCRBdBqBbacuYXt5+7IvRQis2fynTI3btyIoUOHIjc3F9WrV8f27dvh4+PzyMer1Wqo1Wr97czMTGMsk4jMzN7LdzFr0wWcu5UJpQI48V4PuDvZyr0sIrNl8jsUXbp0QUxMDA4cOIBevXphyJAhSEpKeuTjZ82aBXd3d/0HB4MRUWlnbmbghaWH8MLSwzh3q/gPDq0A7mbny7wyIvNm8gmFs7Mz6tatizZt2mDp0qVQqVRYunTpIx8/ffp0ZGRk6D8SEhKMuFoiMlUJqbmY/MsJPD1/H/ZeToatjQKj29dGgLsDACA1p1DmFRKZN5O/5HE/rVZb5pLG/ezt7WFvb2/EFRGRKUvLKcCCXbFYffAaCjRaAED/ZgF4s0d9BHk54URCGhIz8pGWWyDzSonMm6wJRXZ
"text/plain": [
"<Figure size 600x400 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAhMAAAGiCAYAAABd3URpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB0zElEQVR4nO3dd3iTZfcH8G+S7k0L3RMotECBQqEUEJCNbKrIUKagvogC6vsTF8MB6KsIirhYgggiG1myRXbLHgUKpXvQPdM0Ob8/0jwQoNCStE/G+VxXL+mTJ8nJMW1O7+e+zy0hIgJjjDHG2FOSih0AY4wxxowbFxOMMcYY0wkXE4wxxhjTCRcTjDHGGNMJFxOMMcYY0wkXE4wxxhjTCRcTjDHGGNMJFxOMMcYY0wkXE4wxxhjTCRcTjDEtEokEs2fP1utjjhs3DoGBgXp9zLrUrVs3tGjRQuwwGDNYXEwwVgvi4+Px6quvomHDhrCxsYGTkxM6deqERYsWobS0VOzwakVqaipmz56Nc+fOiR0KAHUBIJFInvil78KJMXNkIXYAjJmav/76Cy+88AKsra0xZswYtGjRAuXl5Th69CjeffddXL58GT/99JPYYepdamoq5syZg8DAQLRu3Vrrtp9//hkqlapO4/nggw/wyiuvCN+fPn0aixcvxvvvv4/Q0FDheMuWLes0LsZMERcTjOnR7du3MWLECAQEBODAgQPw8vISbpsyZQpu3ryJv/76S8QIxWFpaVnnz9mrVy+t721sbLB48WL06tUL3bp1q/N4GDNlfJmDMT364osvUFRUhGXLlmkVEhqNGzfGW2+9JXxfUVGBTz75BI0aNYK1tTUCAwPx/vvvQy6Xa90vMDAQAwYMwKFDhxAREQFbW1uEhYXh0KFDAIBNmzYhLCwMNjY2aNu2Lc6ePat1/3HjxsHBwQG3bt1Cnz59YG9vD29vb8ydOxfV2Tg4JSUFEyZMgIeHB6ytrdG8eXMsX75cuP3QoUNo164dAGD8+PHCJYSVK1cKz//gnIni4mK8/fbb8PPzg7W1NZo2bYr//e9/D8UjkUjwxhtvYMuWLWjRooXw/Lt3735i3NXx/fffo3nz5rC2toa3tzemTJmCvLy8J95v7969sLOzw8iRI1FRUQEAuHbtGp5//nm4urrCxsYGERER2LZtm9b9Vq5cCYlEgn///RczZsxAgwYNYG9vj6FDhyIrK0vr3DNnzqBPnz6oX78+bG1tERQUhAkTJujldTOmV8QY0xsfHx9q2LBhtc8fO3YsAaDnn3+elixZQmPGjCEANGTIEK3zAgICqGnTpuTl5UWzZ8+mhQsXko+PDzk4ONCaNWvI39+f5s+fT/PnzydnZ2dq3LgxKZVKreexsbGh4OBgevnll+m7776jAQMGEAD66KOPtJ4LAM2aNUv4Pj09nXx9fcnPz4/mzp1LS5cupUGDBhEAWrhwoXDO3LlzCQBNnjyZVq9eTatXr6b4+Hjh+QMCAoTHVKlU1L17d5JIJPTKK6/Qd999RwMHDiQANG3atIfiadWqFXl5edEnn3xC33zzDTVs2JDs7Ozo7t271c71hg0bCAAdPHhQODZr1iwCQD179qRvv/2W3njjDZLJZNSuXTsqLy8XzuvatSs1b95c+H779u1kbW1NY8aMoYqKCiIiunTpEjk7O1OzZs1owYIF9N1331GXLl1IIpHQpk2bhPuuWLGCAFB4eDh1796dvv32W3r77bdJJpPR8OHDhfMyMjKoXr161KRJE/ryyy/p559/pg8++IBCQ0Or/ZoZqytcTDCmJ/n5+QSABg8eXK3zz507RwDolVde0Tr+zjvvEAA6cOCAcCwgIIAA0LFjx4Rje/bsIQBka2tLd+7cEY7/+OOPD31oaoqWqVOnCsdUKhX179+frKysKCsrSzj+YDExceJE8vLyeuiDe8SIEeTs7EwlJSVERHT69GkCQCtWrHjotT5YTGzZsoUA0Keffqp13vPPP08SiYRu3rypFY+VlZXWsfPnzxMA+vbbbx96rqo8WExkZmaSlZUV9e7dW6vw+u677wgALV++XDh2fzGxceNGsrS0pEmTJmndr0ePHhQWFkZlZWXCMZVKRR07dqTg4GDhmKaY6NmzJ6lUKuH49OnTSSaTUV5eHhERbd68mQDQ6dOnq/0aGRMLX+ZgTE8KCgoAAI6OjtU6f+fOnQCAGTNmaB1/++23AeChuRXNmjVDVFSU8H1kZCQAoHv37vD393/o+K1btx56zjfeeEP4t+byQXl5Ofbt2/fIGIkIGzduxMCBA0FEuHv3rvDVp08f5OfnIzY2tlqv9347d+6ETCbDm2++qXX87bffBhFh165dWsd79uyJRo0aCd+3bNkSTk5Oj3yN1bVv3z6Ul5dj2rRpkErv/SqcNGkSnJycHjm35ffff8eLL76IV199FT/++KNwv5ycHBw4cADDhw9HYWGhkKPs7Gz06dMHN27cQEpKitZjTZ48GRKJRPj+mWeegVKpxJ07dwAALi4uAIAdO3ZAoVA89etkrC5wMcGYnjg5OQEACgsLq3X+nTt3IJVK0bhxY63jnp6ecHFxET5UNO4vGADA2dkZAODn5/fI47m5uVrHpVIpGjZsqHWsSZMmAICEhIRHxpiVlYW8vDz89NNPaNCggdbX+PHjAQCZmZlPeqkPuXPnDry9vR8qvDSrLJ702gGgXr16D73GmsYAAE2bNtU6bmVlhYYNGz4Uw+3bt/HSSy8hOjoa3377rVYhcPPmTRARPvroo4fyNGvWLAAP5+nB11SvXj0A9/6/de3aFdHR0ZgzZw7q16+PwYMHY8WKFQ/Np2HMEPBqDsb0xMnJCd7e3rh06VKN7nf/h9LjyGSyGh2nakysfBLNcs6XXnoJY8eOfeQ5dbG0sjZfY3V5eXnBy8sLO3fuxJkzZxARESHcpsnTO++8gz59+jzy/g8WjU96TRKJBH/++SdOnDiB7du3Y8+ePZgwYQK++uornDhxAg4ODvp4WYzpBRcTjOnRgAED8NNPP+H48eNalyQeJSAgACqVCjdu3NDqe5CRkYG8vDwEBAToNTaVSoVbt24JoxEAcP36dQCosjtlgwYN4OjoCKVSiZ49ez728atbFAHq175v3z4UFhZqjU5cu3ZNuL22aZ4jLi5Oa8SmvLwct2/ffuj12tjYYMeOHejevTv69u2Lw4cPo3nz5gAg3N/S0vKJeaqpDh06oEOHDvjss8+wdu1ajB49GuvWrdPqocGY2PgyB2N69N///hf29vZ45ZVXkJGR8dDt8fHxWLRoEQDgueeeAwB88803Wud8/fXXAID+/fvrPb7vvvtO+DcR4bvvvoOlpSV69OjxyPNlMhmio6OxcePGR4643L+U0d7eHgCqtazyueeeg1Kp1IoHABYuXAiJRIJ+/fpV5+XopGfPnrCyssLixYu1RjiWLVuG/Pz8R+bf2dkZe/bsgbu7O3r16oX4+HgAgLu7O7p164Yff/wRaWlpD93vwSWf1ZGbm/vQyIumGRhf6mCGhkcmGNOjRo0aYe3atXjxxRcRGhqq1QHz2LFj2LBhA8aNGwcAaNWqFcaOHYuffvoJeXl56Nq1K06dOoVVq1ZhyJAhePbZZ/Uam42NDXbv3o2xY8ciMjISu3btwl9//YX3338fDRo0qPJ+8+fPx8GDBxEZGYlJkyahWbNmyMnJQWxsLPbt24ecnBzhtbu4uOCHH36Ao6Mj7O3tERkZiaCgoIcec+DAgXj22WfxwQcfICEhAa1atcLevXuxdetWTJs2TWuyZW1p0KABZs6ciTlz5qBv374YNGgQ4uLi8P3336Ndu3Z46aWXHnm/+vXr4++//0bnzp3Rs2dPHD16FD4+PliyZAk6d+6MsLAwTJo0CQ0bNkRGRgaOHz+O5ORknD9/vkbxrVq1Ct9//z2GDh2KRo0aobCwED///DOcnJyEQpQxgyHWMhLGTNn169dp0qRJFBgYSFZWVuTo6EidOnWib7/9VmvpoEKhoDlz5lBQUBBZWlqSn58fzZw5U+scIvXS0P79+z/0PABoypQpWsdu375NAOjLL78Ujo0dO5b
"text/plain": [
"<Figure size 600x400 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAhMAAAGiCAYAAABd3URpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABvb0lEQVR4nO3dd3iTVfsH8G9GFx0p3XRRdoEWKLsUENlDZMrLUFmCCqKA45WfKIoK4vsqIiL6IpShOJAhIkNAQGRvKKNsWjqhpbtJ0+T8/igJVEBakvTJ+H6uq5f2SfLkzk3a3D3POfeRCSEEiIiIiB6RXOoAiIiIyLaxmCAiIiKTsJggIiIik7CYICIiIpOwmCAiIiKTsJggIiIik7CYICIiIpOwmCAiIiKTsJggIiIik7CYICKT7dy5EzKZDDt37qyS55PJZHjppZeq5LmI6OFYTBDZKJlMVqGvinzAz5o1C+vWrbOZeInIuiilDoCIHs2KFSvKfb98+XJs3br1nuMNGzZ86LlmzZqFwYMHo3///uYMsRxzxktE1oXFBJGNevrpp8t9v3//fmzduvWe49bC1uIloorjZQ4iO1ZYWIhXX30VYWFhcHFxQYMGDfDf//4Xd28WLJPJUFhYiGXLlhkvNYwaNQoAcO3aNUyYMAENGjSAm5sbfH198dRTT+Hq1auSxfsgH3zwAeRyOebPn288tmnTJnTo0AHu7u7w9PREnz59cPr06XKPGzVqFDw8PJCSkoL+/fvDw8MD/v7+eO2116DT6crd94cffkCLFi3g6ekJLy8vREdHY968eeZ58UQ2jCMTRHZKCIEnn3wSO3bswNixY9GsWTNs2bIFr7/+OlJSUjB37lwAZZcfnnvuObRu3Rrjx48HANSpUwcAcOjQIezduxdDhw5FaGgorl69ioULF6JTp044c+YMqlWrVuXx3s/06dMxa9YsfP311xg3bpzxdY0cORI9evTAnDlzUFRUhIULF6J9+/Y4duwYIiIijI/X6XTo0aMH2rRpg//+97/Ytm0bPvnkE9SpUwcvvvgiAGDr1q0YNmwYunTpgjlz5gAAzp49iz179uCVV14xWx6IbJIgIrswceJEcfeP9Lp16wQA8cEHH5S73+DBg4VMJhMXL140HnN3dxcjR46855xFRUX3HNu3b58AIJYvX248tmPHDgFA7Nixo0riBSAmTpwohBDi1VdfFXK5XCxdutR4e35+vvD29hbjxo0rd6709HShUqnKHR85cqQAIGbOnFnuvjExMaJFixbG71955RXh5eUlSktLK/waiRwFL3MQ2amNGzdCoVDg5ZdfLnf81VdfhRACmzZteug53NzcjP+v1WqRlZWFunXrwtvbG0ePHpU0XiEEXnrpJcybNw/ffvstRo4cabxt69atyMnJwbBhw3Dz5k3jl0KhQJs2bbBjx457nv+FF14o932HDh1w+fJl4/fe3t4oLCzE1q1bzfFyiewKL3MQ2alr164hODgYnp6e5Y4bVktcu3btoecoLi7G7NmzER8fj5SUlHJzF3JzcyWNd/ny5SgoKMDChQsxbNiwcrdduHABANC5c+f7PpeXl1e5711dXeHv71/uWPXq1XHr1i3j9xMmTMBPP/2EXr16ISQkBN27d8eQIUPQs2fPSrxKIvvEYoKIHmjSpEmIj4/H5MmTERsbC5VKBZlMhqFDh0Kv10saW1xcHI4fP44vvvgCQ4YMgY+Pj/E2Q2wrVqxAUFDQPY9VKsv/6lMoFA99voCAABw/fhxbtmzBpk2bsGnTJsTHx+PZZ5/FsmXLTHw1RLaNxQSRnapZsya2bduG/Pz8cn/tnzt3zni7gUwmu+85fv75Z4wcORKffPKJ8ZharUZOTo6k8QJA3bp18fHHH6NTp07o2bMntm/fbnycYQJpQEAAunbtarYYnZ2d0bdvX/Tt2xd6vR4TJkzA119/jbfffht169Y12/MQ2RrOmSCyU71794ZOp8MXX3xR7vjcuXMhk8nQq1cv4zF3d/f7FggKheKeZZnz58+/Z8lkVcdr0KRJE2zcuBFnz55F3759UVxcDADo0aMHvLy8MGvWLGi12nsed+PGjUrHl5WVVe57uVyOJk2aAAA0Gk2lz0dkTzgyQWSn+vbti8cffxxvvfUWrl69iqZNm+L333/HL7/8gsmTJxv/egeAFi1aYNu2bfj0008RHByMWrVqoU2bNnjiiSewYsUKqFQqNGrUCPv27cO2bdvg6+srabx3a9u2LX755Rf07t0bgwcPxrp16+Dl5YWFCxfimWeeQfPmzTF06FD4+/sjKSkJv/32G+Li4u4pWh7mueeeQ3Z2Njp37ozQ0FBcu3YN8+fPR7Nmzdi1k0jStSREZDZ/X2opRNkSySlTpojg4GDh5OQk6tWrJ/7zn/8IvV5f7n7nzp0THTt2FG5ubgKAcZnorVu3xOjRo4Wfn5/w8PAQPXr0EOfOnRM1a9Yst5TUHEtDKxMv7loaavDLL78IpVIp/vWvfwmdTmeMq0ePHkKlUglXV1dRp04dMWrUKHH48GHj40aOHCnc3d3viW/GjBnl4vv5559F9+7dRUBAgHB2dhbh4eHi+eefF2lpaRV+zUT2SiZEBVrLERERET0A50wQERGRSVhMEBERkUlYTBAREZFJWEwQERGRSVhMEBERkUlYTBAREZFJ7L5plV6vR2pqKjw9PR/YMpiIiIjuJYRAfn4+goODIZc/ePzB7ouJ1NRUhIWFSR0GERGRzUpOTkZoaOgDb7f7YsKw8U9ycvI92w4TERHRg+Xl5SEsLKzc5nv3Y/fFhOHShpeXF4sJIiKiR/CwaQKcgElEREQmYTFBREREJmExQURERCZhMUFEREQmYTFBREREJmExQURERCZhMUFEREQmYTFBREREJmExQUREZKLLNwqw5uh1CCGkDkUSdt8Bk4iIyJJ0eoFR8YeQlF0EPw8XdKzvL3VIVY4jE0RERCbYeiYDSdlFAICjSbckjkYaLCaIiIhMsGTPFeP/J6TkSRiJdFhMEBERPaKElFwcvJJt/P50aq6E0UiHxQQREdEjMoxKdIkMgEwGpOWqcbNAI3FUVY/FBBER0SPIzFPj1xOpAIBJXeqhlp87AOB0quNd6mAxQURE9Ai+3X8NWp1A83BvNAvzRnSICkDZpQ9Hw2KCiIioktRaHb49kAQAGNu+NgAgKpjFBBEREVXQ+uOpyC4sQbDKFT0aBwIAGod4AQASHHASJosJIiKiShBCGCdejmwXAaWi7KO08e2RieTsYuQWaSWLTwosJoiIiCph76UsnEvPh5uTAkNbhRuPq9ycEO5TDYDjLRFlMUFERFQJS/4qG5V4qmUoVNWcyt0W5aCXOlhMEBERVdCVm4XYfi4TADCqXcQ9tzc2TsJ0rOWhLCaIiIgqaOntuRKdIwNQ29/jntsddXkoiwkiIqIKyC3WYtWR6wCAMXG17nufxsFllzku3yxEvtpxJmGymCAiIqqAHw8loahEhwaBnoir63vf+/h6uCBY5QoAOJuWX5XhSYrFBBER0UOU6vRYtvcaAGBM+wjIZLIH3rexA17qkLSY0Ol0ePvtt1GrVi24ubmhTp06eP/99yGEMN5HCIF33nkHNWrUgJubG7p27YoLFy5IGDURETma389kICWnGD7uzujXLOQf72vshOlAKzokLSbmzJmDhQsX4osvvsDZs2cxZ84cfPzxx5g/f77xPh9//DE+//xzfPXVVzhw4ADc3d3Ro0cPqNVqCSMnIiJHYlgOOqJNOFydFP94X8Py0NMOtKJDKeWT7927F/369UOfPn0AABEREfj+++9x8OBBAGWjEp999hmmT5+Ofv36AQCWL1+OwMBArFu3DkOHDpUsdiIicgwnknNw+NotOClkeKZtzYfeP+r2ZY4LmfkoLtHBzfmfiw97IOnIRLt27bB9+3acP38eAHDixAn89ddf6NWrFwDgypUrSE9PR9euXY2PUalUaNOmDfbt2ydJzERE5FgMrbP7NglGgJfrQ+8f6OUKf08X6AVwNt0xRickHZl48803kZeXh8jISCgUCuh0Onz44YcYMWIEACA9PR0
"text/plain": [
"<Figure size 600x400 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Helper function to create a graph using matplotlib.\n",
"def plot(data, title):\n",
Fix `make docs_build` and related scripts (#7276) **Description: a description of the change** Fixed `make docs_build` and related scripts which caused errors. There are several changes. First, I made the build of the documentation and the API Reference into two separate commands. This is because it takes less time to build. The commands for documents are `make docs_build`, `make docs_clean`, and `make docs_linkcheck`. The commands for API Reference are `make api_docs_build`, `api_docs_clean`, and `api_docs_linkcheck`. It looked like `docs/.local_build.sh` could be used to build the documentation, so I used that. Since `.local_build.sh` was also building API Rerefence internally, I removed that process. `.local_build.sh` also added some Bash options to stop in error or so. Futher more added `cd "${SCRIPT_DIR}"` at the beginning so that the script will work no matter which directory it is executed in. `docs/api_reference/api_reference.rst` is removed, because which is generated by `docs/api_reference/create_api_rst.py`, and added it to .gitignore. Finally, the description of CONTRIBUTING.md was modified. **Issue: the issue # it fixes (if applicable)** https://github.com/hwchase17/langchain/issues/6413 **Dependencies: any dependencies required for this change** `nbdoc` was missing in group docs so it was added. I installed it with the `poetry add --group docs nbdoc` command. I am concerned if any modifications are needed to poetry.lock. I would greatly appreciate it if you could pay close attention to this file during the review. **Tag maintainer** - General / Misc / if you don't know who to tag: @baskaryan If this PR needs any additional changes, I'll be happy to make them! --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-07-12 02:05:14 +00:00
" data = json.loads(data)\n",
"\n",
" # Extract x and y values from the data\n",
" timestamps = [item[\"time\"] for item in data]\n",
" dates = [dt.datetime.fromtimestamp(ts) for ts in timestamps]\n",
" y = [item[\"value\"] for item in data]\n",
"\n",
" plt.rcParams[\"figure.figsize\"] = [6, 4]\n",
" plt.subplots_adjust(bottom=0.2)\n",
" plt.xticks(rotation=25)\n",
" ax = plt.gca()\n",
" xfmt = md.DateFormatter(\"%Y-%m-%d %H:%M:%S\")\n",
" ax.xaxis.set_major_formatter(xfmt)\n",
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
"\n",
Fix `make docs_build` and related scripts (#7276) **Description: a description of the change** Fixed `make docs_build` and related scripts which caused errors. There are several changes. First, I made the build of the documentation and the API Reference into two separate commands. This is because it takes less time to build. The commands for documents are `make docs_build`, `make docs_clean`, and `make docs_linkcheck`. The commands for API Reference are `make api_docs_build`, `api_docs_clean`, and `api_docs_linkcheck`. It looked like `docs/.local_build.sh` could be used to build the documentation, so I used that. Since `.local_build.sh` was also building API Rerefence internally, I removed that process. `.local_build.sh` also added some Bash options to stop in error or so. Futher more added `cd "${SCRIPT_DIR}"` at the beginning so that the script will work no matter which directory it is executed in. `docs/api_reference/api_reference.rst` is removed, because which is generated by `docs/api_reference/create_api_rst.py`, and added it to .gitignore. Finally, the description of CONTRIBUTING.md was modified. **Issue: the issue # it fixes (if applicable)** https://github.com/hwchase17/langchain/issues/6413 **Dependencies: any dependencies required for this change** `nbdoc` was missing in group docs so it was added. I installed it with the `poetry add --group docs nbdoc` command. I am concerned if any modifications are needed to poetry.lock. I would greatly appreciate it if you could pay close attention to this file during the review. **Tag maintainer** - General / Misc / if you don't know who to tag: @baskaryan If this PR needs any additional changes, I'll be happy to make them! --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-07-12 02:05:14 +00:00
" # Create the plot\n",
" plt.plot(dates, y)\n",
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
"\n",
Fix `make docs_build` and related scripts (#7276) **Description: a description of the change** Fixed `make docs_build` and related scripts which caused errors. There are several changes. First, I made the build of the documentation and the API Reference into two separate commands. This is because it takes less time to build. The commands for documents are `make docs_build`, `make docs_clean`, and `make docs_linkcheck`. The commands for API Reference are `make api_docs_build`, `api_docs_clean`, and `api_docs_linkcheck`. It looked like `docs/.local_build.sh` could be used to build the documentation, so I used that. Since `.local_build.sh` was also building API Rerefence internally, I removed that process. `.local_build.sh` also added some Bash options to stop in error or so. Futher more added `cd "${SCRIPT_DIR}"` at the beginning so that the script will work no matter which directory it is executed in. `docs/api_reference/api_reference.rst` is removed, because which is generated by `docs/api_reference/create_api_rst.py`, and added it to .gitignore. Finally, the description of CONTRIBUTING.md was modified. **Issue: the issue # it fixes (if applicable)** https://github.com/hwchase17/langchain/issues/6413 **Dependencies: any dependencies required for this change** `nbdoc` was missing in group docs so it was added. I installed it with the `poetry add --group docs nbdoc` command. I am concerned if any modifications are needed to poetry.lock. I would greatly appreciate it if you could pay close attention to this file during the review. **Tag maintainer** - General / Misc / if you don't know who to tag: @baskaryan If this PR needs any additional changes, I'll be happy to make them! --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-07-12 02:05:14 +00:00
" # Set labels and title\n",
" plt.xlabel(\"Time\")\n",
" plt.ylabel(\"Value\")\n",
" plt.title(title)\n",
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
"\n",
Fix `make docs_build` and related scripts (#7276) **Description: a description of the change** Fixed `make docs_build` and related scripts which caused errors. There are several changes. First, I made the build of the documentation and the API Reference into two separate commands. This is because it takes less time to build. The commands for documents are `make docs_build`, `make docs_clean`, and `make docs_linkcheck`. The commands for API Reference are `make api_docs_build`, `api_docs_clean`, and `api_docs_linkcheck`. It looked like `docs/.local_build.sh` could be used to build the documentation, so I used that. Since `.local_build.sh` was also building API Rerefence internally, I removed that process. `.local_build.sh` also added some Bash options to stop in error or so. Futher more added `cd "${SCRIPT_DIR}"` at the beginning so that the script will work no matter which directory it is executed in. `docs/api_reference/api_reference.rst` is removed, because which is generated by `docs/api_reference/create_api_rst.py`, and added it to .gitignore. Finally, the description of CONTRIBUTING.md was modified. **Issue: the issue # it fixes (if applicable)** https://github.com/hwchase17/langchain/issues/6413 **Dependencies: any dependencies required for this change** `nbdoc` was missing in group docs so it was added. I installed it with the `poetry add --group docs nbdoc` command. I am concerned if any modifications are needed to poetry.lock. I would greatly appreciate it if you could pay close attention to this file during the review. **Tag maintainer** - General / Misc / if you don't know who to tag: @baskaryan If this PR needs any additional changes, I'll be happy to make them! --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-07-12 02:05:14 +00:00
" plt.show()\n",
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
"\n",
"\n",
"response = client.search_ts(\"__name__\", \"latency\", 0, int(time.time()))\n",
"plot(response.text, \"Latency\")\n",
"\n",
"response = client.search_ts(\"__name__\", \"error\", 0, int(time.time()))\n",
"plot(response.text, \"Errors\")\n",
"\n",
"response = client.search_ts(\"__name__\", \"prompt_tokens\", 0, int(time.time()))\n",
"plot(response.text, \"Prompt Tokens\")\n",
"\n",
"response = client.search_ts(\"__name__\", \"completion_tokens\", 0, int(time.time()))\n",
"plot(response.text, \"Completion Tokens\")\n",
"\n",
"response = client.search_ts(\"__name__\", \"total_tokens\", 0, int(time.time()))\n",
Fix `make docs_build` and related scripts (#7276) **Description: a description of the change** Fixed `make docs_build` and related scripts which caused errors. There are several changes. First, I made the build of the documentation and the API Reference into two separate commands. This is because it takes less time to build. The commands for documents are `make docs_build`, `make docs_clean`, and `make docs_linkcheck`. The commands for API Reference are `make api_docs_build`, `api_docs_clean`, and `api_docs_linkcheck`. It looked like `docs/.local_build.sh` could be used to build the documentation, so I used that. Since `.local_build.sh` was also building API Rerefence internally, I removed that process. `.local_build.sh` also added some Bash options to stop in error or so. Futher more added `cd "${SCRIPT_DIR}"` at the beginning so that the script will work no matter which directory it is executed in. `docs/api_reference/api_reference.rst` is removed, because which is generated by `docs/api_reference/create_api_rst.py`, and added it to .gitignore. Finally, the description of CONTRIBUTING.md was modified. **Issue: the issue # it fixes (if applicable)** https://github.com/hwchase17/langchain/issues/6413 **Dependencies: any dependencies required for this change** `nbdoc` was missing in group docs so it was added. I installed it with the `poetry add --group docs nbdoc` command. I am concerned if any modifications are needed to poetry.lock. I would greatly appreciate it if you could pay close attention to this file during the review. **Tag maintainer** - General / Misc / if you don't know who to tag: @baskaryan If this PR needs any additional changes, I'll be happy to make them! --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-07-12 02:05:14 +00:00
"plot(response.text, \"Total Tokens\")"
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
]
},
{
"cell_type": "markdown",
"id": "c3d61822-1781-4bc6-97a2-2abc5c2b2e75",
"metadata": {},
"source": [
"## Full text query on prompt or prompt outputs."
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "a0f051f0-e2bc-44e7-8dfb-bfd5bbd0fc9f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Results for normandy : [{\"time\":1686821979,\"fields\":{\"prompt\":\"In what country is Normandy located?\"},\"text\":\"In what country is Normandy located?\"},{\"time\":1686821982,\"fields\":{\"prompt_response\":\"\\n\\nNormandy is located in France.\"},\"text\":\"\\n\\nNormandy is located in France.\"},{\"time\":1686821984,\"fields\":{\"prompt_response\":\"\\n\\nThe Normans first settled in Normandy in the late 9th century.\"},\"text\":\"\\n\\nThe Normans first settled in Normandy in the late 9th century.\"},{\"time\":1686821993,\"fields\":{\"prompt\":\"Who gave their name to Normandy in the 1000's and 1100's\"},\"text\":\"Who gave their name to Normandy in the 1000's and 1100's\"},{\"time\":1686821997,\"fields\":{\"prompt_response\":\"\\n\\nThe Normans, a people from northern France, gave their name to Normandy in the 1000s and 1100s. The Normans were descended from Viking settlers who had come to the region in the late 800s.\"},\"text\":\"\\n\\nThe Normans, a people from northern France, gave their name to Normandy in the 1000s and 1100s. The Normans were descended from Viking settlers who had come to the region in the late 800s.\"}]\n",
"===\n",
"Results for king charles III : [{\"time\":1686821998,\"fields\":{\"prompt\":\"Who did King Charles III swear fealty to?\"},\"text\":\"Who did King Charles III swear fealty to?\"},{\"time\":1686822000,\"fields\":{\"prompt_response\":\"\\n\\nKing Charles III swore fealty to Pope Innocent III.\"},\"text\":\"\\n\\nKing Charles III swore fealty to Pope Innocent III.\"}]\n"
]
}
],
"source": [
"# Search for a particular prompt text.\n",
"query = \"normandy\"\n",
"response = client.search_log(query, 0, int(time.time()))\n",
"print(\"Results for\", query, \":\", response.text)\n",
"\n",
"print(\"===\")\n",
"\n",
"query = \"king charles III\"\n",
"response = client.search_log(\"king charles III\", 0, int(time.time()))\n",
Fix `make docs_build` and related scripts (#7276) **Description: a description of the change** Fixed `make docs_build` and related scripts which caused errors. There are several changes. First, I made the build of the documentation and the API Reference into two separate commands. This is because it takes less time to build. The commands for documents are `make docs_build`, `make docs_clean`, and `make docs_linkcheck`. The commands for API Reference are `make api_docs_build`, `api_docs_clean`, and `api_docs_linkcheck`. It looked like `docs/.local_build.sh` could be used to build the documentation, so I used that. Since `.local_build.sh` was also building API Rerefence internally, I removed that process. `.local_build.sh` also added some Bash options to stop in error or so. Futher more added `cd "${SCRIPT_DIR}"` at the beginning so that the script will work no matter which directory it is executed in. `docs/api_reference/api_reference.rst` is removed, because which is generated by `docs/api_reference/create_api_rst.py`, and added it to .gitignore. Finally, the description of CONTRIBUTING.md was modified. **Issue: the issue # it fixes (if applicable)** https://github.com/hwchase17/langchain/issues/6413 **Dependencies: any dependencies required for this change** `nbdoc` was missing in group docs so it was added. I installed it with the `poetry add --group docs nbdoc` command. I am concerned if any modifications are needed to poetry.lock. I would greatly appreciate it if you could pay close attention to this file during the review. **Tag maintainer** - General / Misc / if you don't know who to tag: @baskaryan If this PR needs any additional changes, I'll be happy to make them! --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-07-12 02:05:14 +00:00
"print(\"Results for\", query, \":\", response.text)"
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
]
},
{
"cell_type": "markdown",
"id": "4b171074-c775-48e0-a4b3-f550e2c8eccb",
"metadata": {},
"source": [
"## Stop infino server"
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "147663cb-b88f-4cfb-9726-7231dbec7cc1",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"infino-example\n"
]
}
],
"source": [
"!docker rm -f infino-example"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "86f36c49-53a3-460d-b74b-995cda7726b3",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
Infino integration for simplified logs, metrics & search across LLM data & token usage (#6218) ### Integration of Infino with LangChain for Enhanced Observability This PR aims to integrate [Infino](https://github.com/infinohq/infino), an open source observability platform written in rust for storing metrics and logs at scale, with LangChain, providing users with a streamlined and efficient method of tracking and recording LangChain experiments. By incorporating Infino into LangChain, users will be able to gain valuable insights and easily analyze the behavior of their language models. #### Please refer to the following files related to integration: - `InfinoCallbackHandler`: A [callback handler](https://github.com/naman-modi/langchain/blob/feature/infino-integration/langchain/callbacks/infino_callback.py) specifically designed for storing chain responses within Infino. - Example `infino.ipynb` file: A comprehensive notebook named [infino.ipynb](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/modules/callbacks/integrations/infino.ipynb) has been included to guide users on effectively leveraging Infino for tracking LangChain requests. - [Integration Doc](https://github.com/naman-modi/langchain/blob/feature/infino-integration/docs/extras/ecosystem/integrations/infino.mdx) for Infino integration. By integrating Infino, LangChain users will gain access to powerful visualization and debugging capabilities. Infino enables easy tracking of inputs, outputs, token usage, execution time of LLMs. This comprehensive observability ensures a deeper understanding of individual executions and facilitates effective debugging. Co-authors: @vinaykakade @savannahar68 --------- Co-authored-by: Vinay Kakade <vinaykakade@gmail.com>
2023-06-21 08:38:20 +00:00
}
},
"nbformat": 4,
"nbformat_minor": 5
}