You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
langchain/libs/langchain/dev.Dockerfile

61 lines
2.0 KiB
Plaintext

Update dev container (#6189) Fixes https://github.com/hwchase17/langchain/issues/6172 As described in https://github.com/hwchase17/langchain/issues/6172, I'd love to help update the dev container in this project. **Summary of changes:** - Dev container now builds (the current container in this repo won't build for me) - Dockerfile updates - Update image to our [currently-maintained Python image](https://github.com/devcontainers/images/tree/main/src/python/.devcontainer) (`mcr.microsoft.com/devcontainers/python`) rather than the deprecated image from vscode-dev-containers - Move Dockerfile to root of repo - in order for `COPY` to work properly, it needs the files (in this case, `pyproject.toml` and `poetry.toml`) in the same directory - devcontainer.json updates - Removed `customizations` and `remoteUser` since they should be covered by the updated image in the Dockerfile - Update comments - Update docker-compose.yaml to properly point to updated Dockerfile - Add a .gitattributes to avoid line ending conversions, which can result in hundreds of pending changes ([info](https://code.visualstudio.com/docs/devcontainers/tips-and-tricks#_resolving-git-line-ending-issues-in-containers-resulting-in-many-modified-files)) - Add a README in the .devcontainer folder and info on the dev container in the contributing.md **Outstanding questions:** - Is it expected for `poetry install` to take some time? It takes about 30 minutes for this dev container to finish building in a Codespace, but a user should only have to experience this once. Through some online investigation, this doesn't seem unusual - Versions of poetry newer than 1.3.2 failed every time - based on some of the guidance in contributing.md and other online resources, it seemed changing poetry versions might be a good solution. 1.3.2 is from Jan 2023 --------- Co-authored-by: bamurtaugh <brmurtau@microsoft.com> Co-authored-by: Samruddhi Khandale <samruddhikhandale@github.com>
12 months ago
# This is a Dockerfile for the Development Container
# Use the Python base image
ARG VARIANT="3.11-bullseye"
Update dev container (#6189) Fixes https://github.com/hwchase17/langchain/issues/6172 As described in https://github.com/hwchase17/langchain/issues/6172, I'd love to help update the dev container in this project. **Summary of changes:** - Dev container now builds (the current container in this repo won't build for me) - Dockerfile updates - Update image to our [currently-maintained Python image](https://github.com/devcontainers/images/tree/main/src/python/.devcontainer) (`mcr.microsoft.com/devcontainers/python`) rather than the deprecated image from vscode-dev-containers - Move Dockerfile to root of repo - in order for `COPY` to work properly, it needs the files (in this case, `pyproject.toml` and `poetry.toml`) in the same directory - devcontainer.json updates - Removed `customizations` and `remoteUser` since they should be covered by the updated image in the Dockerfile - Update comments - Update docker-compose.yaml to properly point to updated Dockerfile - Add a .gitattributes to avoid line ending conversions, which can result in hundreds of pending changes ([info](https://code.visualstudio.com/docs/devcontainers/tips-and-tricks#_resolving-git-line-ending-issues-in-containers-resulting-in-many-modified-files)) - Add a README in the .devcontainer folder and info on the dev container in the contributing.md **Outstanding questions:** - Is it expected for `poetry install` to take some time? It takes about 30 minutes for this dev container to finish building in a Codespace, but a user should only have to experience this once. Through some online investigation, this doesn't seem unusual - Versions of poetry newer than 1.3.2 failed every time - based on some of the guidance in contributing.md and other online resources, it seemed changing poetry versions might be a good solution. 1.3.2 is from Jan 2023 --------- Co-authored-by: bamurtaugh <brmurtau@microsoft.com> Co-authored-by: Samruddhi Khandale <samruddhikhandale@github.com>
12 months ago
FROM mcr.microsoft.com/devcontainers/python:0-${VARIANT} AS langchain-dev-base
USER vscode
# Define the version of Poetry to install (default is 1.4.2)
# Define the directory of python virtual environment
ARG PYTHON_VIRTUALENV_HOME=/home/vscode/langchain-py-env \
Update dev container (#6189) Fixes https://github.com/hwchase17/langchain/issues/6172 As described in https://github.com/hwchase17/langchain/issues/6172, I'd love to help update the dev container in this project. **Summary of changes:** - Dev container now builds (the current container in this repo won't build for me) - Dockerfile updates - Update image to our [currently-maintained Python image](https://github.com/devcontainers/images/tree/main/src/python/.devcontainer) (`mcr.microsoft.com/devcontainers/python`) rather than the deprecated image from vscode-dev-containers - Move Dockerfile to root of repo - in order for `COPY` to work properly, it needs the files (in this case, `pyproject.toml` and `poetry.toml`) in the same directory - devcontainer.json updates - Removed `customizations` and `remoteUser` since they should be covered by the updated image in the Dockerfile - Update comments - Update docker-compose.yaml to properly point to updated Dockerfile - Add a .gitattributes to avoid line ending conversions, which can result in hundreds of pending changes ([info](https://code.visualstudio.com/docs/devcontainers/tips-and-tricks#_resolving-git-line-ending-issues-in-containers-resulting-in-many-modified-files)) - Add a README in the .devcontainer folder and info on the dev container in the contributing.md **Outstanding questions:** - Is it expected for `poetry install` to take some time? It takes about 30 minutes for this dev container to finish building in a Codespace, but a user should only have to experience this once. Through some online investigation, this doesn't seem unusual - Versions of poetry newer than 1.3.2 failed every time - based on some of the guidance in contributing.md and other online resources, it seemed changing poetry versions might be a good solution. 1.3.2 is from Jan 2023 --------- Co-authored-by: bamurtaugh <brmurtau@microsoft.com> Co-authored-by: Samruddhi Khandale <samruddhikhandale@github.com>
12 months ago
POETRY_VERSION=1.3.2
ENV POETRY_VIRTUALENVS_IN_PROJECT=false \
POETRY_NO_INTERACTION=true
# Install Poetry outside of the v`irtual environment to avoid conflicts
RUN python3 -m pip install --user pipx && \
python3 -m pipx ensurepath && \
pipx install poetry==${POETRY_VERSION}
# Create a Python virtual environment for the project
RUN python3 -m venv ${PYTHON_VIRTUALENV_HOME} && \
$PYTHON_VIRTUALENV_HOME/bin/pip install --upgrade pip
ENV PATH="$PYTHON_VIRTUALENV_HOME/bin:$PATH" \
VIRTUAL_ENV=$PYTHON_VIRTUALENV_HOME
# Setup for bash
RUN poetry completions bash >> /home/vscode/.bash_completion && \
echo "export PATH=$PYTHON_VIRTUALENV_HOME/bin:$PATH" >> ~/.bashrc
# Set the working directory for the app
WORKDIR /workspaces/langchain
# Use a multi-stage build to install dependencies
FROM langchain-dev-base AS langchain-dev-dependencies
ARG PYTHON_VIRTUALENV_HOME
# Copy only the dependency files for installation
infra: Resolve the endless dependency resolution during the build of `dev.Dockerfile` by copying `poetry.lock` (#19465) ## Description This PR proposes a modification to the `libs/langchain/dev.Dockerfile` configuration to copy the `libs/langchain/poetry.lock` into the working directory. The change aims to address the issue where the Poetry install command, the last command in the `dev.Dockerfile`, takes excessively long hours, and to ensure the reproducibility of the poetry environment in the devcontainer. ## Problem The `dev.Dockerfile`, prepared for development environments such as `.devcontainer`, encounters an unending dependency resolution when attempting the Poetry installation. ### Steps to Reproduce Execute the following build command: ```bash docker build -f libs/langchain/dev.Dockerfile . ``` ### Current Behavior The Docker build process gets stuck at the following step, which, in my experience, did not conclude even after an entire night: ``` => [langchain-dev-dependencies 4/6] COPY libs/community/ ../community/ 0.9s => [langchain-dev-dependencies 5/6] COPY libs/text-splitters/ ../text-splitters/ 0.0s => [langchain-dev-dependencies 6/6] RUN poetry install --no-interaction --no-ansi --with dev,test,docs 12.3s => => # Updating dependencies => => # Resolving dependencies... ``` ### Expected Behavior The Docker build completes in a realistic timeframe. By applying this PR, the build finishes within a few minutes. ### Analysis The complexity of LangChain's dependencies has reached a point where Poetry is required to resolve dependencies akin to threading a needle. Consequently, poetry install fails to complete in a practical timeframe. ## Solution The solution for dependency resolution is already recorded in `libs/langchain/poetry.lock`, so we can use it. When copying `project.toml` and `poetry.toml`, the `poetry.lock` located in the same directory should also be copied. ```diff # Copy only the dependency files for installation -COPY libs/langchain/pyproject.toml libs/langchain/poetry.toml ./ +COPY libs/langchain/pyproject.toml libs/langchain/poetry.toml libs/langchain/poetry.lock ./ ``` ## Note I am not intimately familiar with the historical context of the `dev.Dockerfile` and thus do not know why `poetry.lock` has not been copied until now. It might have been an oversight, or perhaps dependency resolution used to complete quickly even without the `poetry.lock` file in the past. However, if there are deliberate reasons why copying `poetry.lock` is not advisable, please just close this PR.
2 months ago
COPY libs/langchain/pyproject.toml libs/langchain/poetry.toml libs/langchain/poetry.lock ./
# Copy the langchain library for installation
COPY libs/langchain/ libs/langchain/
# Copy the core library for installation
COPY libs/core ../core
# Copy the community library for installation
COPY libs/community/ ../community/
text-splitters, infra: fix `libs/langchain/dev.Dockerfile` so that the `text-splitter` directory is copied before poetry installation (#19214) ## Description This PR modifies the settings in `libs/langchain/dev.Dockerfile` to ensure that the `text-splitters` directory is copied before the poetry installation process begins. Without this modification, the `docker build` command fails for `dev.Dockerfile`, preventing the setup of some development environments, including `.devcontainer`. ## Bug Details ### Repro Run the following command: ```bash docker build -f libs/langchain/dev.Dockerfile . ``` ### Current Behavior The docker build command fails, raising the following error: ``` ... => [langchain-dev-dependencies 4/5] COPY libs/community/ ../community/ 0.4s => ERROR [langchain-dev-dependencies 5/5] RUN poetry install --no-interaction --no-ansi --with dev,test,docs 1.1s ------ > [langchain-dev-dependencies 5/5] RUN poetry install --no-interaction --no-ansi --with dev,test,docs: #13 0.970 #13 0.970 Directory ../text-splitters does not exist ------ executor failed running [/bin/sh -c poetry install --no-interaction --no-ansi --with dev,test,docs]: exit code: 1 ``` ### Expected Behavior The `docker build` command successfully completes without the poetry error. ### Analysis The error occurs because the `text-splitters` directory is not copied into the build environment, unlike the other packages under the `libs` directory. I suspect that the `COPY` setting was overlooked since `text-splitters` was separated in a recent PR. ## Fix Add the following lines to the `libs/langchain/dev.Dockerfile`: ```dockerfile # Copy the text-splitters library for installation COPY libs/text-splitters/ ../text-splitters/ ```
2 months ago
# Copy the text-splitters library for installation
COPY libs/text-splitters/ ../text-splitters/
# Copy the partners library for installation
COPY libs/partners ../partners/
# Install the Poetry dependencies (this layer will be cached as long as the dependencies don't change)
RUN poetry install --no-interaction --no-ansi --with dev,test,docs