Commit Graph

83 Commits

Author SHA1 Message Date
XUEYANZ
f97d3a76e7
Update CONTRIBUTING.md (#9817)
<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
- Tag maintainer: for a quicker response, tag the relevant maintainer
(see below),
- Twitter handle: we announce bigger features on Twitter. If your PR
gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.

See contribution guidelines for more information on how to write/run
tests, lint, etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. These live is docs/extras
directory.

If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->

Hi LangChain :) Thank you for such a great project! 
I was going through the CONTRIBUTING.md and found a few minor issues.
2023-08-28 09:38:34 -04:00
Predrag Gruevski
c06f34fa35
Use new Python setup approach for scheduled tests. (#9626)
Using the same new unified Python setup as the regular tests and the
lint job, as set up in #9625.
2023-08-22 16:07:53 -04:00
Predrag Gruevski
83986ea98a
Cache poetry install + unify Python/Poetry setup for lint and test jobs. (#9625)
With this PR:
- All lint and test jobs use the exact same Python + Poetry installation
approach, instead of lints doing it one way and tests doing it another
way.
- The Poetry installation itself is cached, which saves ~15s per run.
- We no longer pass shell commands as workflow arguments to a workflow
that just runs them in a shell. This makes our actions more resilient to
shell code injection.

If y'all like this approach, I can modify the scheduled tests workflow
and the release workflow to use this too.
2023-08-22 15:59:22 -04:00
seamusp
f3ba9ce7f4
Remove -E all from installation instructions (#9573)
Update installation instructions to only install test dependencies rather than all dependencies.

---------

Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
2023-08-22 14:57:58 -04:00
Predrag Gruevski
35812d0096
Set up concurrency groups and workflow cancelation in CI. (#9564)
If another push to the same PR or branch happens while its CI is still
running, cancel the earlier run in favor of the next run.

There's no point in testing an outdated version of the code. GitHub only
allows a limited number of job runners to be active at the same time, so
it's better to cancel pointless jobs early so that more useful jobs can
run sooner.
2023-08-22 14:21:26 -04:00
Predrag Gruevski
3c7cc4d440
Test experimental package with langchain on master branch. (#9621)
It's possible that langchain-experimental works fine with the latest
*published* langchain, but is broken with the langchain on `master`.
Unfortunately, you can see this is currently the case — this is why this
PR also includes a minor fix for the `langchain` package itself.

We want to catch situations like that *before* releasing a new
langchain, hence this test.
2023-08-22 13:35:21 -04:00
Predrag Gruevski
acb54d8b9d
Reduce cache timeouts to ensure faster builds on timeout. (#9619)
The current timeouts are too long, and mean that if the GitHub cache
decides to act up, jobs get bogged down for 15min at a time. This has
happened 2-3 times already this week -- a tiny fraction of our total
workflows but really annoying when it happens to you. We can do better.

Installing deps on cache miss takes about ~4min, so it's not worth
waiting more than 4min for the deps cache. The black and mypy caches
save 1 and 2min, respectively, so wait only up to that long to download
them.
2023-08-22 12:11:38 -04:00
Predrag Gruevski
a1e89aa8d5
Explicitly add the contents: write permission for publishing releases. (#9617) 2023-08-22 08:38:18 -07:00
Predrag Gruevski
c75e1aa5ed
Eliminate special-casing from test CI workflows. (#9562)
The previous approach was relying on `_test.yml` taking an input
parameter, and then doing almost completely orthogonal things for each
parameter value. I've separated out each of those test situations as its
own job or workflow file, which eliminated all the special-casing and,
in my opinion, improved maintainability by making it much more obvious
what code runs when.
2023-08-22 11:36:52 -04:00
Predrag Gruevski
6c308aabae
Use the GitHub-suggested safer pattern for shell interpolation. (#9567)
Using `${{ }}` to construct shell commands is risky, since the `${{ }}`
interpolation runs first and ignores shell quoting rules. This means
that shell commands that look safely quoted, like `echo "${{
github.event.issue.title }}"`, are actually vulnerable to shell
injection.

More details here:
https://github.blog/2023-08-09-four-tips-to-keep-your-github-actions-workflows-secure/
2023-08-21 17:59:10 -04:00
Predrag Gruevski
2a3758a98e
Reminder to not report security issues as "bug" type issues. (#9554)
Updated the issue template that pops up when users open a new issue.
2023-08-21 15:48:33 -04:00
Predrag Gruevski
9f08d29bc8
Use PyPI Trusted Publishing to publish langchain packages. (#9467)
Trusted Publishing is the current best practice for publishing Python
packages. Rather than long-lived secret keys, it uses OpenID Connect
(OIDC) to allow our GitHub runner to directly authenticate itself to
PyPI and get a short-lived publishing token. This locks down publishing
quite a bit:
- There's no long-lived publish key to steal anymore.
- Publishing is *only* allowed via the *specifically designated* GitHub
workflow in the designated repo.

It also is operationally easier: no keys means there's nothing that
needs to be periodically rotated, nothing to worry about leaking, and
nobody can accidentally publish a release from their laptop because they
happened to have PyPI keys set up.

After this gets merged, we'll need to configure PyPI to start expecting
trusted publishing. It's only a few clicks and should only take a
minute; instructions are here:
https://docs.pypi.org/trusted-publishers/adding-a-publisher/

More info:
- https://blog.pypi.org/posts/2023-04-20-introducing-trusted-publishers/
- https://github.com/pypa/gh-action-pypi-publish
2023-08-21 14:44:29 -04:00
Predrag Gruevski
249752e8ee
Require manually triggering release workflows. (#9552) 2023-08-21 13:54:44 -04:00
Predrag Gruevski
875ea4b4c6
Fix conditional that erroneously always runs. (#9543)
The input it means to test for is `"libs/langchain"` and not
`"langchain"`.
2023-08-21 13:24:33 -04:00
Predrag Gruevski
a7eba8b006
Release on push to master instead of on closed PRs targeting it. (#9544)
This is safer than the prior approach, since it's safe by default: the
release workflows never get triggered for non-merged PRs, so there's no
possibility of a buggy conditional accidentally letting a workflow
proceed when it shouldn't have.

The only loss is that publishing no longer requires a `release` label on
the merged PR that bumps the version. We can add a separate CI step that
enforces that part as a condition for merging into `master`, if
desirable.
2023-08-21 12:57:40 -04:00
Predrag Gruevski
a03003f5fd
Upgrade CI poetry version to 1.5.1. (#9479)
Poetry v1.5.1 was released on May 29, almost 3 months ago. Probably a
safe upgrade.
2023-08-21 10:35:56 -04:00
Yuki Miyake
85a1c6d0b7
🐛 fix unexpected run of release workflow (#9494)
I have discovered a bug located within `.github/workflows/_release.yml`
which is the primary cause of continuous integration (CI) errors. The
problem can be solved; therefore, I have constructed a PR to address the
issue.

## The Issue

Access the following link to view the exact errors: [Langhain Release
Workflow](https://github.com/langchain-ai/langchain/actions/workflows/langchain_release.yml)

The instances of these errors take place for **each PR** that updates
`pyproject.toml`, excluding those specifically associated with bumping
PRs.

See below for the specific error message:

```
Error: Error 422: Validation Failed: {"resource":"Release","code":"already_exists","field":"tag_name"}
```

An image of the error can be viewed here:

![Image](https://github.com/langchain-ai/langchain/assets/13769670/13125f73-9b53-49b7-a83e-653bb01a1da1)

The `_release.yml` document contains the following if-condition:

```yaml
    if: |
        ${{ github.event.pull_request.merged == true }}
        && ${{ contains(github.event.pull_request.labels.*.name, 'release') }}
```

## The Root Cause

The above job constantly runs as the `if-condition` is always identified
as `true`.

## The Logic

The `if-condition` can be defined as `if: ${{ b1 }} && ${{ b2 }}`, where
`b1` and `b2` are boolean values. However, in terms of condition
evaluation with GitHub Actions, `${{ false }}` is identified as a string
value, thereby rendering it as truthy as per the [official
documentation](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idif).

I have run some tests regarding this behavior within my forked
repository. You can consult my [debug
PR](https://github.com/zawakin/langchain/pull/1) for reference.

Here is the result of the tests:

|If-Condition|Outcome|
|:--:|:--:|
|`if: true && ${{ false }}`|Execution|
|`if: ${{ false }}` |Skipped|
|`if: true && false` |Skipped|
|`if: false`|Skipped|
|`if: ${{ true && false }}` |Skipped|

In view of the first and second results, we can infer that `${{ false
}}` can only be interpreted as `true` for conditions composed of some
expressions.
It is consistent that the condition of `if: ${{ inputs.working-directory
== 'libs/langchain' }}` works.

It is surprised to be skipped for the second case but it seems the spec
of GitHub Actions 😓

Anyway, the PR would fix these errors, I believe 👍 

Could you review this? @hwchase17 or @shoelsch , who is the author of
[PR](https://github.com/langchain-ai/langchain/pull/360).
2023-08-21 10:34:03 -04:00
Predrag Gruevski
ade683c589
Rely on WORKDIR env var to avoid ugly ternary operators in workflows. (#9456)
Ternary operators in GitHub Actions syntax are pretty ugly and hard to
read: `inputs.working-directory == '' && '.' ||
inputs.working-directory` means "if the condition is true, use `'.'` and
otherwise use the expression after the `||`".

This PR performs the ternary as few times as possible, assigning its
outcome to an env var we can then reuse as needed.
2023-08-18 12:55:33 -04:00
Predrag Gruevski
8976483f3a
Lint only on the min and max supported Python versions. (#9450)
Only lint on the min and max supported Python versions.

It's extremely unlikely that there's a lint issue on any version in
between that doesn't show up on the min or max versions.

GitHub rate-limits how many jobs can be running at any one time.
Starting new jobs is also relatively slow, so linting on fewer versions
makes CI faster.
2023-08-18 10:26:38 -04:00
Predrag Gruevski
463019ac3e
Cache black formatting information across CI runs. (#9413)
Save and persist `black`'s formatted files cache across CI runs.

Around a ~20s win, 21s -> 2s. Most cases should be close to this best
case scenario, since most PRs don't modify most files — and this PR
makes sure we don't re-check files that haven't changed.

Before:

![image](https://github.com/langchain-ai/langchain/assets/2348618/6c5670c5-be70-4a18-aa2a-ece5e4425d1e)

After:

![image](https://github.com/langchain-ai/langchain/assets/2348618/37810d27-c611-4f76-b9bd-e827cefbaa0a)
2023-08-18 09:49:50 -04:00
Predrag Gruevski
0dd2c21089
Do not bust poetry install cache when manually installing pydantic v2. (#9407)
Using `poetry add` to install `pydantic@2.1` was also causing poetry to
change its lockfile. This prevented dependency caching from working:
- When attempting to restore a cache, it would hash the lockfile in git
and use it as part of the cache key. Say this is a cache miss.
- Then, it would attempt to save the cache -- but the lockfile will have
changed, so the cache key would be *different* than the key in the
lookup. So the cache save would succeed, but to a key that cannot be
looked up in the next run -- meaning we never get a cache hit.

In addition to busting the cache, the lockfile update itself is also
non-trivially long, over 30s:

![image](https://github.com/langchain-ai/langchain/assets/2348618/d84d3b56-484d-45eb-818d-54126a094a40)

This PR fixes the problems by using `pip` to perform the installation,
avoiding the lockfile change.
2023-08-17 18:23:00 -04:00
Predrag Gruevski
8f2d321dd0
Cache .mypy_cache across lint runs. (#9405)
Preserve the `.mypy_cache` directory across lint runs, to avoid having
to re-parse all dependencies and their type information.

Approximately a 1min perf win for CI.

Before:

![image](https://github.com/langchain-ai/langchain/assets/2348618/6524f2a9-efc0-4588-a94c-69914b98b382)

After:

![image](https://github.com/langchain-ai/langchain/assets/2348618/dd0af954-4dc9-43d3-8544-25846616d41d)
2023-08-17 13:53:59 -04:00
Predrag Gruevski
7e63270e04
Ensure the in-project venv gets cached in CI tests. (#9336)
The previous caching configuration was attempting to cache poetry venvs
created in the default shared virtualenvs directory. However, all
langchain packages use `in-project = true` for their poetry virtualenv
setup, which moves the venv inside the package itself instead. This
meant that poetry venvs were not being cached at all.

This PR ensures that the venv gets cached by adding the in-project venv
directory to the cached directories list.

It also makes sure that the cache key *only* includes the lockfile being
installed, as opposed to *all lockfiles* (unnecessary cache misses) or
just the *top-level lockfile* (cache hits when it shouldn't).
2023-08-17 11:47:22 -04:00
Predrag Gruevski
f2560188ec
Cache linting venv on CI. (#9342)
Ensure that we cache the linting virtualenv as well as the pip cache for
the `pip install -e langchain` step.

This is a win of about 60-90s overall.

Before:

![image](https://github.com/langchain-ai/langchain/assets/2348618/f55f8398-2c3a-4112-bad3-2c646d186183)

After:

![image](https://github.com/langchain-ai/langchain/assets/2348618/984a9529-2431-41b4-97e5-7f5dd7742651)
2023-08-17 11:46:58 -04:00
Bagatur
995ef8a7fc
unpin pydantic (#9356) 2023-08-17 01:55:46 -07:00
Bagatur
3eccd72382
pin pydantic (#9274)
don't want default to be v2 yet
2023-08-15 15:02:28 -07:00
Eugene Yurtsev
a091b4bf4c
Update testing workflow to test with both pydantic versions (#9206)
* PR updates test.yml to test with both pydantic versions
* Code should be refactored to make it easier to do testing in matrix
format w/ packages
* Added steps to assert that pydantic version in the environment is as
expected
2023-08-15 13:21:11 -04:00
Bagatur
641cb80c9d
update pr temp (#9062) 2023-08-10 11:10:06 -07:00
Bagatur
206f809366
fix sched ci (more) (#9056) 2023-08-10 10:39:29 -07:00
Bagatur
e5db8a16c0
Bagatur/fix sched (#9054) 2023-08-10 09:34:44 -07:00
Bagatur
e162fd418a
fix sched ci (#9053) 2023-08-10 09:29:46 -07:00
Bagatur
269f85b7b7
scheduled gha fix (#8977) 2023-08-09 09:44:25 -07:00
Bagatur
95cf7de112
scheduled tests GHA (#8879)
Adding scheduled daily GHA that runs marked integration tests. To start
just marking some tests in test_openai
2023-08-08 14:55:25 -07:00
Harrison Chase
2448043b84
bump and fix (#8441) 2023-07-28 17:16:51 -07:00
Harrison Chase
cddd8ae83d
update release yml (#8364)
only do the step that tags and adds release notes if its langchain
2023-07-27 16:49:04 -07:00
William FH
01a9b06400
Add api cross ref linking (#8275)
Example of how it would show up in our python docs:


![image](https://github.com/langchain-ai/langchain/assets/13333726/0f0a88cc-ba4a-4778-bc47-118c66807f15)


Examples added to the reference docs:

https://api.python.langchain.com/en/wfh-api_crosslink/vectorstores/langchain.vectorstores.chroma.Chroma.html#langchain.vectorstores.chroma.Chroma


![image](https://github.com/langchain-ai/langchain/assets/13333726/dcd150de-cb56-4d42-b49a-a76a002a5a52)
2023-07-26 12:38:58 -07:00
Harrison Chase
8dcabd9205
bump releases rc0 (#8097) 2023-07-21 13:54:57 -07:00
Harrison Chase
0faba034b1
add experimental release action (#8096) 2023-07-21 13:38:35 -07:00
Harrison Chase
344cbd9c90
update contributor guide (#8088) 2023-07-21 12:01:05 -07:00
Harrison Chase
da04760de1
Harrison/move experimental (#8084) 2023-07-21 10:36:28 -07:00
Harrison Chase
f35db9f43e
(WIP) set up experimental (#7959) 2023-07-21 09:20:24 -07:00
Bagatur
25a2bdfb70
add pr template instructions (#7904) 2023-07-18 13:22:28 -07:00
Yaroslav Halchenko
0d92a7f357
codespell: workflow, config + some (quite a few) typos fixed (#6785)
Probably the most  boring PR to review ;)

Individual commits might be easier to digest

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>
2023-07-12 16:20:08 -04:00
os1ma
2667ddc686
Fix make docs_build and related scripts (#7276)
**Description: a description of the change**

Fixed `make docs_build` and related scripts which caused errors. There
are several changes.

First, I made the build of the documentation and the API Reference into
two separate commands. This is because it takes less time to build. The
commands for documents are `make docs_build`, `make docs_clean`, and
`make docs_linkcheck`. The commands for API Reference are `make
api_docs_build`, `api_docs_clean`, and `api_docs_linkcheck`.

It looked like `docs/.local_build.sh` could be used to build the
documentation, so I used that. Since `.local_build.sh` was also building
API Rerefence internally, I removed that process. `.local_build.sh` also
added some Bash options to stop in error or so. Futher more added `cd
"${SCRIPT_DIR}"` at the beginning so that the script will work no matter
which directory it is executed in.

`docs/api_reference/api_reference.rst` is removed, because which is
generated by `docs/api_reference/create_api_rst.py`, and added it to
.gitignore.

Finally, the description of CONTRIBUTING.md was modified.

**Issue: the issue # it fixes (if applicable)**

https://github.com/hwchase17/langchain/issues/6413

**Dependencies: any dependencies required for this change**

`nbdoc` was missing in group docs so it was added. I installed it with
the `poetry add --group docs nbdoc` command. I am concerned if any
modifications are needed to poetry.lock. I would greatly appreciate it
if you could pay close attention to this file during the review.

**Tag maintainer**
- General / Misc / if you don't know who to tag: @baskaryan

If this PR needs any additional changes, I'll be happy to make them!

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-07-11 22:05:14 -04:00
Kazuki Maeda
5c3fe8b0d1
Enhance Makefile with 'format_diff' Option and Improved Readability (#7394)
### Description:

This PR introduces a new option format_diff to the existing Makefile.
This option allows us to apply the formatting tools (Black and isort)
only to the changed Python and ipynb files since the last commit. This
will make our development process more efficient as we only format the
codes that we modify. Along with this change, comments were added to
make the Makefile more understandable and maintainable.

### Issue:

N/A

### Dependencies:

Add dependency to black.

### Tag maintainer:

@baskaryan

### Twitter handle:

[kzk_maeda](https://twitter.com/kzk_maeda)

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-07-11 21:03:17 -04:00
Bagatur
fad2c7e5e0
update pr tmpl (#7095) 2023-07-03 13:34:03 -06:00
Davis Chase
00a7403236
update pr tmpl (#6552) 2023-06-21 10:03:52 -07:00
Brigit Murtaugh
ccd916babe
Update dev container (#6189)
Fixes https://github.com/hwchase17/langchain/issues/6172

As described in https://github.com/hwchase17/langchain/issues/6172, I'd
love to help update the dev container in this project.

**Summary of changes:**
- Dev container now builds (the current container in this repo won't
build for me)
- Dockerfile updates
- Update image to our [currently-maintained Python
image](https://github.com/devcontainers/images/tree/main/src/python/.devcontainer)
(`mcr.microsoft.com/devcontainers/python`) rather than the deprecated
image from vscode-dev-containers
- Move Dockerfile to root of repo - in order for `COPY` to work
properly, it needs the files (in this case, `pyproject.toml` and
`poetry.toml`) in the same directory
- devcontainer.json updates
- Removed `customizations` and `remoteUser` since they should be covered
by the updated image in the Dockerfile
     - Update comments
- Update docker-compose.yaml to properly point to updated Dockerfile
- Add a .gitattributes to avoid line ending conversions, which can
result in hundreds of pending changes
([info](https://code.visualstudio.com/docs/devcontainers/tips-and-tricks#_resolving-git-line-ending-issues-in-containers-resulting-in-many-modified-files))
- Add a README in the .devcontainer folder and info on the dev container
in the contributing.md

**Outstanding questions:**
- Is it expected for `poetry install` to take some time? It takes about
30 minutes for this dev container to finish building in a Codespace, but
a user should only have to experience this once. Through some online
investigation, this doesn't seem unusual
- Versions of poetry newer than 1.3.2 failed every time - based on some
of the guidance in contributing.md and other online resources, it seemed
changing poetry versions might be a good solution. 1.3.2 is from Jan
2023

---------

Co-authored-by: bamurtaugh <brmurtau@microsoft.com>
Co-authored-by: Samruddhi Khandale <samruddhikhandale@github.com>
2023-06-16 15:42:14 -07:00
Davis Chase
87e502c6bc
Doc refactor (#6300)
Co-authored-by: jacoblee93 <jacoblee93@gmail.com>
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
2023-06-16 11:52:56 -07:00
Zander Chase
2c9619bc1d
Remove from PR template (#6018) 2023-06-11 19:34:26 -07:00