Commit Graph

46 Commits

Author SHA1 Message Date
kahkeng
4a8f5cdf4b
Add alternative token-based text splitter (#816)
This does not involve a separator, and will naively chunk input text at
the appropriate boundaries in token space.

This is helpful if we have strict token length limits that we need to
strictly follow the specified chunk size, and we can't use aggressive
separators like spaces to guarantee the absence of long strings.

CharacterTextSplitter will let these strings through without splitting
them, which could cause overflow errors downstream.

Splitting at arbitrary token boundaries is not ideal but is hopefully
mitigated by having a decent overlap quantity. Also this results in
chunks which has exact number of tokens desired, instead of sometimes
overcounting if we concatenate shorter strings.

Potentially also helps with #528.
2023-02-02 19:55:13 -08:00
Harrison Chase
23d5f64bda
Harrison/ngram example (#846)
Co-authored-by: Sean Spriggens <ssprigge@syr.edu>
2023-02-02 09:44:42 -08:00
Harrison Chase
d564308e0f
rfc: instruct embeddings (#811)
Co-authored-by: seanaedmiston <seane999@gmail.com>
2023-02-02 08:44:02 -08:00
Harrison Chase
7b4882a2f4
Harrison/tf embeddings (#817)
Co-authored-by: Ryohei Kuroki <10434946+yakigac@users.noreply.github.com>
2023-01-31 00:00:08 -08:00
dham
e04b063ff4
add faiss local saving/loading (#676)
- This uses the faiss built-in `write_index` and `load_index` to save
and load faiss indexes locally
- Also fixes #674
- The save/load functions also use the faiss library, so I refactored
the dependency into a function
2023-01-21 16:08:14 -08:00
Harrison Chase
0b204d8c21
Harrison/quadrant (#665)
Co-authored-by: Kacper Łukawski <kacperlukawski@users.noreply.github.com>
2023-01-20 09:45:01 -08:00
Harrison Chase
4d4cff0530
Harrison/cohere experimental (#638)
Co-authored-by: inyourhead <44607279+xettrisomeman@users.noreply.github.com>
2023-01-17 22:28:55 -08:00
Harrison Chase
ffc7e04d44
Harrison/wolfram alpha (#579)
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2023-01-11 05:52:19 -08:00
Harrison Chase
0072686aab
Harrison/new search engine (#477)
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2022-12-30 08:06:57 -05:00
Harrison Chase
f8b605293f
Harrison/improve memory (#432)
add AI prefix

add new type of memory

Co-authored-by: Jason <chisanch@usc.edu>
2022-12-27 08:23:51 -05:00
Harrison Chase
cf98f219f9
Harrison/tools exp (#372) 2022-12-18 21:51:23 -05:00
Harrison Chase
3474f39e21
Harrison/improve cache (#368)
make it so everything goes through generate, which removes the need for
two types of caches
2022-12-18 16:22:42 -05:00
Harrison Chase
a7084ad6e4
Harrison/version 0040 (#366) 2022-12-17 07:53:22 -08:00
mrbean
50257fce59
Support Streaming Tokens from OpenAI (#364)
https://github.com/hwchase17/langchain/issues/363

@hwchase17 how much does this make you want to cry?
2022-12-17 07:02:58 -08:00
mrbean
fe6695b9e7
Add HuggingFacePipeline LLM (#353)
https://github.com/hwchase17/langchain/issues/354

Add support for running your own HF pipeline locally. This would allow
you to get a lot more dynamic with what HF features and models you
support since you wouldn't be beholden to what is hosted in HF hub. You
could also do stuff with HF Optimum to quantize your models and stuff to
get pretty fast inference even running on a laptop.
2022-12-17 07:00:04 -08:00
Harrison Chase
9bb7195085
Harrison/llm saving (#331)
Co-authored-by: Akash Samant <70665700+asamant21@users.noreply.github.com>
2022-12-13 06:46:01 -08:00
Harrison Chase
3ca2c8d6c5
allow passing of stop params into openai (#232) 2022-11-30 22:20:13 -08:00
Harrison Chase
ca2394028f
move search to not be a chain (#226) 2022-11-29 20:07:44 -08:00
Andrew Gleave
ea67c049f0
Support SQL statements that return no results (#222)
Adds support for statements such as insert, update etc which do not
return any rows.

`engine.execute` is deprecated and so execution has been updated to use
`connection.exec_driver_sql` as-per:


https://docs.sqlalchemy.org/en/14/core/connections.html#sqlalchemy.engine.Engine.execute
2022-11-29 08:28:45 -08:00
Harrison Chase
1b9b8efbc9
pal chain (#207)
from https://arxiv.org/pdf/2211.10435.pdf
2022-11-28 21:38:34 -08:00
Harrison Chase
b94244eb12
nits (#210)
use json.dump

move test to integration tests (since it requires huggingface_hub)
2022-11-27 13:03:09 -08:00
Bagatur
b90e25f786
Add HuggingFace Hub Embeddings (#125)
Add support for calling HuggingFace embedding models
using the HuggingFaceHub Inference API. New class mirrors
the existing HuggingFaceHub LLM implementation. Currently
only supports 'sentence-transformers' models.

Closes #86
2022-11-27 00:24:59 -08:00
Harrison Chase
ae9c6257fe
Harrison/arbitrary params (#186) 2022-11-24 20:01:20 -08:00
Harrison Chase
d3a7429f61
(WIP) agents (#171) 2022-11-22 06:16:26 -08:00
Samantha Whitmore
315b0c09c6
wip: add method for both docstore and embeddings (#119)
this will break atm but wanted to get thoughts on implementation.

1. should add() be on docstore interface?
2. should InMemoryDocstore change to take a list of documents as init?
(makes this slightly easier to implement in FAISS -- if we think it is
less clean then could expose a method to get the number of documents
currently in the dict, and perform the logic of creating the necessary
dictionary in the FAISS.add_texts method.

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
2022-11-20 16:23:58 -08:00
Harrison Chase
c02eb199b6
add few shot example (#148) 2022-11-19 20:32:45 -08:00
Harrison Chase
9f223e6ccc
Harrison/fix lint (#138) 2022-11-14 08:55:59 -08:00
Delip Rao
76cecf8165
A fix for Jupyter environment variable issue (#135)
- fixes the Jupyter environment variable issues mentioned in issue #134 
- fixes format/lint issues in some unrelated files (from make
format/lint)


![image](https://user-images.githubusercontent.com/347398/201599322-090af858-362d-4d69-bf59-208aea65419a.png)
2022-11-14 08:34:01 -08:00
Harrison Chase
f23b3ceb49
consolidate run functions (#126)
consolidating logic for when a chain is able to run with single input
text, single output text

open to feedback on naming, logic, usefulness
2022-11-13 18:14:35 -08:00
Harrison Chase
d87e73ddb1
huggingface tokenizer (#75) 2022-11-13 09:37:44 -08:00
Harrison Chase
e43534d41c
add integration with manifest (#62) 2022-11-10 11:24:11 -08:00
tomeras91
d8734ce5ad
Add AI21 LLMs (#99)
Integrate AI21 /complete API into langchain, to allow access to Jurassic
models.
2022-11-10 08:12:28 -08:00
Samantha Whitmore
a0780cc930
OptimizedPrompt -- k-shot example choice backed by semantic search (#91) 2022-11-09 21:15:42 -08:00
Delip Rao
3ee6e332dd
Implements NLTK and Spacy-based TextSplitters (#103)
This PR is for Issue #88 

- [x] `make format`
- [x] `make lint`
- [x] `make tests`
2022-11-09 20:45:30 -08:00
issam9
28282ad099
Issam9/cohere embeddings (#105)
Add support for cohere embeddings
2022-11-09 13:44:27 -08:00
Delip Rao
95dd2f140e
Make Integration Tests "work" again (#106)
This fixes Issue #104 

The tests for HF Embeddings is skipped because of the segfault issue
mentioned there. Perhaps, a new issue should be created for that?
2022-11-09 13:26:58 -08:00
Harrison Chase
b9f61390e9
add text2text generation (#93)
fixes issue #90
2022-11-08 18:08:46 -08:00
Samantha Whitmore
efbc03bda8
NLPCloud client integration (#81)
lots of kwargs! generation docs here:
https://docs.nlpcloud.com/#generation

This somewhat breaks the paradigm introduced in LLM base class as the
stop sequence isn't a list, and should rightfully be introduced at the
time of initialization of the class, along with the other kwargs that
depend on its presence (e.g. remove_end_sequence, etc.) curious if you'd
want to refactor LLM base class to take out stop as a specific named
kwarg?
2022-11-08 06:24:23 -08:00
issam9
990cd821cc
Issam/hf embeddings (#68)
Add support of HuggingFace embedding models
2022-11-07 05:46:44 -08:00
Harrison Chase
76aff023d7
FAISS and embedding support (#48)
also adds embeddings and an in memory docstore
2022-11-01 21:29:39 -07:00
Harrison Chase
e982cf4b2e
Harrison/update docstore (#47)
change docstore interface
2022-10-31 21:18:52 -07:00
Harrison Chase
af81e9ca9c
add sql database (#35) 2022-10-27 23:21:47 -07:00
Harrison Chase
ce7b14b843
Harrison/add react chain (#24)
from https://arxiv.org/abs/2210.03629

still need to think if docstore abstraction makes sense
2022-10-26 21:02:23 -07:00
Harrison Chase
020c42dcae
Harrison/add huggingface hub (#23)
Add support for huggingface hub

I could not find a good way to enforce stop tokens over the huggingface
hub api - that needs to hopefully be cleaned up in the future
2022-10-25 22:00:33 -07:00
Harrison Chase
d2fdcba29d
fix test name (#22) 2022-10-25 20:22:16 -07:00
Harrison Chase
18aeb72012 initial commit 2022-10-24 14:51:15 -07:00