Commit Graph

73 Commits

Author SHA1 Message Date
Harrison Chase
7aba18ea77
Harrison/docs cleanup (#2633) 2023-04-09 12:55:22 -07:00
Davit Buniatyan
aaac7071a3
Deep Lake retriever example analyzing Twitter the-algorithm source code (#2602)
Improvements to Deep Lake Vector Store
- much faster view loading of embeddings after filters with
`fetch_chunks=True`
- 2x faster ingestion
- use np.float32 for embeddings to save 2x storage, LZ4 compression for
text and metadata storage (saves up to 4x storage for text data)
- user defined functions as filters

Docs
- Added retriever full example for analyzing twitter the-algorithm
source code with GPT4
- Added a use case for code analysis (please let us know your thoughts
how we can improve it)

---------

Co-authored-by: Davit Buniatyan <d@activeloop.ai>
2023-04-09 12:29:47 -07:00
Harrison Chase
a31c9511e8
Harrison/redis improvements (#2528)
Co-authored-by: Tyler Hutcherson <tyler.hutcherson@redis.com>
2023-04-06 23:21:22 -07:00
Harrison Chase
5c64b86ba3
Harrison/weaviate retriever (#2524)
Co-authored-by: Erika Cardenas <110841617+erika-cardenas@users.noreply.github.com>
2023-04-06 22:27:37 -07:00
Timon Ruban
f0926bad9f
Fix docstring in indexes/getting-started (#2452)
Fixed a letter. That's all.
2023-04-06 12:48:08 -07:00
Davit Buniatyan
b4914888a7
Deep Lake upgrade to include attribute search, distance metrics, returning scores and MMR (#2455)
### Features include

- Metadata based embedding search
- Choice of distance metric function (`L2` for Euclidean, `L1` for
Nuclear, `max` L-infinity distance, `cos` for cosine similarity, 'dot'
for dot product. Defaults to `L2`
- Returning scores
- Max Marginal Relevance Search
- Deleting samples from the dataset

### Notes
- Added numerous tests, let me know if you would like to shorten them or
make smarter

---------

Co-authored-by: Davit Buniatyan <d@activeloop.ai>
2023-04-06 12:47:33 -07:00
Sam Weaver
2ffb90b161
Extend opensearch to better support existing instances (#2500) (#2509)
Closes #2500.
2023-04-06 12:45:56 -07:00
Harrison Chase
00bc8df640
Harrison/tfidf retriever (#2440) 2023-04-05 07:36:49 -07:00
Harrison Chase
af7f20fa42
Harrison/elastic search (#2419) 2023-04-04 21:29:06 -07:00
Harrison Chase
41832042cc
Harrison/pinecone hybrid (#2405) 2023-04-04 14:09:57 -07:00
Harrison Chase
2b975de94d
add metal retriever (#2244) 2023-04-04 12:17:13 -07:00
Harrison Chase
e90d007db3
Harrison/msg files (#2375)
Co-authored-by: Sahil Masand <masand.sahil@gmail.com>
Co-authored-by: Sahil Masand <masands@cbh.com.au>
2023-04-04 06:48:34 -07:00
Kacper Łukawski
585f60a5aa
Qdrant update to 1.1.1 & docs polishing (#2388)
This PR updates Qdrant to 1.1.1 and introduces local mode, so there is
no need to spin up the Qdrant server. By that occasion, the Qdrant
example notebooks also got updated, covering more cases and answering
some commonly asked questions. All the Qdrant's integration tests were
switched to local mode, so no Docker container is required to launch
them.
2023-04-04 06:48:21 -07:00
Bhanu K
3fb4997ad8
Persist database regardless of notebook or script context (#2351)
`persist()` is required even if it's invoked in a script.

Without this, an error is thrown:

```
chromadb.errors.NoIndexException: Index is not initialized
```
2023-04-03 14:21:17 -07:00
Kevin Huang
e4cfaa5680
Introduces SeleniumURLLoader for JavaScript-Dependent Web Page Data Retrieval (#2291)
### Summary
This PR introduces a `SeleniumURLLoader` which, similar to
`UnstructuredURLLoader`, loads data from URLs. However, it utilizes
`selenium` to fetch page content, enabling it to work with
JavaScript-rendered pages. The `unstructured` library is also employed
for loading the HTML content.

### Testing
```bash
pip install selenium
pip install unstructured
```

```python
from langchain.document_loaders import SeleniumURLLoader

urls = [
    "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    "https://goo.gl/maps/NDSHwePEyaHMFGwh8"
]

loader = SeleniumURLLoader(urls=urls)
data = loader.load()
```
2023-04-02 14:05:00 -07:00
Ambuj Pawar
b4b7e8a54d
Fix typo in documentation: vectorstore-retriever.ipynb (#2306)
There is a typo in the documentation. 
Fixed it!
2023-04-02 13:48:05 -07:00
Frank Liu
134fc87e48
Add Zilliz example (#2288)
Add Zilliz example
2023-04-02 13:38:20 -07:00
Harrison Chase
e57b045402
bump version to 128 (#2236) 2023-03-31 11:16:21 -07:00
Harrison Chase
2eeaccf01c
Harrison/apify (#2215)
Co-authored-by: Jiří Moravčík <jiri.moravcik@gmail.com>
2023-03-30 20:58:14 -07:00
Alex Stachowiak
e6a9ee64b3
Update vectorstore-retriever.ipynb (#2210) 2023-03-30 20:51:46 -07:00
Matt Robinson
3dfe1cf60e
feat: document loader for epublications (#2202)
### Summary

Adds a new document loader for processing e-publications. Works with
`unstructured>=0.5.4`. You need to have
[`pandoc`](https://pandoc.org/installing.html) installed for this loader
to work.

### Testing

```python
from langchain.document_loaders import UnstructuredEPubLoader

loader = UnstructuredEPubLoader("winter-sports.epub", mode="elements")
data = loader.load()
data[0]
```
2023-03-30 20:45:31 -07:00
Ikko Eltociear Ashimine
a4a1ee6b5d
Update huggingface_length_function.ipynb (#2203)
HuggingFace -> Hugging Face
2023-03-30 20:43:58 -07:00
Harrison Chase
1c03205cc2
embedding docs (#2200) 2023-03-30 08:34:14 -07:00
Max Caldwell
3dc49a04a3
[Documents] Updated Figma docs and added example (#2172)
- Current docs are pointing to the wrong module, fixed
- Added some explanation on how to find the necessary parameters
- Added chat-based codegen example w/ retrievers

Picture of the new page:
![Screenshot 2023-03-29 at 20-11-29 Figma — 🦜🔗 LangChain 0 0
126](https://user-images.githubusercontent.com/2172753/228719338-c7ec5b11-01c2-4378-952e-38bc809f217b.png)

Please let me know if you'd like any tweaks! I wasn't sure if the
example was too heavy for the page or not but decided "hey, I probably
would want to see it" and so included it.

Co-authored-by: maxtheman <max@maxs-mbp.lan>
2023-03-29 22:11:45 -07:00
Chase Adams
b5449a866d
docs: tiny fix on docs verbiage (#2124)
Changed `RecursiveCharaterTextSplitter` =>
`RecursiveCharacterTextSplitter`. GH's diff doesn't handle the long
string well.
2023-03-28 22:56:29 -07:00
Harrison Chase
a5bf8c9b9d
Harrison/aleph alpha embeddings (#2117)
Co-authored-by: Piotr Mazurek <piotr635@gmail.com>
Co-authored-by: PiotrMazurek <piotr.mazurek@aleph-alpha.com>
2023-03-28 15:18:03 -07:00
Harrison Chase
410bf37fb8
Harrison/big query (#2100)
Co-authored-by: lu-cashmoney <lucas.corley@gmail.com>
2023-03-28 08:17:22 -07:00
Stéphane Busso
0bee219cb3
feat: Add Notion database document loader (#2056)
This PR adds Notion DB loader for langchain. 

It reads content from pages within a Notion Database. It uses the Notion
API to query the database and read the pages. It also reads the metadata
from the pages and stores it in the Document object.
2023-03-28 08:07:09 -07:00
Harrison Chase
d5825bd3e8
Harrison/whatsapp loader (#2085)
Co-authored-by: Moshe <hello@moshemalka.me>
2023-03-27 23:43:45 -07:00
Harrison Chase
f74a1bebf5
Harrison/duckdb (#2064)
Co-authored-by: Trent Hauck <trent@trenthauck.com>
2023-03-27 19:51:34 -07:00
Harrison Chase
76ecca4d53
redis retriever (#2060) 2023-03-27 19:51:23 -07:00
Harrison Chase
30e3b31b04
Harrison/document cleanup (#2062)
Co-authored-by: Delip Rao <delip@users.noreply.github.com>
2023-03-27 16:32:55 -07:00
Harrison Chase
a0cd6672aa
Harrison/site map (#2061)
Co-authored-by: Tim Asp <707699+timothyasp@users.noreply.github.com>
2023-03-27 16:28:08 -07:00
Harrison Chase
705431aecc
big docs refactor (#1978)
Co-authored-by: Ankush Gola <ankush.gola@gmail.com>
2023-03-26 19:49:46 -07:00
Harrison Chase
6ec5780547
add docs for openai retriever ingest (#1969) 2023-03-24 08:24:33 -07:00
Harrison Chase
47d37db2d2
WIP: Harrison/base retriever (#1765) 2023-03-24 07:46:49 -07:00
Harrison Chase
8990122d5d
retrievers interface (#1948) 2023-03-23 19:00:38 -07:00
Harrison Chase
50626a10ee
Hx23840 feat/add redisearch vectorstore (#1909)
Co-authored-by: Peter <peter.shi@alephf.com>
Co-authored-by: Peter Shi <42536066+hx23840@users.noreply.github.com>
2023-03-22 19:57:56 -07:00
Sean Zheng
15b5a08f4b
Update how_to_guides.rst (#1893)
Adding OpenSearch examples
2023-03-22 14:30:43 -07:00
Harrison Chase
f299bd1416
clean up sagemaker nb (#1875) 2023-03-21 22:00:08 -07:00
Philipp Schmid
064be93edf
[Embeddings] Add SageMaker Endpoint Embedding class (#1859)
# What does this PR do? 

This PR adds similar to `llms` a SageMaker-powered `embeddings` class.
This is helpful if you want to leverage Hugging Face models on SageMaker
for creating your indexes.

I added a example into the
[docs/modules/indexes/examples/embeddings.ipynb](https://github.com/hwchase17/langchain/compare/master...philschmid:add-sm-embeddings?expand=1#diff-e82629e2894974ec87856aedd769d4bdfe400314b03734f32bee5990bc7e8062)
document. The example currently includes some `_### TEMPORARY: Showing
how to deploy a SageMaker Endpoint from a Hugging Face model ###_ ` code
showing how you can deploy a sentence-transformers to SageMaker and then
run the methods of the embeddings class.

@hwchase17 please let me know if/when i should remove the `_###
TEMPORARY: Showing how to deploy a SageMaker Endpoint from a Hugging
Face model ###_` in the description i linked to a detail blog on how to
deploy a Sentence Transformers so i think we don't need to include those
steps here.

I also reused the `ContentHandlerBase` from
`langchain.llms.sagemaker_endpoint` and changed the output type to `any`
since it is depending on the implementation.
2023-03-21 21:51:48 -07:00
anupam-tiwari
86822d1cc2
Fixes the import typo in the vector db text generator notebook (#1874)
Fixes the import typo in the vector db text generator notebook for the
chroma library

Co-authored-by: Anupam <anupam@10-16-252-145.dynapool.wireless.nyu.edu>
2023-03-21 21:48:26 -07:00
Tomoko Uchida
b706966ebc
Add setup instruction in Getting Started for Indexing (#1847)
`VectorstoreIndexCreator` [uses Chroma as the vectorstore by
default](1c22657256/langchain/indexes/vectorstore.py (L49)).
It may be helpful to add a short note for the setup.

You can see how the notebook looks here.

https://github.com/mocobeta/langchain/blob/feat/add-setup-instruction-to-index-getting-started/docs/modules/indexes/getting_started.ipynb
2023-03-21 09:06:35 -07:00
Harrison Chase
1c22657256
Harrison/faiss merge (#1843)
Co-authored-by: Ting Su <ting.su.1995@outlook.com>
2023-03-20 22:54:08 -07:00
Harrison Chase
d5d50c39e6
Harrison/azure embeddings (#1787)
Co-authored-by: Hemant <4627288+ghaccount@users.noreply.github.com>
2023-03-19 10:42:33 -07:00
Harrison Chase
96ebe98dc2
Harrison/latex splitter (#1738)
Co-authored-by: Aidan Holland <thehappydinoa@gmail.com>
Co-authored-by: Jan de Boer <44832123+Janldeboer@users.noreply.github.com>
2023-03-17 08:10:27 -07:00
Piyush Jain
1279c8de39
Fixed typo, clarified language (#1682) 2023-03-15 08:00:11 -07:00
Jithin James
6f4f771897
docs: add path to state_of_the_union.txt in indexes/getting_started page (#1691)
add the state_of_the_union.txt file so that its easier to follow through
with the example.

---------

Co-authored-by: Jithin James <jjmachan@pop-os.localdomain>
2023-03-15 07:59:47 -07:00
Harrison Chase
0b29e68c17
Harrison/pgvector (#1679)
Co-authored-by: Aman Kumar <krsingh.aman@gmail.com>
2023-03-14 21:13:58 -07:00
Harrison Chase
4d7fdb8957
Harrison/gml save (#1676)
Co-authored-by: Satoru Sakamoto <51464932+satoru814@users.noreply.github.com>
2023-03-14 20:00:22 -07:00