Commit Graph

68 Commits

Author SHA1 Message Date
Idan
897b4ef2cd Fixed a bug with reading md files 2023-06-23 14:57:29 +03:00
Anton Larin
84168e22d0 add missing variable after testin and minor fixes. 2023-06-17 16:09:22 +02:00
Alex
f6b6c2e9a3 Update requirements.txt 2023-06-16 01:36:45 +01:00
dependabot[bot]
8a98789be1
Bump transformers from 4.26.0 to 4.30.0 in /scripts
Bumps [transformers](https://github.com/huggingface/transformers) from 4.26.0 to 4.30.0.
- [Release notes](https://github.com/huggingface/transformers/releases)
- [Commits](https://github.com/huggingface/transformers/compare/v4.26.0...v4.30.0)

---
updated-dependencies:
- dependency-name: transformers
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-06-15 16:53:39 +00:00
Anton Larin
6d5b698c39 fix arc53/DocsGPT#199 2023-06-03 11:04:04 +02:00
Anton Larin
dd9f1abcea fix arc53/DocsGPT#199 2023-06-03 11:03:44 +02:00
Anton Larin
b4bd34fb96 fix arc53/DocsGPT#199 2023-06-03 10:58:31 +02:00
Nazih Kalo
da5d62cc1c updating the bulk ingest file metadata to account for parsers that output lists 2023-05-19 10:29:18 -07:00
Alex
e49dd0cc6a metadata on ingestion 2023-05-17 21:41:24 +01:00
Alex
059ffe09ea
Merge pull request #232 from larinam/lint
Lint
2023-05-15 13:53:09 +01:00
GH Action - Upstream Sync
ce6f0dab56 Merge branch 'main' of https://github.com/arc53/DocsGPT 2023-05-15 12:05:18 +00:00
Alex
3001688e0e Update requirements.txt 2023-05-15 12:46:39 +01:00
dependabot[bot]
b28676d52c
Bump flask from 2.2.2 to 2.2.5 in /scripts
Bumps [flask](https://github.com/pallets/flask) from 2.2.2 to 2.2.5.
- [Release notes](https://github.com/pallets/flask/releases)
- [Changelog](https://github.com/pallets/flask/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/flask/compare/2.2.2...2.2.5)

---
updated-dependencies:
- dependency-name: flask
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-05-15 11:00:19 +00:00
Anton Larin
962becb9a5
Linting
* validate python formatting on every build with Ruff
* fix lint warnings
2023-05-13 10:36:17 +02:00
Anton Larin
168648e789 Proper PEP8 formatting 2023-05-12 12:02:25 +02:00
dependabot[bot]
80dfdd1cb9
Bump redis from 4.5.3 to 4.5.4 in /scripts
Bumps [redis](https://github.com/redis/redis-py) from 4.5.3 to 4.5.4.
- [Release notes](https://github.com/redis/redis-py/releases)
- [Changelog](https://github.com/redis/redis-py/blob/master/CHANGES)
- [Commits](https://github.com/redis/redis-py/compare/v4.5.3...v4.5.4)

---
updated-dependencies:
- dependency-name: redis
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-30 11:49:21 +00:00
dependabot[bot]
b7f1a94ba4
Bump redis from 4.5.1 to 4.5.3 in /scripts
Bumps [redis](https://github.com/redis/redis-py) from 4.5.1 to 4.5.3.
- [Release notes](https://github.com/redis/redis-py/releases)
- [Changelog](https://github.com/redis/redis-py/blob/master/CHANGES)
- [Commits](https://github.com/redis/redis-py/compare/v4.5.1...v4.5.3)

---
updated-dependencies:
- dependency-name: redis
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-27 21:34:50 +00:00
Pavel
ce8f0ef9e1
Merge pull request #168 from arc53/feature/backend-uploads
Feature/backend uploads
2023-03-14 19:09:37 +04:00
Pavel
c9e1c326f5 - index.plk 2023-03-14 17:56:42 +04:00
Pavel
4532b6cd8c print minus 2023-03-14 17:49:57 +04:00
Pavel
53424a5c19 Added cli commands 2023-03-14 17:33:19 +04:00
Pavel
b6c02c850a token ingeest 2023-03-14 13:32:29 +04:00
Alex
20a0800aa7 Create test_ingestion.py 2023-03-13 17:37:01 +00:00
Pavel
bac25112b7 v1 2023-03-13 19:14:33 +04:00
Alex
1d2162705d uploads backend first 2023-03-13 14:20:03 +00:00
Alex
ac0224b687 mdx format 2023-03-08 23:16:20 +00:00
Alex
0799728000 Create requirements.txt 2023-03-08 11:44:02 +00:00
Alex
1f02f3b376 Update rst_parser.py 2023-03-08 11:32:44 +00:00
Alex
f7d7244588 chunks rst 2023-03-08 00:07:53 +00:00
Alex
63e35fbdbc env vars 2023-03-01 14:16:11 +00:00
Alex
1af4ca2340
Merge pull request #129 from arc53/code-ingestion
Code_to_dict
2023-02-25 13:52:14 +00:00
Pavel
2c364d3c00 Code_to_dict
3 languages added, works well with python. Java and Js require additional revieving
2023-02-25 17:37:33 +04:00
Alex
5c2a537393
Merge pull request #116 from arc53/code-ingestion
Code ingestion
2023-02-22 18:46:50 +00:00
Pavel
0fb28e5213 Calc + structure 2023-02-22 21:19:13 +04:00
Manan
524e0f6f01 fix | Chunk creation error when
title not the first element in HTML
2023-02-22 20:20:54 +05:30
Manan
16eb503e36 Added HTML Support. read, clean-up, filter return 2023-02-21 23:06:00 +05:30
Manan
e8baa46eb6 Merge branch 'main' of https://github.com/arc53/DocsGPT into main 2023-02-21 22:11:57 +05:30
Alex
c92f5dba32 New docs gen 2023-02-20 17:15:18 +00:00
Alex
962be4d8ec
Merge pull request #109 from arc53/main
updates
2023-02-20 16:53:15 +00:00
Manan
d0b472ad38 Implemented html_parser: cleaning & chunk creation 2023-02-19 01:53:16 +05:30
Alex
8a5e1e8d98 cleanups 2023-02-15 20:29:09 +00:00
Alex
4d1ff8238d switching between llms 2023-02-15 18:40:23 +00:00
Alex
f9fe3f2f48 Merge branch 'main' into custom-llm 2023-02-15 14:42:57 +00:00
EricGao888
aeac186484 Add retry strategy to increase stability 2023-02-15 17:29:39 +08:00
Pavel
d57c7b0296 -y-description 2023-02-15 13:10:30 +04:00
冯不游
b83589a308 feat: add support for directory list
example: `python ingest.py --dir inputs1 --dir another --dir ../inputs`,
the outputs will be in `outputs/input_folder_name/`
2023-02-15 02:30:39 +08:00
Alex
d4ede13747 Merge branch 'main' into code-ingestion 2023-02-14 16:47:56 +00:00
Alex
5883ce2685
Merge pull request #87 from arc53/ingest-cli
Ingest cli
2023-02-14 16:20:05 +00:00
Pavel
af20c7298a new-ingest
Ingest with a CLI
2023-02-14 19:37:07 +04:00
冯不游
636783ca8a fix: avoid second error issue 2023-02-14 22:29:17 +08:00