Compare commits

..

No commits in common. 'main' and '0.9.0' have entirely different histories.
main ... 0.9.0

@ -1,15 +0,0 @@
# To get started with Dependabot version updates, you'll need to specify which
# package ecosystems to update and where the package manifests are located.
# Please see the documentation for all configuration options:
# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
version: 2
updates:
- package-ecosystem: "pip" # See documentation for possible values
directory: "/application" # Location of package manifests
schedule:
interval: "weekly"
- package-ecosystem: "npm" # See documentation for possible values
directory: "/frontend" # Location of package manifests
schedule:
interval: "weekly"

@ -13,6 +13,7 @@ jobs:
permissions: permissions:
contents: read contents: read
packages: write packages: write
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v3
@ -35,6 +36,7 @@ jobs:
username: ${{ github.repository_owner }} username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }} password: ${{ secrets.GITHUB_TOKEN }}
# Runs a single command using the runners shell
- name: Build and push Docker images to docker.io and ghcr.io - name: Build and push Docker images to docker.io and ghcr.io
uses: docker/build-push-action@v4 uses: docker/build-push-action@v4
with: with:

@ -8,11 +8,11 @@ on:
jobs: jobs:
deploy: deploy:
if: github.repository == 'arc53/DocsGPT'
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions: permissions:
contents: read contents: read
packages: write packages: write
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v3
@ -40,7 +40,7 @@ jobs:
uses: docker/build-push-action@v4 uses: docker/build-push-action@v4
with: with:
file: './frontend/Dockerfile' file: './frontend/Dockerfile'
platforms: linux/amd64, linux/arm64 platforms: linux/amd64
context: ./frontend context: ./frontend
push: true push: true
tags: | tags: |

@ -4,7 +4,6 @@ on:
- pull_request_target - pull_request_target
jobs: jobs:
triage: triage:
if: github.repository == 'arc53/DocsGPT'
permissions: permissions:
contents: read contents: read
pull-requests: write pull-requests: write

@ -6,7 +6,7 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
strategy: strategy:
matrix: matrix:
python-version: ["3.11"] python-version: ["3.9", "3.10", "3.11"]
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }} - name: Set up Python ${{ matrix.python-version }}
@ -21,7 +21,7 @@ jobs:
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Test with pytest and generate coverage report - name: Test with pytest and generate coverage report
run: | run: |
python -m pytest --cov=application --cov-report=xml python -m pytest --cov=application --cov=scripts --cov=extensions --cov-report=xml
- name: Upload coverage reports to Codecov - name: Upload coverage reports to Codecov
if: github.event_name == 'pull_request' && matrix.python-version == '3.11' if: github.event_name == 'pull_request' && matrix.python-version == '3.11'
uses: codecov/codecov-action@v3 uses: codecov/codecov-action@v3

2
.gitignore vendored

@ -172,5 +172,5 @@ application/vectors/
node_modules/ node_modules/
.vscode/settings.json .vscode/settings.json
/models/ models/
model/ model/

@ -1,36 +0,0 @@
# **🎉 Join the Hacktoberfest with DocsGPT and win a Free T-shirt and other prizes! 🎉**
Welcome, contributors! We're excited to announce that DocsGPT is participating in Hacktoberfest. Get involved by submitting meaningful pull requests.
All contributors with accepted PRs will receive a cool Holopin! 🤩 (Watch out for a reply in your PR to collect it).
### 🏆 Top 50 contributors will recieve a special T-shirt
### 🏆 LLM Document analysis by LexEU competition:
A separate competition is available for those sumbit best new retrieval method that will analyze a Document using EU laws.
You will find more information about it on 1st of October.
## 📜 Here's How to Contribute:
```text
🛠️ Code: This is the golden ticket! Make meaningful contributions through PRs.
🧩 API extention: Build an app utilising DocsGPT API. We prefer submissions that showcase original ideas and turn the API into an AI agent.
Non-Code Contributions:
📚 Wiki: Improve our documentation, Create a guide or change existing documentation.
🖥️ Design: Improve the UI/UX or design a new feature.
📝 Blogging or Content Creation: Write articles or create videos to showcase DocsGPT or highlight your contributions!
```
### 📝 Guidelines for Pull Requests:
- Familiarize yourself with the current contributions and our [Roadmap](https://github.com/orgs/arc53/projects/2).
- Before contributing we highly advise that you check existing [issues](https://github.com/arc53/DocsGPT/issues) or [create](https://github.com/arc53/DocsGPT/issues/new/choose) an issue and wait to get assigned.
- Once you are finished with your contribution, please fill in this [form](https://airtable.com/appikMaJwdHhC1SDP/pagoblCJ9W29wf6Hf/form).
- Refer to the [Documentation](https://docs.docsgpt.cloud/).
- Feel free to join our [Discord](https://discord.gg/n5BX8dh8rU) server. We're here to help newcomers, so don't hesitate to jump in! Join us [here](https://discord.gg/n5BX8dh8rU).
Thank you very much for considering contributing to DocsGPT during Hacktoberfest! 🙏 Your contributions (not just simple typo) could earn you a stylish new t-shirt and other prizes as a token of our appreciation. 🎁 Join us, and let's code together! 🚀

@ -7,9 +7,9 @@
</p> </p>
<p align="left"> <p align="left">
<strong><a href="https://www.docsgpt.cloud/">DocsGPT</a></strong> is a cutting-edge open-source solution that streamlines the process of finding information in the project documentation. With its integration of the powerful <strong>GPT</strong> models, developers can easily ask questions about a project and receive accurate answers. <strong><a href="https://docsgpt.arc53.com/">DocsGPT</a></strong> is a cutting-edge open-source solution that streamlines the process of finding information in the project documentation. With its integration of the powerful <strong>GPT</strong> models, developers can easily ask questions about a project and receive accurate answers.
Say goodbye to time-consuming manual searches, and let <strong><a href="https://www.docsgpt.cloud/">DocsGPT</a></strong> help you quickly find the information you need. Try it out and see how it revolutionizes your project documentation experience. Contribute to its development and be a part of the future of AI-powered assistance. Say goodbye to time-consuming manual searches, and let <strong><a href="https://docsgpt.arc53.com/">DocsGPT</a></strong> help you quickly find the information you need. Try it out and see how it revolutionizes your project documentation experience. Contribute to its development and be a part of the future of AI-powered assistance.
</p> </p>
<div align="center"> <div align="center">
@ -23,13 +23,11 @@ Say goodbye to time-consuming manual searches, and let <strong><a href="https://
</div> </div>
### 🎃 [Hacktoberfest Prizes, Rules & Q&A](https://github.com/arc53/DocsGPT/blob/main/HACKTOBERFEST.md) 🎃
### Production Support / Help for Companies: ### Production Support / Help for Companies:
We're eager to provide personalized assistance when deploying your DocsGPT to a live environment. We're eager to provide personalized assistance when deploying your DocsGPT to a live environment.
- [Book Enterprise / teams Demo :wave:](https://cal.com/arc53/docsgpt-demo-b2b?date=2024-09-27&month=2024-09) - [Book Demo :wave:](https://airtable.com/appdeaL0F1qV8Bl2C/shrrJF1Ll7btCJRbP)
- [Send Email :email:](mailto:contact@arc53.com?subject=DocsGPT%20support%2Fsolutions) - [Send Email :email:](mailto:contact@arc53.com?subject=DocsGPT%20support%2Fsolutions)
![video-example-of-docs-gpt](https://d3dg1063dc54p9.cloudfront.net/videos/demov3.gif) ![video-example-of-docs-gpt](https://d3dg1063dc54p9.cloudfront.net/videos/demov3.gif)
@ -48,23 +46,23 @@ You can find our roadmap [here](https://github.com/orgs/arc53/projects/2). Pleas
If you don't have enough resources to run it, you can use bitsnbytes to quantize. If you don't have enough resources to run it, you can use bitsnbytes to quantize.
## End to End AI Framework for Information Retrieval ## Features
![Architecture chart](https://github.com/user-attachments/assets/fc6a7841-ddfc-45e6-b5a0-d05fe648cbe2) ![Main features of DocsGPT showcasing six main features](https://user-images.githubusercontent.com/17906039/220427472-2644cff4-7666-46a5-819f-fc4a521f63c7.png)
## Useful Links ## Useful Links
- :mag: :fire: [Cloud Version](https://app.docsgpt.cloud/) - :mag: :fire: [Live preview](https://docsgpt.arc53.com/)
- :speech_balloon: :tada: [Join our Discord](https://discord.gg/n5BX8dh8rU) - :speech_balloon: :tada: [Join our Discord](https://discord.gg/n5BX8dh8rU)
- :books: :sunglasses: [Guides](https://docs.docsgpt.cloud/) - :books: :sunglasses: [Guides](https://docs.docsgpt.co.uk/)
- :couple: [Interested in contributing?](https://github.com/arc53/DocsGPT/blob/main/CONTRIBUTING.md) - :couple: [Interested in contributing?](https://github.com/arc53/DocsGPT/blob/main/CONTRIBUTING.md)
- :file_folder: :rocket: [How to use any other documentation](https://docs.docsgpt.cloud/Guides/How-to-train-on-other-documentation) - :file_folder: :rocket: [How to use any other documentation](https://docs.docsgpt.co.uk/Guides/How-to-train-on-other-documentation)
- :house: :closed_lock_with_key: [How to host it locally (so all data will stay on-premises)](https://docs.docsgpt.cloud/Guides/How-to-use-different-LLM) - :house: :closed_lock_with_key: [How to host it locally (so all data will stay on-premises)](https://docs.docsgpt.co.uk/Guides/How-to-use-different-LLM)
## Project Structure ## Project Structure
@ -87,7 +85,7 @@ On Mac OS or Linux, write:
It will install all the dependencies and allow you to download the local model, use OpenAI or use our LLM API. It will install all the dependencies and allow you to download the local model, use OpenAI or use our LLM API.
Otherwise, refer to this Guide for Windows: Otherwise, refer to this Guide:
1. Download and open this repository with `git clone https://github.com/arc53/DocsGPT.git` 1. Download and open this repository with `git clone https://github.com/arc53/DocsGPT.git`
2. Create a `.env` file in your root directory and set the env variables and `VITE_API_STREAMING` to true or false, depending on whether you want streaming answers or not. 2. Create a `.env` file in your root directory and set the env variables and `VITE_API_STREAMING` to true or false, depending on whether you want streaming answers or not.

@ -1,88 +1,31 @@
# Builder Stage FROM python:3.11-slim-bullseye as builder
FROM ubuntu:24.04 as builder
# Tiktoken requires Rust toolchain, so build it in a separate stage
ENV DEBIAN_FRONTEND=noninteractive RUN apt-get update && apt-get install -y gcc curl
RUN apt-get install -y wget unzip
RUN apt-get update && \ RUN wget https://d3dg1063dc54p9.cloudfront.net/models/embeddings/mpnet-base-v2.zip
apt-get install -y software-properties-common && \ RUN unzip mpnet-base-v2.zip -d model
add-apt-repository ppa:deadsnakes/ppa && \ RUN rm mpnet-base-v2.zip
# Install necessary packages and Python RUN curl https://sh.rustup.rs -sSf | sh -s -- -y && apt-get install --reinstall libc6-dev -y
apt-get update && \ ENV PATH="/root/.cargo/bin:${PATH}"
apt-get install -y --no-install-recommends gcc wget unzip libc6-dev python3.11 python3.11-distutils python3.11-venv && \ RUN pip install --upgrade pip && pip install tiktoken==0.5.2
rm -rf /var/lib/apt/lists/*
# Verify Python installation and setup symlink
RUN if [ -f /usr/bin/python3.11 ]; then \
ln -s /usr/bin/python3.11 /usr/bin/python; \
else \
echo "Python 3.11 not found"; exit 1; \
fi
# Download and unzip the model
RUN wget https://d3dg1063dc54p9.cloudfront.net/models/embeddings/mpnet-base-v2.zip && \
unzip mpnet-base-v2.zip -d model && \
rm mpnet-base-v2.zip
# Install Rust
RUN wget -q -O - https://sh.rustup.rs | sh -s -- -y
# Clean up to reduce container size
RUN apt-get remove --purge -y wget unzip && apt-get autoremove -y && rm -rf /var/lib/apt/lists/*
# Copy requirements.txt
COPY requirements.txt . COPY requirements.txt .
RUN pip install -r requirements.txt
# Setup Python virtual environment
RUN python3.11 -m venv /venv
# Activate virtual environment and install Python packages
ENV PATH="/venv/bin:$PATH"
# Install Python packages FROM python:3.11-slim-bullseye
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir tiktoken && \
pip install --no-cache-dir -r requirements.txt
# Final Stage # Copy pre-built packages and binaries from builder stage
FROM ubuntu:24.04 as final COPY --from=builder /usr/local/ /usr/local/
RUN apt-get update && \
apt-get install -y software-properties-common && \
add-apt-repository ppa:deadsnakes/ppa && \
# Install Python
apt-get update && apt-get install -y --no-install-recommends python3.11 && \
ln -s /usr/bin/python3.11 /usr/bin/python && \
rm -rf /var/lib/apt/lists/*
# Set working directory
WORKDIR /app WORKDIR /app
# Create a non-root user: `appuser` (Feel free to choose a name)
RUN groupadd -r appuser && \
useradd -r -g appuser -d /app -s /sbin/nologin -c "Docker image user" appuser
# Copy the virtual environment and model from the builder stage
COPY --from=builder /venv /venv
COPY --from=builder /model /app/model COPY --from=builder /model /app/model
# Copy your application code
COPY . /app/application COPY . /app/application
ENV FLASK_APP=app.py
ENV FLASK_DEBUG=true
# Change the ownership of the /app directory to the appuser
RUN mkdir -p /app/application/inputs/local
RUN chown -R appuser:appuser /app
# Set environment variables
ENV FLASK_APP=app.py \
FLASK_DEBUG=true \
PATH="/venv/bin:$PATH"
# Expose the port the app runs on
EXPOSE 7091 EXPOSE 7091
# Switch to non-root user
USER appuser
# Start Gunicorn
CMD ["gunicorn", "-w", "2", "--timeout", "120", "--bind", "0.0.0.0:7091", "application.wsgi:app"] CMD ["gunicorn", "-w", "2", "--timeout", "120", "--bind", "0.0.0.0:7091", "application.wsgi:app"]

@ -1,7 +1,6 @@
import asyncio import asyncio
import os import os
import sys from flask import Blueprint, request, Response
from flask import Blueprint, request, Response, current_app
import json import json
import datetime import datetime
import logging import logging
@ -9,22 +8,22 @@ import traceback
from pymongo import MongoClient from pymongo import MongoClient
from bson.objectid import ObjectId from bson.objectid import ObjectId
from bson.dbref import DBRef
from application.core.settings import settings from application.core.settings import settings
from application.llm.llm_creator import LLMCreator from application.llm.llm_creator import LLMCreator
from application.retriever.retriever_creator import RetrieverCreator from application.retriever.retriever_creator import RetrieverCreator
from application.error import bad_request from application.error import bad_request
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
mongo = MongoClient(settings.MONGO_URI) mongo = MongoClient(settings.MONGO_URI)
db = mongo["docsgpt"] db = mongo["docsgpt"]
conversations_collection = db["conversations"] conversations_collection = db["conversations"]
sources_collection = db["sources"] vectors_collection = db["vectors"]
prompts_collection = db["prompts"] prompts_collection = db["prompts"]
api_key_collection = db["api_keys"] api_key_collection = db["api_keys"]
user_logs_collection = db["user_logs"]
answer = Blueprint("answer", __name__) answer = Blueprint("answer", __name__)
gpt_model = "" gpt_model = ""
@ -76,29 +75,25 @@ def run_async_chain(chain, question, chat_history):
def get_data_from_api_key(api_key): def get_data_from_api_key(api_key):
data = api_key_collection.find_one({"key": api_key}) data = api_key_collection.find_one({"key": api_key})
# # Raise custom exception if the API key is not found
if data is None: if data is None:
raise Exception("Invalid API Key, please generate new key", 401) return bad_request(401, "Invalid API key")
if "retriever" not in data:
data["retriever"] = None
if "source" in data and isinstance(data["source"], DBRef):
source_doc = db.dereference(data["source"])
data["source"] = str(source_doc["_id"])
if "retriever" in source_doc:
data["retriever"] = source_doc["retriever"]
else:
data["source"] = {}
return data return data
def get_retriever(source_id: str): def get_vectorstore(data):
doc = sources_collection.find_one({"_id": ObjectId(source_id)}) if "active_docs" in data:
if doc is None: if data["active_docs"].split("/")[0] == "default":
raise Exception("Source document does not exist", 404) vectorstore = ""
retriever_name = None if "retriever" not in doc else doc["retriever"] elif data["active_docs"].split("/")[0] == "local":
return retriever_name vectorstore = "indexes/" + data["active_docs"]
else:
vectorstore = "vectors/" + data["active_docs"]
if data["active_docs"] == "default":
vectorstore = ""
else:
vectorstore = ""
vectorstore = os.path.join("application", vectorstore)
return vectorstore
def is_azure_configured(): def is_azure_configured():
@ -176,181 +171,101 @@ def get_prompt(prompt_id):
return prompt return prompt
def complete_stream( def complete_stream(question, retriever, conversation_id, user_api_key):
question, retriever, conversation_id, user_api_key, isNoneDoc=False
):
try: response_full = ""
response_full = "" source_log_docs = []
source_log_docs = [] answer = retriever.gen()
answer = retriever.gen() for line in answer:
sources = retriever.search() if "answer" in line:
for source in sources: response_full += str(line["answer"])
if("text" in source): data = json.dumps(line)
source["text"] = source["text"][:100].strip()+"..."
if(len(sources) > 0):
data = json.dumps({"type":"source","source":sources})
yield f"data: {data}\n\n" yield f"data: {data}\n\n"
for line in answer: elif "source" in line:
if "answer" in line: source_log_docs.append(line["source"])
response_full += str(line["answer"])
data = json.dumps(line)
yield f"data: {data}\n\n"
elif "source" in line:
source_log_docs.append(line["source"])
if isNoneDoc:
for doc in source_log_docs:
doc["source"] = "None"
llm = LLMCreator.create_llm( llm = LLMCreator.create_llm(
settings.LLM_NAME, api_key=settings.API_KEY, user_api_key=user_api_key settings.LLM_NAME, api_key=settings.API_KEY, user_api_key=user_api_key
) )
if user_api_key is None: conversation_id = save_conversation(
conversation_id = save_conversation( conversation_id, question, response_full, source_log_docs, llm
conversation_id, question, response_full, source_log_docs, llm )
)
# send data.type = "end" to indicate that the stream has ended as json
data = json.dumps({"type": "id", "id": str(conversation_id)})
yield f"data: {data}\n\n"
retriever_params = retriever.get_params() # send data.type = "end" to indicate that the stream has ended as json
user_logs_collection.insert_one( data = json.dumps({"type": "id", "id": str(conversation_id)})
{ yield f"data: {data}\n\n"
"action": "stream_answer", data = json.dumps({"type": "end"})
"level": "info", yield f"data: {data}\n\n"
"user": "local",
"api_key": user_api_key,
"question": question,
"response": response_full,
"sources": source_log_docs,
"retriever_params": retriever_params,
"timestamp": datetime.datetime.now(datetime.timezone.utc),
}
)
data = json.dumps({"type": "end"})
yield f"data: {data}\n\n"
except Exception as e:
print("\033[91merr", str(e), file=sys.stderr)
data = json.dumps(
{
"type": "error",
"error": "Please try again later. We apologize for any inconvenience.",
"error_exception": str(e),
}
)
yield f"data: {data}\n\n"
return
@answer.route("/stream", methods=["POST"]) @answer.route("/stream", methods=["POST"])
def stream(): def stream():
try: data = request.get_json()
data = request.get_json() # get parameter from url question
question = data["question"] question = data["question"]
if "history" not in data: if "history" not in data:
history = [] history = []
else: else:
history = data["history"] history = data["history"]
history = json.loads(history) history = json.loads(history)
if "conversation_id" not in data: if "conversation_id" not in data:
conversation_id = None conversation_id = None
else: else:
conversation_id = data["conversation_id"] conversation_id = data["conversation_id"]
if "prompt_id" in data: if "prompt_id" in data:
prompt_id = data["prompt_id"] prompt_id = data["prompt_id"]
else: else:
prompt_id = "default" prompt_id = "default"
if "selectedDocs" in data and data["selectedDocs"] is None: if "selectedDocs" in data and data["selectedDocs"] is None:
chunks = 0 chunks = 0
elif "chunks" in data: elif "chunks" in data:
chunks = int(data["chunks"]) chunks = int(data["chunks"])
else: else:
chunks = 2 chunks = 2
if "token_limit" in data:
token_limit = data["token_limit"]
else:
token_limit = settings.DEFAULT_MAX_HISTORY
## retriever can be "brave_search, duckduck_search or classic"
retriever_name = data["retriever"] if "retriever" in data else "classic"
# check if active_docs or api_key is set prompt = get_prompt(prompt_id)
if "api_key" in data:
data_key = get_data_from_api_key(data["api_key"])
chunks = int(data_key["chunks"])
prompt_id = data_key["prompt_id"]
source = {"active_docs": data_key["source"]}
retriever_name = data_key["retriever"] or retriever_name
user_api_key = data["api_key"]
elif "active_docs" in data: # check if active_docs is set
source = {"active_docs": data["active_docs"]}
retriever_name = get_retriever(data["active_docs"]) or retriever_name
user_api_key = None
else: if "api_key" in data:
source = {} data_key = get_data_from_api_key(data["api_key"])
user_api_key = None source = {"active_docs": data_key["source"]}
user_api_key = data["api_key"]
elif "active_docs" in data:
source = {"active_docs": data["active_docs"]}
user_api_key = None
else:
source = {}
user_api_key = None
current_app.logger.info( if (
f"/stream - request_data: {data}, source: {source}", source["active_docs"].split("/")[0] == "default"
extra={"data": json.dumps({"request_data": data, "source": source})}, or source["active_docs"].split("/")[0] == "local"
) ):
retriever_name = "classic"
else:
retriever_name = source["active_docs"]
prompt = get_prompt(prompt_id) retriever = RetrieverCreator.create_retriever(
retriever_name,
question=question,
source=source,
chat_history=history,
prompt=prompt,
chunks=chunks,
gpt_model=gpt_model,
user_api_key=user_api_key,
)
retriever = RetrieverCreator.create_retriever( return Response(
retriever_name, complete_stream(
question=question, question=question,
source=source, retriever=retriever,
chat_history=history, conversation_id=conversation_id,
prompt=prompt,
chunks=chunks,
token_limit=token_limit,
gpt_model=gpt_model,
user_api_key=user_api_key, user_api_key=user_api_key,
) ),
mimetype="text/event-stream",
return Response( )
complete_stream(
question=question,
retriever=retriever,
conversation_id=conversation_id,
user_api_key=user_api_key,
isNoneDoc=data.get("isNoneDoc"),
),
mimetype="text/event-stream",
)
except ValueError:
message = "Malformed request body"
print("\033[91merr", str(message), file=sys.stderr)
return Response(
error_stream_generate(message),
status=400,
mimetype="text/event-stream",
)
except Exception as e:
current_app.logger.error(
f"/stream - error: {str(e)} - traceback: {traceback.format_exc()}",
extra={"error": str(e), "traceback": traceback.format_exc()},
)
message = e.args[0]
status_code = 400
# # Custom exceptions with two arguments, index 1 as status code
if len(e.args) >= 2:
status_code = e.args[1]
return Response(
error_stream_generate(message),
status=status_code,
mimetype="text/event-stream",
)
def error_stream_generate(err_response):
data = json.dumps({"type": "error", "error": err_response})
yield f"data: {data}\n\n"
@answer.route("/api/answer", methods=["POST"]) @answer.route("/api/answer", methods=["POST"])
@ -374,38 +289,27 @@ def api_answer():
chunks = int(data["chunks"]) chunks = int(data["chunks"])
else: else:
chunks = 2 chunks = 2
if "token_limit" in data:
token_limit = data["token_limit"]
else:
token_limit = settings.DEFAULT_MAX_HISTORY
## retriever can be brave_search, duckduck_search or classic prompt = get_prompt(prompt_id)
retriever_name = data["retriever"] if "retriever" in data else "classic"
# use try and except to check for exception # use try and except to check for exception
try: try:
# check if the vectorstore is set # check if the vectorstore is set
if "api_key" in data: if "api_key" in data:
data_key = get_data_from_api_key(data["api_key"]) data_key = get_data_from_api_key(data["api_key"])
chunks = int(data_key["chunks"])
prompt_id = data_key["prompt_id"]
source = {"active_docs": data_key["source"]} source = {"active_docs": data_key["source"]}
retriever_name = data_key["retriever"] or retriever_name
user_api_key = data["api_key"] user_api_key = data["api_key"]
elif "active_docs" in data:
source = {"active_docs": data["active_docs"]}
retriever_name = get_retriever(data["active_docs"]) or retriever_name
user_api_key = None
else: else:
source = {} source = {data}
user_api_key = None user_api_key = None
prompt = get_prompt(prompt_id) if (
source["active_docs"].split("/")[0] == "default"
current_app.logger.info( or source["active_docs"].split("/")[0] == "local"
f"/api/answer - request_data: {data}, source: {source}", ):
extra={"data": json.dumps({"request_data": data, "source": source})}, retriever_name = "classic"
) else:
retriever_name = source["active_docs"]
retriever = RetrieverCreator.create_retriever( retriever = RetrieverCreator.create_retriever(
retriever_name, retriever_name,
@ -414,7 +318,6 @@ def api_answer():
chat_history=history, chat_history=history,
prompt=prompt, prompt=prompt,
chunks=chunks, chunks=chunks,
token_limit=token_limit,
gpt_model=gpt_model, gpt_model=gpt_model,
user_api_key=user_api_key, user_api_key=user_api_key,
) )
@ -426,56 +329,32 @@ def api_answer():
elif "answer" in line: elif "answer" in line:
response_full += line["answer"] response_full += line["answer"]
if data.get("isNoneDoc"):
for doc in source_log_docs:
doc["source"] = "None"
llm = LLMCreator.create_llm( llm = LLMCreator.create_llm(
settings.LLM_NAME, api_key=settings.API_KEY, user_api_key=user_api_key settings.LLM_NAME, api_key=settings.API_KEY, user_api_key=user_api_key
) )
result = {"answer": response_full, "sources": source_log_docs} result = {"answer": response_full, "sources": source_log_docs}
result["conversation_id"] = str( result["conversation_id"] = save_conversation(
save_conversation( conversation_id, question, response_full, source_log_docs, llm
conversation_id, question, response_full, source_log_docs, llm
)
)
retriever_params = retriever.get_params()
user_logs_collection.insert_one(
{
"action": "api_answer",
"level": "info",
"user": "local",
"api_key": user_api_key,
"question": question,
"response": response_full,
"sources": source_log_docs,
"retriever_params": retriever_params,
"timestamp": datetime.datetime.now(datetime.timezone.utc),
}
) )
return result return result
except Exception as e: except Exception as e:
current_app.logger.error( # print whole traceback
f"/api/answer - error: {str(e)} - traceback: {traceback.format_exc()}", traceback.print_exc()
extra={"error": str(e), "traceback": traceback.format_exc()}, print(str(e))
)
return bad_request(500, str(e)) return bad_request(500, str(e))
@answer.route("/api/search", methods=["POST"]) @answer.route("/api/search", methods=["POST"])
def api_search(): def api_search():
data = request.get_json() data = request.get_json()
# get parameter from url question
question = data["question"] question = data["question"]
if "chunks" in data:
chunks = int(data["chunks"])
else:
chunks = 2
if "api_key" in data: if "api_key" in data:
data_key = get_data_from_api_key(data["api_key"]) data_key = get_data_from_api_key(data["api_key"])
chunks = int(data_key["chunks"]) source = {"active_docs": data_key["source"]}
source = {"active_docs":data_key["source"]}
user_api_key = data["api_key"] user_api_key = data["api_key"]
elif "active_docs" in data: elif "active_docs" in data:
source = {"active_docs": data["active_docs"]} source = {"active_docs": data["active_docs"]}
@ -483,20 +362,18 @@ def api_search():
else: else:
source = {} source = {}
user_api_key = None user_api_key = None
if "chunks" in data:
if "retriever" in data: chunks = int(data["chunks"])
retriever_name = data["retriever"]
else: else:
chunks = 2
if (
source["active_docs"].split("/")[0] == "default"
or source["active_docs"].split("/")[0] == "local"
):
retriever_name = "classic" retriever_name = "classic"
if "token_limit" in data:
token_limit = data["token_limit"]
else: else:
token_limit = settings.DEFAULT_MAX_HISTORY retriever_name = source["active_docs"]
current_app.logger.info(
f"/api/answer - request_data: {data}, source: {source}",
extra={"data": json.dumps({"request_data": data, "source": source})},
)
retriever = RetrieverCreator.create_retriever( retriever = RetrieverCreator.create_retriever(
retriever_name, retriever_name,
@ -505,28 +382,8 @@ def api_search():
chat_history=[], chat_history=[],
prompt="default", prompt="default",
chunks=chunks, chunks=chunks,
token_limit=token_limit,
gpt_model=gpt_model, gpt_model=gpt_model,
user_api_key=user_api_key, user_api_key=user_api_key,
) )
docs = retriever.search() docs = retriever.search()
retriever_params = retriever.get_params()
user_logs_collection.insert_one(
{
"action": "api_search",
"level": "info",
"user": "local",
"api_key": user_api_key,
"question": question,
"sources": docs,
"retriever_params": retriever_params,
"timestamp": datetime.datetime.now(datetime.timezone.utc),
}
)
if data.get("isNoneDoc"):
for doc in docs:
doc["source"] = "None"
return docs return docs

@ -3,23 +3,18 @@ import datetime
from flask import Blueprint, request, send_from_directory from flask import Blueprint, request, send_from_directory
from pymongo import MongoClient from pymongo import MongoClient
from werkzeug.utils import secure_filename from werkzeug.utils import secure_filename
from bson.objectid import ObjectId
from application.core.settings import settings
from application.core.settings import settings
mongo = MongoClient(settings.MONGO_URI) mongo = MongoClient(settings.MONGO_URI)
db = mongo["docsgpt"] db = mongo["docsgpt"]
conversations_collection = db["conversations"] conversations_collection = db["conversations"]
sources_collection = db["sources"] vectors_collection = db["vectors"]
current_dir = os.path.dirname(
os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
)
internal = Blueprint("internal", __name__) current_dir = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
internal = Blueprint('internal', __name__)
@internal.route("/api/download", methods=["get"]) @internal.route("/api/download", methods=["get"])
def download_file(): def download_file():
user = secure_filename(request.args.get("user")) user = secure_filename(request.args.get("user"))
@ -29,6 +24,7 @@ def download_file():
return send_from_directory(save_dir, filename, as_attachment=True) return send_from_directory(save_dir, filename, as_attachment=True)
@internal.route("/api/upload_index", methods=["POST"]) @internal.route("/api/upload_index", methods=["POST"])
def upload_index_files(): def upload_index_files():
"""Upload two files(index.faiss, index.pkl) to the user's folder.""" """Upload two files(index.faiss, index.pkl) to the user's folder."""
@ -38,14 +34,7 @@ def upload_index_files():
if "name" not in request.form: if "name" not in request.form:
return {"status": "no name"} return {"status": "no name"}
job_name = secure_filename(request.form["name"]) job_name = secure_filename(request.form["name"])
tokens = secure_filename(request.form["tokens"]) save_dir = os.path.join(current_dir, "indexes", user, job_name)
retriever = secure_filename(request.form["retriever"])
id = secure_filename(request.form["id"])
type = secure_filename(request.form["type"])
remote_data = request.form["remote_data"] if "remote_data" in request.form else None
sync_frequency = secure_filename(request.form["sync_frequency"])
save_dir = os.path.join(current_dir, "indexes", str(id))
if settings.VECTOR_STORE == "faiss": if settings.VECTOR_STORE == "faiss":
if "file_faiss" not in request.files: if "file_faiss" not in request.files:
print("No file part") print("No file part")
@ -65,40 +54,16 @@ def upload_index_files():
os.makedirs(save_dir) os.makedirs(save_dir)
file_faiss.save(os.path.join(save_dir, "index.faiss")) file_faiss.save(os.path.join(save_dir, "index.faiss"))
file_pkl.save(os.path.join(save_dir, "index.pkl")) file_pkl.save(os.path.join(save_dir, "index.pkl"))
# create entry in vectors_collection
existing_entry = sources_collection.find_one({"_id": ObjectId(id)}) vectors_collection.insert_one(
if existing_entry: {
sources_collection.update_one( "user": user,
{"_id": ObjectId(id)}, "name": job_name,
{ "language": job_name,
"$set": { "location": save_dir,
"user": user, "date": datetime.datetime.now().strftime("%d/%m/%Y %H:%M:%S"),
"name": job_name, "model": settings.EMBEDDINGS_NAME,
"language": job_name, "type": "local",
"date": datetime.datetime.now().strftime("%d/%m/%Y %H:%M:%S"), }
"model": settings.EMBEDDINGS_NAME, )
"type": type,
"tokens": tokens,
"retriever": retriever,
"remote_data": remote_data,
"sync_frequency": sync_frequency,
}
},
)
else:
sources_collection.insert_one(
{
"_id": ObjectId(id),
"user": user,
"name": job_name,
"language": job_name,
"date": datetime.datetime.now().strftime("%d/%m/%Y %H:%M:%S"),
"model": settings.EMBEDDINGS_NAME,
"type": type,
"tokens": tokens,
"retriever": retriever,
"remote_data": remote_data,
"sync_frequency": sync_frequency,
}
)
return {"status": "ok"} return {"status": "ok"}

File diff suppressed because it is too large Load Diff

@ -1,38 +1,12 @@
from datetime import timedelta from application.worker import ingest_worker, remote_worker
from application.celery import celery
from application.celery_init import celery
from application.worker import ingest_worker, remote_worker, sync_worker
@celery.task(bind=True) @celery.task(bind=True)
def ingest(self, directory, formats, name_job, filename, user): def ingest(self, directory, formats, name_job, filename, user):
resp = ingest_worker(self, directory, formats, name_job, filename, user) resp = ingest_worker(self, directory, formats, name_job, filename, user)
return resp return resp
@celery.task(bind=True) @celery.task(bind=True)
def ingest_remote(self, source_data, job_name, user, loader): def ingest_remote(self, source_data, job_name, user, loader):
resp = remote_worker(self, source_data, job_name, user, loader) resp = remote_worker(self, source_data, job_name, user, loader)
return resp return resp
@celery.task(bind=True)
def schedule_syncs(self, frequency):
resp = sync_worker(self, frequency)
return resp
@celery.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
sender.add_periodic_task(
timedelta(days=1),
schedule_syncs.s("daily"),
)
sender.add_periodic_task(
timedelta(weeks=1),
schedule_syncs.s("weekly"),
)
sender.add_periodic_task(
timedelta(days=30),
schedule_syncs.s("monthly"),
)

@ -1,19 +1,17 @@
import platform import platform
import dotenv import dotenv
from application.celery_init import celery from application.celery import celery
from flask import Flask, request, redirect from flask import Flask, request, redirect
from application.core.settings import settings from application.core.settings import settings
from application.api.user.routes import user from application.api.user.routes import user
from application.api.answer.routes import answer from application.api.answer.routes import answer
from application.api.internal.routes import internal from application.api.internal.routes import internal
from application.core.logging_config import setup_logging
if platform.system() == "Windows": if platform.system() == "Windows":
import pathlib import pathlib
pathlib.PosixPath = pathlib.WindowsPath pathlib.PosixPath = pathlib.WindowsPath
dotenv.load_dotenv() dotenv.load_dotenv()
setup_logging()
app = Flask(__name__) app = Flask(__name__)
app.register_blueprint(user) app.register_blueprint(user)

@ -1,15 +1,9 @@
from celery import Celery from celery import Celery
from application.core.settings import settings from application.core.settings import settings
from celery.signals import setup_logging
def make_celery(app_name=__name__): def make_celery(app_name=__name__):
celery = Celery(app_name, broker=settings.CELERY_BROKER_URL, backend=settings.CELERY_RESULT_BACKEND) celery = Celery(app_name, broker=settings.CELERY_BROKER_URL, backend=settings.CELERY_RESULT_BACKEND)
celery.conf.update(settings) celery.conf.update(settings)
return celery return celery
@setup_logging.connect
def config_loggers(*args, **kwargs):
from application.core.logging_config import setup_logging
setup_logging()
celery = make_celery() celery = make_celery()

@ -1,22 +0,0 @@
from logging.config import dictConfig
def setup_logging():
dictConfig({
'version': 1,
'formatters': {
'default': {
'format': '[%(asctime)s] %(levelname)s in %(module)s: %(message)s',
}
},
"handlers": {
"console": {
"class": "logging.StreamHandler",
"stream": "ext://sys.stdout",
"formatter": "default",
}
},
'root': {
'level': 'INFO',
'handlers': ['console'],
},
})

@ -15,10 +15,9 @@ class Settings(BaseSettings):
CELERY_RESULT_BACKEND: str = "redis://localhost:6379/1" CELERY_RESULT_BACKEND: str = "redis://localhost:6379/1"
MONGO_URI: str = "mongodb://localhost:27017/docsgpt" MONGO_URI: str = "mongodb://localhost:27017/docsgpt"
MODEL_PATH: str = os.path.join(current_dir, "models/docsgpt-7b-f16.gguf") MODEL_PATH: str = os.path.join(current_dir, "models/docsgpt-7b-f16.gguf")
DEFAULT_MAX_HISTORY: int = 150 TOKENS_MAX_HISTORY: int = 150
MODEL_TOKEN_LIMITS: dict = {"gpt-3.5-turbo": 4096, "claude-2": 1e5}
UPLOAD_FOLDER: str = "inputs" UPLOAD_FOLDER: str = "inputs"
VECTOR_STORE: str = "faiss" # "faiss" or "elasticsearch" or "qdrant" or "milvus" VECTOR_STORE: str = "faiss" # "faiss" or "elasticsearch" or "qdrant"
RETRIEVERS_ENABLED: list = ["classic_rag", "duckduck_search"] # also brave_search RETRIEVERS_ENABLED: list = ["classic_rag", "duckduck_search"] # also brave_search
API_URL: str = "http://localhost:7091" # backend url for celery worker API_URL: str = "http://localhost:7091" # backend url for celery worker
@ -29,7 +28,6 @@ class Settings(BaseSettings):
OPENAI_API_VERSION: Optional[str] = None # azure openai api version OPENAI_API_VERSION: Optional[str] = None # azure openai api version
AZURE_DEPLOYMENT_NAME: Optional[str] = None # azure deployment name for answering AZURE_DEPLOYMENT_NAME: Optional[str] = None # azure deployment name for answering
AZURE_EMBEDDINGS_DEPLOYMENT_NAME: Optional[str] = None # azure deployment name for embeddings AZURE_EMBEDDINGS_DEPLOYMENT_NAME: Optional[str] = None # azure deployment name for embeddings
OPENAI_BASE_URL: Optional[str] = None # openai base url for open ai compatable models
# elasticsearch # elasticsearch
ELASTIC_CLOUD_ID: Optional[str] = None # cloud id for elasticsearch ELASTIC_CLOUD_ID: Optional[str] = None # cloud id for elasticsearch
@ -62,11 +60,6 @@ class Settings(BaseSettings):
QDRANT_PATH: Optional[str] = None QDRANT_PATH: Optional[str] = None
QDRANT_DISTANCE_FUNC: str = "Cosine" QDRANT_DISTANCE_FUNC: str = "Cosine"
# Milvus vectorstore config
MILVUS_COLLECTION_NAME: Optional[str] = "docsgpt"
MILVUS_URI: Optional[str] = "./milvus_local.db" # milvus lite version as default
MILVUS_TOKEN: Optional[str] = ""
BRAVE_SEARCH_API_KEY: Optional[str] = None BRAVE_SEARCH_API_KEY: Optional[str] = None
FLASK_DEBUG_MODE: bool = False FLASK_DEBUG_MODE: bool = False

@ -1,30 +1,9 @@
from application.llm.base import BaseLLM from application.llm.base import BaseLLM
from application.core.settings import settings from application.core.settings import settings
import threading
class LlamaSingleton:
_instances = {}
_lock = threading.Lock() # Add a lock for thread synchronization
@classmethod
def get_instance(cls, llm_name):
if llm_name not in cls._instances:
try:
from llama_cpp import Llama
except ImportError:
raise ImportError(
"Please install llama_cpp using pip install llama-cpp-python"
)
cls._instances[llm_name] = Llama(model_path=llm_name, n_ctx=2048)
return cls._instances[llm_name]
@classmethod
def query_model(cls, llm, prompt, **kwargs):
with cls._lock:
return llm(prompt, **kwargs)
class LlamaCpp(BaseLLM): class LlamaCpp(BaseLLM):
def __init__( def __init__(
self, self,
api_key=None, api_key=None,
@ -33,23 +12,41 @@ class LlamaCpp(BaseLLM):
*args, *args,
**kwargs, **kwargs,
): ):
global llama
try:
from llama_cpp import Llama
except ImportError:
raise ImportError(
"Please install llama_cpp using pip install llama-cpp-python"
)
super().__init__(*args, **kwargs) super().__init__(*args, **kwargs)
self.api_key = api_key self.api_key = api_key
self.user_api_key = user_api_key self.user_api_key = user_api_key
self.llama = LlamaSingleton.get_instance(llm_name) llama = Llama(model_path=llm_name, n_ctx=2048)
def _raw_gen(self, baseself, model, messages, stream=False, **kwargs): def _raw_gen(self, baseself, model, messages, stream=False, **kwargs):
context = messages[0]["content"] context = messages[0]["content"]
user_question = messages[-1]["content"] user_question = messages[-1]["content"]
prompt = f"### Instruction \n {user_question} \n ### Context \n {context} \n ### Answer \n" prompt = f"### Instruction \n {user_question} \n ### Context \n {context} \n ### Answer \n"
result = LlamaSingleton.query_model(self.llama, prompt, max_tokens=150, echo=False)
result = llama(prompt, max_tokens=150, echo=False)
# import sys
# print(result['choices'][0]['text'].split('### Answer \n')[-1], file=sys.stderr)
return result["choices"][0]["text"].split("### Answer \n")[-1] return result["choices"][0]["text"].split("### Answer \n")[-1]
def _raw_gen_stream(self, baseself, model, messages, stream=True, **kwargs): def _raw_gen_stream(self, baseself, model, messages, stream=True, **kwargs):
context = messages[0]["content"] context = messages[0]["content"]
user_question = messages[-1]["content"] user_question = messages[-1]["content"]
prompt = f"### Instruction \n {user_question} \n ### Context \n {context} \n ### Answer \n" prompt = f"### Instruction \n {user_question} \n ### Context \n {context} \n ### Answer \n"
result = LlamaSingleton.query_model(self.llama, prompt, max_tokens=150, echo=False, stream=stream)
result = llama(prompt, max_tokens=150, echo=False, stream=stream)
# import sys
# print(list(result), file=sys.stderr)
for item in result: for item in result:
for choice in item["choices"]: for choice in item["choices"]:
yield choice["text"] yield choice["text"]

@ -2,23 +2,25 @@ from application.llm.base import BaseLLM
from application.core.settings import settings from application.core.settings import settings
class OpenAILLM(BaseLLM): class OpenAILLM(BaseLLM):
def __init__(self, api_key=None, user_api_key=None, *args, **kwargs): def __init__(self, api_key=None, user_api_key=None, *args, **kwargs):
global openai
from openai import OpenAI from openai import OpenAI
super().__init__(*args, **kwargs) super().__init__(*args, **kwargs)
if settings.OPENAI_BASE_URL: self.client = OpenAI(
self.client = OpenAI( api_key=api_key,
api_key=api_key, )
base_url=settings.OPENAI_BASE_URL
)
else:
self.client = OpenAI(api_key=api_key)
self.api_key = api_key self.api_key = api_key
self.user_api_key = user_api_key self.user_api_key = user_api_key
def _get_openai(self):
# Import openai when needed
import openai
return openai
def _raw_gen( def _raw_gen(
self, self,
baseself, baseself,
@ -71,3 +73,8 @@ class AzureOpenAILLM(OpenAILLM):
api_base=settings.OPENAI_API_BASE, api_base=settings.OPENAI_API_BASE,
deployment_name=settings.AZURE_DEPLOYMENT_NAME, deployment_name=settings.AZURE_DEPLOYMENT_NAME,
) )
def _get_openai(self):
openai = super()._get_openai()
return openai

@ -3,6 +3,7 @@
Contains parser for html files. Contains parser for html files.
""" """
import re
from pathlib import Path from pathlib import Path
from typing import Dict, Union from typing import Dict, Union
@ -17,8 +18,66 @@ class HTMLParser(BaseParser):
return {} return {}
def parse_file(self, file: Path, errors: str = "ignore") -> Union[str, list[str]]: def parse_file(self, file: Path, errors: str = "ignore") -> Union[str, list[str]]:
from langchain_community.document_loaders import BSHTMLLoader """Parse file.
loader = BSHTMLLoader(file) Returns:
data = loader.load() Union[str, List[str]]: a string or a List of strings.
return data """
try:
from unstructured.partition.html import partition_html
from unstructured.staging.base import convert_to_isd
from unstructured.cleaners.core import clean
except ImportError:
raise ValueError("unstructured package is required to parse HTML files.")
# Using the unstructured library to convert the html to isd format
# isd sample : isd = [
# {"text": "My Title", "type": "Title"},
# {"text": "My Narrative", "type": "NarrativeText"}
# ]
with open(file, "r", encoding="utf-8") as fp:
elements = partition_html(file=fp)
isd = convert_to_isd(elements)
# Removing non ascii charactwers from isd_el['text']
for isd_el in isd:
isd_el['text'] = isd_el['text'].encode("ascii", "ignore").decode()
# Removing all the \n characters from isd_el['text'] using regex and replace with single space
# Removing all the extra spaces from isd_el['text'] using regex and replace with single space
for isd_el in isd:
isd_el['text'] = re.sub(r'\n', ' ', isd_el['text'], flags=re.MULTILINE | re.DOTALL)
isd_el['text'] = re.sub(r"\s{2,}", " ", isd_el['text'], flags=re.MULTILINE | re.DOTALL)
# more cleaning: extra_whitespaces, dashes, bullets, trailing_punctuation
for isd_el in isd:
clean(isd_el['text'], extra_whitespace=True, dashes=True, bullets=True, trailing_punctuation=True)
# Creating a list of all the indexes of isd_el['type'] = 'Title'
title_indexes = [i for i, isd_el in enumerate(isd) if isd_el['type'] == 'Title']
# Creating 'Chunks' - List of lists of strings
# each list starting with isd_el['type'] = 'Title' and all the data till the next 'Title'
# Each Chunk can be thought of as an individual set of data, which can be sent to the model
# Where Each Title is grouped together with the data under it
Chunks = [[]]
final_chunks = list(list())
for i, isd_el in enumerate(isd):
if i in title_indexes:
Chunks.append([])
Chunks[-1].append(isd_el['text'])
# Removing all the chunks with sum of length of all the strings in the chunk < 25
# TODO: This value can be an user defined variable
for chunk in Chunks:
# sum of length of all the strings in the chunk
sum = 0
sum += len(str(chunk))
if sum < 25:
Chunks.remove(chunk)
else:
# appending all the approved chunks to final_chunks as a single string
final_chunks.append(" ".join([str(item) for item in chunk]))
return final_chunks

@ -1,10 +1,9 @@
import os import os
from retry import retry import tiktoken
from application.core.settings import settings
from application.vectorstore.vector_creator import VectorCreator from application.vectorstore.vector_creator import VectorCreator
from application.core.settings import settings
from retry import retry
# from langchain_community.embeddings import HuggingFaceEmbeddings # from langchain_community.embeddings import HuggingFaceEmbeddings
@ -12,22 +11,28 @@ from application.vectorstore.vector_creator import VectorCreator
# from langchain_community.embeddings import CohereEmbeddings # from langchain_community.embeddings import CohereEmbeddings
def num_tokens_from_string(string: str, encoding_name: str) -> int:
# Function to convert string to tokens and estimate user cost.
encoding = tiktoken.get_encoding(encoding_name)
num_tokens = len(encoding.encode(string))
total_price = ((num_tokens / 1000) * 0.0004)
return num_tokens, total_price
@retry(tries=10, delay=60) @retry(tries=10, delay=60)
def store_add_texts_with_retry(store, i, id): def store_add_texts_with_retry(store, i):
# add source_id to the metadata
i.metadata["source_id"] = str(id)
store.add_texts([i.page_content], metadatas=[i.metadata]) store.add_texts([i.page_content], metadatas=[i.metadata])
# store_pine.add_texts([i.page_content], metadatas=[i.metadata]) # store_pine.add_texts([i.page_content], metadatas=[i.metadata])
def call_openai_api(docs, folder_name, id, task_status): def call_openai_api(docs, folder_name, task_status):
# Function to create a vector store from the documents and save it to disk # Function to create a vector store from the documents and save it to disk.
# create output folder if it doesn't exist
if not os.path.exists(f"{folder_name}"): if not os.path.exists(f"{folder_name}"):
os.makedirs(f"{folder_name}") os.makedirs(f"{folder_name}")
from tqdm import tqdm from tqdm import tqdm
c1 = 0 c1 = 0
if settings.VECTOR_STORE == "faiss": if settings.VECTOR_STORE == "faiss":
docs_init = [docs[0]] docs_init = [docs[0]]
@ -35,34 +40,26 @@ def call_openai_api(docs, folder_name, id, task_status):
store = VectorCreator.create_vectorstore( store = VectorCreator.create_vectorstore(
settings.VECTOR_STORE, settings.VECTOR_STORE,
docs_init=docs_init, docs_init = docs_init,
source_id=f"{folder_name}", path=f"{folder_name}",
embeddings_key=os.getenv("EMBEDDINGS_KEY"), embeddings_key=os.getenv("EMBEDDINGS_KEY")
) )
else: else:
store = VectorCreator.create_vectorstore( store = VectorCreator.create_vectorstore(
settings.VECTOR_STORE, settings.VECTOR_STORE,
source_id=str(id), path=f"{folder_name}",
embeddings_key=os.getenv("EMBEDDINGS_KEY"), embeddings_key=os.getenv("EMBEDDINGS_KEY")
) )
store.delete_index()
# Uncomment for MPNet embeddings # Uncomment for MPNet embeddings
# model_name = "sentence-transformers/all-mpnet-base-v2" # model_name = "sentence-transformers/all-mpnet-base-v2"
# hf = HuggingFaceEmbeddings(model_name=model_name) # hf = HuggingFaceEmbeddings(model_name=model_name)
# store = FAISS.from_documents(docs_test, hf) # store = FAISS.from_documents(docs_test, hf)
s1 = len(docs) s1 = len(docs)
for i in tqdm( for i in tqdm(docs, desc="Embedding 🦖", unit="docs", total=len(docs),
docs, bar_format='{l_bar}{bar}| Time Left: {remaining}'):
desc="Embedding 🦖",
unit="docs",
total=len(docs),
bar_format="{l_bar}{bar}| Time Left: {remaining}",
):
try: try:
task_status.update_state( task_status.update_state(state='PROGRESS', meta={'current': int((c1 / s1) * 100)})
state="PROGRESS", meta={"current": int((c1 / s1) * 100)} store_add_texts_with_retry(store, i)
)
store_add_texts_with_retry(store, i, id)
except Exception as e: except Exception as e:
print(e) print(e)
print("Error on ", i) print("Error on ", i)
@ -73,3 +70,25 @@ def call_openai_api(docs, folder_name, id, task_status):
c1 += 1 c1 += 1
if settings.VECTOR_STORE == "faiss": if settings.VECTOR_STORE == "faiss":
store.save_local(f"{folder_name}") store.save_local(f"{folder_name}")
def get_user_permission(docs, folder_name):
# Function to ask user permission to call the OpenAI api and spend their OpenAI funds.
# Here we convert the docs list to a string and calculate the number of OpenAI tokens the string represents.
# docs_content = (" ".join(docs))
docs_content = ""
for doc in docs:
docs_content += doc.page_content
tokens, total_price = num_tokens_from_string(string=docs_content, encoding_name="cl100k_base")
# Here we print the number of tokens and the approx user cost with some visually appealing formatting.
print(f"Number of Tokens = {format(tokens, ',d')}")
print(f"Approx Cost = ${format(total_price, ',.2f')}")
# Here we check for user permission before calling the API.
user_input = input("Price Okay? (Y/N) \n").lower()
if user_input == "y":
call_openai_api(docs, folder_name)
elif user_input == "":
call_openai_api(docs, folder_name)
else:
print("The API was not called. No money was spent.")

@ -5,7 +5,7 @@ from application.parser.remote.base import BaseRemote
class CrawlerLoader(BaseRemote): class CrawlerLoader(BaseRemote):
def __init__(self, limit=10): def __init__(self, limit=10):
from langchain_community.document_loaders import WebBaseLoader from langchain.document_loaders import WebBaseLoader
self.loader = WebBaseLoader # Initialize the document loader self.loader = WebBaseLoader # Initialize the document loader
self.limit = limit # Set the limit for the number of pages to scrape self.limit = limit # Set the limit for the number of pages to scrape

@ -5,7 +5,7 @@ from application.parser.remote.base import BaseRemote
class SitemapLoader(BaseRemote): class SitemapLoader(BaseRemote):
def __init__(self, limit=20): def __init__(self, limit=20):
from langchain_community.document_loaders import WebBaseLoader from langchain.document_loaders import WebBaseLoader
self.loader = WebBaseLoader self.loader = WebBaseLoader
self.limit = limit # Adding limit to control the number of URLs to process self.limit = limit # Adding limit to control the number of URLs to process

@ -1,32 +1,22 @@
from application.parser.remote.base import BaseRemote from application.parser.remote.base import BaseRemote
from langchain_community.document_loaders import WebBaseLoader
headers = {
"User-Agent": "Mozilla/5.0",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*"
";q=0.8",
"Accept-Language": "en-US,en;q=0.5",
"Referer": "https://www.google.com/",
"DNT": "1",
"Connection": "keep-alive",
"Upgrade-Insecure-Requests": "1",
}
class WebLoader(BaseRemote): class WebLoader(BaseRemote):
def __init__(self): def __init__(self):
from langchain.document_loaders import WebBaseLoader
self.loader = WebBaseLoader self.loader = WebBaseLoader
def load_data(self, inputs): def load_data(self, inputs):
urls = inputs urls = inputs
if isinstance(urls, str): if isinstance(urls, str):
urls = [urls] urls = [urls] # Convert string to list if a single URL is passed
documents = [] documents = []
for url in urls: for url in urls:
try: try:
loader = self.loader([url], header_template=headers) loader = self.loader([url]) # Process URLs one by one
documents.extend(loader.load()) documents.extend(loader.load())
except Exception as e: except Exception as e:
print(f"Error processing URL {url}: {e}") print(f"Error processing URL {url}: {e}")
continue continue # Continue with the next URL if an error occurs
return documents return documents

@ -1,85 +1,35 @@
anthropic==0.34.2 anthropic==0.12.0
boto3==1.34.153 boto3==1.34.6
beautifulsoup4==4.12.3
celery==5.3.6 celery==5.3.6
dataclasses-json==0.6.7 dataclasses_json==0.6.3
docx2txt==0.8 docx2txt==0.8
duckduckgo-search==6.2.6 duckduckgo-search==5.3.0
ebooklib==0.18 EbookLib==0.18
elastic-transport==8.15.0 elasticsearch==8.12.0
elasticsearch==8.15.1
escodegen==1.0.11 escodegen==1.0.11
esprima==4.0.1 esprima==4.0.1
esutils==1.0.1 faiss-cpu==1.7.4
Flask==3.0.3 Flask==3.0.1
faiss-cpu==1.8.0.post1 gunicorn==21.2.0
gunicorn==23.0.0 html2text==2020.1.16
html2text==2024.2.26
javalang==0.13.0 javalang==0.13.0
jinja2==3.1.4 langchain==0.1.4
jiter==0.5.0 langchain-openai==0.0.5
jmespath==1.0.1 nltk==3.8.1
joblib==1.4.2 openapi3_parser==1.1.16
jsonpatch==1.33 pandas==2.2.0
jsonpointer==3.0.0 pydantic_settings==2.1.0
jsonschema==4.23.0 pymongo==4.6.3
jsonschema-spec==0.2.4 PyPDF2==3.0.1
jsonschema-specifications==2023.7.1
kombu==5.4.2
langchain==0.3.0
langchain-community==0.3.0
langchain-core==0.3.2
langchain-openai==0.2.0
langchain-text-splitters==0.3.0
langsmith==0.1.125
lazy-object-proxy==1.10.0
lxml==5.3.0
markupsafe==2.1.5
marshmallow==3.22.0
mpmath==1.3.0
multidict==6.1.0
mypy-extensions==1.0.0
networkx==3.3
numpy==1.26.4
openai==1.46.1
openapi-schema-validator==0.6.2
openapi-spec-validator==0.6.0
openapi3-parser==1.1.18
orjson==3.10.7
packaging==24.1
pandas==2.2.3
pathable==0.4.3
pillow==10.4.0
portalocker==2.10.1
prance==23.6.21.0
primp==0.6.2
prompt-toolkit==3.0.47
protobuf==5.28.2
py==1.11.0
pydantic==2.9.2
pydantic-core==2.23.4
pydantic-settings==2.4.0
pymongo==4.8.0
pypdf2==3.0.1
python-dateutil==2.9.0.post0
python-dotenv==1.0.1 python-dotenv==1.0.1
qdrant-client==1.11.0 qdrant-client==1.8.2
redis==5.0.1 redis==5.0.1
referencing==0.30.2 Requests==2.31.0
regex==2024.9.11
requests==2.32.3
retry==0.9.2 retry==0.9.2
sentence-transformers==3.0.1 sentence-transformers
tiktoken==0.7.0 tiktoken==0.5.2
tokenizers==0.19.1 torch==2.1.2
torch==2.4.1 tqdm==4.66.1
tqdm==4.66.5 transformers==4.36.2
transformers==4.44.2 unstructured==0.12.2
typing-extensions==4.12.2 Werkzeug==3.0.1
typing-inspect==0.9.0
tzdata==2024.2
urllib3==2.2.3
vine==5.1.0
wcwidth==0.2.13
werkzeug==3.0.4
yarl==1.11.1

@ -12,7 +12,3 @@ class BaseRetriever(ABC):
@abstractmethod @abstractmethod
def search(self, *args, **kwargs): def search(self, *args, **kwargs):
pass pass
@abstractmethod
def get_params(self):
pass

@ -2,7 +2,7 @@ import json
from application.retriever.base import BaseRetriever from application.retriever.base import BaseRetriever
from application.core.settings import settings from application.core.settings import settings
from application.llm.llm_creator import LLMCreator from application.llm.llm_creator import LLMCreator
from application.utils import num_tokens_from_string from application.utils import count_tokens
from langchain_community.tools import BraveSearch from langchain_community.tools import BraveSearch
@ -15,7 +15,6 @@ class BraveRetSearch(BaseRetriever):
chat_history, chat_history,
prompt, prompt,
chunks=2, chunks=2,
token_limit=150,
gpt_model="docsgpt", gpt_model="docsgpt",
user_api_key=None, user_api_key=None,
): ):
@ -25,16 +24,6 @@ class BraveRetSearch(BaseRetriever):
self.prompt = prompt self.prompt = prompt
self.chunks = chunks self.chunks = chunks
self.gpt_model = gpt_model self.gpt_model = gpt_model
self.token_limit = (
token_limit
if token_limit
< settings.MODEL_TOKEN_LIMITS.get(
self.gpt_model, settings.DEFAULT_MAX_HISTORY
)
else settings.MODEL_TOKEN_LIMITS.get(
self.gpt_model, settings.DEFAULT_MAX_HISTORY
)
)
self.user_api_key = user_api_key self.user_api_key = user_api_key
def _get_data(self): def _get_data(self):
@ -78,10 +67,13 @@ class BraveRetSearch(BaseRetriever):
self.chat_history.reverse() self.chat_history.reverse()
for i in self.chat_history: for i in self.chat_history:
if "prompt" in i and "response" in i: if "prompt" in i and "response" in i:
tokens_batch = num_tokens_from_string(i["prompt"]) + num_tokens_from_string( tokens_batch = count_tokens(i["prompt"]) + count_tokens(
i["response"] i["response"]
) )
if tokens_current_history + tokens_batch < self.token_limit: if (
tokens_current_history + tokens_batch
< settings.TOKENS_MAX_HISTORY
):
tokens_current_history += tokens_batch tokens_current_history += tokens_batch
messages_combine.append( messages_combine.append(
{"role": "user", "content": i["prompt"]} {"role": "user", "content": i["prompt"]}
@ -101,15 +93,3 @@ class BraveRetSearch(BaseRetriever):
def search(self): def search(self):
return self._get_data() return self._get_data()
def get_params(self):
return {
"question": self.question,
"source": self.source,
"chat_history": self.chat_history,
"prompt": self.prompt,
"chunks": self.chunks,
"token_limit": self.token_limit,
"gpt_model": self.gpt_model,
"user_api_key": self.user_api_key
}

@ -1,9 +1,10 @@
import os
from application.retriever.base import BaseRetriever from application.retriever.base import BaseRetriever
from application.core.settings import settings from application.core.settings import settings
from application.vectorstore.vector_creator import VectorCreator from application.vectorstore.vector_creator import VectorCreator
from application.llm.llm_creator import LLMCreator from application.llm.llm_creator import LLMCreator
from application.utils import num_tokens_from_string from application.utils import count_tokens
class ClassicRAG(BaseRetriever): class ClassicRAG(BaseRetriever):
@ -15,28 +16,32 @@ class ClassicRAG(BaseRetriever):
chat_history, chat_history,
prompt, prompt,
chunks=2, chunks=2,
token_limit=150,
gpt_model="docsgpt", gpt_model="docsgpt",
user_api_key=None, user_api_key=None,
): ):
self.question = question self.question = question
self.vectorstore = source['active_docs'] if 'active_docs' in source else None self.vectorstore = self._get_vectorstore(source=source)
self.chat_history = chat_history self.chat_history = chat_history
self.prompt = prompt self.prompt = prompt
self.chunks = chunks self.chunks = chunks
self.gpt_model = gpt_model self.gpt_model = gpt_model
self.token_limit = (
token_limit
if token_limit
< settings.MODEL_TOKEN_LIMITS.get(
self.gpt_model, settings.DEFAULT_MAX_HISTORY
)
else settings.MODEL_TOKEN_LIMITS.get(
self.gpt_model, settings.DEFAULT_MAX_HISTORY
)
)
self.user_api_key = user_api_key self.user_api_key = user_api_key
def _get_vectorstore(self, source):
if "active_docs" in source:
if source["active_docs"].split("/")[0] == "default":
vectorstore = ""
elif source["active_docs"].split("/")[0] == "local":
vectorstore = "indexes/" + source["active_docs"]
else:
vectorstore = "vectors/" + source["active_docs"]
if source["active_docs"] == "default":
vectorstore = ""
else:
vectorstore = ""
vectorstore = os.path.join("application", vectorstore)
return vectorstore
def _get_data(self): def _get_data(self):
if self.chunks == 0: if self.chunks == 0:
docs = [] docs = []
@ -45,18 +50,14 @@ class ClassicRAG(BaseRetriever):
settings.VECTOR_STORE, self.vectorstore, settings.EMBEDDINGS_KEY settings.VECTOR_STORE, self.vectorstore, settings.EMBEDDINGS_KEY
) )
docs_temp = docsearch.search(self.question, k=self.chunks) docs_temp = docsearch.search(self.question, k=self.chunks)
print(docs_temp)
docs = [ docs = [
{ {
"title": i.metadata.get( "title": (
"title", i.metadata.get("post_title", i.page_content) i.metadata["title"].split("/")[-1]
).split("/")[-1], if i.metadata
"text": i.page_content, else i.page_content
"source": (
i.metadata.get("source")
if i.metadata.get("source")
else "local"
), ),
"text": i.page_content,
} }
for i in docs_temp for i in docs_temp
] ]
@ -81,10 +82,13 @@ class ClassicRAG(BaseRetriever):
self.chat_history.reverse() self.chat_history.reverse()
for i in self.chat_history: for i in self.chat_history:
if "prompt" in i and "response" in i: if "prompt" in i and "response" in i:
tokens_batch = num_tokens_from_string(i["prompt"]) + num_tokens_from_string( tokens_batch = count_tokens(i["prompt"]) + count_tokens(
i["response"] i["response"]
) )
if tokens_current_history + tokens_batch < self.token_limit: if (
tokens_current_history + tokens_batch
< settings.TOKENS_MAX_HISTORY
):
tokens_current_history += tokens_batch tokens_current_history += tokens_batch
messages_combine.append( messages_combine.append(
{"role": "user", "content": i["prompt"]} {"role": "user", "content": i["prompt"]}
@ -104,15 +108,3 @@ class ClassicRAG(BaseRetriever):
def search(self): def search(self):
return self._get_data() return self._get_data()
def get_params(self):
return {
"question": self.question,
"source": self.vectorstore,
"chat_history": self.chat_history,
"prompt": self.prompt,
"chunks": self.chunks,
"token_limit": self.token_limit,
"gpt_model": self.gpt_model,
"user_api_key": self.user_api_key
}

@ -1,7 +1,7 @@
from application.retriever.base import BaseRetriever from application.retriever.base import BaseRetriever
from application.core.settings import settings from application.core.settings import settings
from application.llm.llm_creator import LLMCreator from application.llm.llm_creator import LLMCreator
from application.utils import num_tokens_from_string from application.utils import count_tokens
from langchain_community.tools import DuckDuckGoSearchResults from langchain_community.tools import DuckDuckGoSearchResults
from langchain_community.utilities import DuckDuckGoSearchAPIWrapper from langchain_community.utilities import DuckDuckGoSearchAPIWrapper
@ -15,7 +15,6 @@ class DuckDuckSearch(BaseRetriever):
chat_history, chat_history,
prompt, prompt,
chunks=2, chunks=2,
token_limit=150,
gpt_model="docsgpt", gpt_model="docsgpt",
user_api_key=None, user_api_key=None,
): ):
@ -25,16 +24,6 @@ class DuckDuckSearch(BaseRetriever):
self.prompt = prompt self.prompt = prompt
self.chunks = chunks self.chunks = chunks
self.gpt_model = gpt_model self.gpt_model = gpt_model
self.token_limit = (
token_limit
if token_limit
< settings.MODEL_TOKEN_LIMITS.get(
self.gpt_model, settings.DEFAULT_MAX_HISTORY
)
else settings.MODEL_TOKEN_LIMITS.get(
self.gpt_model, settings.DEFAULT_MAX_HISTORY
)
)
self.user_api_key = user_api_key self.user_api_key = user_api_key
def _parse_lang_string(self, input_string): def _parse_lang_string(self, input_string):
@ -95,10 +84,13 @@ class DuckDuckSearch(BaseRetriever):
self.chat_history.reverse() self.chat_history.reverse()
for i in self.chat_history: for i in self.chat_history:
if "prompt" in i and "response" in i: if "prompt" in i and "response" in i:
tokens_batch = num_tokens_from_string(i["prompt"]) + num_tokens_from_string( tokens_batch = count_tokens(i["prompt"]) + count_tokens(
i["response"] i["response"]
) )
if tokens_current_history + tokens_batch < self.token_limit: if (
tokens_current_history + tokens_batch
< settings.TOKENS_MAX_HISTORY
):
tokens_current_history += tokens_batch tokens_current_history += tokens_batch
messages_combine.append( messages_combine.append(
{"role": "user", "content": i["prompt"]} {"role": "user", "content": i["prompt"]}
@ -118,15 +110,3 @@ class DuckDuckSearch(BaseRetriever):
def search(self): def search(self):
return self._get_data() return self._get_data()
def get_params(self):
return {
"question": self.question,
"source": self.source,
"chat_history": self.chat_history,
"prompt": self.prompt,
"chunks": self.chunks,
"token_limit": self.token_limit,
"gpt_model": self.gpt_model,
"user_api_key": self.user_api_key
}

@ -5,16 +5,15 @@ from application.retriever.brave_search import BraveRetSearch
class RetrieverCreator: class RetrieverCreator:
retrievers = { retievers = {
'classic': ClassicRAG, 'classic': ClassicRAG,
'duckduck_search': DuckDuckSearch, 'duckduck_search': DuckDuckSearch,
'brave_search': BraveRetSearch, 'brave_search': BraveRetSearch
'default': ClassicRAG
} }
@classmethod @classmethod
def create_retriever(cls, type, *args, **kwargs): def create_retriever(cls, type, *args, **kwargs):
retiever_class = cls.retrievers.get(type.lower()) retiever_class = cls.retievers.get(type.lower())
if not retiever_class: if not retiever_class:
raise ValueError(f"No retievers class found for type {type}") raise ValueError(f"No retievers class found for type {type}")
return retiever_class(*args, **kwargs) return retiever_class(*args, **kwargs)

@ -2,7 +2,7 @@ import sys
from pymongo import MongoClient from pymongo import MongoClient
from datetime import datetime from datetime import datetime
from application.core.settings import settings from application.core.settings import settings
from application.utils import num_tokens_from_string from application.utils import count_tokens
mongo = MongoClient(settings.MONGO_URI) mongo = MongoClient(settings.MONGO_URI)
db = mongo["docsgpt"] db = mongo["docsgpt"]
@ -24,9 +24,9 @@ def update_token_usage(user_api_key, token_usage):
def gen_token_usage(func): def gen_token_usage(func):
def wrapper(self, model, messages, stream, **kwargs): def wrapper(self, model, messages, stream, **kwargs):
for message in messages: for message in messages:
self.token_usage["prompt_tokens"] += num_tokens_from_string(message["content"]) self.token_usage["prompt_tokens"] += count_tokens(message["content"])
result = func(self, model, messages, stream, **kwargs) result = func(self, model, messages, stream, **kwargs)
self.token_usage["generated_tokens"] += num_tokens_from_string(result) self.token_usage["generated_tokens"] += count_tokens(result)
update_token_usage(self.user_api_key, self.token_usage) update_token_usage(self.user_api_key, self.token_usage)
return result return result
@ -36,14 +36,14 @@ def gen_token_usage(func):
def stream_token_usage(func): def stream_token_usage(func):
def wrapper(self, model, messages, stream, **kwargs): def wrapper(self, model, messages, stream, **kwargs):
for message in messages: for message in messages:
self.token_usage["prompt_tokens"] += num_tokens_from_string(message["content"]) self.token_usage["prompt_tokens"] += count_tokens(message["content"])
batch = [] batch = []
result = func(self, model, messages, stream, **kwargs) result = func(self, model, messages, stream, **kwargs)
for r in result: for r in result:
batch.append(r) batch.append(r)
yield r yield r
for line in batch: for line in batch:
self.token_usage["generated_tokens"] += num_tokens_from_string(line) self.token_usage["generated_tokens"] += count_tokens(line)
update_token_usage(self.user_api_key, self.token_usage) update_token_usage(self.user_api_key, self.token_usage)
return wrapper return wrapper

@ -1,22 +1,6 @@
import tiktoken from transformers import GPT2TokenizerFast
_encoding = None
def get_encoding(): def count_tokens(string):
global _encoding tokenizer = GPT2TokenizerFast.from_pretrained('gpt2')
if _encoding is None: return len(tokenizer(string)['input_ids'])
_encoding = tiktoken.get_encoding("cl100k_base")
return _encoding
def num_tokens_from_string(string: str) -> int:
encoding = get_encoding()
num_tokens = len(encoding.encode(string))
return num_tokens
def count_tokens_docs(docs):
docs_content = ""
for doc in docs:
docs_content += doc.page_content
tokens = num_tokens_from_string(docs_content)
return tokens

@ -1,55 +1,13 @@
from abc import ABC, abstractmethod from abc import ABC, abstractmethod
import os import os
from sentence_transformers import SentenceTransformer from langchain_community.embeddings import (
HuggingFaceEmbeddings,
CohereEmbeddings,
HuggingFaceInstructEmbeddings,
)
from langchain_openai import OpenAIEmbeddings from langchain_openai import OpenAIEmbeddings
from application.core.settings import settings from application.core.settings import settings
class EmbeddingsWrapper:
def __init__(self, model_name, *args, **kwargs):
self.model = SentenceTransformer(model_name, config_kwargs={'allow_dangerous_deserialization': True}, *args, **kwargs)
self.dimension = self.model.get_sentence_embedding_dimension()
def embed_query(self, query: str):
return self.model.encode(query).tolist()
def embed_documents(self, documents: list):
return self.model.encode(documents).tolist()
def __call__(self, text):
if isinstance(text, str):
return self.embed_query(text)
elif isinstance(text, list):
return self.embed_documents(text)
else:
raise ValueError("Input must be a string or a list of strings")
class EmbeddingsSingleton:
_instances = {}
@staticmethod
def get_instance(embeddings_name, *args, **kwargs):
if embeddings_name not in EmbeddingsSingleton._instances:
EmbeddingsSingleton._instances[embeddings_name] = EmbeddingsSingleton._create_instance(
embeddings_name, *args, **kwargs
)
return EmbeddingsSingleton._instances[embeddings_name]
@staticmethod
def _create_instance(embeddings_name, *args, **kwargs):
embeddings_factory = {
"openai_text-embedding-ada-002": OpenAIEmbeddings,
"huggingface_sentence-transformers/all-mpnet-base-v2": lambda: EmbeddingsWrapper("sentence-transformers/all-mpnet-base-v2"),
"huggingface_sentence-transformers-all-mpnet-base-v2": lambda: EmbeddingsWrapper("sentence-transformers/all-mpnet-base-v2"),
"huggingface_hkunlp/instructor-large": lambda: EmbeddingsWrapper("hkunlp/instructor-large"),
}
if embeddings_name in embeddings_factory:
return embeddings_factory[embeddings_name](*args, **kwargs)
else:
return EmbeddingsWrapper(embeddings_name, *args, **kwargs)
class BaseVectorStore(ABC): class BaseVectorStore(ABC):
def __init__(self): def __init__(self):
pass pass
@ -62,28 +20,37 @@ class BaseVectorStore(ABC):
return settings.OPENAI_API_BASE and settings.OPENAI_API_VERSION and settings.AZURE_DEPLOYMENT_NAME return settings.OPENAI_API_BASE and settings.OPENAI_API_VERSION and settings.AZURE_DEPLOYMENT_NAME
def _get_embeddings(self, embeddings_name, embeddings_key=None): def _get_embeddings(self, embeddings_name, embeddings_key=None):
embeddings_factory = {
"openai_text-embedding-ada-002": OpenAIEmbeddings,
"huggingface_sentence-transformers/all-mpnet-base-v2": HuggingFaceEmbeddings,
"huggingface_hkunlp/instructor-large": HuggingFaceInstructEmbeddings,
"cohere_medium": CohereEmbeddings
}
if embeddings_name not in embeddings_factory:
raise ValueError(f"Invalid embeddings_name: {embeddings_name}")
if embeddings_name == "openai_text-embedding-ada-002": if embeddings_name == "openai_text-embedding-ada-002":
if self.is_azure_configured(): if self.is_azure_configured():
os.environ["OPENAI_API_TYPE"] = "azure" os.environ["OPENAI_API_TYPE"] = "azure"
embedding_instance = EmbeddingsSingleton.get_instance( embedding_instance = embeddings_factory[embeddings_name](
embeddings_name,
model=settings.AZURE_EMBEDDINGS_DEPLOYMENT_NAME model=settings.AZURE_EMBEDDINGS_DEPLOYMENT_NAME
) )
else: else:
embedding_instance = EmbeddingsSingleton.get_instance( embedding_instance = embeddings_factory[embeddings_name](
embeddings_name,
openai_api_key=embeddings_key openai_api_key=embeddings_key
) )
elif embeddings_name == "cohere_medium":
embedding_instance = embeddings_factory[embeddings_name](
cohere_api_key=embeddings_key
)
elif embeddings_name == "huggingface_sentence-transformers/all-mpnet-base-v2": elif embeddings_name == "huggingface_sentence-transformers/all-mpnet-base-v2":
if os.path.exists("./model/all-mpnet-base-v2"): embedding_instance = embeddings_factory[embeddings_name](
embedding_instance = EmbeddingsSingleton.get_instance( #model_name="./model/all-mpnet-base-v2",
embeddings_name="./model/all-mpnet-base-v2", model_kwargs={"device": "cpu"},
) )
else:
embedding_instance = EmbeddingsSingleton.get_instance(
embeddings_name,
)
else: else:
embedding_instance = EmbeddingsSingleton.get_instance(embeddings_name) embedding_instance = embeddings_factory[embeddings_name]()
return embedding_instance return embedding_instance

@ -9,9 +9,9 @@ import elasticsearch
class ElasticsearchStore(BaseVectorStore): class ElasticsearchStore(BaseVectorStore):
_es_connection = None # Class attribute to hold the Elasticsearch connection _es_connection = None # Class attribute to hold the Elasticsearch connection
def __init__(self, source_id, embeddings_key, index_name=settings.ELASTIC_INDEX): def __init__(self, path, embeddings_key, index_name=settings.ELASTIC_INDEX):
super().__init__() super().__init__()
self.source_id = source_id.replace("application/indexes/", "").rstrip("/") self.path = path.replace("application/indexes/", "").rstrip("/")
self.embeddings_key = embeddings_key self.embeddings_key = embeddings_key
self.index_name = index_name self.index_name = index_name
@ -81,7 +81,7 @@ class ElasticsearchStore(BaseVectorStore):
embeddings = self._get_embeddings(settings.EMBEDDINGS_NAME, self.embeddings_key) embeddings = self._get_embeddings(settings.EMBEDDINGS_NAME, self.embeddings_key)
vector = embeddings.embed_query(question) vector = embeddings.embed_query(question)
knn = { knn = {
"filter": [{"match": {"metadata.source_id.keyword": self.source_id}}], "filter": [{"match": {"metadata.store.keyword": self.path}}],
"field": "vector", "field": "vector",
"k": k, "k": k,
"num_candidates": 100, "num_candidates": 100,
@ -100,7 +100,7 @@ class ElasticsearchStore(BaseVectorStore):
} }
} }
], ],
"filter": [{"match": {"metadata.source_id.keyword": self.source_id}}], "filter": [{"match": {"metadata.store.keyword": self.path}}],
} }
}, },
"rank": {"rrf": {}}, "rank": {"rrf": {}},
@ -209,4 +209,5 @@ class ElasticsearchStore(BaseVectorStore):
def delete_index(self): def delete_index(self):
self._es_connection.delete_by_query(index=self.index_name, query={"match": { self._es_connection.delete_by_query(index=self.index_name, query={"match": {
"metadata.source_id.keyword": self.source_id}},) "metadata.store.keyword": self.path}},)

@ -1,22 +1,12 @@
from langchain_community.vectorstores import FAISS from langchain_community.vectorstores import FAISS
from application.vectorstore.base import BaseVectorStore from application.vectorstore.base import BaseVectorStore
from application.core.settings import settings from application.core.settings import settings
import os
def get_vectorstore(path):
if path:
vectorstore = "indexes/"+path
vectorstore = os.path.join("application", vectorstore)
else:
vectorstore = os.path.join("application")
return vectorstore
class FaissStore(BaseVectorStore): class FaissStore(BaseVectorStore):
def __init__(self, source_id, embeddings_key, docs_init=None): def __init__(self, path, embeddings_key, docs_init=None):
super().__init__() super().__init__()
self.path = get_vectorstore(source_id) self.path = path
embeddings = self._get_embeddings(settings.EMBEDDINGS_NAME, embeddings_key) embeddings = self._get_embeddings(settings.EMBEDDINGS_NAME, embeddings_key)
if docs_init: if docs_init:
self.docsearch = FAISS.from_documents( self.docsearch = FAISS.from_documents(
@ -24,8 +14,7 @@ class FaissStore(BaseVectorStore):
) )
else: else:
self.docsearch = FAISS.load_local( self.docsearch = FAISS.load_local(
self.path, embeddings, self.path, embeddings
allow_dangerous_deserialization=True
) )
self.assert_embedding_dimensions(embeddings) self.assert_embedding_dimensions(embeddings)
@ -48,10 +37,10 @@ class FaissStore(BaseVectorStore):
""" """
if settings.EMBEDDINGS_NAME == "huggingface_sentence-transformers/all-mpnet-base-v2": if settings.EMBEDDINGS_NAME == "huggingface_sentence-transformers/all-mpnet-base-v2":
try: try:
word_embedding_dimension = embeddings.dimension word_embedding_dimension = embeddings.client[1].word_embedding_dimension
except AttributeError as e: except AttributeError as e:
raise AttributeError("'dimension' attribute not found in embeddings instance. Make sure the embeddings object is properly initialized.") from e raise AttributeError("word_embedding_dimension not found in embeddings.client[1]") from e
docsearch_index_dimension = self.docsearch.index.d docsearch_index_dimension = self.docsearch.index.d
if word_embedding_dimension != docsearch_index_dimension: if word_embedding_dimension != docsearch_index_dimension:
raise ValueError(f"Embedding dimension mismatch: embeddings.dimension ({word_embedding_dimension}) " + raise ValueError(f"word_embedding_dimension ({word_embedding_dimension}) " +
f"!= docsearch index dimension ({docsearch_index_dimension})") f"!= docsearch_index_word_embedding_dimension ({docsearch_index_dimension})")

@ -1,37 +0,0 @@
from typing import List, Optional
from uuid import uuid4
from application.core.settings import settings
from application.vectorstore.base import BaseVectorStore
class MilvusStore(BaseVectorStore):
def __init__(self, path: str = "", embeddings_key: str = "embeddings"):
super().__init__()
from langchain_milvus import Milvus
connection_args = {
"uri": settings.MILVUS_URI,
"token": settings.MILVUS_TOKEN,
}
self._docsearch = Milvus(
embedding_function=self._get_embeddings(settings.EMBEDDINGS_NAME, embeddings_key),
collection_name=settings.MILVUS_COLLECTION_NAME,
connection_args=connection_args,
)
self._path = path
def search(self, question, k=2, *args, **kwargs):
return self._docsearch.similarity_search(query=question, k=k, filter={"path": self._path} *args, **kwargs)
def add_texts(self, texts: List[str], metadatas: Optional[List[dict]], *args, **kwargs):
ids = [str(uuid4()) for _ in range(len(texts))]
return self._docsearch.add_texts(texts=texts, metadatas=metadatas, ids=ids, *args, **kwargs)
def save_local(self, *args, **kwargs):
pass
def delete_index(self, *args, **kwargs):
pass

@ -1,12 +1,11 @@
from application.core.settings import settings
from application.vectorstore.base import BaseVectorStore from application.vectorstore.base import BaseVectorStore
from application.core.settings import settings
from application.vectorstore.document_class import Document from application.vectorstore.document_class import Document
class MongoDBVectorStore(BaseVectorStore): class MongoDBVectorStore(BaseVectorStore):
def __init__( def __init__(
self, self,
source_id: str = "", path: str = "",
embeddings_key: str = "embeddings", embeddings_key: str = "embeddings",
collection: str = "documents", collection: str = "documents",
index_name: str = "vector_search_index", index_name: str = "vector_search_index",
@ -19,7 +18,7 @@ class MongoDBVectorStore(BaseVectorStore):
self._embedding_key = embedding_key self._embedding_key = embedding_key
self._embeddings_key = embeddings_key self._embeddings_key = embeddings_key
self._mongo_uri = settings.MONGO_URI self._mongo_uri = settings.MONGO_URI
self._source_id = source_id.replace("application/indexes/", "").rstrip("/") self._path = path.replace("application/indexes/", "").rstrip("/")
self._embedding = self._get_embeddings(settings.EMBEDDINGS_NAME, embeddings_key) self._embedding = self._get_embeddings(settings.EMBEDDINGS_NAME, embeddings_key)
try: try:
@ -34,6 +33,7 @@ class MongoDBVectorStore(BaseVectorStore):
self._database = self._client[database] self._database = self._client[database]
self._collection = self._database[collection] self._collection = self._database[collection]
def search(self, question, k=2, *args, **kwargs): def search(self, question, k=2, *args, **kwargs):
query_vector = self._embedding.embed_query(question) query_vector = self._embedding.embed_query(question)
@ -45,7 +45,9 @@ class MongoDBVectorStore(BaseVectorStore):
"limit": k, "limit": k,
"numCandidates": k * 10, "numCandidates": k * 10,
"index": self._index_name, "index": self._index_name,
"filter": {"source_id": {"$eq": self._source_id}}, "filter": {
"store": {"$eq": self._path}
}
} }
} }
] ]
@ -66,27 +68,25 @@ class MongoDBVectorStore(BaseVectorStore):
if not texts: if not texts:
return [] return []
embeddings = self._embedding.embed_documents(texts) embeddings = self._embedding.embed_documents(texts)
to_insert = [ to_insert = [
{self._text_key: t, self._embedding_key: embedding, **m} {self._text_key: t, self._embedding_key: embedding, **m}
for t, m, embedding in zip(texts, metadatas, embeddings) for t, m, embedding in zip(texts, metadatas, embeddings)
] ]
# insert the documents in MongoDB Atlas
insert_result = self._collection.insert_many(to_insert) insert_result = self._collection.insert_many(to_insert)
return insert_result.inserted_ids return insert_result.inserted_ids
def add_texts( def add_texts(self,
self,
texts, texts,
metadatas=None, metadatas = None,
ids=None, ids = None,
refresh_indices=True, refresh_indices = True,
create_index_if_not_exists=True, create_index_if_not_exists = True,
bulk_kwargs=None, bulk_kwargs = None,
**kwargs, **kwargs,):
):
# dims = self._embedding.client[1].word_embedding_dimension #dims = self._embedding.client[1].word_embedding_dimension
# # check if index exists # # check if index exists
# if create_index_if_not_exists: # if create_index_if_not_exists:
# # check if index exists # # check if index exists
@ -123,4 +123,4 @@ class MongoDBVectorStore(BaseVectorStore):
return result_ids return result_ids
def delete_index(self, *args, **kwargs): def delete_index(self, *args, **kwargs):
self._collection.delete_many({"source_id": self._source_id}) self._collection.delete_many({"store": self._path})

@ -5,12 +5,12 @@ from qdrant_client import models
class QdrantStore(BaseVectorStore): class QdrantStore(BaseVectorStore):
def __init__(self, source_id: str = "", embeddings_key: str = "embeddings"): def __init__(self, path: str = "", embeddings_key: str = "embeddings"):
self._filter = models.Filter( self._filter = models.Filter(
must=[ must=[
models.FieldCondition( models.FieldCondition(
key="metadata.source_id", key="metadata.store",
match=models.MatchValue(value=source_id.replace("application/indexes/", "").rstrip("/")), match=models.MatchValue(value=path.replace("application/indexes/", "").rstrip("/")),
) )
] ]
) )

@ -1,6 +1,5 @@
from application.vectorstore.faiss import FaissStore from application.vectorstore.faiss import FaissStore
from application.vectorstore.elasticsearch import ElasticsearchStore from application.vectorstore.elasticsearch import ElasticsearchStore
from application.vectorstore.milvus import MilvusStore
from application.vectorstore.mongodb import MongoDBVectorStore from application.vectorstore.mongodb import MongoDBVectorStore
from application.vectorstore.qdrant import QdrantStore from application.vectorstore.qdrant import QdrantStore
@ -11,7 +10,6 @@ class VectorCreator:
"elasticsearch": ElasticsearchStore, "elasticsearch": ElasticsearchStore,
"mongodb": MongoDBVectorStore, "mongodb": MongoDBVectorStore,
"qdrant": QdrantStore, "qdrant": QdrantStore,
"milvus": MilvusStore,
} }
@classmethod @classmethod

@ -1,31 +1,30 @@
import logging
import os import os
import shutil import shutil
import string import string
import zipfile import zipfile
from collections import Counter
from urllib.parse import urljoin from urllib.parse import urljoin
import nltk
import requests import requests
from bson.objectid import ObjectId
from pymongo import MongoClient
from application.core.settings import settings from application.core.settings import settings
from application.parser.file.bulk import SimpleDirectoryReader from application.parser.file.bulk import SimpleDirectoryReader
from application.parser.open_ai_func import call_openai_api
from application.parser.remote.remote_creator import RemoteCreator from application.parser.remote.remote_creator import RemoteCreator
from application.parser.open_ai_func import call_openai_api
from application.parser.schema.base import Document from application.parser.schema.base import Document
from application.parser.token_func import group_split from application.parser.token_func import group_split
from application.utils import count_tokens_docs
mongo = MongoClient(settings.MONGO_URI) try:
db = mongo["docsgpt"] nltk.download("punkt", quiet=True)
sources_collection = db["sources"] nltk.download("averaged_perceptron_tagger", quiet=True)
except FileExistsError:
pass
# Define a function to extract metadata from a given filename. # Define a function to extract metadata from a given filename.
def metadata_from_filename(title): def metadata_from_filename(title):
return {"title": title} store = "/".join(title.split("/")[1:3])
return {"title": title, "store": store}
# Define a function to generate a random string of a given length. # Define a function to generate a random string of a given length.
@ -37,7 +36,6 @@ current_dir = os.path.dirname(
os.path.dirname(os.path.dirname(os.path.abspath(__file__))) os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
) )
def extract_zip_recursive(zip_path, extract_to, current_depth=0, max_depth=5): def extract_zip_recursive(zip_path, extract_to, current_depth=0, max_depth=5):
""" """
Recursively extract zip files with a limit on recursion depth. Recursively extract zip files with a limit on recursion depth.
@ -49,10 +47,10 @@ def extract_zip_recursive(zip_path, extract_to, current_depth=0, max_depth=5):
max_depth (int): Maximum allowed depth of recursion to prevent infinite loops. max_depth (int): Maximum allowed depth of recursion to prevent infinite loops.
""" """
if current_depth > max_depth: if current_depth > max_depth:
logging.warning(f"Reached maximum recursion depth of {max_depth}") print(f"Reached maximum recursion depth of {max_depth}")
return return
with zipfile.ZipFile(zip_path, "r") as zip_ref: with zipfile.ZipFile(zip_path, 'r') as zip_ref:
zip_ref.extractall(extract_to) zip_ref.extractall(extract_to)
os.remove(zip_path) # Remove the zip file after extracting os.remove(zip_path) # Remove the zip file after extracting
@ -66,9 +64,7 @@ def extract_zip_recursive(zip_path, extract_to, current_depth=0, max_depth=5):
# Define the main function for ingesting and processing documents. # Define the main function for ingesting and processing documents.
def ingest_worker( def ingest_worker(self, directory, formats, name_job, filename, user):
self, directory, formats, name_job, filename, user, retriever="classic"
):
""" """
Ingest and process documents. Ingest and process documents.
@ -79,7 +75,6 @@ def ingest_worker(
name_job (str): Name of the job for this ingestion task. name_job (str): Name of the job for this ingestion task.
filename (str): Name of the file to be ingested. filename (str): Name of the file to be ingested.
user (str): Identifier for the user initiating the ingestion. user (str): Identifier for the user initiating the ingestion.
retriever (str): Type of retriever to use for processing the documents.
Returns: Returns:
dict: Information about the completed ingestion task, including input parameters and a "limited" flag. dict: Information about the completed ingestion task, including input parameters and a "limited" flag.
@ -99,13 +94,17 @@ def ingest_worker(
max_tokens = 1250 max_tokens = 1250
recursion_depth = 2 recursion_depth = 2
full_path = os.path.join(directory, user, name_job) full_path = os.path.join(directory, user, name_job)
import sys
logging.info(f"Ingest file: {full_path}", extra={"user": user, "job": name_job})
print(full_path, file=sys.stderr)
# check if API_URL env variable is set # check if API_URL env variable is set
file_data = {"name": name_job, "file": filename, "user": user} file_data = {"name": name_job, "file": filename, "user": user}
response = requests.get( response = requests.get(
urljoin(settings.API_URL, "/api/download"), params=file_data urljoin(settings.API_URL, "/api/download"), params=file_data
) )
# check if file is in the response
print(response, file=sys.stderr)
file = response.content file = response.content
if not os.path.exists(full_path): if not os.path.exists(full_path):
@ -115,9 +114,7 @@ def ingest_worker(
# check if file is .zip and extract it # check if file is .zip and extract it
if filename.endswith(".zip"): if filename.endswith(".zip"):
extract_zip_recursive( extract_zip_recursive(os.path.join(full_path, filename), full_path, 0, recursion_depth)
os.path.join(full_path, filename), full_path, 0, recursion_depth
)
self.update_state(state="PROGRESS", meta={"current": 1}) self.update_state(state="PROGRESS", meta={"current": 1})
@ -138,26 +135,17 @@ def ingest_worker(
) )
docs = [Document.to_langchain_format(raw_doc) for raw_doc in raw_docs] docs = [Document.to_langchain_format(raw_doc) for raw_doc in raw_docs]
id = ObjectId()
call_openai_api(docs, full_path, id, self) call_openai_api(docs, full_path, self)
tokens = count_tokens_docs(docs)
self.update_state(state="PROGRESS", meta={"current": 100}) self.update_state(state="PROGRESS", meta={"current": 100})
if sample: if sample:
for i in range(min(5, len(raw_docs))): for i in range(min(5, len(raw_docs))):
logging.info(f"Sample document {i}: {raw_docs[i]}") print(raw_docs[i].text)
# get files from outputs/inputs/index.faiss and outputs/inputs/index.pkl # get files from outputs/inputs/index.faiss and outputs/inputs/index.pkl
# and send them to the server (provide user and name in form) # and send them to the server (provide user and name in form)
file_data = { file_data = {"name": name_job, "user": user}
"name": name_job,
"user": user,
"tokens": tokens,
"retriever": retriever,
"id": str(id),
"type": "local",
}
if settings.VECTOR_STORE == "faiss": if settings.VECTOR_STORE == "faiss":
files = { files = {
"file_faiss": open(full_path + "/index.faiss", "rb"), "file_faiss": open(full_path + "/index.faiss", "rb"),
@ -166,6 +154,9 @@ def ingest_worker(
response = requests.post( response = requests.post(
urljoin(settings.API_URL, "/api/upload_index"), files=files, data=file_data urljoin(settings.API_URL, "/api/upload_index"), files=files, data=file_data
) )
response = requests.get(
urljoin(settings.API_URL, "/api/delete_old?path=" + full_path)
)
else: else:
response = requests.post( response = requests.post(
urljoin(settings.API_URL, "/api/upload_index"), data=file_data urljoin(settings.API_URL, "/api/upload_index"), data=file_data
@ -184,18 +175,8 @@ def ingest_worker(
} }
def remote_worker( def remote_worker(self, source_data, name_job, user, loader, directory="temp"):
self, # sample = False
source_data,
name_job,
user,
loader,
directory="temp",
retriever="classic",
sync_frequency="never",
operation_mode="upload",
doc_id=None,
):
token_check = True token_check = True
min_tokens = 150 min_tokens = 150
max_tokens = 1250 max_tokens = 1250
@ -203,12 +184,12 @@ def remote_worker(
if not os.path.exists(full_path): if not os.path.exists(full_path):
os.makedirs(full_path) os.makedirs(full_path)
self.update_state(state="PROGRESS", meta={"current": 1}) self.update_state(state="PROGRESS", meta={"current": 1})
logging.info(
f"Remote job: {full_path}",
extra={"user": user, "job": name_job, source_data: source_data},
)
# source_data {"data": [url]} for url type task just urls
# Use RemoteCreator to load data from URL
remote_loader = RemoteCreator.create_loader(loader) remote_loader = RemoteCreator.create_loader(loader)
raw_docs = remote_loader.load_data(source_data) raw_docs = remote_loader.load_data(source_data)
@ -218,95 +199,26 @@ def remote_worker(
max_tokens=max_tokens, max_tokens=max_tokens,
token_check=token_check, token_check=token_check,
) )
# docs = [Document.to_langchain_format(raw_doc) for raw_doc in raw_docs] # docs = [Document.to_langchain_format(raw_doc) for raw_doc in raw_docs]
tokens = count_tokens_docs(docs)
if operation_mode == "upload": call_openai_api(docs, full_path, self)
id = ObjectId()
call_openai_api(docs, full_path, id, self)
elif operation_mode == "sync":
if not doc_id or not ObjectId.is_valid(doc_id):
raise ValueError("doc_id must be provided for sync operation.")
id = ObjectId(doc_id)
call_openai_api(docs, full_path, id, self)
self.update_state(state="PROGRESS", meta={"current": 100}) self.update_state(state="PROGRESS", meta={"current": 100})
# Proceed with uploading and cleaning as in the original function # Proceed with uploading and cleaning as in the original function
file_data = { file_data = {"name": name_job, "user": user}
"name": name_job,
"user": user,
"tokens": tokens,
"retriever": retriever,
"id": str(id),
"type": loader,
"remote_data": source_data,
"sync_frequency": sync_frequency,
}
if settings.VECTOR_STORE == "faiss": if settings.VECTOR_STORE == "faiss":
files = { files = {
"file_faiss": open(full_path + "/index.faiss", "rb"), "file_faiss": open(full_path + "/index.faiss", "rb"),
"file_pkl": open(full_path + "/index.pkl", "rb"), "file_pkl": open(full_path + "/index.pkl", "rb"),
} }
requests.post( requests.post(
urljoin(settings.API_URL, "/api/upload_index"), files=files, data=file_data urljoin(settings.API_URL, "/api/upload_index"), files=files, data=file_data
) )
requests.get(urljoin(settings.API_URL, "/api/delete_old?path=" + full_path))
else: else:
requests.post(urljoin(settings.API_URL, "/api/upload_index"), data=file_data) requests.post(urljoin(settings.API_URL, "/api/upload_index"), data=file_data)
shutil.rmtree(full_path) shutil.rmtree(full_path)
return {"urls": source_data, "name_job": name_job, "user": user, "limited": False} return {"urls": source_data, "name_job": name_job, "user": user, "limited": False}
def sync(
self,
source_data,
name_job,
user,
loader,
sync_frequency,
retriever,
doc_id=None,
directory="temp",
):
try:
remote_worker(
self,
source_data,
name_job,
user,
loader,
directory,
retriever,
sync_frequency,
"sync",
doc_id,
)
except Exception as e:
return {"status": "error", "error": str(e)}
return {"status": "success"}
def sync_worker(self, frequency):
sync_counts = Counter()
sources = sources_collection.find()
for doc in sources:
if doc.get("sync_frequency") == frequency:
name = doc.get("name")
user = doc.get("user")
source_type = doc.get("type")
source_data = doc.get("remote_data")
retriever = doc.get("retriever")
doc_id = str(doc.get("_id"))
resp = sync(
self, source_data, name, user, source_type, frequency, retriever, doc_id
)
sync_counts["total_sync_count"] += 1
sync_counts[
"sync_success" if resp["status"] == "success" else "sync_failure"
] += 1
return {
key: sync_counts[key]
for key in ["total_sync_count", "sync_success", "sync_failure"]
}

@ -1,3 +1,5 @@
version: "3.9"
services: services:
frontend: frontend:
build: ./frontend build: ./frontend

@ -1,3 +1,5 @@
version: "3.9"
services: services:
redis: redis:

@ -1,8 +1,8 @@
version: "3.9"
services: services:
frontend: frontend:
build: ./frontend build: ./frontend
volumes:
- ./frontend/src:/app/src
environment: environment:
- VITE_API_HOST=http://localhost:7091 - VITE_API_HOST=http://localhost:7091
- VITE_API_STREAMING=$VITE_API_STREAMING - VITE_API_STREAMING=$VITE_API_STREAMING

@ -1,3 +1,5 @@
version: "3.9"
services: services:
frontend: frontend:
build: ./frontend build: ./frontend

@ -1,3 +1,5 @@
version: "3.9"
services: services:
frontend: frontend:
build: ./frontend build: ./frontend
@ -30,7 +32,7 @@ services:
worker: worker:
build: ./application build: ./application
command: celery -A application.app.celery worker -l INFO -B command: celery -A application.app.celery worker -l INFO
environment: environment:
- API_KEY=$API_KEY - API_KEY=$API_KEY
- EMBEDDINGS_KEY=$API_KEY - EMBEDDINGS_KEY=$API_KEY

2339
docs/package-lock.json generated

File diff suppressed because it is too large Load Diff

@ -7,8 +7,8 @@
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"@vercel/analytics": "^1.1.1", "@vercel/analytics": "^1.1.1",
"docsgpt": "^0.4.1", "docsgpt": "^0.3.7",
"next": "^14.2.12", "next": "^14.0.4",
"nextra": "^2.13.2", "nextra": "^2.13.2",
"nextra-theme-docs": "^2.13.2", "nextra-theme-docs": "^2.13.2",
"react": "^18.2.0", "react": "^18.2.0",

@ -1,10 +0,0 @@
{
"API-docs": {
"title": "🗂️️ API-docs",
"href": "/API/API-docs"
},
"api-key-guide": {
"title": "🔐 API Keys guide",
"href": "/API/api-key-guide"
}
}

@ -1,100 +0,0 @@
# Self-hosting DocsGPT on Kubernetes
This guide will walk you through deploying DocsGPT on Kubernetes.
## Prerequisites
Ensure you have the following installed before proceeding:
- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
- Access to a Kubernetes cluster
## Folder Structure
The `k8s` folder contains the necessary deployment and service configuration files:
- `deployments/`
- `services/`
- `docsgpt-secrets.yaml`
## Deployment Instructions
1. **Clone the Repository**
```sh
git clone https://github.com/arc53/DocsGPT.git
cd docsgpt/k8s
```
2. **Configure Secrets (optional)**
Ensure that you have all the necessary secrets in `docsgpt-secrets.yaml`. Update it with your secrets before applying if you want. By default we will use qdrant as a vectorstore and public docsgpt llm as llm for inference.
3. **Apply Kubernetes Deployments**
Deploy your DocsGPT resources using the following commands:
```sh
kubectl apply -f deployments/
```
4. **Apply Kubernetes Services**
Set up your services using the following commands:
```sh
kubectl apply -f services/
```
5. **Apply Secrets**
Apply the secret configurations:
```sh
kubectl apply -f docsgpt-secrets.yaml
```
6. **Substitute API URL**
After deploying the services, you need to update the environment variable `VITE_API_HOST` in your deployment file `deployments/docsgpt-deploy.yaml` with the actual endpoint URL created by your `docsgpt-api-service`.
```sh
kubectl get services/docsgpt-api-service -o jsonpath='{.status.loadBalancer.ingress[0].ip}' | xargs -I {} sed -i "s|<your-api-endpoint>|{}|g" deployments/docsgpt-deploy.yaml
```
7. **Rerun Deployment**
After making the changes, reapply the deployment configuration to update the environment variables:
```sh
kubectl apply -f deployments/
```
## Verifying the Deployment
To verify if everything is set up correctly, you can run the following:
```sh
kubectl get pods
kubectl get services
```
Ensure that the pods are running and the services are available.
## Accessing DocsGPT
To access DocsGPT, you need to find the external IP address of the frontend service. You can do this by running:
```sh
kubectl get services/docsgpt-frontend-service | awk 'NR>1 {print "http://" $4}'
```
## Troubleshooting
If you encounter any issues, you can check the logs of the pods for more details:
```sh
kubectl logs <pod-name>
```
Replace `<pod-name>` with the actual name of your DocsGPT pod.

@ -67,3 +67,62 @@ To run the setup on Windows, you have two options: using the Windows Subsystem f
These steps should help you set up and run the project on Windows using either WSL or Git Bash/Command Prompt. These steps should help you set up and run the project on Windows using either WSL or Git Bash/Command Prompt.
**Important:** Ensure that Docker is installed and properly configured on your Windows system for these steps to work. **Important:** Ensure that Docker is installed and properly configured on your Windows system for these steps to work.
For WINDOWS:
To run the given setup on Windows, you can use the Windows Subsystem for Linux (WSL) or a Git Bash terminal to execute similar commands. Here are the steps adapted for Windows:
Option 1: Using Windows Subsystem for Linux (WSL):
1. Install WSL if you haven't already. You can follow the official Microsoft documentation for installation: (https://learn.microsoft.com/en-us/windows/wsl/install).
2. After setting up WSL, open the WSL terminal.
3. Clone the repository and create the `.env` file:
```bash
git clone https://github.com/arc53/DocsGPT.git
cd DocsGPT
echo "API_KEY=Yourkey" > .env
echo "VITE_API_STREAMING=true" >> .env
```
4. Run the following command to start the setup with Docker Compose:
```bash
./run-with-docker-compose.sh
```
5. Open your web browser and navigate to http://localhost:5173/.
6. To stop the setup, just press **Ctrl + C** in the WSL terminal.
Option 2: Using Git Bash or Command Prompt (CMD):
1. Install Git for Windows if you haven't already. You can download it from the official website: (https://gitforwindows.org/).
2. Open Git Bash or Command Prompt.
3. Clone the repository and create the `.env` file:
```bash
git clone https://github.com/arc53/DocsGPT.git
cd DocsGPT
echo "API_KEY=Yourkey" > .env
echo "VITE_API_STREAMING=true" >> .env
```
4. Run the following command to start the setup with Docker Compose:
```bash
./run-with-docker-compose.sh
```
5. Open your web browser and navigate to http://localhost:5173/.
6. To stop the setup, just press **Ctrl + C** in the Git Bash or Command Prompt terminal.
These steps should help you set up and run the project on Windows using either WSL or Git Bash/Command Prompt. Make sure you have Docker installed and properly configured on your Windows system for this to work.
### Chrome Extension
#### Installing the Chrome extension:
To enhance your DocsGPT experience, you can install the DocsGPT Chrome extension. Here's how:
1. In the DocsGPT GitHub repository, click on the **Code** button and select **Download ZIP**.
2. Unzip the downloaded file to a location you can easily access.
3. Open the Google Chrome browser and click on the three dots menu (upper right corner).
4. Select **More Tools** and then **Extensions**.
5. Turn on the **Developer mode** switch in the top right corner of the **Extensions page**.
6. Click on the **Load unpacked** button.
7. Select the **Chrome** folder where the DocsGPT files have been unzipped (docsgpt-main > extensions > chrome).
8. The extension should now be added to Google Chrome and can be managed on the Extensions page.
9. To disable or remove the extension, simply turn off the toggle switch on the extension card or click the **Remove** button.

@ -10,9 +10,5 @@
"Railway-Deploying": { "Railway-Deploying": {
"title": "🚂Deploying on Railway", "title": "🚂Deploying on Railway",
"href": "/Deploying/Railway-Deploying" "href": "/Deploying/Railway-Deploying"
},
"Kubernetes-Deploying": {
"title": "☸Deploying on Kubernetes",
"href": "/Deploying/Kubernetes-Deploying"
} }
} }

@ -0,0 +1,6 @@
{
"API-docs": {
"title": "🗂️️ API-docs",
"href": "/Developing/API-docs"
}
}

@ -1,34 +0,0 @@
import {Steps} from 'nextra/components'
import { Callout } from 'nextra/components'
## Chrome Extension Setup Guide
To enhance your DocsGPT experience, you can install the DocsGPT Chrome extension. Here's how:
<Steps >
### Step 1
In the DocsGPT GitHub repository, click on the **Code** button and select **Download ZIP**.
### Step 2
Unzip the downloaded file to a location you can easily access.
### Step 3
Open the Google Chrome browser and click on the three dots menu (upper right corner).
### Step 4
Select **More Tools** and then **Extensions**.
### Step 5
Turn on the **Developer mode** switch in the top right corner of the **Extensions page**.
### Step 6
Click on the **Load unpacked** button.
### Step 7
7. Select the **Chrome** folder where the DocsGPT files have been unzipped (docsgpt-main > extensions > chrome).
### Step 8
The extension should now be added to Google Chrome and can be managed on the Extensions page.
### Step 9
To disable or remove the extension, simply turn off the toggle switch on the extension card or click the **Remove** button.
</Steps>

@ -7,8 +7,8 @@
"title": "🏗️ Widget setup", "title": "🏗️ Widget setup",
"href": "/Extensions/react-widget" "href": "/Extensions/react-widget"
}, },
"Chrome-extension": { "api-key-guide": {
"title": "🌐 Chrome Extension", "title": "🔐 API Keys guide",
"href": "/Extensions/Chrome-extension" "href": "/Extensions/api-key-guide"
} }
} }

@ -14,7 +14,7 @@ Before creating your first API key, you must upload the document that will be li
After uploading your document, you can obtain an API key either through the graphical user interface or via an API call: After uploading your document, you can obtain an API key either through the graphical user interface or via an API call:
- **Graphical User Interface:** Navigate to the Settings section of the DocsGPT web app, find the API Keys option, and press 'Create New' to generate your key. - **Graphical User Interface:** Navigate to the Settings section of the DocsGPT web app, find the API Keys option, and press 'Create New' to generate your key.
- **API Call:** Alternatively, you can use the `/api/create_api_key` endpoint to create a new API key. For detailed instructions, visit [DocsGPT API Documentation](https://docs.docsgpt.cloud/API/API-docs#8-apicreate_api_key). - **API Call:** Alternatively, you can use the `/api/create_api_key` endpoint to create a new API key. For detailed instructions, visit [DocsGPT API Documentation](https://docs.docsgpt.co.uk/Developing/API-docs#8-apicreate_api_key).
### Understanding Key Variables ### Understanding Key Variables

@ -17,31 +17,25 @@ Now, you can use the widget in your component like this :
```jsx ```jsx
<DocsGPTWidget <DocsGPTWidget
apiHost="https://your-docsgpt-api.com" apiHost="https://your-docsgpt-api.com"
selectDocs="local/docs.zip"
apiKey="" apiKey=""
avatar = "https://d3dg1063dc54p9.cloudfront.net/cute-docsgpt.png" avatar = "https://d3dg1063dc54p9.cloudfront.net/cute-docsgpt.png",
title = "Get AI assistance" title = "Get AI assistance",
description = "DocsGPT's AI Chatbot is here to help" description = "DocsGPT's AI Chatbot is here to help",
heroTitle = "Welcome to DocsGPT !" heroTitle = "Welcome to DocsGPT !",
heroDescription="This chatbot is built with DocsGPT and utilises GenAI, heroDescription="This chatbot is built with DocsGPT and utilises GenAI,
please review important information using sources." please review important information using sources."
theme = "dark"
buttonIcon = "https://your-icon"
buttonBg = "#222327"
/> />
``` ```
To tailor the widget to your needs, you can configure the following props in your component: DocsGPTWidget takes 8 **props** with default fallback values:
1. `apiHost` — The URL of your DocsGPT API. 1. `apiHost` — The URL of your DocsGPT API.
2. `theme` — Allows to select your specific theme (dark or light). 2. `selectDocs` — The documentation source that you want to use for your widget (e.g. `default` or `local/docs1.zip`).
3. `apiKey` — Usually, it's empty. 3. `apiKey` — Usually, it's empty.
4. `avatar`: Specifies the URL of the avatar or image representing the chatbot. 4. `avatar`: Specifies the URL of the avatar or image representing the chatbot.
5. `title`: Sets the title text displayed in the chatbot interface. 5. `title`: Sets the title text displayed in the chatbot interface.
6. `description`: Provides a brief description of the chatbot's purpose or functionality. 6. `description`: Provides a brief description of the chatbot's purpose or functionality.
7. `heroTitle`: Displays a welcome title when users interact with the chatbot. 7. `heroTitle`: Displays a welcome title when users interact with the chatbot.
8. `heroDescription`: Provide additional introductory text or information about the chatbot's capabilities. 8. `heroDescription`: Provide additional introductory text or information about the chatbot's capabilities.
9. `buttonIcon`: Specifies the url of the icon image for the widget.
10. `buttonBg`: Allows to specify the Background color of the widget.
11. `size`: Sets the size of the widget ( small, medium).
### How to use DocsGPTWidget with [Nextra](https://nextra.site/) (Next.js + MDX) ### How to use DocsGPTWidget with [Nextra](https://nextra.site/) (Next.js + MDX)
Install your widget as described above and then go to your `pages/` folder and create a new file `_app.js` with the following content: Install your widget as described above and then go to your `pages/` folder and create a new file `_app.js` with the following content:
@ -57,69 +51,6 @@ export default function MyApp({ Component, pageProps }) {
) )
} }
``` ```
### How to use DocsGPTWidget with HTML
```html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta http-equiv="X-UA-Compatible" content="ie=edge" />
<title>HTML + CSS</title>
<link rel="stylesheet" href="styles.css" />
</head>
<body>
<h1>This is a simple HTML + CSS template!</h1>
<div id="app"></div>
<!-- Include the widget script from dist/modern or dist/legacy -->
<script
src="https://unpkg.com/docsgpt/dist/modern/main.js"
type="module"
></script>
<script type="module">
window.onload = function () {
renderDocsGPTWidget("app", {
apiKey: "",
size: "medium",
});
};
</script>
</body>
</html>
```
To link the widget to your api and your documents you can pass parameters to the renderDocsGPTWidget('div id', { parameters }).
```html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>DocsGPT Widget</title>
<script src="https://unpkg.com/docsgpt/dist/modern/main.js" type="module"></script>
</head>
<body>
<div id="app"></div>
<!-- Include the widget script from dist/modern or dist/legacy -->
<script type="module">
window.onload = function() {
renderDocsGPTWidget('app', {
apiHost: 'http://localhost:7001',
apiKey:"",
avatar: 'https://d3dg1063dc54p9.cloudfront.net/cute-docsgpt.png',
title: 'Get AI assistance',
description: "DocsGPT's AI Chatbot is here to help",
heroTitle: 'Welcome to DocsGPT!',
heroDescription: 'This chatbot is built with DocsGPT and utilises GenAI, please review important information using sources.',
theme:"dark",
buttonIcon:"https://your-icon",
buttonBg:"#222327"
});
}
</script>
</body>
</html>
```
For more information about React, refer to this [link here](https://react.dev/learn) For more information about React, refer to this [link here](https://react.dev/learn)

@ -1,25 +1,10 @@
import Image from 'next/image'
# Customizing the Main Prompt # Customizing the Main Prompt
Customizing the main prompt for DocsGPT gives you the ability to tailor the AI's responses to your specific requirements. By modifying the prompt text, you can achieve more accurate and relevant answers. Here's how you can do it: Customizing the main prompt for DocsGPT gives you the ability to tailor the AI's responses to your specific requirements. By modifying the prompt text, you can achieve more accurate and relevant answers. Here's how you can do it:
1. Navigate to `SideBar -> Settings`. 1. Navigate to `/application/prompts/combine_prompt.txt`.
2.In Settings select the `Active Prompt` now you will be able to see various prompts style.x
3.Click on the `edit icon` on the prompt of your choice and you will be able to see the current prompt for it,you can now customise the prompt as per your choice.
### Video Demo
<Image src="/prompts.gif" alt="prompts" width={800} height={500} />
2. Open the `combine_prompt.txt` file and modify the prompt text to suit your needs. You can experiment with different phrasings and structures to observe how the model responds. The main prompt serves as guidance to the AI model on how to generate responses.
## Example Prompt Modification ## Example Prompt Modification

@ -0,0 +1,63 @@
## How to train on other documentation
This AI can utilize any documentation, but it requires preparation for similarity search. Follow these steps to get your documentation ready:
**Step 1: Prepare Your Documentation**
![video-example-of-how-to-do-it](https://d3dg1063dc54p9.cloudfront.net/videos/how-to-vectorise.gif)
Start by going to `/scripts/` folder.
If you open this file, you will see that it uses RST files from the folder to create a `index.faiss` and `index.pkl`.
It currently uses OPENAI to create the vector store, so make sure your documentation is not too large. Using Pandas cost me around $3-$4.
You can typically find documentation on GitHub in the `docs/` folder for most open-source projects.
### 1. Find documentation in .rst/.md format and create a folder with it in your scripts directory.
- Name it `inputs/`.
- Put all your .rst/.md files in there.
- The search is recursive, so you don't need to flatten them.
If there are no .rst/.md files, convert whatever you find to a .txt file and feed it. (Don't forget to change the extension in the script).
### Step 2: Configure Your OpenAI API Key
1. Create a .env file in the scripts/ folder.
- Add your OpenAI API key inside: OPENAI_API_KEY=<your-api-key>.
### Step 3: Run the Ingestion Script
`python ingest.py ingest`
It will provide you with the estimated cost.
### Step 4: Move `index.faiss` and `index.pkl` generated in `scripts/output` to `application/` folder.
### Step 5: Run the Web App
Once you run it, it will use new context relevant to your documentation.Make sure you select default in the dropdown in the UI.
## Customization
You can learn more about options while running ingest.py by running:
- Make sure you select 'default' from the dropdown in the UI.
## Customization
You can learn more about options while running ingest.py by executing:
`python ingest.py --help`
| Options | |
|:--------------------------------:|:------------------------------------------------------------------------------------------------------------------------------:|
| **ingest** | Runs 'ingest' function, converting documentation to Faiss plus Index format |
| --dir TEXT | List of paths to directory for index creation. E.g. --dir inputs --dir inputs2 [default: inputs] |
| --file TEXT | File paths to use (Optional; overrides directory) E.g. --files inputs/1.md --files inputs/2.md |
| --recursive / --no-recursive | Whether to recursively search in subdirectories [default: recursive] |
| --limit INTEGER | Maximum number of files to read |
| --formats TEXT | List of required extensions (list with .) Currently supported: .rst, .md, .pdf, .docx, .csv, .epub, .html [default: .rst, .md] |
| --exclude / --no-exclude | Whether to exclude hidden files (dotfiles) [default: exclude] |
| -y, --yes | Whether to skip price confirmation |
| --sample / --no-sample | Whether to output sample of the first 5 split documents. [default: no-sample] |
| --token-check / --no-token-check | Whether to group small documents and split large. Improves semantics. [default: token-check] |
| --min_tokens INTEGER | Minimum number of tokens to not group. [default: 150] |
| --max_tokens INTEGER | Maximum number of tokens to not split. [default: 2000] |
| | |
| **convert** | Creates documentation in .md format from source code |
| --dir TEXT | Path to a directory with source code. E.g. --dir inputs [default: inputs] |
| --formats TEXT | Source code language from which to create documentation. Supports py, js and java. E.g. --formats py [default: py] |

@ -1,44 +0,0 @@
import { Callout } from 'nextra/components'
import Image from 'next/image'
import { Steps } from 'nextra/components'
## How to train on other documentation
Training on other documentation sources can greatly enhance the versatility and depth of DocsGPT's knowledge. By incorporating diverse materials, you can broaden the AI's understanding and improve its ability to generate insightful responses across a range of topics. Here's a step-by-step guide on how to effectively train DocsGPT on additional documentation sources:
**Get your document ready**:
Make sure you have the document on which you want to train on ready with you on the device which you are using .You can also use links to the documentation to train on.
<Callout type="warning" emoji="⚠️">
Note: The document should be either of the given file formats .pdf, .txt, .rst, .docx, .md, .zip and limited to 25mb.You can also train using the link of the documentation.
</Callout>
### Video Demo
<Image src="/docs.gif" alt="prompts" width={800} height={500} />
<Steps>
### Step1
Navigate to the sidebar where you will find `Source Docs` option,here you will find 3 options built in which are default,Web Search and None.
### Step 2
Click on the `Upload icon` just beside the source docs options,now borwse and upload the document which you want to train on or select the `remote` option if you have to insert the link of the documentation.
### Step 3
Now you will be able to see the name of the file uploaded under the Uploaded Files ,now click on `Train`,once you click on train it might take some time to train on the document. You will be able to see the `Training progress` and once the training is completed you can click the `finish` button and there you go your docuemnt is uploaded.
### Step 4
Go to `New chat` and from the side bar select the document you uploaded under the `Source Docs` and go ahead with your chat, now you can ask qestions regarding the document you uploaded and you will get the effective answer based on it.
</Steps>

@ -0,0 +1,48 @@
# Setting Up Local Language Models for Your App
Your app relies on two essential models: Embeddings and Text Generation. While OpenAI's default models work seamlessly, you have the flexibility to switch providers or even run the models locally.
## Step 1: Configure Environment Variables
Navigate to the `.env` file or set the following environment variables:
```env
LLM_NAME=<your Text Generation model>
API_KEY=<API key for Text Generation>
EMBEDDINGS_NAME=<LLM for Embeddings>
EMBEDDINGS_KEY=<API key for Embeddings>
VITE_API_STREAMING=<true or false>
```
You can omit the keys if users provide their own. Ensure you set `LLM_NAME` and `EMBEDDINGS_NAME`.
## Step 2: Choose Your Models
**Options for `LLM_NAME`:**
- openai ([More details](https://platform.openai.com/docs/models))
- anthropic ([More details](https://docs.anthropic.com/claude/reference/selecting-a-model))
- manifest ([More details](https://python.langchain.com/docs/integrations/llms/manifest))
- cohere ([More details](https://docs.cohere.com/docs/llmu))
- llama.cpp ([More details](https://python.langchain.com/docs/integrations/llms/llamacpp))
- huggingface (Arc53/DocsGPT-7B by default)
- sagemaker ([Mode details](https://aws.amazon.com/sagemaker/))
Note: for huggingface you can choose any model inside application/llm/huggingface.py or pass llm_name on init, loads
**Options for `EMBEDDINGS_NAME`:**
- openai_text-embedding-ada-002
- huggingface_sentence-transformers/all-mpnet-base-v2
- huggingface_hkunlp/instructor-large
- cohere_medium
If you want to be completely local, set `EMBEDDINGS_NAME` to `huggingface_sentence-transformers/all-mpnet-base-v2`.
For llama.cpp Download the required model and place it in the `models/` folder.
Alternatively, for local Llama setup, run `setup.sh` and choose option 1. The script handles the DocsGPT model addition.
## Step 3: Local Hosting for Privacy
If working with sensitive data, host everything locally by setting `LLM_NAME`, llama.cpp or huggingface, use any model available on Hugging Face, for llama.cpp you need to convert it into gguf format.
That's it! Your app is now configured for local and private hosting, ensuring optimal security for critical data.

@ -1,49 +0,0 @@
import { Callout } from 'nextra/components'
import Image from 'next/image'
import { Steps } from 'nextra/components'
# Setting Up Local Language Models for Your App
Setting up local language models for your app can significantly enhance its capabilities, enabling it to understand and generate text in multiple languages without relying on external APIs. By integrating local language models, you can improve privacy, reduce latency, and ensure continuous functionality even in offline environments. Here's a comprehensive guide on how to set up local language models for your application:
## Steps:
### For cloud version LLM change:
<Steps >
### Step 1
Visit the chat screen and you will be to see the default LLM selected.
### Step 2
Click on it and you will get a drop down of various LLM's available to choose.
### Step 3
Choose the LLM of your choice.
</Steps>
### Video Demo
<Image src="/llms.gif" alt="prompts" width={800} height={500} />
### For Open source llm change:
<Steps >
### Step 1
For open source you have to edit .env file with LLM_NAME with their desired LLM name.
### Step 2
All the supported LLM providers are here application/llm and you can check what env variable are needed for each
List of latest supported LLMs are https://github.com/arc53/DocsGPT/blob/main/application/llm/llm_creator.py
### Step 3
Visit application/llm and select the file of your selected llm and there you will find the speicifc requirements needed to be filled in order to use it,i.e API key of that llm.
</Steps>
### For OpenAI-Compatible Endpoints:
DocsGPT supports the use of OpenAI-compatible endpoints through base URL substitution. This feature allows you to use alternative AI models or services that implement the OpenAI API interface.
Set the OPENAI_BASE_URL in your environment. You can change .env file with OPENAI_BASE_URL with the desired base URL or docker-compose.yml file and add the environment variable to the backend container.
> Make sure you have the right API_KEY and correct LLM_NAME.

@ -1,6 +1,6 @@
{ {
"Customising-prompts": { "Customising-prompts": {
"title": "💻 Customising Prompts", "title": "🏗️ Customising Prompts",
"href": "/Guides/Customising-prompts" "href": "/Guides/Customising-prompts"
}, },
"How-to-train-on-other-documentation": { "How-to-train-on-other-documentation": {
@ -8,7 +8,7 @@
"href": "/Guides/How-to-train-on-other-documentation" "href": "/Guides/How-to-train-on-other-documentation"
}, },
"How-to-use-different-LLM": { "How-to-use-different-LLM": {
"title": "🤖 How to use different LLM's", "title": "⚙️ How to use different LLM's",
"href": "/Guides/How-to-use-different-LLM" "href": "/Guides/How-to-use-different-LLM"
}, },
"My-AI-answers-questions-using-external-knowledge": { "My-AI-answers-questions-using-external-knowledge": {

@ -4,7 +4,7 @@ export default function MyApp({ Component, pageProps }) {
return ( return (
<> <>
<Component {...pageProps} /> <Component {...pageProps} />
<DocsGPTWidget apiKey="d61a020c-ac8f-4f23-bb98-458e4da3c240" theme="dark" /> <DocsGPTWidget apiKey="d61a020c-ac8f-4f23-bb98-458e4da3c240" />
</> </>
) )
} }

@ -2,16 +2,14 @@
title: 'Home' title: 'Home'
--- ---
import { Cards, Card } from 'nextra/components' import { Cards, Card } from 'nextra/components'
import Image from 'next/image'
import deployingGuides from './Deploying/_meta.json'; import deployingGuides from './Deploying/_meta.json';
import developingGuides from './API/_meta.json'; import developingGuides from './Developing/_meta.json';
import extensionGuides from './Extensions/_meta.json'; import extensionGuides from './Extensions/_meta.json';
import mainGuides from './Guides/_meta.json'; import mainGuides from './Guides/_meta.json';
export const allGuides = { export const allGuides = {
...deployingGuides, ...deployingGuides,
...developingGuides, ...developingGuides,
@ -23,12 +21,9 @@ export const allGuides = {
DocsGPT 🦖 is an innovative open-source tool designed to simplify the retrieval of information from project documentation using advanced GPT models 🤖. Eliminate lengthy manual searches 🔍 and enhance your documentation experience with DocsGPT, and consider contributing to its AI-powered future 🚀. DocsGPT 🦖 is an innovative open-source tool designed to simplify the retrieval of information from project documentation using advanced GPT models 🤖. Eliminate lengthy manual searches 🔍 and enhance your documentation experience with DocsGPT, and consider contributing to its AI-powered future 🚀.
![video-example-of-docs-gpt](https://d3dg1063dc54p9.cloudfront.net/videos/demov3.gif)
Try it yourself: [https://docsgpt.arc53.com/](https://docsgpt.arc53.com/)
<Image src="/homevideo.gif" alt="homedemo" width={800} height={500}/>
Try it yourself: [https://www.docsgpt.cloud/](https://www.docsgpt.cloud/)
<Cards <Cards
num={3} num={3}

Binary file not shown.

Before

Width:  |  Height:  |  Size: 839 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 23 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 500 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 974 KiB

@ -107,12 +107,12 @@
} }
}, },
"node_modules/braces": { "node_modules/braces": {
"version": "3.0.3", "version": "3.0.2",
"resolved": "https://registry.npmjs.org/braces/-/braces-3.0.3.tgz", "resolved": "https://registry.npmjs.org/braces/-/braces-3.0.2.tgz",
"integrity": "sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA==", "integrity": "sha512-b8um+L1RzM3WDSzvhm6gIz1yfTbBt6YTlcEKAvsmqCZZFw46z626lVj9j1yEPW33H5H+lBQpZMP1k8l+78Ha0A==",
"dev": true, "dev": true,
"dependencies": { "dependencies": {
"fill-range": "^7.1.1" "fill-range": "^7.0.1"
}, },
"engines": { "engines": {
"node": ">=8" "node": ">=8"
@ -260,9 +260,9 @@
} }
}, },
"node_modules/fill-range": { "node_modules/fill-range": {
"version": "7.1.1", "version": "7.0.1",
"resolved": "https://registry.npmjs.org/fill-range/-/fill-range-7.1.1.tgz", "resolved": "https://registry.npmjs.org/fill-range/-/fill-range-7.0.1.tgz",
"integrity": "sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg==", "integrity": "sha512-qOo9F+dMUmC2Lcb4BbVvnKJxTPjCm+RRpe4gDuGrzkL7mEVl/djYSu2OdQ2Pa302N4oqkSg9ir6jaLWJ2USVpQ==",
"dev": true, "dev": true,
"dependencies": { "dependencies": {
"to-regex-range": "^5.0.1" "to-regex-range": "^5.0.1"
@ -884,12 +884,12 @@
"dev": true "dev": true
}, },
"braces": { "braces": {
"version": "3.0.3", "version": "3.0.2",
"resolved": "https://registry.npmjs.org/braces/-/braces-3.0.3.tgz", "resolved": "https://registry.npmjs.org/braces/-/braces-3.0.2.tgz",
"integrity": "sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA==", "integrity": "sha512-b8um+L1RzM3WDSzvhm6gIz1yfTbBt6YTlcEKAvsmqCZZFw46z626lVj9j1yEPW33H5H+lBQpZMP1k8l+78Ha0A==",
"dev": true, "dev": true,
"requires": { "requires": {
"fill-range": "^7.1.1" "fill-range": "^7.0.1"
} }
}, },
"camelcase-css": { "camelcase-css": {
@ -1000,9 +1000,9 @@
} }
}, },
"fill-range": { "fill-range": {
"version": "7.1.1", "version": "7.0.1",
"resolved": "https://registry.npmjs.org/fill-range/-/fill-range-7.1.1.tgz", "resolved": "https://registry.npmjs.org/fill-range/-/fill-range-7.0.1.tgz",
"integrity": "sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg==", "integrity": "sha512-qOo9F+dMUmC2Lcb4BbVvnKJxTPjCm+RRpe4gDuGrzkL7mEVl/djYSu2OdQ2Pa302N4oqkSg9ir6jaLWJ2USVpQ==",
"dev": true, "dev": true,
"requires": { "requires": {
"to-regex-range": "^5.0.1" "to-regex-range": "^5.0.1"

@ -1,5 +1,6 @@
# DocsGPT react widget # DocsGPT react widget
This widget will allow you to embed a DocsGPT assistant in your React app. This widget will allow you to embed a DocsGPT assistant in your React app.
## Installation ## Installation
@ -10,8 +11,6 @@ npm install docsgpt
## Usage ## Usage
### React
```javascript ```javascript
import { DocsGPTWidget } from "docsgpt"; import { DocsGPTWidget } from "docsgpt";
@ -27,82 +26,22 @@ To link the widget to your api and your documents you can pass parameters to the
const App = () => { const App = () => {
return <DocsGPTWidget return <DocsGPTWidget
apiHost="https://your-docsgpt-api.com" apiHost = 'http://localhost:7001',
apiKey="" selectDocs = 'default',
avatar = "https://d3dg1063dc54p9.cloudfront.net/cute-docsgpt.png" apiKey = '',
title = "Get AI assistance" avatar = 'https://d3dg1063dc54p9.cloudfront.net/cute-docsgpt.png',
description = "DocsGPT's AI Chatbot is here to help" title = 'Get AI assistance',
heroTitle = "Welcome to DocsGPT !" description = 'DocsGPT\'s AI Chatbot is here to help',
heroDescription="This chatbot is built with DocsGPT and utilises GenAI, heroTitle = 'Welcome to DocsGPT !',
please review important information using sources." heroDescription='This chatbot is built with DocsGPT and utilises GenAI, please review important information using sources.'
theme = "dark" />;
buttonIcon = "https://your-icon"
buttonBg = "#222327"
/>;
}; };
``` ```
### Html
```html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>DocsGPT Widget</title>
</head>
<body>
<div id="app"></div>
<!-- Include the widget script from dist/modern or dist/legacy -->
<script src="https://unpkg.com/docsgpt/dist/modern/main.js" type="module"></script>
<script type="module">
window.onload = function() {
renderDocsGPTWidget('app');
}
</script>
</body>
</html>
```
To link the widget to your api and your documents you can pass parameters to the **renderDocsGPTWidget('div id', { parameters })**.
```html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>DocsGPT Widget</title>
</head>
<body>
<div id="app"></div>
<!-- Include the widget script from dist/modern or dist/legacy -->
<script src="https://unpkg.com/docsgpt/dist/modern/main.js" type="module"></script>
<script type="module">
window.onload = function() {
renderDocsGPTWidget('app', {
apiHost: 'http://localhost:7001',
apiKey:"",
avatar: 'https://d3dg1063dc54p9.cloudfront.net/cute-docsgpt.png',
title: 'Get AI assistance',
description: "DocsGPT's AI Chatbot is here to help",
heroTitle: 'Welcome to DocsGPT!',
heroDescription: 'This chatbot is built with DocsGPT and utilises GenAI, please review important information using sources.',
theme:"dark",
buttonIcon:"https://your-icon.svg",
buttonBg:"#222327"
});
}
</script>
</body>
</html>
```
## Our github ## Our github
[DocsGPT](https://github.com/arc53/DocsGPT) [DocsGPT](https://github.com/arc53/DocsGPT)
You can find the source code in the extensions/react-widget folder. You can find the source code in the extensions/react-widget folder.

File diff suppressed because it is too large Load Diff

@ -1,6 +1,6 @@
{ {
"name": "docsgpt", "name": "docsgpt",
"version": "0.4.2", "version": "0.3.7",
"private": false, "private": false,
"description": "DocsGPT 🦖 is an innovative open-source tool designed to simplify the retrieval of information from project documentation using advanced GPT models 🤖.", "description": "DocsGPT 🦖 is an innovative open-source tool designed to simplify the retrieval of information from project documentation using advanced GPT models 🤖.",
"source": "./src/index.html", "source": "./src/index.html",
@ -11,18 +11,6 @@
"dist", "dist",
"package.json" "package.json"
], ],
"targets": {
"modern": {
"engines": {
"browsers": "Chrome 80"
}
},
"legacy": {
"engines": {
"browsers": "> 0.5%, last 2 versions, not dead"
}
}
},
"@parcel/resolver-default": { "@parcel/resolver-default": {
"packageExports": true "packageExports": true
}, },
@ -30,8 +18,7 @@
"styled-components": "^5" "styled-components": "^5"
}, },
"scripts": { "scripts": {
"build": "parcel build src/main.tsx --public-url ./", "build": "parcel build src/index.ts",
"build:react": "parcel build src/index.ts",
"dev": "parcel src/index.html -p 3000", "dev": "parcel src/index.html -p 3000",
"test": "jest", "test": "jest",
"lint": "eslint", "lint": "eslint",
@ -40,19 +27,22 @@
}, },
"dependencies": { "dependencies": {
"@babel/plugin-transform-flow-strip-types": "^7.23.3", "@babel/plugin-transform-flow-strip-types": "^7.23.3",
"@bpmn-io/snarkdown": "^2.2.0",
"@parcel/resolver-glob": "^2.12.0", "@parcel/resolver-glob": "^2.12.0",
"@parcel/transformer-svg-react": "^2.12.0", "@parcel/transformer-svg-react": "^2.12.0",
"@parcel/transformer-typescript-tsc": "^2.12.0", "@parcel/transformer-typescript-tsc": "^2.12.0",
"@parcel/validator-typescript": "^2.12.0", "@parcel/validator-typescript": "^2.12.0",
"@radix-ui/react-icons": "^1.3.0", "@radix-ui/react-icons": "^1.3.0",
"@types/react": "^18.2.61",
"@types/react-dom": "^18.2.19",
"class-variance-authority": "^0.7.0", "class-variance-authority": "^0.7.0",
"clsx": "^2.1.0", "clsx": "^2.1.0",
"dompurify": "^3.1.5", "dompurify": "^3.0.9",
"flow-bin": "^0.229.2", "flow-bin": "^0.229.2",
"i": "^0.3.7", "i": "^0.3.7",
"install": "^0.13.0", "install": "^0.13.0",
"markdown-it": "^14.1.0",
"npm": "^10.5.0", "npm": "^10.5.0",
"parcel": "^2.12.0",
"react": "^18.2.0", "react": "^18.2.0",
"react-dom": "^18.2.0", "react-dom": "^18.2.0",
"styled-components": "^6.1.8" "styled-components": "^6.1.8"
@ -64,11 +54,7 @@
"@parcel/packager-ts": "^2.12.0", "@parcel/packager-ts": "^2.12.0",
"@parcel/transformer-typescript-types": "^2.12.0", "@parcel/transformer-typescript-types": "^2.12.0",
"@types/dompurify": "^3.0.5", "@types/dompurify": "^3.0.5",
"@types/markdown-it": "^14.1.2",
"@types/react": "^18.3.3",
"@types/react-dom": "^18.3.0",
"babel-loader": "^8.0.4", "babel-loader": "^8.0.4",
"parcel": "^2.12.0",
"process": "^0.11.10", "process": "^0.11.10",
"typescript": "^5.3.3" "typescript": "^5.3.3"
}, },

@ -1,43 +0,0 @@
#!/bin/bash
## chmod +x publish.sh - to upgrade ownership
set -e
cat package.json >> package_copy.json
cat package-lock.json >> package-lock_copy.json
publish_package() {
PACKAGE_NAME=$1
BUILD_COMMAND=$2
# Update package name in package.json
jq --arg name "$PACKAGE_NAME" '.name=$name' package.json > temp.json && mv temp.json package.json
# Remove 'target' key if the package name is 'docsgpt-react'
if [ "$PACKAGE_NAME" = "docsgpt-react" ]; then
jq 'del(.targets)' package.json > temp.json && mv temp.json package.json
fi
if [ -d "dist" ]; then
echo "Deleting existing dist directory..."
rm -rf dist
fi
npm version patch
npm run "$BUILD_COMMAND"
# Publish to npm
npm publish
# Clean up
mv package_copy.json package.json
mv package-lock_copy.json package-lock.json
echo "Published ${PACKAGE_NAME}"
}
# Publish docsgpt package
publish_package "docsgpt" "build"
# Publish docsgpt-react package
publish_package "docsgpt-react" "build:react"
rm -rf package_copy.json
rm -rf package-lock_copy.json
echo "---Process completed---"

@ -1,4 +0,0 @@
<svg width="14" height="14" viewBox="0 0 16 16" fill="current" xmlns="http://www.w3.org/2000/svg">
<path d="M6.37776 10.1001V12.9C6.37776 13.457 6.599 13.9911 6.99282 14.3849C7.38664 14.7788 7.92077 15 8.47772 15L11.2777 8.70011V1.00025H3.38181C3.04419 0.996436 2.71656 1.11477 2.45929 1.33344C2.20203 1.55212 2.03246 1.8564 1.98184 2.19023L1.01585 8.49012C0.985398 8.69076 0.998931 8.89563 1.05551 9.09053C1.1121 9.28543 1.21038 9.46569 1.34355 9.61884C1.47671 9.77198 1.64159 9.89434 1.82674 9.97744C2.01189 10.0605 2.2129 10.1024 2.41583 10.1001H6.37776ZM11.2777 1.00025H13.1466C13.5428 0.993247 13.9277 1.13195 14.2284 1.39002C14.5291 1.64809 14.7245 2.00758 14.7776 2.40023V7.30014C14.7245 7.69279 14.5291 8.05227 14.2284 8.31035C13.9277 8.56842 13.5428 8.70712 13.1466 8.70011H11.2777" fill="none"/>
<path d="M11.2777 8.70011L8.47772 15C7.92077 15 7.38664 14.7788 6.99282 14.3849C6.599 13.9911 6.37776 13.457 6.37776 12.9V10.1001H2.41583C2.2129 10.1024 2.01189 10.0605 1.82674 9.97744C1.64159 9.89434 1.47671 9.77198 1.34355 9.61884C1.21038 9.46569 1.1121 9.28543 1.05551 9.09053C0.998931 8.89563 0.985398 8.69076 1.01585 8.49012L1.98184 2.19023C2.03246 1.8564 2.20203 1.55212 2.45929 1.33344C2.71656 1.11477 3.04419 0.996436 3.38181 1.00025H11.2777M11.2777 8.70011V1.00025M11.2777 8.70011H13.1466C13.5428 8.70712 13.9277 8.56842 14.2284 8.31035C14.5291 8.05227 14.7245 7.69279 14.7776 7.30014V2.40023C14.7245 2.00758 14.5291 1.64809 14.2284 1.39002C13.9277 1.13195 13.5428 0.993247 13.1466 1.00025H11.2777" stroke="current" stroke-width="1.4" stroke-linecap="round" stroke-linejoin="round"/>
</svg>

Before

Width:  |  Height:  |  Size: 1.6 KiB

@ -1,4 +0,0 @@
<svg width="14" height="14" viewBox="0 0 16 16" fill="current" xmlns="http://www.w3.org/2000/svg">
<path d="M9.39995 5.89997V3.09999C9.39995 2.54304 9.1787 2.0089 8.78487 1.61507C8.39105 1.22125 7.85691 1 7.29996 1L4.49998 7.29996V14.9999H12.3959C12.7336 15.0037 13.0612 14.8854 13.3185 14.6667C13.5757 14.448 13.7453 14.1437 13.7959 13.8099L14.7619 7.50996C14.7924 7.30931 14.7788 7.10444 14.7222 6.90954C14.6657 6.71464 14.5674 6.53437 14.4342 6.38123C14.301 6.22808 14.1362 6.10572 13.951 6.02262C13.7659 5.93952 13.5649 5.89767 13.3619 5.89997H9.39995ZM4.49998 14.9999H2.39999C2.02869 14.9999 1.6726 14.8524 1.41005 14.5899C1.1475 14.3273 1 13.9712 1 13.5999V8.69995C1 8.32865 1.1475 7.97256 1.41005 7.71001C1.6726 7.44746 2.02869 7.29996 2.39999 7.29996H4.49998" fill="none"/>
<path d="M4.49998 7.29996L7.29996 1C7.85691 1 8.39105 1.22125 8.78487 1.61507C9.1787 2.0089 9.39995 2.54304 9.39995 3.09999V5.89997H13.3619C13.5649 5.89767 13.7659 5.93952 13.951 6.02262C14.1362 6.10572 14.301 6.22808 14.4342 6.38123C14.5674 6.53437 14.6657 6.71464 14.7223 6.90954C14.7788 7.10444 14.7924 7.30931 14.7619 7.50996L13.7959 13.8099C13.7453 14.1437 13.5757 14.448 13.3185 14.6667C13.0612 14.8854 12.7336 15.0037 12.3959 14.9999H4.49998M4.49998 7.29996V14.9999M4.49998 7.29996H2.39999C2.02869 7.29996 1.6726 7.44746 1.41005 7.71001C1.1475 7.97256 1 8.32865 1 8.69995V13.5999C1 13.9712 1.1475 14.3273 1.41005 14.5899C1.6726 14.8524 2.02869 14.9999 2.39999 14.9999H4.49998" stroke="current" stroke-width="1.39999" stroke-linecap="round" stroke-linejoin="round"/>
</svg>

Before

Width:  |  Height:  |  Size: 1.5 KiB

@ -0,0 +1,7 @@
<svg width="36" height="36" viewBox="0 0 18 18" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M4.37891 9.44824H7.75821" stroke="white" stroke-width="1.68965" stroke-linecap="round" stroke-linejoin="round"/>
<path d="M11.1377 9.44824H12.8273" stroke="white" stroke-width="1.68965" stroke-linecap="round" stroke-linejoin="round"/>
<path d="M4.37891 6.06934H6.06856" stroke="white" stroke-width="1.68965" stroke-linecap="round" stroke-linejoin="round"/>
<path d="M9.44824 6.06934H12.8276" stroke="white" stroke-width="1.68965" stroke-linecap="round" stroke-linejoin="round"/>
<path d="M16.2069 11.1379C16.2069 11.5861 16.0289 12.0158 15.712 12.3327C15.3951 12.6496 14.9654 12.8276 14.5172 12.8276H4.37931L1 16.2069V2.68965C1 2.24153 1.17802 1.81176 1.49489 1.49489C1.81176 1.17802 2.24153 1 2.68965 1H14.5172C14.9654 1 15.3951 1.17802 15.712 1.49489C16.0289 1.81176 16.2069 2.24153 16.2069 2.68965V11.1379Z" stroke="white" stroke-width="1.68965" stroke-linecap="round" stroke-linejoin="round"/>
</svg>

After

Width:  |  Height:  |  Size: 1009 B

@ -1,40 +1,12 @@
"use client"; "use client";
import React, { useRef } from 'react' import { Fragment, useEffect, useRef, useState } from 'react'
import DOMPurify from 'dompurify';
import styled, { keyframes, createGlobalStyle } from 'styled-components';
import { PaperPlaneIcon, RocketIcon, ExclamationTriangleIcon, Cross2Icon } from '@radix-ui/react-icons'; import { PaperPlaneIcon, RocketIcon, ExclamationTriangleIcon, Cross2Icon } from '@radix-ui/react-icons';
import { FEEDBACK, MESSAGE_TYPE, Query, Status, WidgetProps } from '../types/index'; import { MESSAGE_TYPE, Query, Status } from '../types/index';
import { fetchAnswerStreaming, sendFeedback } from '../requests/streamingApi'; import MessageIcon from '../assets/message.svg'
import { ThemeProvider } from 'styled-components'; import { fetchAnswerStreaming } from '../requests/streamingApi';
import Like from "../assets/like.svg" import styled, { keyframes, createGlobalStyle } from 'styled-components';
import Dislike from "../assets/dislike.svg" import snarkdown from '@bpmn-io/snarkdown';
import MarkdownIt from 'markdown-it'; import { sanitize } from 'dompurify';
const themes = {
dark: {
bg: '#222327',
text: '#fff',
primary: {
text: "#FAFAFA",
bg: '#222327'
},
secondary: {
text: "#A1A1AA",
bg: "#38383b"
}
},
light: {
bg: '#fff',
text: '#000',
primary: {
text: "#222327",
bg: "#fff"
},
secondary: {
text: "#A1A1AA",
bg: "#F6F6F6"
}
}
}
const GlobalStyles = createGlobalStyle` const GlobalStyles = createGlobalStyle`
.response pre { .response pre {
padding: 8px; padding: 8px;
@ -43,7 +15,6 @@ const GlobalStyles = createGlobalStyle`
border-radius: 6px; border-radius: 6px;
overflow-x: auto; overflow-x: auto;
background-color: #1B1C1F; background-color: #1B1C1F;
color: #fff !important;
} }
.response h1{ .response h1{
font-size: 20px; font-size: 20px;
@ -54,64 +25,40 @@ const GlobalStyles = createGlobalStyle`
.response h3{ .response h3{
font-size: 16px; font-size: 16px;
} }
.response p{
margin:0px;
}
.response code:not(pre code){ .response code:not(pre code){
border-radius: 6px; border-radius: 6px;
padding: 1px 3px 1px 3px; padding: 1px 3px 1px 3px;
font-size: 12px; font-size: 12px;
display: inline-block; display: inline-block;
background-color: #646464; background-color: #646464;
color: #fff !important;
}
.response code {
white-space: pre-wrap !important;
line-break: loose !important;
} }
`; `;
const Overlay = styled.div` const WidgetContainer = styled.div`
position: fixed;
top: 0;
left: 0;
width: 100%;
height: 100%;
background-color: rgba(0, 0, 0, 0.5);
z-index: 999;
transition: opacity 0.5s;
`
const WidgetContainer = styled.div<{ modal: boolean }>`
display: block; display: block;
position: fixed; position: fixed;
right: ${props => props.modal ? '50%' : '10px'}; right: 10px;
bottom: ${props => props.modal ? '50%' : '10px'}; bottom: 10px;
z-index: 1000; z-index: 1000;
display: flex; display: flex;
${props => props.modal &&
"transform : translate(50%,50%);"
}
flex-direction: column; flex-direction: column;
align-items: center; align-items: center;
text-align: left; text-align: left;
@media only screen and (max-width: 768px) {
max-height: 100vh !important;
overflow: auto;
}
`; `;
const StyledContainer = styled.div` const StyledContainer = styled.div`
display: flex; display: block;
position: relative; position: relative;
flex-direction: column;
justify-content: center;
bottom: 0; bottom: 0;
left: 0; left: 0;
width: 352px;
height: 407px;
max-height: 407px;
border-radius: 0.75rem; border-radius: 0.75rem;
background-color: ${props => props.theme.primary.bg}; background-color: #222327;
font-family: sans-serif; font-family: sans-serif;
box-shadow: 0 1px 2px rgba(0, 0, 0, 0.05), 0 2px 4px rgba(0, 0, 0, 0.1); box-shadow: 0 1px 2px rgba(0, 0, 0, 0.05), 0 2px 4px rgba(0, 0, 0, 0.1);
transition: visibility 0.3s, opacity 0.3s; transition: visibility 0.3s, opacity 0.3s;
`; `;
const FloatingButton = styled.div<{ bgcolor: string }>` const FloatingButton = styled.div`
position: fixed; position: fixed;
display: flex; display: flex;
z-index: 500; z-index: 500;
@ -122,7 +69,7 @@ const FloatingButton = styled.div<{ bgcolor: string }>`
width: 5rem; width: 5rem;
height: 5rem; height: 5rem;
border-radius: 9999px; border-radius: 9999px;
background: ${props => props.bgcolor}; background-image: linear-gradient(to bottom right, #5AF0EC, #E80D9D);
box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1); box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);
cursor: pointer; cursor: pointer;
&:hover { &:hover {
@ -172,62 +119,39 @@ const ContentWrapper = styled.div`
const Title = styled.h3` const Title = styled.h3`
font-size: 1rem; font-size: 1rem;
font-weight: normal; font-weight: normal;
color: ${props => props.theme.primary.text}; color: #FAFAFA;
margin-top: 0; margin-top: 0;
margin-bottom: 0.25rem; margin-bottom: 0.25rem;
`; `;
const Description = styled.p` const Description = styled.p`
font-size: 0.85rem; font-size: 0.85rem;
color: ${props => props.theme.secondary.text}; color: #A1A1AA;
margin-top: 0; margin-top: 0;
`; `;
const Conversation = styled.div`
const Conversation = styled.div<{ size: string }>` height: 16rem;
min-height: 250px;
max-width: 968px;
height: ${props => props.size === 'large' ? '75vh' : props.size === 'medium' ? '70vh' : '320px'};
width: ${props => props.size === 'large' ? '60vw' : props.size === 'medium' ? '28vw' : '400px'};
padding-inline: 0.5rem; padding-inline: 0.5rem;
border-radius: 0.375rem; border-radius: 0.375rem;
text-align: left; text-align: left;
overflow-y: auto; overflow-y: auto;
scrollbar-width: thin; scrollbar-width: thin;
scrollbar-color: #4a4a4a transparent; /* thumb color track color */ scrollbar-color: #4a4a4a transparent; /* thumb color track color */
@media only screen and (max-width: 768px) {
width: 90vw !important;
}
@media only screen and (min-width:768px ) and (max-width: 1280px) {
width:${props => props.size === 'large' ? '90vw' : props.size === 'medium' ? '60vw' : '400px'} !important;
}
`;
const Feedback = styled.div`
background-color: transparent;
font-weight: normal;
gap: 12px;
display: flex;
padding: 6px;
clear: both;
`; `;
const MessageBubble = styled.div<{ type: MESSAGE_TYPE }>` const MessageBubble = styled.div<{ type: MESSAGE_TYPE }>`
display: block; display: flex;
font-size: 16px; font-size: 16px;
position: relative; justify-content: ${props => props.type === 'QUESTION' ? 'flex-end' : 'flex-start'};
width: 100%;; margin: 0.5rem;
float: right;
margin: 0rem;
&:hover ${Feedback} * {
visibility: visible !important;
}
`; `;
const Message = styled.div<{ type: MESSAGE_TYPE }>` const Message = styled.p<{ type: MESSAGE_TYPE }>`
background: ${props => props.type === 'QUESTION' ? background: ${props => props.type === 'QUESTION' ?
'linear-gradient(to bottom right, #8860DB, #6D42C5)' : 'linear-gradient(to bottom right, #8860DB, #6D42C5)' :
props.theme.secondary.bg}; '#38383b'};
color: ${props => props.type === 'ANSWER' ? props.theme.primary.text : '#fff'}; color: #ffff;
border: none; border: none;
float: ${props => props.type === 'QUESTION' ? 'right' : 'left'}; max-width: 80%;
max-width: ${props => props.type === 'ANSWER' ? '100%' : '80'};
overflow: auto; overflow: auto;
margin: 4px; margin: 4px;
display: block; display: block;
@ -265,31 +189,35 @@ const DotAnimation = styled.div`
const Delay = styled(DotAnimation) <{ delay: number }>` const Delay = styled(DotAnimation) <{ delay: number }>`
animation-delay: ${props => props.delay + 'ms'}; animation-delay: ${props => props.delay + 'ms'};
`; `;
const PromptContainer = styled.form<{ size: string }>` const PromptContainer = styled.form`
background-color: transparent; background-color: transparent;
height: ${props => props.size == 'large' ? '60px' : '40px'}; height: 36px;
margin: 16px; position: absolute;
bottom: 25px;
left: 24px;
right: 24px;
display: flex; display: flex;
justify-content: space-evenly; justify-content: space-evenly;
`; `;
const StyledInput = styled.input` const StyledInput = styled.input`
width: 100%; width: 260px;
height: 36px;
border: 1px solid #686877; border: 1px solid #686877;
padding-left: 12px; padding-left: 12px;
background-color: transparent; background-color: transparent;
font-size: 16px; font-size: 16px;
border-radius: 6px; border-radius: 6px;
color: ${props => props.theme.text}; color: #ffff;
outline: none; outline: none;
`; `;
const StyledButton = styled.button<{ size: string }>` const StyledButton = styled.button`
display: flex; display: flex;
justify-content: center; justify-content: center;
align-items: center; align-items: center;
background-image: linear-gradient(to bottom right, #5AF0EC, #E80D9D); background-image: linear-gradient(to bottom right, #5AF0EC, #E80D9D);
border-radius: 6px; border-radius: 6px;
min-width: ${props => props.size === 'large' ? '60px' : '36px'}; width: 36px;
height: ${props => props.size === 'large' ? '60px' : '36px'}; height: 36px;
margin-left:8px; margin-left:8px;
padding: 0px; padding: 0px;
border: none; border: none;
@ -310,14 +238,13 @@ const HeroContainer = styled.div`
align-items: middle; align-items: middle;
transform: translate(-50%, -50%); transform: translate(-50%, -50%);
width: 80%; width: 80%;
max-width: 500px;
background-image: linear-gradient(to bottom right, #5AF0EC, #ff1bf4); background-image: linear-gradient(to bottom right, #5AF0EC, #ff1bf4);
border-radius: 10px; border-radius: 10px;
margin: 0 auto; margin: 0 auto;
padding: 2px; padding: 2px;
`; `;
const HeroWrapper = styled.div` const HeroWrapper = styled.div`
background-color: ${props => props.theme.primary.bg}; background-color: #222327;
border-radius: 10px; border-radius: 10px;
font-weight: normal; font-weight: normal;
padding: 6px; padding: 6px;
@ -325,23 +252,23 @@ const HeroWrapper = styled.div`
justify-content: space-between; justify-content: space-between;
` `
const HeroTitle = styled.h3` const HeroTitle = styled.h3`
color: ${props => props.theme.text}; color: #fff;
font-size: 17px;
margin-bottom: 5px; margin-bottom: 5px;
padding: 2px; padding: 2px;
`; `;
const HeroDescription = styled.p` const HeroDescription = styled.p`
color: ${props => props.theme.text}; color: #fff;
font-size: 14px; font-size: 14px;
line-height: 1.5; line-height: 1.5;
`; `;
const Hero = ({ title, description }: { title: string, description: string }) => {
const Hero = ({ title, description, theme }: { title: string, description: string, theme: string }) => {
return ( return (
<> <>
<HeroContainer> <HeroContainer>
<HeroWrapper> <HeroWrapper>
<IconWrapper style={{ marginTop: '12px' }}> <IconWrapper style={{ marginTop: '8px' }}>
<RocketIcon color={theme === 'light' ? 'black' : 'white'} width={20} height={20} /> <RocketIcon color='white' width={20} height={20} />
</IconWrapper> </IconWrapper>
<div> <div>
<HeroTitle>{title}</HeroTitle> <HeroTitle>{title}</HeroTitle>
@ -356,28 +283,22 @@ const Hero = ({ title, description, theme }: { title: string, description: strin
}; };
export const DocsGPTWidget = ({ export const DocsGPTWidget = ({
apiHost = 'https://gptcloud.arc53.com', apiHost = 'https://gptcloud.arc53.com',
selectDocs = 'default',
apiKey = '82962c9a-aa77-4152-94e5-a4f84fd44c6a', apiKey = '82962c9a-aa77-4152-94e5-a4f84fd44c6a',
avatar = 'https://d3dg1063dc54p9.cloudfront.net/cute-docsgpt.png', avatar = 'https://d3dg1063dc54p9.cloudfront.net/cute-docsgpt.png',
title = 'Get AI assistance', title = 'Get AI assistance',
description = 'DocsGPT\'s AI Chatbot is here to help', description = 'DocsGPT\'s AI Chatbot is here to help',
heroTitle = 'Welcome to DocsGPT !', heroTitle = 'Welcome to DocsGPT !',
heroDescription = 'This chatbot is built with DocsGPT and utilises GenAI, please review important information using sources.', heroDescription = 'This chatbot is built with DocsGPT and utilises GenAI, please review important information using sources.'
size = 'small', }) => {
theme = 'dark',
buttonIcon = 'https://d3dg1063dc54p9.cloudfront.net/widget/message.svg',
buttonBg = 'linear-gradient(to bottom right, #5AF0EC, #E80D9D)',
collectFeedback = true
}: WidgetProps) => {
const [prompt, setPrompt] = React.useState('');
const [status, setStatus] = React.useState<Status>('idle');
const [queries, setQueries] = React.useState<Query[]>([])
const [conversationId, setConversationId] = React.useState<string | null>(null)
const [open, setOpen] = React.useState<boolean>(false)
const [eventInterrupt, setEventInterrupt] = React.useState<boolean>(false); //click or scroll by user while autoScrolling
const isBubbleHovered = useRef<boolean>(false)
const endMessageRef = React.useRef<HTMLDivElement | null>(null);
const md = new MarkdownIt();
const [prompt, setPrompt] = useState('');
const [status, setStatus] = useState<Status>('idle');
const [queries, setQueries] = useState<Query[]>([])
const [conversationId, setConversationId] = useState<string | null>(null)
const [open, setOpen] = useState<boolean>(false)
const [eventInterrupt, setEventInterrupt] = useState<boolean>(false); //click or scroll by user while autoScrolling
const endMessageRef = useRef<HTMLDivElement | null>(null);
const handleUserInterrupt = () => { const handleUserInterrupt = () => {
(status === 'loading') && setEventInterrupt(true); (status === 'loading') && setEventInterrupt(true);
} }
@ -394,40 +315,11 @@ export const DocsGPTWidget = ({
const lastChild = element?.children?.[element.children.length - 1] const lastChild = element?.children?.[element.children.length - 1]
lastChild && scrollToBottom(lastChild) lastChild && scrollToBottom(lastChild)
}; };
React.useEffect(() => {
useEffect(() => {
!eventInterrupt && scrollToBottom(endMessageRef.current); !eventInterrupt && scrollToBottom(endMessageRef.current);
}, [queries.length, queries[queries.length - 1]?.response]); }, [queries.length, queries[queries.length - 1]?.response]);
async function handleFeedback(feedback: FEEDBACK, index: number) {
let query = queries[index]
if (!query.response)
return;
if (query.feedback != feedback) {
sendFeedback({
question: query.prompt,
answer: query.response,
feedback: feedback,
apikey: apiKey
}, apiHost)
.then(res => {
if (res.status == 200) {
query.feedback = feedback;
setQueries((prev: Query[]) => {
return prev.map((q, i) => (i === index ? query : q));
});
}
})
.catch(err => console.log("Connection failed",err))
}
else {
delete query.feedback;
setQueries((prev: Query[]) => {
return prev.map((q, i) => (i === index ? query : q));
});
}
}
async function stream(question: string) { async function stream(question: string) {
setStatus('loading') setStatus('loading')
try { try {
@ -436,24 +328,19 @@ export const DocsGPTWidget = ({
question: question, question: question,
apiKey: apiKey, apiKey: apiKey,
apiHost: apiHost, apiHost: apiHost,
selectedDocs: selectDocs,
history: queries, history: queries,
conversationId: conversationId, conversationId: conversationId,
onEvent: (event: MessageEvent) => { onEvent: (event: MessageEvent) => {
const data = JSON.parse(event.data); const data = JSON.parse(event.data);
// check if the 'end' event has been received // check if the 'end' event has been received
if (data.type === 'end') { if (data.type === 'end') {
// set status to 'idle'
setStatus('idle'); setStatus('idle');
}
else if (data.type === 'id') { } else if (data.type === 'id') {
setConversationId(data.id) setConversationId(data.id)
} } else {
else if (data.type === 'error') {
const updatedQueries = [...queries];
updatedQueries[updatedQueries.length - 1].error = data.error;
setQueries(updatedQueries);
setStatus('idle')
}
else {
const result = data.answer; const result = data.answer;
const streamingResponse = queries[queries.length - 1].response ? queries[queries.length - 1].response : ''; const streamingResponse = queries[queries.length - 1].response ? queries[queries.length - 1].response : '';
const updatedQueries = [...queries]; const updatedQueries = [...queries];
@ -465,7 +352,7 @@ export const DocsGPTWidget = ({
); );
} catch (error) { } catch (error) {
const updatedQueries = [...queries]; const updatedQueries = [...queries];
updatedQueries[updatedQueries.length - 1].error = 'Something went wrong !' updatedQueries[updatedQueries.length - 1].error = 'error'
setQueries(updatedQueries); setQueries(updatedQueries);
setStatus('idle') setStatus('idle')
//setEventInterrupt(false) //setEventInterrupt(false)
@ -484,21 +371,16 @@ export const DocsGPTWidget = ({
event.currentTarget.src = "https://d3dg1063dc54p9.cloudfront.net/cute-docsgpt.png"; event.currentTarget.src = "https://d3dg1063dc54p9.cloudfront.net/cute-docsgpt.png";
}; };
return ( return (
<ThemeProvider theme={themes[theme]}> <>
{open && size === 'large' && <WidgetContainer>
<Overlay onClick={() => {
setOpen(false)
}} />
}
<FloatingButton bgcolor={buttonBg} onClick={() => setOpen(!open)} hidden={open}>
<img style={{ maxHeight: '4rem', maxWidth: '4rem' }} src={buttonIcon} />
</FloatingButton>
<WidgetContainer modal={size == 'large'}>
<GlobalStyles /> <GlobalStyles />
{!open && <FloatingButton onClick={() => setOpen(true)} hidden={open}>
<MessageIcon style={{ marginTop: '8px' }} />
</FloatingButton>}
{open && <StyledContainer> {open && <StyledContainer>
<div> <div>
<CancelButton onClick={() => setOpen(false)}> <CancelButton onClick={() => setOpen(false)}>
<Cross2Icon width={24} height={24} color={theme === 'light' ? 'black' : 'white'} /> <Cross2Icon width={24} height={24} color='white' />
</CancelButton> </CancelButton>
<Header> <Header>
<IconWrapper> <IconWrapper>
@ -510,11 +392,11 @@ export const DocsGPTWidget = ({
</ContentWrapper> </ContentWrapper>
</Header> </Header>
</div> </div>
<Conversation size={size} onWheel={handleUserInterrupt} onTouchMove={handleUserInterrupt}> <Conversation onWheel={handleUserInterrupt} onTouchMove={handleUserInterrupt}>
{ {
queries.length > 0 ? queries?.map((query, index) => { queries.length > 0 ? queries?.map((query, index) => {
return ( return (
<React.Fragment key={index}> <Fragment key={index}>
{ {
query.prompt && <MessageBubble type='QUESTION'> query.prompt && <MessageBubble type='QUESTION'>
<Message <Message
@ -525,34 +407,13 @@ export const DocsGPTWidget = ({
</MessageBubble> </MessageBubble>
} }
{ {
query.response ? <MessageBubble onMouseOver={() => { isBubbleHovered.current = true }} type='ANSWER'> query.response ? <MessageBubble type='ANSWER'>
<Message <Message
type='ANSWER' type='ANSWER'
ref={(index === queries.length - 1) ? endMessageRef : null} ref={(index === queries.length - 1) ? endMessageRef : null}
> >
<div <div className="response" dangerouslySetInnerHTML={{ __html: sanitize(snarkdown(query.response)) }} />
className="response"
dangerouslySetInnerHTML={{ __html: DOMPurify.sanitize(md.render(query.response)) }}
/>
</Message> </Message>
{collectFeedback &&
<Feedback>
<Like
style={{
stroke: query.feedback == 'LIKE' ? '#8860DB' : '#c0c0c0',
visibility: query.feedback == 'LIKE' ? 'visible' : 'hidden'
}}
fill='none'
onClick={() => handleFeedback("LIKE", index)} />
<Dislike
style={{
stroke: query.feedback == 'DISLIKE' ? '#ed8085' : '#c0c0c0',
visibility: query.feedback == 'DISLIKE' ? 'visible' : 'hidden'
}}
fill='none'
onClick={() => handleFeedback("DISLIKE", index)} />
</Feedback>}
</MessageBubble> </MessageBubble>
: <div> : <div>
{ {
@ -562,7 +423,7 @@ export const DocsGPTWidget = ({
</IconWrapper> </IconWrapper>
<div> <div>
<h5 style={{ margin: 2 }}>Network Error</h5> <h5 style={{ margin: 2 }}>Network Error</h5>
<span style={{ margin: 2, fontSize: '13px' }}>{query.error}</span> <span style={{ margin: 2, fontSize: '13px' }}>Something went wrong !</span>
</div> </div>
</ErrorAlert> </ErrorAlert>
: <MessageBubble type='ANSWER'> : <MessageBubble type='ANSWER'>
@ -575,25 +436,24 @@ export const DocsGPTWidget = ({
} }
</div> </div>
} }
</React.Fragment>) </Fragment>)
}) })
: <Hero title={heroTitle} description={heroDescription} theme={theme} /> : <Hero title={heroTitle} description={heroDescription} />
} }
</Conversation> </Conversation>
<PromptContainer <PromptContainer
size={size}
onSubmit={handleSubmit}> onSubmit={handleSubmit}>
<StyledInput <StyledInput
value={prompt} onChange={(event) => setPrompt(event.target.value)} value={prompt} onChange={(event) => setPrompt(event.target.value)}
type='text' placeholder="What do you want to do?" /> type='text' placeholder="What do you want to do?" />
<StyledButton <StyledButton
size={size} disabled={prompt.length == 0 || status !== 'idle'}>
disabled={prompt.trim().length == 0 || status !== 'idle'}>
<PaperPlaneIcon width={15} height={15} color='white' /> <PaperPlaneIcon width={15} height={15} color='white' />
</StyledButton> </StyledButton>
</PromptContainer> </PromptContainer>
</StyledContainer>} </StyledContainer>}
</WidgetContainer> </WidgetContainer>
</ThemeProvider> </>
) )
} }

@ -9,11 +9,5 @@
<body> <body>
<div id="app"></div> <div id="app"></div>
<script type="module" src="main.tsx"></script> <script type="module" src="main.tsx"></script>
<script type="module" src="../dist/main.js"></script>
<script type="module">
window.onload = function() {
renderDocsGPTWidget('app');
}
</script>
</body> </body>
</html> </html>

@ -1,12 +1,6 @@
import React from 'react';
import { createRoot } from 'react-dom/client'; import { createRoot } from 'react-dom/client';
import { DocsGPTWidget } from './components/DocsGPTWidget'; import App from './App.tsx';
import React from 'react';
const root = createRoot(document.getElementById('app') as HTMLElement);
if (typeof window !== 'undefined') { root.render(<App />);
const renderWidget = (elementId: string, props = {}) => {
const root = createRoot(document.getElementById(elementId) as HTMLElement);
root.render(<DocsGPTWidget {...props} />);
};
(window as any).renderDocsGPTWidget = renderWidget;
}
export { DocsGPTWidget };

@ -1,106 +1,92 @@
import { FEEDBACK } from "@/types";
interface HistoryItem { interface HistoryItem {
prompt: string; prompt: string;
response?: string; response?: string;
} }
interface FetchAnswerStreamingProps { interface FetchAnswerStreamingProps {
question?: string; question?: string;
apiKey?: string; apiKey?: string;
selectedDocs?: string; selectedDocs?: string;
history?: HistoryItem[]; history?: HistoryItem[];
conversationId?: string | null; conversationId?: string | null;
apiHost?: string; apiHost?: string;
onEvent?: (event: MessageEvent) => void; onEvent?: (event: MessageEvent) => void;
} }
interface FeedbackPayload {
question: string;
answer: string;
apikey: string;
feedback: FEEDBACK;
}
export function fetchAnswerStreaming({ export function fetchAnswerStreaming({
question = '', question = '',
apiKey = '', apiKey = '',
history = [], selectedDocs = '',
conversationId = null, history = [],
apiHost = '', conversationId = null,
onEvent = () => { console.log("Event triggered, but no handler provided."); } apiHost = '',
}: FetchAnswerStreamingProps): Promise<void> { onEvent = () => {console.log("Event triggered, but no handler provided.");}
return new Promise<void>((resolve, reject) => { }: FetchAnswerStreamingProps): Promise<void> {
const body = { let docPath = 'default';
question: question, if (selectedDocs) {
history: JSON.stringify(history), docPath = selectedDocs;
conversation_id: conversationId, }
model: 'default',
api_key: apiKey
};
fetch(apiHost + '/stream', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify(body),
})
.then((response) => {
if (!response.body) throw Error('No response body');
const reader = response.body.getReader(); return new Promise<void>((resolve, reject) => {
const decoder = new TextDecoder('utf-8'); const body = {
let counterrr = 0; question: question,
const processStream = ({ api_key: apiKey,
done, embeddings_key: apiKey,
value, active_docs: docPath,
}: ReadableStreamReadResult<Uint8Array>) => { history: JSON.stringify(history),
if (done) { conversation_id: conversationId,
resolve(); model: 'default'
return; };
}
counterrr += 1; fetch(apiHost + '/stream', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify(body),
})
.then((response) => {
if (!response.body) throw Error('No response body');
const chunk = decoder.decode(value); const reader = response.body.getReader();
const decoder = new TextDecoder('utf-8');
let counterrr = 0;
const processStream = ({
done,
value,
}: ReadableStreamReadResult<Uint8Array>) => {
if (done) {
resolve();
return;
}
const lines = chunk.split('\n'); counterrr += 1;
for (let line of lines) { const chunk = decoder.decode(value);
if (line.trim() == '') {
continue;
}
if (line.startsWith('data:')) {
line = line.substring(5);
}
const messageEvent = new MessageEvent('message', { const lines = chunk.split('\n');
data: line,
});
onEvent(messageEvent); // handle each message for (let line of lines) {
} if (line.trim() == '') {
continue;
}
if (line.startsWith('data:')) {
line = line.substring(5);
}
reader.read().then(processStream).catch(reject); const messageEvent = new MessageEvent('message', {
}; data: line,
});
reader.read().then(processStream).catch(reject); onEvent(messageEvent); // handle each message
}) }
.catch((error) => {
console.error('Connection failed:', error);
reject(error);
});
});
}
reader.read().then(processStream).catch(reject);
};
export const sendFeedback = (payload: FeedbackPayload,apiHost:string): Promise<Response> => { reader.read().then(processStream).catch(reject);
return fetch(`${apiHost}/api/feedback`, { })
method: 'POST', .catch((error) => {
headers: { console.error('Connection failed:', error);
'Content-Type': 'application/json' reject(error);
}, });
body: JSON.stringify({ });
question: payload.question, }
answer: payload.answer,
feedback: payload.feedback,
api_key:payload.apikey
}),
});
};

@ -1,7 +1,7 @@
export type MESSAGE_TYPE = 'QUESTION' | 'ANSWER' | 'ERROR'; export type MESSAGE_TYPE = 'QUESTION' | 'ANSWER' | 'ERROR';
export type Status = 'idle' | 'loading' | 'failed'; export type Status = 'idle' | 'loading' | 'failed';
export type FEEDBACK = 'LIKE' | 'DISLIKE'; export type FEEDBACK = 'LIKE' | 'DISLIKE';
export type THEME = 'light' | 'dark';
export interface Query { export interface Query {
prompt: string; prompt: string;
response?: string; response?: string;
@ -11,17 +11,3 @@ export interface Query {
conversationId?: string | null; conversationId?: string | null;
title?: string | null; title?: string | null;
} }
export interface WidgetProps {
apiHost?: string;
apiKey?: string;
avatar?: string;
title?: string;
description?: string;
heroTitle?: string;
heroDescription?: string;
size?: 'small' | 'medium' | 'large';
theme?:THEME,
buttonIcon?:string;
buttonBg?:string;
collectFeedback?:boolean
}

@ -152,12 +152,12 @@
} }
}, },
"node_modules/braces": { "node_modules/braces": {
"version": "3.0.3", "version": "3.0.2",
"resolved": "https://registry.npmjs.org/braces/-/braces-3.0.3.tgz", "resolved": "https://registry.npmjs.org/braces/-/braces-3.0.2.tgz",
"integrity": "sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA==", "integrity": "sha512-b8um+L1RzM3WDSzvhm6gIz1yfTbBt6YTlcEKAvsmqCZZFw46z626lVj9j1yEPW33H5H+lBQpZMP1k8l+78Ha0A==",
"dev": true, "dev": true,
"dependencies": { "dependencies": {
"fill-range": "^7.1.1" "fill-range": "^7.0.1"
}, },
"engines": { "engines": {
"node": ">=8" "node": ">=8"
@ -294,9 +294,9 @@
} }
}, },
"node_modules/fill-range": { "node_modules/fill-range": {
"version": "7.1.1", "version": "7.0.1",
"resolved": "https://registry.npmjs.org/fill-range/-/fill-range-7.1.1.tgz", "resolved": "https://registry.npmjs.org/fill-range/-/fill-range-7.0.1.tgz",
"integrity": "sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg==", "integrity": "sha512-qOo9F+dMUmC2Lcb4BbVvnKJxTPjCm+RRpe4gDuGrzkL7mEVl/djYSu2OdQ2Pa302N4oqkSg9ir6jaLWJ2USVpQ==",
"dev": true, "dev": true,
"dependencies": { "dependencies": {
"to-regex-range": "^5.0.1" "to-regex-range": "^5.0.1"

@ -1,17 +1,13 @@
<!DOCTYPE html> <!DOCTYPE html>
<html lang="en"> <html lang="en">
<head>
<head> <meta charset="UTF-8" />
<meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="viewport" content="width=device-width, initial-scale=1.0,viewport-fit=cover" /> <title>DocsGPT 🦖</title>
<meta name="apple-mobile-web-app-capable" content="yes"> <link rel="shortcut icon" type="image/x-icon" href="/favicon.ico" />
<title>DocsGPT 🦖</title> </head>
<link rel="shortcut icon" type="image/x-icon" href="/favicon.ico" /> <body>
</head> <div id="root" class="h-screen"></div>
<script type="module" src="/src/main.tsx"></script>
<body> </body>
<div id="root" class="h-screen"></div>
<script type="module" src="/src/main.tsx"></script>
</body>
</html> </html>

File diff suppressed because it is too large Load Diff

@ -19,49 +19,43 @@
] ]
}, },
"dependencies": { "dependencies": {
"@reduxjs/toolkit": "^2.2.7", "@reduxjs/toolkit": "^1.9.2",
"@vercel/analytics": "^0.1.10", "@vercel/analytics": "^0.1.10",
"chart.js": "^4.4.4",
"i18next": "^23.15.1",
"i18next-browser-languagedetector": "^8.0.0",
"prop-types": "^15.8.1",
"react": "^18.2.0", "react": "^18.2.0",
"react-chartjs-2": "^5.2.0",
"react-copy-to-clipboard": "^5.1.0", "react-copy-to-clipboard": "^5.1.0",
"react-dom": "^18.3.1", "react-dom": "^18.2.0",
"react-dropzone": "^14.2.3", "react-dropzone": "^14.2.3",
"react-i18next": "^15.0.2", "react-markdown": "^8.0.7",
"react-markdown": "^9.0.1",
"react-redux": "^8.0.5", "react-redux": "^8.0.5",
"react-router-dom": "^6.8.1", "react-router-dom": "^6.8.1",
"react-syntax-highlighter": "^15.5.0", "react-syntax-highlighter": "^15.5.0",
"remark-gfm": "^4.0.0" "remark-gfm": "^3.0.0"
}, },
"devDependencies": { "devDependencies": {
"@types/react": "^18.0.27", "@types/react": "^18.0.27",
"@types/react-dom": "^18.3.0", "@types/react-dom": "^18.0.10",
"@types/react-syntax-highlighter": "^15.5.13", "@types/react-syntax-highlighter": "^15.5.6",
"@typescript-eslint/eslint-plugin": "^5.51.0", "@typescript-eslint/eslint-plugin": "^5.51.0",
"@typescript-eslint/parser": "^5.62.0", "@typescript-eslint/parser": "^5.51.0",
"@vitejs/plugin-react": "^4.3.1", "@vitejs/plugin-react": "^4.2.1",
"autoprefixer": "^10.4.13", "autoprefixer": "^10.4.13",
"eslint": "^8.57.1", "eslint": "^8.33.0",
"eslint-config-prettier": "^9.1.0", "eslint-config-prettier": "^8.6.0",
"eslint-config-standard-with-typescript": "^34.0.0", "eslint-config-standard-with-typescript": "^34.0.0",
"eslint-plugin-import": "^2.30.0", "eslint-plugin-import": "^2.27.5",
"eslint-plugin-n": "^15.7.0", "eslint-plugin-n": "^15.6.1",
"eslint-plugin-prettier": "^5.2.1", "eslint-plugin-prettier": "^4.2.1",
"eslint-plugin-promise": "^6.6.0", "eslint-plugin-promise": "^6.1.1",
"eslint-plugin-react": "^7.35.0", "eslint-plugin-react": "^7.32.2",
"eslint-plugin-unused-imports": "^2.0.0", "eslint-plugin-unused-imports": "^2.0.0",
"husky": "^8.0.0", "husky": "^8.0.0",
"lint-staged": "^15.2.10", "lint-staged": "^13.1.1",
"postcss": "^8.4.41", "postcss": "^8.4.31",
"prettier": "^3.3.3", "prettier": "^2.8.4",
"prettier-plugin-tailwindcss": "^0.6.6", "prettier-plugin-tailwindcss": "^0.2.2",
"tailwindcss": "^3.4.11", "tailwindcss": "^3.2.4",
"typescript": "^4.9.5", "typescript": "^4.9.5",
"vite": "^5.4.6", "vite": "^5.0.13",
"vite-plugin-svgr": "^4.2.0" "vite-plugin-svgr": "^4.2.0"
} }
} }

@ -7,45 +7,29 @@ import { inject } from '@vercel/analytics';
import { useMediaQuery } from './hooks'; import { useMediaQuery } from './hooks';
import { useState } from 'react'; import { useState } from 'react';
import Setting from './settings'; import Setting from './settings';
import './locale/i18n';
import { Outlet } from 'react-router-dom';
import { SharedConversation } from './conversation/SharedConversation';
import { useDarkTheme } from './hooks';
inject(); inject();
function MainLayout() { export default function App() {
const { isMobile } = useMediaQuery(); const { isMobile } = useMediaQuery();
const [navOpen, setNavOpen] = useState(!isMobile); const [navOpen, setNavOpen] = useState(!isMobile);
return ( return (
<div className="dark:bg-raisin-black relative h-screen overflow-auto"> <div className="min-h-full min-w-full dark:bg-raisin-black">
<Navigation navOpen={navOpen} setNavOpen={setNavOpen} /> <Navigation navOpen={navOpen} setNavOpen={setNavOpen} />
<div <div
className={`h-[calc(100dvh-64px)] sm:h-screen ${ className={`transition-all duration-200 ${
!isMobile !isMobile
? `ml-0 ${!navOpen ? 'md:mx-auto lg:mx-auto' : 'md:ml-72'}` ? `ml-0 ${!navOpen ? '-mt-5 md:mx-auto lg:mx-auto' : 'md:ml-72'}`
: 'ml-0 md:ml-16' : 'ml-0 md:ml-16'
}`} }`}
> >
<Outlet /> <Routes>
</div> <Route path="/" element={<Conversation />} />
</div>
);
}
export default function App() {
useDarkTheme();
return (
<div className="h-full relative overflow-auto">
<Routes>
<Route element={<MainLayout />}>
<Route index element={<Conversation />} />
<Route path="/about" element={<About />} /> <Route path="/about" element={<About />} />
<Route path="*" element={<PageNotFound />} />
<Route path="/settings" element={<Setting />} /> <Route path="/settings" element={<Setting />} />
</Route> </Routes>
<Route path="/share/:identifier" element={<SharedConversation />} /> </div>
<Route path="/*" element={<PageNotFound />} />
</Routes>
</div> </div>
); );
} }

@ -1,52 +1,191 @@
import { Fragment } from 'react'; import { useDarkTheme, useMediaQuery } from './hooks';
import DocsGPT3 from './assets/cute_docsgpt3.svg'; import DocsGPT3 from './assets/cute_docsgpt3.svg';
import { useTranslation } from 'react-i18next';
export default function Hero({ export default function Hero({ className = '' }: { className?: string }) {
handleQuestion, // const isMobile = window.innerWidth <= 768;
}: { const { isMobile } = useMediaQuery();
handleQuestion: ({ const [isDarkTheme] = useDarkTheme();
question,
isRetry,
}: {
question: string;
isRetry?: boolean;
}) => void;
}) {
const { t } = useTranslation();
const demos = t('demo', { returnObjects: true }) as Array<{
header: string;
query: string;
}>;
return ( return (
<div <div
className={`pt-20 sm:pt-0 pb-6 sm:pb-12 flex h-full w-full flex-col text-black-1000 dark:text-bright-gray sm:w-full px-2 sm:px-0`} className={`mt-14 mb-32 flex flex-col text-black-1000 dark:text-bright-gray lg:mt-6`}
> >
<div className="flex h-full w-full flex-col items-center justify-center"> <div className=" mb-2 flex items-center justify-center sm:mb-10">
<div className="flex items-center"> <p className="mr-2 text-4xl font-semibold">DocsGPT</p>
<span className="p-0 text-4xl font-semibold">DocsGPT</span> <img className="mb-2 h-14" src={DocsGPT3} alt="DocsGPT" />
<img className="mb-1 inline w-14 p-0" src={DocsGPT3} alt="docsgpt" />
</div>
<div className="mb-4 flex flex-col items-center justify-center dark:text-white"></div>
</div> </div>
<div className="mb-16 grid w-full grid-cols-1 items-center gap-4 self-center text-xs sm:w-auto sm:gap-6 md:mb-0 md:text-sm lg:grid-cols-2"> {isMobile ? (
{demos?.map( <p className="mb-3 text-center leading-6">
(demo: { header: string; query: string }, key: number) => Welcome to <span className="font-bold">DocsGPT</span>, your technical
demo.header && documentation assistant! Start by entering your query in the input
demo.query && ( field below, and we&apos;ll provide you with the most relevant
<Fragment key={key}> answers.
<button </p>
onClick={() => handleQuestion({ question: demo.query })} ) : (
className="w-full rounded-full border-2 border-silver px-6 py-4 text-left hover:border-gray-4000 dark:hover:border-gray-3000 xl:min-w-[24vw]" <>
> <p className="mb-3 text-center leading-6">
<p className="mb-1 font-semibold text-black dark:text-silver"> Welcome to DocsGPT, your technical documentation assistant!
{demo.header} </p>
</p> <p className="mb-3 text-center leading-6">
<span className="text-gray-400">{demo.query}</span> Enter a query related to the information in the documentation you
</button> selected to receive
</Fragment> <br /> and we will provide you with the most relevant answers.
), </p>
)} <p className="mb-3 text-center leading-6">
Start by entering your query in the input field below and we will do
the rest!
</p>
</>
)}
<div
className={`mt-0 flex flex-wrap items-center justify-center gap-2 sm:mt-1 sm:gap-4 md:gap-4 lg:gap-0`}
>
{/* first */}
<div className="h-auto rounded-[50px] bg-gradient-to-l from-[#6EE7B7]/70 via-[#3B82F6] to-[#9333EA]/50 p-1 dark:from-[#D16FF8] dark:via-[#48E6E0] dark:to-[#C85EF6] lg:h-60 lg:rounded-tr-none lg:rounded-br-none">
<div
className={`h-full rounded-[45px] bg-white dark:bg-dark-charcoal p-${
isMobile ? '3.5' : '6 py-8'
} lg:rounded-tr-none lg:rounded-br-none`}
>
{/* Add Mobile check here */}
{isMobile ? (
<div className="flex justify-center">
<img
src={
isDarkTheme ? '/message-text-dark.svg' : '/message-text.svg'
}
alt="lock"
className="h-[24px] w-[24px] "
/>
<h2 className="mb-0 pl-1 text-lg font-bold">
Chat with Your Data
</h2>
</div>
) : (
<>
<img
src={
isDarkTheme ? '/message-text-dark.svg' : '/message-text.svg'
}
alt="lock"
className="h-[24px] w-[24px]"
/>
<h2 className="mt-2 mb-3 text-lg font-bold">
Chat with Your Data
</h2>
</>
)}
<p
className={
isMobile
? `w-[250px] text-center text-xs text-gray-500 dark:text-bright-gray`
: `w-[250px] text-xs text-gray-500 dark:text-bright-gray`
}
>
DocsGPT will use your data to answer questions. Whether its
documentation, source code, or Microsoft files, DocsGPT allows you
to have interactive conversations and find answers based on the
provided data.
</p>
</div>
</div>
{/* second */}
<div className="h-auto rounded-[50px] bg-gradient-to-r from-[#6EE7B7]/70 via-[#3B82F6] to-[#9333EA]/50 p-1 dark:from-[#D16FF8] dark:via-[#48E6E0] dark:to-[#C85EF6] lg:h-60 lg:rounded-none lg:py-1 lg:px-0">
<div
className={`h-full rounded-[45px] bg-white dark:bg-dark-charcoal p-${
isMobile ? '3.5' : '6 py-6'
} lg:rounded-none`}
>
{/* Add Mobile check here */}
{isMobile ? (
<div className="flex justify-center ">
<img
src={isDarkTheme ? '/lock-dark.svg' : '/lock.svg'}
alt="lock"
className="h-[24px] w-[24px]"
/>
<h2 className="mb-0 pl-1 text-lg font-bold">
Secure Data Storage
</h2>
</div>
) : (
<>
<img
src={isDarkTheme ? '/lock-dark.svg' : '/lock.svg'}
alt="lock"
className="h-[24px] w-[24px]"
/>
<h2 className="mt-2 mb-3 text-lg font-bold">
Secure Data Storage
</h2>
</>
)}
<p
className={
isMobile
? `w-[250px] text-center text-xs text-gray-500 dark:text-bright-gray`
: `w-[250px] text-xs text-gray-500 dark:text-bright-gray`
}
>
The security of your data is our top priority. DocsGPT ensures the
utmost protection for your sensitive information. With secure data
storage and privacy measures in place, you can trust that your
data is kept safe and confidential.
</p>
</div>
</div>
{/* third */}
<div className="h-auto rounded-[50px] bg-gradient-to-l from-[#6EE7B7]/70 via-[#3B82F6] to-[#9333EA]/50 p-1 dark:from-[#D16FF8] dark:via-[#48E6E0] dark:to-[#C85EF6] lg:h-60 lg:rounded-tl-none lg:rounded-bl-none ">
<div
className={`firefox h-full rounded-[45px] bg-white dark:bg-dark-charcoal p-${
isMobile ? '3.5' : '6 px-6 '
} lg:rounded-tl-none lg:rounded-bl-none`}
>
{/* Add Mobile check here */}
{isMobile ? (
<div className="flex justify-center">
<img
src={
isDarkTheme
? 'message-programming-dark.svg'
: '/message-programming.svg'
}
alt="lock"
className="h-[24px] w-[24px]"
/>
<h2 className="mb-0 pl-1 text-lg font-bold">
Open Source Code
</h2>
</div>
) : (
<>
<img
src={
isDarkTheme
? '/message-programming-dark.svg'
: '/message-programming.svg'
}
alt="lock"
className="h-[24px] w-[24px]"
/>
<h2 className="mt-2 mb-3 text-lg font-bold">
Open Source Code
</h2>
</>
)}
<p
className={
isMobile
? `w-[250px] text-center text-xs text-gray-500 dark:text-bright-gray`
: `w-[250px] text-xs text-gray-500 dark:text-bright-gray`
}
>
DocsGPT is built on open source principles, promoting transparency
and collaboration. The source code is freely available, enabling
developers to contribute, enhance, and customize the app to meet
their specific needs.
</p>
</div>
</div>
</div> </div>
</div> </div>
); );

@ -1,5 +1,5 @@
import * as React from 'react'; import * as React from 'react';
import { useTranslation } from 'react-i18next';
interface ModalProps { interface ModalProps {
handleSubmit: () => void; handleSubmit: () => void;
isCancellable: boolean; isCancellable: boolean;
@ -8,11 +8,8 @@ interface ModalProps {
modalState: string; modalState: string;
isError: boolean; isError: boolean;
errorMessage?: string; errorMessage?: string;
textDelete?: boolean;
} }
const Modal = (props: ModalProps) => { const Modal = (props: ModalProps) => {
const { t } = useTranslation();
return ( return (
<div <div
className={`${ className={`${
@ -20,20 +17,20 @@ const Modal = (props: ModalProps) => {
} absolute z-30 h-screen w-screen bg-gray-alpha`} } absolute z-30 h-screen w-screen bg-gray-alpha`}
> >
{props.render()} {props.render()}
<div className=" mx-auto flex w-[90vw] max-w-lg flex-row-reverse rounded-b-lg bg-white pb-5 pr-5 shadow-lg dark:bg-outer-space"> <div className=" mx-auto flex w-[90vw] max-w-lg flex-row-reverse rounded-b-lg bg-white pb-5 pr-5 shadow-lg">
<div> <div>
<button <button
onClick={() => props.handleSubmit()} onClick={() => props.handleSubmit()}
className="ml-auto h-10 w-20 rounded-3xl bg-violet-800 text-white transition-all hover:bg-violet-700 dark:text-silver" className="ml-auto h-10 w-20 rounded-3xl bg-violet-800 text-white transition-all hover:bg-violet-700"
> >
{props.textDelete ? 'Delete' : 'Save'} Save
</button> </button>
{props.isCancellable && ( {props.isCancellable && (
<button <button
onClick={() => props.handleCancel && props.handleCancel()} onClick={() => props.handleCancel && props.handleCancel()}
className="cursor-pointer rounded-3xl px-5 py-2 text-sm font-medium hover:bg-gray-100 dark:bg-transparent dark:text-light-gray dark:hover:bg-[#767183]/50" className="ml-5 h-10 w-20 rounded-lg border border-violet-700 bg-white text-violet-800 transition-all hover:bg-violet-700 hover:text-white"
> >
{t('cancel')} Cancel
</button> </button>
)} )}
</div> </div>

@ -1,78 +1,75 @@
import { useEffect, useRef, useState } from 'react'; import { useEffect, useRef, useState } from 'react';
import { useTranslation } from 'react-i18next';
import { useDispatch, useSelector } from 'react-redux'; import { useDispatch, useSelector } from 'react-redux';
import { NavLink, useNavigate } from 'react-router-dom'; import { NavLink, useNavigate } from 'react-router-dom';
import PropTypes from 'prop-types';
import conversationService from './api/services/conversationService';
import userService from './api/services/userService';
import Add from './assets/add.svg';
import DocsGPT3 from './assets/cute_docsgpt3.svg'; import DocsGPT3 from './assets/cute_docsgpt3.svg';
import Documentation from './assets/documentation.svg';
import DocumentationDark from './assets/documentation-dark.svg';
import Discord from './assets/discord.svg'; import Discord from './assets/discord.svg';
import DiscordDark from './assets/discord-dark.svg';
import Expand from './assets/expand.svg'; import Expand from './assets/expand.svg';
import Github from './assets/github.svg'; import Github from './assets/github.svg';
import GithubDark from './assets/github-dark.svg';
import Hamburger from './assets/hamburger.svg'; import Hamburger from './assets/hamburger.svg';
import HamburgerDark from './assets/hamburger-dark.svg';
import Info from './assets/info.svg'; import Info from './assets/info.svg';
import InfoDark from './assets/info-dark.svg';
import SettingGear from './assets/settingGear.svg'; import SettingGear from './assets/settingGear.svg';
import Twitter from './assets/TwitterX.svg'; import SettingGearDark from './assets/settingGear-dark.svg';
import Add from './assets/add.svg';
import UploadIcon from './assets/upload.svg'; import UploadIcon from './assets/upload.svg';
import SourceDropdown from './components/SourceDropdown'; import { ActiveState } from './models/misc';
import {
setConversation,
updateConversationId,
} from './conversation/conversationSlice';
import ConversationTile from './conversation/ConversationTile';
import { useDarkTheme, useMediaQuery, useOutsideAlerter } from './hooks';
import useDefaultDocument from './hooks/useDefaultDocument';
import DeleteConvModal from './modals/DeleteConvModal';
import { ActiveState, Doc } from './models/misc';
import APIKeyModal from './preferences/APIKeyModal'; import APIKeyModal from './preferences/APIKeyModal';
import { getConversations, getDocs } from './preferences/preferenceApi';
import { import {
selectApiKeyStatus, selectApiKeyStatus,
selectConversationId,
selectConversations,
selectModalStateDeleteConv,
selectSelectedDocs, selectSelectedDocs,
selectSelectedDocsStatus, selectSelectedDocsStatus,
selectSourceDocs, selectSourceDocs,
setConversations,
setModalStateDeleteConv,
setSelectedDocs, setSelectedDocs,
setSourceDocs, selectConversations,
setConversations,
selectConversationId,
} from './preferences/preferenceSlice'; } from './preferences/preferenceSlice';
import {
setConversation,
updateConversationId,
} from './conversation/conversationSlice';
import { useMediaQuery, useOutsideAlerter } from './hooks';
import Upload from './upload/Upload'; import Upload from './upload/Upload';
import { Doc, getConversations } from './preferences/preferenceApi';
import SelectDocsModal from './preferences/SelectDocsModal';
import ConversationTile from './conversation/ConversationTile';
import { useDarkTheme } from './hooks';
import SourceDropdown from './components/SourceDropdown';
interface NavigationProps { interface NavigationProps {
navOpen: boolean; navOpen: boolean;
setNavOpen: React.Dispatch<React.SetStateAction<boolean>>; setNavOpen: React.Dispatch<React.SetStateAction<boolean>>;
} }
/* const NavImage: React.FC<{ const NavImage: React.FC<{
Light: string | undefined; Light: string | undefined;
Dark: string | undefined; Dark: string | undefined;
}> = ({ Light, Dark }) => { }> = ({ Light, Dark }) => {
return ( return (
<> <>
<img src={Dark} alt="icon" className="ml-2 hidden w-5 dark:block " /> <img src={Dark} alt="icon" className="ml-2 hidden w-5 dark:block " />
<img src={Light} alt="icon" className="ml-2 w-5 dark:hidden filter dark:invert" /> <img src={Light} alt="icon" className="ml-2 w-5 dark:hidden " />
</> </>
); );
}; };
NavImage.propTypes = { NavImage.propTypes = {
Light: PropTypes.string, Light: PropTypes.string,
Dark: PropTypes.string, Dark: PropTypes.string,
}; */ };
export default function Navigation({ navOpen, setNavOpen }: NavigationProps) { export default function Navigation({ navOpen, setNavOpen }: NavigationProps) {
const dispatch = useDispatch(); const dispatch = useDispatch();
const docs = useSelector(selectSourceDocs); const docs = useSelector(selectSourceDocs);
const selectedDocs = useSelector(selectSelectedDocs); const selectedDocs = useSelector(selectSelectedDocs);
const conversations = useSelector(selectConversations); const conversations = useSelector(selectConversations);
const modalStateDeleteConv = useSelector(selectModalStateDeleteConv);
const conversationId = useSelector(selectConversationId); const conversationId = useSelector(selectConversationId);
const { isMobile } = useMediaQuery(); const { isMobile } = useMediaQuery();
const [isDarkTheme] = useDarkTheme(); const [isDarkTheme] = useDarkTheme();
const [isDocsListOpen, setIsDocsListOpen] = useState(false); const [isDocsListOpen, setIsDocsListOpen] = useState(false);
const { t } = useTranslation();
const isApiKeySet = useSelector(selectApiKeyStatus); const isApiKeySet = useSelector(selectApiKeyStatus);
const [apiKeyModalState, setApiKeyModalState] = const [apiKeyModalState, setApiKeyModalState] =
@ -86,6 +83,7 @@ export default function Navigation({ navOpen, setNavOpen }: NavigationProps) {
useState<ActiveState>('INACTIVE'); useState<ActiveState>('INACTIVE');
const navRef = useRef(null); const navRef = useRef(null);
const apiHost = import.meta.env.VITE_API_HOST || 'https://docsapi.arc53.com';
const navigate = useNavigate(); const navigate = useNavigate();
@ -94,7 +92,6 @@ export default function Navigation({ navOpen, setNavOpen }: NavigationProps) {
fetchConversations(); fetchConversations();
} }
}, [conversations, dispatch]); }, [conversations, dispatch]);
async function fetchConversations() { async function fetchConversations() {
return await getConversations() return await getConversations()
.then((fetchedConversations) => { .then((fetchedConversations) => {
@ -105,44 +102,38 @@ export default function Navigation({ navOpen, setNavOpen }: NavigationProps) {
}); });
} }
const handleDeleteAllConversations = () => {
conversationService
.deleteAll()
.then(() => {
fetchConversations();
})
.catch((error) => console.error(error));
};
const handleDeleteConversation = (id: string) => { const handleDeleteConversation = (id: string) => {
conversationService fetch(`${apiHost}/api/delete_conversation?id=${id}`, {
.delete(id, {}) method: 'POST',
})
.then(() => { .then(() => {
fetchConversations(); fetchConversations();
}) })
.catch((error) => console.error(error)); .catch((error) => console.error(error));
}; };
const handleDeleteClick = (doc: Doc) => { const handleDeleteClick = (index: number, doc: Doc) => {
userService const docPath = 'indexes/' + 'local' + '/' + doc.name;
.deletePath(doc.id ?? '')
fetch(`${apiHost}/api/delete_old?path=${docPath}`, {
method: 'GET',
})
.then(() => { .then(() => {
return getDocs(); // remove the image element from the DOM
}) const imageElement = document.querySelector(
.then((updatedDocs) => { `#img-${index}`,
dispatch(setSourceDocs(updatedDocs)); ) as HTMLElement;
dispatch( const parentElement = imageElement.parentNode as HTMLElement;
setSelectedDocs( parentElement.parentNode?.removeChild(parentElement);
updatedDocs?.find((doc) => doc.name.toLowerCase() === 'default'),
),
);
}) })
.catch((error) => console.error(error)); .catch((error) => console.error(error));
}; };
const handleConversationClick = (index: string) => { const handleConversationClick = (index: string) => {
conversationService // fetch the conversation from the server and setConversation in the store
.getConversation(index) fetch(`${apiHost}/api/get_single_conversation?id=${index}`, {
method: 'GET',
})
.then((response) => response.json()) .then((response) => response.json())
.then((data) => { .then((data) => {
navigate('/'); navigate('/');
@ -159,8 +150,13 @@ export default function Navigation({ navOpen, setNavOpen }: NavigationProps) {
name: string; name: string;
id: string; id: string;
}) { }) {
await conversationService await fetch(`${apiHost}/api/update_conversation_name`, {
.update(updatedConversation) method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify(updatedConversation),
})
.then((response) => response.json()) .then((response) => response.json())
.then((data) => { .then((data) => {
if (data) { if (data) {
@ -172,12 +168,16 @@ export default function Navigation({ navOpen, setNavOpen }: NavigationProps) {
console.error(err); console.error(err);
}); });
} }
useOutsideAlerter(navRef, () => { useOutsideAlerter(
if (isMobile && navOpen && apiKeyModalState === 'INACTIVE') { navRef,
setNavOpen(false); () => {
setIsDocsListOpen(false); if (isMobile && navOpen && apiKeyModalState === 'INACTIVE') {
} setNavOpen(false);
}, [navOpen, isDocsListOpen, apiKeyModalState]); setIsDocsListOpen(false);
}
},
[navOpen, isDocsListOpen, apiKeyModalState],
);
/* /*
Needed to fix bug where if mobile nav was closed and then window was resized to desktop, nav would still be closed but the button to open would be gone, as per #1 on issue #146 Needed to fix bug where if mobile nav was closed and then window was resized to desktop, nav would still be closed but the button to open would be gone, as per #1 on issue #146
@ -186,7 +186,6 @@ export default function Navigation({ navOpen, setNavOpen }: NavigationProps) {
useEffect(() => { useEffect(() => {
setNavOpen(!isMobile); setNavOpen(!isMobile);
}, [isMobile]); }, [isMobile]);
useDefaultDocument();
return ( return (
<> <>
{!navOpen && ( {!navOpen && (
@ -255,15 +254,13 @@ export default function Navigation({ navOpen, setNavOpen }: NavigationProps) {
className="opacity-80 group-hover:opacity-100" className="opacity-80 group-hover:opacity-100"
/> />
<p className=" text-sm text-dove-gray group-hover:text-neutral-600 dark:text-chinese-silver dark:group-hover:text-bright-gray"> <p className=" text-sm text-dove-gray group-hover:text-neutral-600 dark:text-chinese-silver dark:group-hover:text-bright-gray">
{t('newChat')} New Chat
</p> </p>
</NavLink> </NavLink>
<div className="mb-auto h-[78vh] overflow-y-auto overflow-x-hidden dark:text-white"> <div className="mb-auto h-[56vh] overflow-y-auto overflow-x-hidden dark:text-white">
{conversations && conversations.length > 0 ? ( {conversations && (
<div> <div>
<div className=" my-auto mx-4 mt-2 flex h-6 items-center justify-between gap-4 rounded-3xl"> <p className="ml-6 mt-3 text-sm font-semibold">Chats</p>
<p className="mt-1 ml-4 text-sm font-semibold">{t('chats')}</p>
</div>
<div className="conversations-container"> <div className="conversations-container">
{conversations?.map((conversation) => ( {conversations?.map((conversation) => (
<ConversationTile <ConversationTile
@ -278,14 +275,12 @@ export default function Navigation({ navOpen, setNavOpen }: NavigationProps) {
))} ))}
</div> </div>
</div> </div>
) : (
<></>
)} )}
</div> </div>
<div className="flex h-auto flex-col justify-end text-eerie-black dark:text-white"> <div className="flex h-auto flex-col justify-end text-eerie-black dark:text-white">
<div className="flex flex-col-reverse border-b-[1px] dark:border-b-purple-taupe"> <div className="flex flex-col-reverse border-b-[1px] dark:border-b-purple-taupe">
<div className="relative my-4 mx-4 flex gap-2"> <div className="relative my-4 flex gap-2 px-2">
<SourceDropdown <SourceDropdown
options={docs} options={docs}
selectedDocs={selectedDocs} selectedDocs={selectedDocs}
@ -300,109 +295,91 @@ export default function Navigation({ navOpen, setNavOpen }: NavigationProps) {
onClick={() => setUploadModalState('ACTIVE')} onClick={() => setUploadModalState('ACTIVE')}
></img> ></img>
</div> </div>
<p className="ml-5 mt-3 text-sm font-semibold">{t('sourceDocs')}</p> <p className="ml-6 mt-3 text-sm font-semibold">Source Docs</p>
</div> </div>
<div className="flex flex-col gap-2 border-b-[1px] py-2 dark:border-b-purple-taupe"> <div className="flex flex-col gap-2 border-b-[1px] py-2 dark:border-b-purple-taupe">
<NavLink <NavLink
to="/settings" to="/settings"
className={({ isActive }) => className={({ isActive }) =>
`my-auto mx-4 flex h-9 cursor-pointer gap-4 rounded-3xl hover:bg-gray-100 dark:hover:bg-[#28292E] ${ `my-auto mx-4 flex h-9 cursor-pointer gap-4 rounded-3xl hover:bg-gray-100 dark:hover:bg-purple-taupe ${
isActive ? 'bg-gray-3000 dark:bg-transparent' : '' isActive ? 'bg-gray-3000 dark:bg-transparent' : ''
}` }`
} }
> >
<img <NavImage Light={SettingGear} Dark={SettingGearDark} />
src={SettingGear}
alt="icon"
className="ml-2 w-5 filter dark:invert"
/>
<p className="my-auto text-sm text-eerie-black dark:text-white"> <p className="my-auto text-sm text-eerie-black dark:text-white">
{t('settings.label')} Settings
</p> </p>
</NavLink> </NavLink>
</div> </div>
<div className="flex justify-between gap-2 border-b-[1.5px] py-2 dark:border-b-purple-taupe">
<div className="flex flex-col gap-2 border-b-[1.5px] py-2 dark:border-b-purple-taupe">
<NavLink <NavLink
to="/about" to="/about"
className={({ isActive }) => className={({ isActive }) =>
`my-auto mx-4 flex h-9 cursor-pointer gap-4 rounded-3xl hover:bg-gray-100 dark:hover:bg-[#28292E] ${ `my-auto mx-4 flex h-9 cursor-pointer gap-4 rounded-3xl hover:bg-gray-100 dark:hover:bg-purple-taupe ${
isActive ? 'bg-gray-3000 dark:bg-[#28292E]' : '' isActive ? 'bg-gray-3000 dark:bg-purple-taupe' : ''
}` }`
} }
> >
<img <NavImage Light={Info} Dark={InfoDark} />
src={Info} <p className="my-auto text-sm">About</p>
alt="icon"
className="ml-2 w-5 filter dark:invert"
/>
<p className="my-auto pr-1 text-sm">{t('about')}</p>
</NavLink> </NavLink>
<div className="flex items-center justify-evenly gap-1 px-1">
<NavLink <a
target="_blank" href="https://docs.docsgpt.co.uk/"
to={'https://discord.gg/WHJdfbQDR4'} target="_blank"
className={ rel="noreferrer"
'rounded-full hover:bg-gray-100 dark:hover:bg-[#28292E]' className="my-auto mx-4 flex h-9 cursor-pointer gap-4 rounded-3xl hover:bg-gray-100 dark:hover:bg-purple-taupe"
} >
> <NavImage Light={Documentation} Dark={DocumentationDark} />
<img <p className="my-auto text-sm ">Documentation</p>
src={Discord} </a>
alt="discord" <a
className="m-2 w-6 self-center filter dark:invert" href="https://discord.gg/WHJdfbQDR4"
/> target="_blank"
</NavLink> rel="noreferrer"
<NavLink className="my-auto mx-4 flex h-9 cursor-pointer gap-4 rounded-3xl hover:bg-gray-100 dark:hover:bg-purple-taupe"
target="_blank" >
to={'https://twitter.com/docsgptai'} <NavImage Light={Discord} Dark={DiscordDark} />
className={ {/* <img src={isDarkTheme ? DiscordDark : Discord} alt="discord-link" className="ml-2 w-5" /> */}
'rounded-full hover:bg-gray-100 dark:hover:bg-[#28292E]' <p className="my-auto text-sm">Visit our Discord</p>
} </a>
>
<img <a
src={Twitter} href="https://github.com/arc53/DocsGPT"
alt="x" target="_blank"
className="m-2 w-5 self-center filter dark:invert" rel="noreferrer"
/> className="mx-4 mt-auto flex h-9 cursor-pointer gap-4 rounded-3xl hover:bg-gray-100 dark:hover:bg-purple-taupe"
</NavLink> >
<NavLink <NavImage Light={Github} Dark={GithubDark} />
target="_blank" <p className="my-auto text-sm">Visit our Github</p>
to={'https://github.com/arc53/docsgpt'} </a>
className={
'rounded-full hover:bg-gray-100 dark:hover:bg-[#28292E]'
}
>
<img
src={Github}
alt="github"
className="m-2 w-6 self-center filter dark:invert"
/>
</NavLink>
</div>
</div> </div>
</div> </div>
</div> </div>
<div className="sticky z-10 h-16 w-full border-b-2 bg-gray-50 dark:border-b-purple-taupe dark:bg-chinese-black md:hidden"> <div className="fixed z-10 h-16 w-full border-b-2 bg-gray-50 dark:border-b-purple-taupe dark:bg-chinese-black md:hidden">
<button <button
className="mt-5 ml-6 h-6 w-6 md:hidden" className="mt-5 ml-6 h-6 w-6 md:hidden"
onClick={() => setNavOpen(true)} onClick={() => setNavOpen(true)}
> >
<img <img
src={Hamburger} src={isDarkTheme ? HamburgerDark : Hamburger}
alt="menu toggle" alt="menu toggle"
className="w-7 filter dark:invert" className="w-7"
/> />
</button> </button>
</div> </div>
<SelectDocsModal
modalState={selectedDocsModalState}
setModalState={setSelectedDocsModalState}
isCancellable={isSelectedDocsSet}
/>
<APIKeyModal <APIKeyModal
modalState={apiKeyModalState} modalState={apiKeyModalState}
setModalState={setApiKeyModalState} setModalState={setApiKeyModalState}
isCancellable={isApiKeySet} isCancellable={isApiKeySet}
/> />
<DeleteConvModal
modalState={modalStateDeleteConv}
setModalState={setModalStateDeleteConv}
handleDeleteAllConv={handleDeleteAllConversations}
/>
<Upload <Upload
modalState={uploadModalState} modalState={uploadModalState}
setModalState={setUploadModalState} setModalState={setUploadModalState}

@ -2,11 +2,11 @@ import { Link } from 'react-router-dom';
export default function PageNotFound() { export default function PageNotFound() {
return ( return (
<div className="grid min-h-screen dark:bg-raisin-black"> <div className="mx-5 grid min-h-screen md:mx-36">
<p className="mx-auto my-auto mt-20 flex w-full max-w-6xl flex-col place-items-center gap-6 rounded-3xl bg-gray-100 p-6 text-jet dark:bg-outer-space dark:text-gray-100 lg:p-10 xl:p-16"> <p className="mx-auto my-auto mt-20 flex w-full max-w-6xl flex-col place-items-center gap-6 rounded-3xl bg-gray-100 p-6 text-jet lg:p-10 xl:p-16">
<h1>404</h1> <h1>404</h1>
<p>The page you are looking for does not exist.</p> <p>The page you are looking for does not exist.</p>
<button className="pointer-cursor mr-4 flex cursor-pointer items-center justify-center rounded-full bg-blue-1000 py-2 px-4 text-white transition-colors duration-100 hover:bg-blue-3000"> <button className="pointer-cursor mr-4 flex cursor-pointer items-center justify-center rounded-full bg-blue-1000 py-2 px-4 text-white hover:bg-blue-3000">
<Link to="/">Go Back Home</Link> <Link to="/">Go Back Home</Link>
</button> </button>
</p> </p>

@ -1,69 +0,0 @@
const baseURL = import.meta.env.VITE_API_HOST || 'https://docsapi.arc53.com';
const defaultHeaders = {
'Content-Type': 'application/json',
};
const apiClient = {
get: (url: string, headers = {}, signal?: AbortSignal): Promise<any> =>
fetch(`${baseURL}${url}`, {
method: 'GET',
headers: {
...defaultHeaders,
...headers,
},
signal,
}).then((response) => {
return response;
}),
post: (
url: string,
data: any,
headers = {},
signal?: AbortSignal,
): Promise<any> =>
fetch(`${baseURL}${url}`, {
method: 'POST',
headers: {
...defaultHeaders,
...headers,
},
body: JSON.stringify(data),
signal,
}).then((response) => {
return response;
}),
put: (
url: string,
data: any,
headers = {},
signal?: AbortSignal,
): Promise<any> =>
fetch(`${baseURL}${url}`, {
method: 'PUT',
headers: {
...defaultHeaders,
...headers,
},
body: JSON.stringify(data),
signal,
}).then((response) => {
return response;
}),
delete: (url: string, headers = {}, signal?: AbortSignal): Promise<any> =>
fetch(`${baseURL}${url}`, {
method: 'DELETE',
headers: {
...defaultHeaders,
...headers,
},
signal,
}).then((response) => {
return response;
}),
};
export default apiClient;

@ -1,38 +0,0 @@
const endpoints = {
USER: {
DOCS: '/api/combine',
DOCS_CHECK: '/api/docs_check',
API_KEYS: '/api/get_api_keys',
CREATE_API_KEY: '/api/create_api_key',
DELETE_API_KEY: '/api/delete_api_key',
PROMPTS: '/api/get_prompts',
CREATE_PROMPT: '/api/create_prompt',
DELETE_PROMPT: '/api/delete_prompt',
UPDATE_PROMPT: '/api/update_prompt',
SINGLE_PROMPT: (id: string) => `/api/get_single_prompt?id=${id}`,
DELETE_PATH: (docPath: string) => `/api/delete_old?source_id=${docPath}`,
TASK_STATUS: (task_id: string) => `/api/task_status?task_id=${task_id}`,
MESSAGE_ANALYTICS: '/api/get_message_analytics',
TOKEN_ANALYTICS: '/api/get_token_analytics',
FEEDBACK_ANALYTICS: '/api/get_feedback_analytics',
LOGS: `/api/get_user_logs`,
MANAGE_SYNC: '/api/manage_sync',
},
CONVERSATION: {
ANSWER: '/api/answer',
ANSWER_STREAMING: '/stream',
SEARCH: '/api/search',
FEEDBACK: '/api/feedback',
CONVERSATION: (id: string) => `/api/get_single_conversation?id=${id}`,
CONVERSATIONS: '/api/get_conversations',
SHARE_CONVERSATION: (isPromptable: boolean) =>
`/api/share?isPromptable=${isPromptable}`,
SHARED_CONVERSATION: (identifier: string) =>
`/api/shared_conversation/${identifier}`,
DELETE: (id: string) => `/api/delete_conversation?id=${id}`,
DELETE_ALL: '/api/delete_all_conversations',
UPDATE: '/api/update_conversation_name',
},
};
export default endpoints;

@ -1,32 +0,0 @@
import apiClient from '../client';
import endpoints from '../endpoints';
const conversationService = {
answer: (data: any, signal: AbortSignal): Promise<any> =>
apiClient.post(endpoints.CONVERSATION.ANSWER, data, {}, signal),
answerStream: (data: any, signal: AbortSignal): Promise<any> =>
apiClient.post(endpoints.CONVERSATION.ANSWER_STREAMING, data, {}, signal),
search: (data: any): Promise<any> =>
apiClient.post(endpoints.CONVERSATION.SEARCH, data),
feedback: (data: any): Promise<any> =>
apiClient.post(endpoints.CONVERSATION.FEEDBACK, data),
getConversation: (id: string): Promise<any> =>
apiClient.get(endpoints.CONVERSATION.CONVERSATION(id)),
getConversations: (): Promise<any> =>
apiClient.get(endpoints.CONVERSATION.CONVERSATIONS),
shareConversation: (isPromptable: boolean, data: any): Promise<any> =>
apiClient.post(
endpoints.CONVERSATION.SHARE_CONVERSATION(isPromptable),
data,
),
getSharedConversation: (identifier: string): Promise<any> =>
apiClient.get(endpoints.CONVERSATION.SHARED_CONVERSATION(identifier)),
delete: (id: string, data: any): Promise<any> =>
apiClient.post(endpoints.CONVERSATION.DELETE(id), data),
deleteAll: (): Promise<any> =>
apiClient.get(endpoints.CONVERSATION.DELETE_ALL),
update: (data: any): Promise<any> =>
apiClient.post(endpoints.CONVERSATION.UPDATE, data),
};
export default conversationService;

@ -1,38 +0,0 @@
import apiClient from '../client';
import endpoints from '../endpoints';
const userService = {
getDocs: (): Promise<any> => apiClient.get(endpoints.USER.DOCS),
checkDocs: (data: any): Promise<any> =>
apiClient.post(endpoints.USER.DOCS_CHECK, data),
getAPIKeys: (): Promise<any> => apiClient.get(endpoints.USER.API_KEYS),
createAPIKey: (data: any): Promise<any> =>
apiClient.post(endpoints.USER.CREATE_API_KEY, data),
deleteAPIKey: (data: any): Promise<any> =>
apiClient.post(endpoints.USER.DELETE_API_KEY, data),
getPrompts: (): Promise<any> => apiClient.get(endpoints.USER.PROMPTS),
createPrompt: (data: any): Promise<any> =>
apiClient.post(endpoints.USER.CREATE_PROMPT, data),
deletePrompt: (data: any): Promise<any> =>
apiClient.post(endpoints.USER.DELETE_PROMPT, data),
updatePrompt: (data: any): Promise<any> =>
apiClient.post(endpoints.USER.UPDATE_PROMPT, data),
getSinglePrompt: (id: string): Promise<any> =>
apiClient.get(endpoints.USER.SINGLE_PROMPT(id)),
deletePath: (docPath: string): Promise<any> =>
apiClient.get(endpoints.USER.DELETE_PATH(docPath)),
getTaskStatus: (task_id: string): Promise<any> =>
apiClient.get(endpoints.USER.TASK_STATUS(task_id)),
getMessageAnalytics: (data: any): Promise<any> =>
apiClient.post(endpoints.USER.MESSAGE_ANALYTICS, data),
getTokenAnalytics: (data: any): Promise<any> =>
apiClient.post(endpoints.USER.TOKEN_ANALYTICS, data),
getFeedbackAnalytics: (data: any): Promise<any> =>
apiClient.post(endpoints.USER.FEEDBACK_ANALYTICS, data),
getLogs: (data: any): Promise<any> =>
apiClient.post(endpoints.USER.LOGS, data),
manageSync: (data: any): Promise<any> =>
apiClient.post(endpoints.USER.MANAGE_SYNC, data),
};
export default userService;

@ -1,3 +0,0 @@
<svg width="18" height="18" viewBox="0 0 18 18" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M14.175 0.843262H16.9354L10.9054 7.75269L18 17.1564H12.4457L8.09229 11.4543L3.11657 17.1564H0.353571L6.80271 9.76355L0 0.844547H5.69571L9.62486 6.05555L14.175 0.843262ZM13.2043 15.5004H14.7343L4.86 2.41312H3.21943L13.2043 15.5004Z" fill="#747474"/>
</svg>

Before

Width:  |  Height:  |  Size: 361 B

@ -1,3 +0,0 @@
<svg width="7" height="12" viewBox="0 0 7 12" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M6.29154 4.88202L1.70154 0.29202C1.60896 0.199438 1.49905 0.125998 1.37808 0.0758932C1.25712 0.0257882 1.12747 -2.37536e-07 0.99654 -2.44235e-07C0.86561 -2.50934e-07 0.735961 0.0257882 0.614997 0.0758931C0.494033 0.125998 0.384122 0.199438 0.29154 0.29202C0.198958 0.384602 0.125519 0.494513 0.0754137 0.615477C0.0253086 0.736441 -0.00048069 0.86609 -0.000480695 0.99702C-0.000480701 1.12795 0.0253086 1.2576 0.0754136 1.37856C0.125519 1.49953 0.198958 1.60944 0.29154 1.70202L4.17154 5.59202L0.29154 9.47202C0.198958 9.5646 0.125518 9.67451 0.0754133 9.79548C0.0253082 9.91644 -0.000481091 10.0461 -0.000481097 10.177C-0.000481102 10.3079 0.0253082 10.4376 0.0754132 10.5586C0.125518 10.6795 0.198958 10.7894 0.29154 10.882C0.384121 10.9746 0.494032 11.048 0.614996 11.0981C0.73596 11.1483 0.865609 11.174 0.99654 11.174C1.12747 11.174 1.25712 11.1483 1.37808 11.0981C1.49905 11.048 1.60896 10.9746 1.70154 10.882L6.29154 6.29202C6.38424 6.19951 6.45779 6.08962 6.50797 5.96864C6.55815 5.84767 6.58398 5.71799 6.58398 5.58702C6.58398 5.45605 6.55815 5.32637 6.50797 5.2054C6.45779 5.08442 6.38424 4.97453 6.29154 4.88202Z" fill="#666666"/>
</svg>

Before

Width:  |  Height:  |  Size: 1.2 KiB

@ -1,3 +0,0 @@
<svg width="14" height="15" viewBox="0 0 14 15" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M12.9294 5.375V12.4583C12.9294 12.8341 12.7801 13.1944 12.5145 13.4601C12.2488 13.7257 11.8885 13.875 11.5127 13.875H3.01274C2.63701 13.875 2.27668 13.7257 2.011 13.4601C1.74532 13.1944 1.59607 12.8341 1.59607 12.4583V2.54167C1.59607 2.16594 1.74532 1.80561 2.011 1.53993C2.27668 1.27426 2.63701 1.125 3.01274 1.125H8.6794M12.9294 5.375V5.25317C12.9293 4.87747 12.78 4.5172 12.5143 4.25158L9.80282 1.54008C9.53721 1.27439 9.17693 1.12508 8.80124 1.125H8.6794M12.9294 5.375H10.0961C9.72035 5.375 9.36001 5.22574 9.09434 4.96007C8.82866 4.69439 8.6794 4.33406 8.6794 3.95833V1.125" stroke="#949494" stroke-width="1.41667" stroke-linecap="round" stroke-linejoin="round"/>
</svg>

Before

Width:  |  Height:  |  Size: 781 B

Some files were not shown because too many files have changed in this diff Show More

Loading…
Cancel
Save