Archives/gpt4all
2
0
Fork 0
mirror of https://github.com/nomic-ai/gpt4all synced 2024-11-08 07:10:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
db528ef1b0
gpt4all/gpt4all-backend/mpt_impl.h

40 lines
1.3 KiB
C
Raw Normal View History

Dlopen backend 5 (#779) Major change to the backend that allows for pluggable versions of llama.cpp/ggml. This was squashed merged from dlopen_backend_5 where the history is preserved.
2023-05-31 21:04:01 +00:00
#ifndef MPT_H_I_KNOW_WHAT_I_AM_DOING_WHEN_INCLUDING_THIS_FILE
#error This file is NOT meant to be included outside of mpt.cpp. Doing so is DANGEROUS. Be sure to know what you are doing before proceeding to #define MPT_H_I_KNOW_WHAT_I_AM_DOING_WHEN_INCLUDING_THIS_FILE
#endif
Scaffolding for the mpt <-> ggml project.
2023-05-05 18:04:32 +00:00
#ifndef MPT_H
#define MPT_H
#include <string>
#include <functional>
#include <vector>
#include "llmodel.h"
Dlopen backend 5 (#779) Major change to the backend that allows for pluggable versions of llama.cpp/ggml. This was squashed merged from dlopen_backend_5 where the history is preserved.
2023-05-31 21:04:01 +00:00
struct MPTPrivate;
Scaffolding for the mpt <-> ggml project.
2023-05-05 18:04:32 +00:00
class MPT : public LLModel {
public:
MPT();
~MPT();
bool loadModel(const std::string &modelPath) override;
bool isModelLoaded() const override;
add requiredMem method to llmodel impls most of these can just shortcut out of the model loading logic llama is a bit worse to deal with because we submodule it so I have to at least parse the hparams, and then I just use the size on disk as an estimate for the mem size (which seems reasonable since we mmap() the llama files anyway)
2023-06-26 19:17:34 +00:00
size_t requiredMem(const std::string &modelPath) override;
Scaffolding for the mpt <-> ggml project.
2023-05-05 18:04:32 +00:00
size_t stateSize() const override;
size_t saveState(uint8_t *dest) const override;
size_t restoreState(const uint8_t *src) override;
void setThreadCount(int32_t n_threads) override;
llmodel: constify LLModel::threadCount()
2023-05-21 20:45:29 +00:00
int32_t threadCount() const override;
Scaffolding for the mpt <-> ggml project.
2023-05-05 18:04:32 +00:00
private:
MPTPrivate *d_ptr;
Backend prompt dedup (#822) * Deduplicated prompt() function code
2023-06-04 12:59:24 +00:00
protected:
Fix up for newer models on reset context. This fixes the model from totally failing after a reset context.
2023-06-04 23:31:00 +00:00
std::vector<Token> tokenize(PromptContext &, const std::string&) const override;
llmodel: change tokenToString to not use string_view (#968) fixes a definite use-after-free and likely avoids some other potential ones - std::string will convert to a std::string_view automatically but as soon as the std::string in question goes out of scope it is already freed and the string_view is pointing at freed memory - this is *mostly* fine if its returning a reference to the tokenizer's internal vocab table but it's, imo, too easy to return a reference to a dynamically constructed string with this as replit is doing (and unfortunately needs to do to convert the internal whitespace replacement symbol back to a space)
2023-06-13 11:14:02 +00:00
std::string tokenToString(Token) const override;
Backend prompt dedup (#822) * Deduplicated prompt() function code
2023-06-04 12:59:24 +00:00
Token sampleToken(PromptContext &ctx) const override;
bool evalTokens(PromptContext &ctx, const std::vector<int32_t> &tokens) const override;
int32_t contextLength() const override;
const std::vector<Token>& endTokens() const override;
Scaffolding for the mpt <-> ggml project.
2023-05-05 18:04:32 +00:00
};
#endif // MPT_H
Reference in New Issue Copy Permalink