Archives/langchain
2
0
Fork 1
mirror of https://github.com/hwchase17/langchain synced 2024-11-18 09:25:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update n_gpu_layers"s description (#16685)

Browse Source 
The `n_gpu_layers` parameter in `llama.cpp` supports the use of `-1`,
which means to offload all layers to the GPU, so the document has been
updated.

Ref:
35918873b4/llama_cpp/server/settings.py (L29C22-L29C117)


35918873b4/llama_cpp/llama.py (L125)
This commit is contained in:
Bob Lin 2024-01-29 08:46:50 +08:00 committed by GitHub
parent 0600998f38
commit 0866a984fe
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 3 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
docs/docs/integrations/llms/llamacpp.ipynb
View File

@ -415,7 +415,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"n_gpu_layers = 40 # Change this value based on your model and your GPU VRAM pool.\n", "n_gpu_layers = -1 # The number of layers to put on the GPU. The rest will be on the CPU. If you don't know how many layers there are, you can use -1 to move all to GPU.\n",
"n_batch = 512 # Should be between 1 and n_ctx, consider the amount of VRAM in your GPU.\n", "n_batch = 512 # Should be between 1 and n_ctx, consider the amount of VRAM in your GPU.\n",
"\n", "\n",
"# Make sure the model path is correct for your system!\n", "# Make sure the model path is correct for your system!\n",
@ -501,7 +501,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"n_gpu_layers = 1 # Change this value based on your model and your GPU VRAM pool.\n", "n_gpu_layers = 1 # The number of layers to put on the GPU. The rest will be on the CPU. If you don't know how many layers there are, you can use -1 to move all to GPU.\n",
"n_batch = 512 # Should be between 1 and n_ctx, consider the amount of RAM of your Apple Silicon Chip.\n", "n_batch = 512 # Should be between 1 and n_ctx, consider the amount of RAM of your Apple Silicon Chip.\n",
"# Make sure the model path is correct for your system!\n", "# Make sure the model path is correct for your system!\n",
"llm = LlamaCpp(\n", "llm = LlamaCpp(\n",
@ -559,7 +559,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"n_gpu_layers = 1 # Metal set to 1 is enough.\n", "n_gpu_layers = 1 # The number of layers to put on the GPU. The rest will be on the CPU. If you don't know how many layers there are, you can use -1 to move all to GPU.\n",
"n_batch = 512 # Should be between 1 and n_ctx, consider the amount of RAM of your Apple Silicon Chip.\n", "n_batch = 512 # Should be between 1 and n_ctx, consider the amount of RAM of your Apple Silicon Chip.\n",
"# Make sure the model path is correct for your system!\n", "# Make sure the model path is correct for your system!\n",
"llm = LlamaCpp(\n", "llm = LlamaCpp(\n",