"Inference speed is a challenge when running models locally (see above).\n",
"\n",
"To minimize latency, it is desiable to run models locally on GPU, which ships with many consumer laptops [e.g., Apple devices](https://www.apple.com/newsroom/2022/06/apple-unveils-m2-with-breakthrough-performance-and-capabilities/).\n",
"To minimize latency, it is desirable to run models locally on GPU, which ships with many consumer laptops [e.g., Apple devices](https://www.apple.com/newsroom/2022/06/apple-unveils-m2-with-breakthrough-performance-and-capabilities/).\n",
"\n",
"And even with GPU, the available GPU memory bandwidth (as noted above) is important.\n",
"\n",
@ -254,7 +254,7 @@
"\n",
"`f16_kv`: whether the model should use half-precision for the key/value cache\n",
"* Value: True\n",
"* Meaning: The model will use half-precision, which can be more memory efficient; Metal only support True."
"* Meaning: The model will use half-precision, which can be more memory efficient; Metal only supports True."
]
},
{
@ -291,7 +291,7 @@
"id": "f56f5168",
"metadata": {},
"source": [
"The console log will show the the below to indicate Metal was enabled properly from steps above:\n",
"The console log will show the below to indicate Metal was enabled properly from steps above:\n",
"- `file_path`: the path the the `.bib` bibtex file\n",
"- `file_path`: the path of the `.bib` bibtex file\n",
"- optional `max_docs`: default=None, i.e. not limit. Use it to limit number of retrieved documents.\n",
"- optional `max_content_chars`: default=4000. Use it to limit the number of characters in a single document.\n",
"- optional `load_extra_meta`: default=False. By default only the most important fields from the bibtex entries: `Published` (publication year), `Title`, `Authors`, `Summary`, `Journal`, `Keywords`, and `URL`. If True, it will also try to load return `entry_id`, `note`, `doi`, and `links` fields. \n",
"The record manager relies on a time-based mechanism to determine what content can be cleaned up (when using `full` or `incremental` cleanup modes).\n",
"\n",
"If two tasks run back-to-back, and the first task finishes before the the clock time changes, then the second task may not be able to clean up content.\n",
"If two tasks run back-to-back, and the first task finishes before the clock time changes, then the second task may not be able to clean up content.\n",
"\n",
"This is unlikely to be an issue in actual settings for the following reasons:\n",
The above way allows you to really simply change the chain_type, but it doesn't provide a ton of flexibility over parameters to that chain type. If you want to control those parameters, you can load the chain directly (as you did in [this notebook](/docs/modules/chains/additional/question_answering.html)) and then pass that directly to the the RetrievalQA chain with the `combine_documents_chain` parameter. For example:
The above way allows you to really simply change the chain_type, but it doesn't provide a ton of flexibility over parameters to that chain type. If you want to control those parameters, you can load the chain directly (as you did in [this notebook](/docs/modules/chains/additional/question_answering.html)) and then pass that directly to the RetrievalQA chain with the `combine_documents_chain` parameter. For example:
omahs
11 months ago
committed by
GitHub