Update huggingface_length_function.ipynb (#2203)

HuggingFace -> Hugging Face
doc
Ikko Eltociear Ashimine 1 year ago committed by GitHub
parent 2d3918c152
commit a4a1ee6b5d
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

@ -5,8 +5,8 @@
"id": "13dc0983",
"metadata": {},
"source": [
"# HuggingFace Length Function\n",
"Most LLMs are constrained by the number of tokens that you can pass in, which is not the same as the number of characters. In order to get a more accurate estimate, we can use HuggingFace tokenizers to count the text length.\n",
"# Hugging Face Length Function\n",
"Most LLMs are constrained by the number of tokens that you can pass in, which is not the same as the number of characters. In order to get a more accurate estimate, we can use Hugging Face tokenizers to count the text length.\n",
"\n",
"1. How the text is split: by character passed in\n",
"2. How the chunk size is measured: by Hugging Face tokenizer"

Loading…
Cancel
Save