Archives/openai-cookbook
2
0
Fork 0
mirror of https://github.com/openai/openai-cookbook synced 2024-11-04 06:00:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

add jtokkit to How_to_count_tokens_with_tiktoken.ipynb

Browse Source
This commit is contained in:
jiangying 2023-04-28 12:24:51 +08:00 committed by GitHub
parent 734577b3dd
commit df3bc473f0
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
examples/How_to_count_tokens_with_tiktoken.ipynb
View File

@ -37,6 +37,7 @@
"For `cl100k_base` and `p50k_base` encodings:\n",
"- Python: [tiktoken](https://github.com/openai/tiktoken/blob/main/README.md)\n",
"- .NET / C#: [SharpToken](https://github.com/dmitry-brazhenko/SharpToken)\n",
"- Java: [jtokkit](https://github.com/knuddelsgmbh/jtokkit)\n",
"\n",
"For `r50k_base` (`gpt2`) encodings, tokenizers are available in many languages.\n",
"- Python: [tiktoken](https://github.com/openai/tiktoken/blob/main/README.md) (or alternatively [GPT2TokenizerFast](https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2TokenizerFast))\n",