From df3bc473f0e4ed18b3d46164d18abec4bc5c01c0 Mon Sep 17 00:00:00 2001 From: jiangying Date: Fri, 28 Apr 2023 12:24:51 +0800 Subject: [PATCH] add jtokkit to How_to_count_tokens_with_tiktoken.ipynb --- examples/How_to_count_tokens_with_tiktoken.ipynb | 1 + 1 file changed, 1 insertion(+) diff --git a/examples/How_to_count_tokens_with_tiktoken.ipynb b/examples/How_to_count_tokens_with_tiktoken.ipynb index 3ef07ff4..b0419418 100644 --- a/examples/How_to_count_tokens_with_tiktoken.ipynb +++ b/examples/How_to_count_tokens_with_tiktoken.ipynb @@ -37,6 +37,7 @@ "For `cl100k_base` and `p50k_base` encodings:\n", "- Python: [tiktoken](https://github.com/openai/tiktoken/blob/main/README.md)\n", "- .NET / C#: [SharpToken](https://github.com/dmitry-brazhenko/SharpToken)\n", + "- Java: [jtokkit](https://github.com/knuddelsgmbh/jtokkit)\n", "\n", "For `r50k_base` (`gpt2`) encodings, tokenizers are available in many languages.\n", "- Python: [tiktoken](https://github.com/openai/tiktoken/blob/main/README.md) (or alternatively [GPT2TokenizerFast](https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2TokenizerFast))\n",