From 4b869a71a3290bd833db16abb204a23f7b8ce0a0 Mon Sep 17 00:00:00 2001 From: aiqinxuancai Date: Sat, 6 May 2023 04:09:05 +0800 Subject: [PATCH] Update How_to_count_tokens_with_tiktoken.ipynb (#368) * Update How_to_count_tokens_with_tiktoken.ipynb add c# tiktoken lib. * Update How_to_count_tokens_with_tiktoken.ipynb fix format. * Update How_to_count_tokens_with_tiktoken.ipynb fix format. --- examples/How_to_count_tokens_with_tiktoken.ipynb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/examples/How_to_count_tokens_with_tiktoken.ipynb b/examples/How_to_count_tokens_with_tiktoken.ipynb index b0419418..e14073dd 100644 --- a/examples/How_to_count_tokens_with_tiktoken.ipynb +++ b/examples/How_to_count_tokens_with_tiktoken.ipynb @@ -1,6 +1,7 @@ { "cells": [ { + "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ @@ -36,7 +37,7 @@ "\n", "For `cl100k_base` and `p50k_base` encodings:\n", "- Python: [tiktoken](https://github.com/openai/tiktoken/blob/main/README.md)\n", - "- .NET / C#: [SharpToken](https://github.com/dmitry-brazhenko/SharpToken)\n", + "- .NET / C#: [SharpToken](https://github.com/dmitry-brazhenko/SharpToken), [TiktokenSharp](https://github.com/aiqinxuancai/TiktokenSharp)\n", "- Java: [jtokkit](https://github.com/knuddelsgmbh/jtokkit)\n", "\n", "For `r50k_base` (`gpt2`) encodings, tokenizers are available in many languages.\n",