Update How_to_count_tokens_with_tiktoken.ipynb (#368)

* Update How_to_count_tokens_with_tiktoken.ipynb

add c# tiktoken lib.

* Update How_to_count_tokens_with_tiktoken.ipynb

fix format.

* Update How_to_count_tokens_with_tiktoken.ipynb

fix format.
pull/399/head
aiqinxuancai 1 year ago committed by GitHub
parent 5008f3a3d3
commit 23bd266fef
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

@ -1,6 +1,7 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
@ -36,7 +37,7 @@
"\n",
"For `cl100k_base` and `p50k_base` encodings:\n",
"- Python: [tiktoken](https://github.com/openai/tiktoken/blob/main/README.md)\n",
"- .NET / C#: [SharpToken](https://github.com/dmitry-brazhenko/SharpToken)\n",
"- .NET / C#: [SharpToken](https://github.com/dmitry-brazhenko/SharpToken), [TiktokenSharp](https://github.com/aiqinxuancai/TiktokenSharp)\n",
"- Java: [jtokkit](https://github.com/knuddelsgmbh/jtokkit)\n",
"\n",
"For `r50k_base` (`gpt2`) encodings, tokenizers are available in many languages.\n",

Loading…
Cancel
Save