Commit Graph

15 Commits (561b3afd6ef9fc08fbb8bbd59ba7377a232238e6)

Author SHA1 Message Date
simonpfish 4a8376573f Remove unnecessary cell outputs 12 months ago
Roger Zurawicki d2b9c2e4cf Add tiktoken-rs to third-party library (#791) 12 months ago
Simón Fishman e87dde9f27 [tiktoken_counting] fix tokenizer name (#741) 1 year ago
Eliah Kagan 1b3ef07d3e Add Tiktokenizer link in "How to count tokens" (#604)
This adds a link to Tiktokenizer webapp as another tool, in
addition to the OpenAI Tokenizer.
1 year ago
richzw 39f1776c1a add tiktoken-go to How_to_count_tokens_with_tiktoken.ipynb (#605) 1 year ago
Ted Sanders 4a565cdebb Ted/update token counting 0613 (#541)
* updates tiktoken counting function

* slight rewording

* updates ChatGPT guide with new models
1 year ago
Avery Khoo a9f8eb3518 Update How_to_count_tokens_with_tiktoken.ipynb (#511)
Add support for gpt-3.5-*-0613 models
1 year ago
aiqinxuancai 4b869a71a3 Update How_to_count_tokens_with_tiktoken.ipynb (#368)
* Update How_to_count_tokens_with_tiktoken.ipynb

add c# tiktoken lib.

* Update How_to_count_tokens_with_tiktoken.ipynb

fix format.

* Update How_to_count_tokens_with_tiktoken.ipynb

fix format.
1 year ago
jiangying df3bc473f0 add jtokkit to How_to_count_tokens_with_tiktoken.ipynb 1 year ago
dmitry-brazhenko 3c53f34e01 added sharptoken as exampel 2 years ago
Ted Sanders f571555953 updates description of token counting function 2 years ago
Ted Sanders 867d6a3afa updates token counting guide 2 years ago
Ted Sanders 6ee7a82ac4 updates token counting function to work with gpt-4 2 years ago
Ted Sanders 5b956ab602 adds examples of counting tokens for the ChatGPT API 2 years ago
Ted Sanders 22baac6106 adds tiktoken example 2 years ago