Bagatur
4d427b2397
Base language model docstrings ( #7104 )
2023-07-07 16:09:10 -04:00
Zander Chase
785502edb3
Add 'get_token_ids' method ( #4784 )
...
Let user inspect the token ids in addition to getting th enumber of tokens
---------
Co-authored-by: Zach Schillaci <40636930+zachschillaci27@users.noreply.github.com>
2023-05-22 13:17:26 +00:00
Ankush Gola
d3ec00b566
Callbacks Refactor [base] ( #3256 )
...
Co-authored-by: Nuno Campos <nuno@boringbits.io>
Co-authored-by: Davis Chase <130488702+dev2049@users.noreply.github.com>
Co-authored-by: Zander Chase <130414180+vowelparrot@users.noreply.github.com>
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
2023-04-30 11:14:09 -07:00
Mike Wang
b588446bf9
[simple][test] Added test case for schema.py ( #3692 )
...
- added unittest for schema.py covering utility functions and token
counting.
- fixed a nit. based on huggingface doc, the tokenizer model is gpt-2.
[link](https://huggingface.co/transformers/v4.8.2/_modules/transformers/models/gpt2/tokenization_gpt2_fast.html )
- make lint && make format, passed on local
- screenshot of new test running result
<img width="1283" alt="Screenshot 2023-04-27 at 9 51 55 PM"
src="https://user-images.githubusercontent.com/62768671/235057441-c0ac3406-9541-453f-ba14-3ebb08656114.png ">
2023-04-28 20:42:24 -07:00
