![[ICO]](/icons/blank.gif){kind=icon}
![[PARENTDIR]](/icons/back.gif){kind=icon}
![[ ]](/icons/unknown.gif){kind=icon}
![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/folder.gif){kind=icon}
| Name | Last modified | Size | Description |
|---|---|---|---|---|
| Parent Directory | - | ||
| TARGETS | 2024-12-02 00:13 | 232 | |
| __init__.py | 2024-12-02 00:13 | 0 | |
| base64.h | 2024-12-02 00:13 | 6.5K | |
| bpe_tokenizer.cpp | 2024-12-02 00:13 | 10K | |
| bpe_tokenizer.h | 2024-12-02 00:13 | 1.6K | |
| targets.bzl | 2024-12-02 00:13 | 2.5K | |
| test/ | 2024-12-02 00:13 | - | |
| tiktoken.cpp | 2024-12-02 00:13 | 14K | |
| tiktoken.h | 2024-12-02 00:13 | 2.8K | |
| tokenizer.h | 2024-12-02 00:13 | 2.0K | |
| tokenizer.py | 2024-12-02 00:13 | 5.2K | |
| utils.py | 2024-12-02 00:13 | 658 | |