This is a tokenizer play ground implemented with python.
- Step by Step
pip install -r requirements.txt
cd src
python app.py- Quick launch
Linux/Mac
bash run.shWindows
double click `run.bat`
t
tokenizer-play-ground
public
1 stars
0 forks
0 issues
Commits
List of commits on branch main.Unverified
97de07e055aa50416047839445721abe52d69fedFix hard code dnabert
tthinkall committed 9 months ago
Unverified
12bfa3f2494b9c55b1d77ca63b4c41da72e898e6Add readme
tthinkall committed 9 months ago
Unverified
60cfe480193e3147995da7ef8f837864125cc5c4Add support for dnabert
tthinkall committed 9 months ago
Unverified
802e1afe5da952c4c8f08b202af5cca898b6c29cUpdate requirments
tthinkall committed 9 months ago
Unverified
30fe9e87f076f38d686976e311a0f6be81de2408Add support for tiktoken
tthinkall committed 9 months ago
Unverified
af9bc58bc8849d07dff0a89351a31d1e0583f7f5Add requirements and run script
tthinkall committed 9 months ago