Skip to content
GitLab

[jsonl] Support other tokenizers.

Merged Administrator requested to merge tokenizer into main

Created by: stephenroller

Patch Description Add support for other HF tokenizers for when we train our own.

Testing steps Started training OPT baseline with new tokenizer.