Glossary
Token
The basic unit a model processes — a fragment of text, or a discrete slice of audio or video. Corpus sizes and some data prices are quoted per token, alongside per-hour pricing for media.
Text tokenizers split words into subword pieces — in English, a token averages roughly three-quarters of a word. Audio and video are tokenized too, into discrete acoustic or visual units, so an hour of media corresponds to a token count that varies by model. Corpus sizes, context windows, and some prices are quoted in tokens.
Why it matters
Knowing an archive’s approximate token equivalent lets an owner compare it with how corpora and deals are sized.