Token

A token is a small unit of text that an AI language model reads and generates. It can be a whole word, a piece of a word, a single character, or a punctuation mark, depending on how the model breaks text apart. Language models do not process raw sentences directly; they first split the text into tokens and work with those units. Understanding tokens explains a lot about how these models behave, how they are priced, and why they have limits on how much text they can handle at once.

The mechanics start with a step called tokenization. Before a model can read your input, a tokenizer chops the text into tokens according to a fixed vocabulary the model learned during training. Common words often become a single token, while rarer or longer words get split into several pieces, and spaces and punctuation are counted too. A rough rule of thumb in English is that a token averages around three-quarters of a word, so a paragraph of a hundred words might be roughly a hundred and thirty tokens. The model then processes these tokens through its network and generates its response one token at a time, each new token chosen based on all the tokens that came before. This token-by-token generation is why you often see AI responses appear to type themselves out gradually.

The word comes from Old English tacen, meaning a sign or a symbol, and it has long been used in computing to mean a discrete unit of meaning within a stream of text. In AI it took on the specific sense of the small text units a model reads and produces, a usage that became standard as language models grew central to the field.

For a business, tokens matter in concrete, practical ways. Most AI services charge by the token, counting both the text you send and the text the model returns, so understanding token usage is understanding your cost. Models also have a context window, a maximum number of tokens they can consider at once, which limits how much document or conversation history you can include in a single request. If you need an AI tool to process long contracts, large knowledge bases, or extended conversations, token limits shape what is technically and financially feasible, and they influence design choices like how much to summarize or how to split work into pieces.

A common mistake is assuming tokens map neatly to words, which leads to miscounting cost and hitting length limits unexpectedly. Because rare words, other languages, and code can tokenize into many more pieces than plain English prose, the same visible length of text can consume very different numbers of tokens. Another nuance is that both input and output count, so a verbose prompt or a request for a very long answer both add to the total. It also helps to know that trimming unnecessary context, rather than padding prompts, keeps usage efficient without sacrificing quality. Tokens connect closely to large language models that operate on them, to the prompts measured in them, and to the fine-tuning and machine learning processes that shape how a model handles them. They are a foundational concept for anyone budgeting for or building with AI in a serious way.