Token counts from the OpenAI tokenizer page don't match what the API bills

Pasted the same prompt into the web tokenizer and into my script using tiktoken. Numbers are off by quite a lot. Which one should I trust when estimating cost, and why would they differ?
1

Leave A Reply

Log in to post an answer.

1 Answer

brittons Nov 04, 2026
dasy_71
Three things stack up. First, encodings differ per model, the web page defaults to one and your script may be using another, use tiktoken.encoding_for_model with the exact model name. Second, the API counts the chat framing, every message costs a few tokens for role and separators plus a couple for the reply priming, so a one line prompt bills noticeably more than its text. Third, system prompts and any tool definitions are counted too. Trust tiktoken with the right encoding for the text, add the overhead, and treat the usage block in the response as the real number.