Token Technologies — History and Evolution: From Physical Tokens to AI Tokens
The word "token" appears in wildly different contexts: bank RSA tokens, OAuth access tokens, JWT tokens, blockchain tokens, and now AI language model tokens. Each evolved from different problems but shares the same fundamental idea — a unit of identity, access, or value that can be issued, verified, and exchanged. This guide traces the complete history of token technologies, explains how each type works, and clarifies the differences between authentication tokens, authorization tokens, blockchain tokens, and AI tokens.
1960s
software security tokens first introduced
JWT
JSON Web Tokens — most widely used web auth standard
AI tokens
text chunks for LLM context window pricing
OAuth 2.0
modern delegated authorization using tokens
The Token Evolution Timeline
Physical Hardware Tokens
Early two-factor authentication using physical devices that generate one-time codes. IBM and government systems used these for secure facility and system access. The principle: something you know (password) + something you have (token).
RSA SecurID and HMAC-OTP
RSA Security introduces SecurID tokens — small keyfobs that display a new 6-digit code every 60 seconds. Counter-based one-time passwords (HOTP) become the standard for enterprise authentication. Time-based OTP (TOTP) follows, forming the basis of modern authenticator apps like Google Authenticator.
Session Tokens and Web Cookies
Web applications use opaque session tokens stored in cookies. The server maintains session state mapped to the token ID. Every request carries the session cookie; the server looks up the session. Stateful — requires server-side storage.
OAuth 1.0 — Delegated Authorization
First standardized delegated authorization protocol. Allows third-party apps to access user data without sharing passwords. Introduces access tokens, request tokens, and the concept of token scopes — granular permissions attached to tokens.
OAuth 2.0 and JWT Standardization
OAuth 2.0 simplifies the authorization framework significantly. JSON Web Tokens (JWT/RFC 7519) emerge as the standard for stateless authentication — all claims are encoded in the token itself, signed by the server. No server session lookup needed for verification.
Bitcoin — Cryptocurrency Tokens
Bitcoin introduces the concept of digital tokens with cryptographic ownership — a token representing value on a distributed ledger. No central authority required. Later, Ethereum (2015) enables programmable tokens via smart contracts — the ERC-20 standard for fungible tokens and ERC-721 for non-fungible tokens (NFTs).
API Keys and Scoped Access Tokens
API keys become universal for developer access to SaaS platforms. The industry settles on short-lived tokens with scope limitations as the security best practice. Personal Access Tokens (PATs) on GitHub, scoped API keys in Stripe, and service account tokens in Google Cloud formalize this pattern.
AI/LLM Tokens — Text Processing Units
Large language models use "tokens" as atomic units of text processing and pricing. Tokenization splits text into sub-word pieces — "unbelievable" becomes 3–4 tokens. Context windows, pricing, and rate limits are all measured in tokens. ~750 words ≈ 1000 tokens for most LLM models.
Types of Tokens in Modern Computing
Authentication Tokens (JWT)
JSON Web Tokens encode identity claims in a signed structure: Header.Payload.Signature. Stateless — servers verify the signature without a database lookup. Standard for REST APIs, microservices, and SPAs. Claims include: user ID, email, roles, expiry (exp), issued-at (iat).
OAuth 2.0 Access Tokens
Grant delegated access to resources without exposing user credentials. Short-lived (15 minutes to 1 hour). Paired with long-lived refresh tokens for renewal. Scope-limited — "read:email" not "full account access." Bearer tokens are the most common format.
Refresh Tokens
Long-lived tokens (days to weeks) used exclusively to obtain new access tokens when the short-lived access token expires. Never sent to resource servers — only to the authorization server. Stored securely (HttpOnly cookies or encrypted storage). Can be revoked to force re-authentication.
API Keys
Long-lived credentials for developer API access. Simpler than OAuth but higher risk if exposed. Should be environment-specific, rotated regularly, and stored in environment variables (never in code). Rate-limited and scoped to specific operations in modern implementations.
CSRF Tokens
One-time tokens embedded in HTML forms to prevent cross-site request forgery attacks. Server generates a unique token per session, includes it in every form, and verifies it on submission. Stateful — server must track the token. Standard web security practice for any state-changing form action.
Blockchain Tokens
Digital assets on blockchain networks representing ownership, utility, or governance rights. Fungible tokens (ERC-20) are interchangeable — like currency. Non-fungible tokens (ERC-721, NFTs) are unique — like ownership certificates. Smart contracts define token rules without requiring a central authority.
AI/LLM Tokens
Sub-word text fragments used by language models for processing. Tokenization uses algorithms like Byte Pair Encoding (BPE) or SentencePiece. Context windows (4K, 32K, 128K, 1M tokens) define how much text a model can process at once. API pricing is per token (input + output separately).
TOTP/HOTP One-Time Tokens
Time-based (TOTP) or counter-based (HOTP) one-time passwords used for two-factor authentication. TOTP changes every 30 seconds based on the current time. Used by Google Authenticator, Authy, and hardware security keys. Standardized in RFC 6238 (TOTP) and RFC 4226 (HOTP).
JWT Anatomy and Structure
JWT structure
A JWT consists of three Base64url-encoded parts separated by dots: HEADER.PAYLOAD.SIGNATURE. The header specifies the algorithm (HS256, RS256, ES256). The payload contains claims (user ID, expiry, roles, email). The signature verifies that the token hasn't been tampered with. JWTs are signed but NOT encrypted by default — the payload is readable to anyone with the token. Decode any JWT at jwt.io.
// Creating a JWT (server-side)
const jwt = require('jsonwebtoken');
const payload = {
sub: 'user_123', // subject (user ID)
email: 'alice@example.com',
roles: ['user', 'admin'],
iat: Math.floor(Date.now() / 1000), // issued at (Unix timestamp)
exp: Math.floor(Date.now() / 1000) + 3600, // expires in 1 hour
};
const token = jwt.sign(payload, process.env.JWT_SECRET, { algorithm: 'HS256' });
// Returns: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyXzEyMyJ9.signature
// Verifying a JWT (server-side, on each protected request)
try {
const decoded = jwt.verify(token, process.env.JWT_SECRET);
console.log(decoded.sub); // 'user_123'
console.log(decoded.roles); // ['user', 'admin']
} catch (err) {
if (err.name === 'TokenExpiredError') {
// Token is expired — require re-login or use refresh token
} else if (err.name === 'JsonWebTokenError') {
// Token is invalid/tampered — reject request
}
}Token Security Best Practices
Short expiry for access tokens
Access tokens should expire in 15–60 minutes. This limits the damage if a token is intercepted. Use refresh tokens for maintaining sessions across token expiry. Never issue access tokens valid for days or weeks.
Store tokens securely
In browsers: store access tokens in memory (JS variable) for best security. If persistence is needed, use HttpOnly cookies (immune to XSS) or sessionStorage (cleared on tab close). Never store tokens in localStorage without XSS protection — malicious scripts can read it.
Use HTTPS always
Tokens transmitted over HTTP can be intercepted. All token-carrying requests must use HTTPS. Implement HSTS (HTTP Strict Transport Security) to prevent protocol downgrade attacks.
Implement token revocation
JWT statelessness makes revocation hard — once issued, a valid JWT cannot be invalidated until expiry. Solutions: use short expiry, maintain a token denylist in Redis, or use opaque tokens that can be revoked instantly. For high-security applications, short-lived JWTs + revocable refresh tokens is the standard pattern.
Scope tokens appropriately
Issue tokens with the minimum permissions needed for the operation. A token for reading user profile shouldn't also have permission to modify it or delete the account. Scope separation reduces the blast radius of a compromised token.
JWT is signed, not encrypted