AI inference,
from A to Z.
13 terms
No matching terms
Try another term or clear the filters.
A
B
C
Cold-start latency
Cold-start latency is the time a newly launched copy of a model takes to load and become ready to serve its first request.
Cost per million tokens
Cost per million tokens is the price an LLM provider charges to process one million tokens, with separate rates for input and output tokens.
E
G
I
M
O
T
Tail latency (p95 / p99 latency)
Tail latency is the response time of the slowest requests in a system, usually reported as the 95th percentile (p95) or 99th percentile (p99).
Time to first token (TTFT)
Time to first token (TTFT) is the time between sending a request to an LLM and receiving the first generated token of its response.