The API name you call does not pin down the model behind it. On October 17, 2026, DeepSeek began serving DeepSeek-V4.1-Flash-1017 under the existing deepseek-flash API name and sent requests for two retired names deepseek-v4-flash and deepseek-v4-flash-vision-exp to DeepSeek-V4.1-Flash-1017. Teams that changed nothing in their code started getting answers from a model they’d never tested.
When that happens, the failure looks like your bug and the usual signals can’t tell you the provider changed something. The model ID in the API response is set by the provider and doesn’t identify the weights or numerical precision that served the request. The model name and version number in your request are not enough either. DeepSeek kept the name deepseek-flash while changing the model behind it, and a version number only fixes the model when the provider guarantees that the ID always refers to the same checkpoint.
This article shows you how to verify which model your inference provider is serving. Start by checking whether the endpoint’s documented version and settings match what your application expects. Then compare the endpoint’s responses with responses from a reference deployment, a copy of the expected model running under known settings with prompts you have used before. These checks can show that the model changed. They cannot prove which model handled a given request, because the same model can answer differently on different hardware and a provider can treat test prompts differently from the rest of your traffic. That proof requires hardware-backed attestation from the provider.
The running example is a coding assistant on a hosted DeepSeek endpoint advertised as serving the 1017 release whose tool calls have started failing. The checks below work out whether the hosted model changed or something else went wrong.
Establish what should be running
The model name in your API request does not fully describe what runs. The provider can change the weights or the way it serves them without changing the name. To establish what should be running, start with recording a checkpoint, which is the exact version of the model weights. Then record the numerical precision, tokenizer, chat template and generation settings. The chat template formats messages and tool definitions into the token sequence the model receives, so a different template changes the model’s input even when your request JSON is identical.
For the coding assistant, compare the hosted endpoint's responses with responses from a reference deployment running the 1017 checkpoint. Comparing against an older checkpoint could reveal differences between model versions rather than a provider problem.
Three causes of changed behavior
Changed behavior usually traces to one of three causes, each with their own next step to investigate further:
- Announced model update: the provider starts serving a newer model under the same API name and announces the change, as in the DeepSeek example. The next step is to test whether the new version still meets your application's requirements.
- Serving misconfiguration: the expected model runs with an incorrect chat template, response parser, or output limit. The next step is to find a request that demonstrates the problem and ask the provider to correct the configuration.
- Undisclosed model change: the provider uses different weights or changes the model’s weights’ numerical precision without telling you. The next step is to save the affected requests and results, and ask the provider to explain what changed.
If those three explanations don’t fit what you see, it’s possible the model provider is engaging in covert model substitution. This is when a model provider deliberately serves a different model from the one requested or advertised. A failed tool call alone does not prove that model substitution happened.
With open weights, you can run the expected model yourself and compare local and hosted responses. That option does not exist for a closed-source model served by its own developer, such as Claude Opus 5.5. Parasail's migration guide covers how to control model versions and serving settings.
Check the endpoint settings and tool calls
Log every request and response with its timestamp, endpoint URL, requested and returned model IDs, and selected provider when the API provides it. Include the model’s reasoning mode, output limit, and provider routing settings. A returned model ID is the provider's claim about what model was used for your request. It does not prove which model or precision you actually got.
Run these checks before you collect responses for statistical tests:
- Version and routing: ask the model provider whether the requested model ID always refers to the same checkpoint and whether requests can be redirected to another provider or model. OpenRouter's provider controls let you select providers and a desired quantization level. While you diagnose, select one provider endpoint and disable automatic fallbacks so every request stays on the endpoint under test.
- Precision: ask what precision the provider uses for the weights and for values computed or cached during inference Quantization uses lower-precision numbers and can affect answers. Output tests can detect some precision changes but cannot identify every quantization scheme, so the precision the provider declares is your starting point. The DeepSeek-V4.1-Flash-1017 configuration already specifies quantization, so ask whether the provider quantizes further.
- Limits and parameters: check the endpoint's supported context length, maximum output tokens, and accepted settings. Check what happens when a request exceeds those limits or uses an unsupported setting. An endpoint may impose lower limits than the model supports.
- Tool calls: check that the response names a tool you supplied and that the call arguments satisfy the named tool's JSON schema. Then check whether the selected tool and argument values match the task.
Kimi Vendor Verifier checks parameters, tool-call argument formats, supported features, and reported prompt-token counts before running benchmarks. Adapt these checks to the documented requirements of the model your provider claims to serve.
For the coding assistant, define a read_file tool that accepts a path argument and ask the assistant to read README.md. Check that the assistant’s response calls read_file with path set to README.md.
If you are investigating a failed tool call from a longer conversation, replay that conversation with the same message history and requested output-token limit. If the basic file-reading test succeeds but the response to the longer conversation is cut off, check whether the hosted endpoint applied the output-token limit you requested.
A published provider comparison such as Parasail vs. DeepInfra only applies to your endpoint if it covers the same model version and settings, so check that before you rely on one.
Compare investigation results against a controlled reference
Create a reference on a deployment whose checkpoint and configuration you can inspect, ideally one you set up locally. When the expected model's weights are available, that means running the model where you control the version and inference settings.
Configure the reference to match the hosted endpoint's documented tokenizer, template, reasoning mode, sampling settings and output limits, then repeat reference requests to measure normal response variation. A configuration mismatch produces differences unrelated to model substitution, and differences in the software that runs the model can still affect results after the configuration matches.
When the expected model’s weights are available, you can run that model in a separate deployment where you control the model version and inference settings. Compare the provider’s responses with responses from this deployment. This gives you a known configuration to test against, although differences in the software used to run the model can still affect the results.
For DeepSeek-V4.1-Flash, DeepSeek publishes two projects that help you build that reference. deepseek-recipe formats API requests into model inputs and converts generated output into API responses. DeepInfer runs the model itself. These projects do not provide DeepSeek’s complete hosted service, so a deployment built with them should not be assumed to behave identically to DeepSeek’s API.
If you cannot run the expected model locally, send identical prompts to two hosted providers and compare repeated responses. Similar responses do not confirm either endpoint's weights, and different responses do not establish which endpoint serves the wrong model.
Test for significant differences across repeated responses
Model Equality Testing sends the same prompts several times each to the hosted API and to a reference deployment, then compares the two sets of responses. The test asks whether API and reference responses differ more than normal sampling variation would explain.
A significant result shows that the hosted and reference outputs differ. It does not show why, or which model the API is serving. A result that is not significant does not prove the weights are identical either. Set the significance level before you collect responses. When the test’s assumptions hold, the significance level caps the rate of false-positives it flags.
Compare log probabilities when the API exposes them
Log probabilities, also called logprobs, express token probabilities on a logarithmic scale. To compare them, have the reference model score the API's returned token sequence using the hosted endpoint's full input and tokenizer. API and reference logprobs are only comparable when both models receive identical preceding context, including system instructions and reasoning tokens. If the API does not return logprobs or does not expose that context, skip direct logprob comparison.
Research has found that inference software, hardware, and request batching can change logprobs even with identical model weights and no sample count or maximum difference between API and reference logprobs reliably verifies model identity across providers. Measure how reference logprobs vary across repeated runs and configuration changes before interpreting a difference between API and reference logprobs.
Use LLM fingerprinting to identify likely models
LLM fingerprinting compares an endpoint's responses to patterns previously recorded from known models. LLMmap reported over 95% identification accuracy across 42 model versions with as few as 8 interactions in its evaluated setting.
Fingerprinting results depend on the candidate models represented in the fingerprint database. A database built before a new model release like DeepSeek-V4.1-Flash-1017 cannot be used to identify that model. Identifying a model family or version does not establish the hosted model's numerical precision.
Ask the provider for attestation to confirm which model handled the request
Output tests can miss a model substitution. A malicious model provider could identify verification requests and send those to the advertised model while using another model for other requests.
Different inference software or hardware can also produce different outputs from the same model. vLLM's reproducibility documentation limits reproducibility guarantees to the same hardware and vLLM version, even with reproducibility settings enabled. Investigate discrepancies without treating every text mismatch as proof of substitution.
Evidence beyond the provider's claim takes hardware-backed attestation that covers the inference software and the model. A trusted execution environment (TEE) uses hardware protections to isolate the inference program and its data from the host operating system. Attestation provides a signed report that your client can check against an approved configuration.
Verifying the model requires an additional check. The protected inference program must confirm that the weights it uses match a trusted cryptographic fingerprint of the expected model. Your client then must verify evidence covering that check and establish an encrypted connection to the same verified program. Together these steps connect the evidence about the model to the program that actually receives your request.
Ask the provider how your client verifies the inference program's identity, how the program checks the loaded weights against a cryptographic hash of the expected weight files, and how your client securely connects to the verified program. Without those answers, an operator could present an attestation report from one deployment while processing requests on another. GPU attestation alone does not identify the model, and NVIDIA distinguishes hardware attestation from workload validation. Even with every check in place, you still trust the hardware, the inference program, your client's attestation verification software and the administrators allowed inside the TEE. Attestation does not establish answer quality.
Without evidence linking your requests to the verified inference program and weights, output comparisons alone cannot establish which weights handled a particular request.
Rerun checks regularly and as service changes
Save a set of representative requests with criteria for a successful response and repeat checks as the service changes. For the coding assistant, the set includes a simple tool call, a conversation requiring several tool calls, a long-context task, and a code change you can test. Keep versions of both the requests and their evaluation criteria. Parasail's OpenCode walkthrough illustrates checking completed work rather than judging how convincing an answer sounds.
If model changes could seriously disrupt your application, rerun the saved requests daily to verify that they still produce successful responses. For less critical applications start by checking representative request outputs weekly. These intervals are recommendations and do not guarantee detection. In addition to scheduled checks, rerun saved requests after a model update announcement, provider routing change, or unexplained changes in quality or cost. Keep earlier results for comparison.
To keep the model version fixed, use an ID guaranteed by the provider to always refer to the same checkpoint. A version number in the name is not enough. Keep the selected provider and settings fixed where supported. When a check on a representative request fails, save the request and endpoint settings and run it on the hosted and reference deployments. Then simplify the request while checking whether the hosted endpoint's failure persists. Update evaluation criteria only after your team understands and accepts the changed model behavior.
Use Artificial Analysis's Endpoint Accuracy Index as a second opinion. The index compares provider endpoints with a self-hosted reference on tool use, reasoning, and long-context tasks, reporting measurement dates and statistical uncertainty. Endpoint accuracy measures task performance, not the identity of the installed weights. Check the index's measurement date and model configuration before applying a score to your endpoint.
If you switch providers, use Parasail's open-weight provider guide to compare options. Test saved requests against the candidate provider before moving traffic.
Run checks on a Parasail endpoint
Choose a model from the Parasail model catalog and record the configuration of an endpoint serving that model. Save representative requests and results. Check settings and tool calls first. If failures remain unexplained, compare the endpoint's responses with responses from a reference deployment.