LLM Inference on CPU with Intel Xeon 6

Large language model inference is not the exclusive domain of GPUs. Our tests on an Intel Xeon 6980P node demonstrate that, when the right CPU architecture is combined with Multiverse's compression technology (CompactifAI), the results are not only functional but surprisingly capable for multiple real-world use cases.

When people talk about language model inference, the conversation almost automatically revolves around the GPU. This is understandable: GPUs have dominated training and, by extension, have become the default reference point for any AI-related task. However, that association has also led to an underestimation of what a properly sized and utilized modern CPU can do in production.

On July 23, 2026, Multiverse Computing published this news, showing that models compressed with CompactifAI, including Llama 3.3 70B, could run on Intel Xeon 6 processors with strong performance and accuracy. Building on that milestone, this article provides an independent practical validation on an Intel Xeon 6980P node, with several models and load scenarios focused on real-world use cases.

The following sections present the results of a practical and methodical exercise: deploying models from the Llama (Meta) family on a single CPU node based on an Intel Xeon 6980P, measuring their real behavior under stress with the vllm bench serve benchmarking tool and, for the Llama 3.1 8B Instruct version, comparing two versions: the original and the one optimized with CompactifAI (Multiverse) compression technology. The data presented below is neither estimated nor projected; it consists of direct measurements on the hardware (no virtual machines or containers were used either).

The conclusion that emerges from this analysis is twofold. First, and more surprisingly, the Intel Xeon 6980P provides a genuinely capable compute foundation for LLMs in real-world use cases. Second, Multiverse's compression technology (CompactifAI) multiplies performance on that foundation and, in the case of 3.3 70B Instruct, makes it viable (the unoptimized original model could not even start).

Image retrieved from here on 8/5/2026

The node evaluated is neither an experimental machine nor a laboratory prototype. The Intel Xeon 6980P is a high-end processor with 2 sockets, 128 physical cores per socket, and 2 threads per core, resulting in a total of 512 vCPUs available to the inference server. Its operating frequency ranges from 800 MHz to 3,900 MHz depending on the load, and the total available RAM reaches 1.5 TB.

This is not a marginal machine that runs out of resources on its first attempt to load a serious model. It is a node with enough memory capacity to comfortably host 8B and 70B parameter models, and enough execution threads to handle multiple requests in parallel. The limitation is not the platform; it is how the platform is used.

The inference server used was vLLM[1], the option Intel recommends for extracting maximum performance in high-throughput scenarios on x86 architectures. This choice is not accidental: vLLM is specifically designed to maximize transformer efficiency when serving requests, and represents the state of the art available for this hardware category.

The tests were run with vllm bench serve , the same tool Intel uses as a reference in its public documentation. The configuration was identical for all models: 200 prompts per run, up to 32 concurrent requests, with no request-rate limit and prompt and instruction lengths fixed by scenario.

The design covers four use cases rooted in real production needs:

Chatbot (128 input tokens, 128 output tokens): short conversations, support, questions, and answers. The key metric is time to first token (TTFT), which determines the perception of immediate responsiveness.

Content creation (128 input tokens, 2,048 output tokens): generation of long-form text, blogs, and descriptions. The focus shifts to the sustained generation cadence (TPOT).

Describe (2,048 input tokens, 128 output tokens): summarization, synthesis, and tagging. It requires the model to process extensive context before responding.

Correct (2,048 input tokens, 2,048 output tokens): extended editing, rewriting, and document improvement. This is the most demanding case in both dimensions.

Of these four, Chatbot and Correct are the most representative for the validation, covering the two ends of the spectrum: fast conversational response versus sustained high-volume generation.

The two central metrics in the analysis are:

TTFT (Time To First Token): the time until the system delivers the first token to the user. It indicates perceived fluency.

TPOT (Time Per Output Token): the average time per token generated during the rest of the response. It describes sustained generation speed.

Intel establishes indicative thresholds: a maximum TTFT of 3 seconds and a maximum TPOT of 100 ms for conversational use cases, and a TTFT of 10 seconds for long-input cases.

Chatbot Case

The Chatbot scenario is the most demanding in terms of perceived latency. Every millisecond counts when the user is waiting for a response to appear on screen.

The most striking data point in this table is not the absolute number, but the gap between the original version and the CompactifAI-optimized version within the same model family and on the same hardware.

The Llama 3.1 8B Slim model achieves a mean TTFT of 742 ms and a TPOT of 63 ms. This is not merely a relative improvement over the original: it is a result that falls within the operating thresholds Intel considers acceptable for a conversational use case (TTFT < 3,000 ms, TPOT < 100 ms). With the properly compressed model, the CPU responds in less than one second and generates each subsequent token in less than 65 ms. That is a smooth and functional user experience.

The original version of the same model, without CompactifAI compression, has a TTFT of almost 2 seconds and a TPOT of 132 ms, already exceeding the recommended threshold. The difference is 2.69 times in TTFT, 2.09 times in TPOT, and 2.15 times in total throughput.

The Slim model reaches 849 tokens per second of throughput in this configuration. For an inference server serving multiple simultaneous users, this volume has direct implications for concurrent request capacity and operating cost per request.

Correct Case

The Correct scenario is deliberately more demanding: inputs and outputs of 2,048 tokens mean that the model must process a long context before beginning generation, and then maintain its cadence throughout an extensive response. It is the use case most similar to real production tasks involving document editing, code review, or content transformation.

Once again, CompactifAI compression technology makes a substantial difference. The Slim model reduces mean TTFT by 2.52 times compared with the original, and improves TPOT by 1.62 times. Output throughput rises from 139 to 229 tokens per second, a 65% improvement in sustained generation capacity.

The most significant point here is the TTFT median. While the mean can be affected by outliers, the median reflects typical behavior. The Slim model achieves a median of 1,693 ms, compared with 4,227 ms for the original. This means that in half of the requests, the response begins in less than 1.7 seconds, even for a task with 2,048 input context tokens.

Image retrieved from here on 8/5/2026

The observed difference between the original models and the Slim models is not simply a reduction in size that sacrifices quality in exchange for speed. Multiverse's compression technology (CompactifAI) works on the model's internal structure to reduce the computational load of each processed token, resulting in simultaneous improvements across all metrics: less time to first token, lower time per generated token, and higher total throughput.

This is especially relevant in the context of the CPU because the x86 architecture imposes memory bandwidth and access latency constraints that do not exist in the same way on a GPU. On the GPU, massive parallelization makes it possible to mask much of that latency. On the CPU, each operation has a higher relative cost, which is why reducing the computational work per token has a more pronounced multiplying effect.

The data confirms this: CompactifAI's relative gain on the CPU is more pronounced than would be expected in a GPU environment, precisely because the CPU benefits disproportionately from any reduction in the model's computational density.

The case of Llama 3.3 70B Instruct Slim deserves specific mention. A 70-billion-parameter model running on a single CPU node, with no accelerators of any kind, reaching 99 tokens per second of total throughput in the Chatbot scenario is, in itself, a result that would have seemed unlikely only a few years ago.

The original version of Llama 3.3 70B does not even complete initialization in the tests, confirming that without compression this model size is not manageable on this platform. With CompactifAI compression, the model not only loads and operates, but also produces measurable and functional results. The TPOT of 531 ms places the experience outside the optimal range for a low-latency interactive chatbot, but within a perfectly valid range for asynchronous use cases, batch processing, analysis tasks, or document review where the user is not waiting in real time.

This contrast is revealing: the difference between a model that does not start and one that operates at an acceptable throughput is precisely CompactifAI compression technology. It is not a hardware change, an operating system optimization, or a serving parameter adjustment: it is the model that changes.

The data makes it possible to identify precisely the scenarios where the Intel Xeon 6980P + CompactifAI Slim models combination offers operationally reasonable performance:

Assistance and conversational support with compact models (8B Slim): with a TTFT of 742 ms and a TPOT of 63 ms in the Chatbot scenario, the user experience is smooth. These values meet the criteria Intel establishes as quality-of-service references for interactive use cases.

Document editing and rewriting (8B Slim, Correct scenario): with a median TTFT of 1,693 ms and output throughput of 229 tok/s, the Slim model is a legitimate tool for productivity workflows where the response does not need to be instantaneous but should still be reasonably fast.

Batch processing and asynchronous analysis (70B Slim): for tasks that do not require a real-time response, the compressed 70B model produces results at an operational pace. Document analysis, report generation, and data enrichment are tasks where throughput matters more than perceived latency.

Environments without available GPUs: in organizations or infrastructures where access to GPUs is not guaranteed or is costly because of availability, the Xeon 6980P with CompactifAI models offers a functional alternative that requires no architectural changes.

The most important lesson from these tests is not only that the CPU can perform inference, but when and with which model. The internal comparison table within the Llama 3.1 8B family is telling:

These gains are consistent and significant in both scenarios. They are not artifacts of a single test condition, nor do they depend on specific configuration parameters: they emerge from the same benchmark with the same infrastructure, simply by changing the model to its CompactifAI-compressed version.

This pattern has a direct implication for any organization evaluating inference platforms: model choice carries as much weight as hardware choice. A model optimized with advanced compression technology on modern CPU hardware can clearly outperform an original model on the same hardware, and that difference cannot be recovered by tuning server parameters.

The tests on the Intel Xeon 6980P leave two clear messages.

The first, and most revealing, is that language model inference on CPU is not a marginal idea or a last-resort solution. A node with 512 vCPUs, 1.5 TB of RAM, and vLLM as its inference server is a legitimate and functional platform for serving language models. The numbers confirm it: with the right model, the CPU delivers response times and throughput that meet the operating criteria of real-world use cases.

The second message is that CompactifAI compression technology (Multiverse) further improves model performance on the CPU. Without compression, the original models exhibit high latency and limited throughput. With CompactifAI Slim versions, the same hardware delivers between 1.6 and 2.7 times better performance across all key metrics, while also enabling 70B parameter models to operate.

CPU + CompactifAI is not merely a technically interesting alternative: it is a combination that opens up real-world use cases on existing infrastructure without depending on GPU availability. And the data is there to prove it.

Llama 4 Scout (Meta) introduces an architectural change compared with the dense models analyzed so far: it is a Mixture-of-Experts (MoE) model, with 17B active parameters per token distributed among 16 experts, out of approximately 109B total parameters. Its memory footprint and the complexity of routing between experts are much greater than those of an equivalent dense model, even though the computation activated per token remains on the order of a 17B model.

As happened with Llama 3.3 70B Instruct, the original uncompressed version, meta-llama/Llama-4-Scout-17B-16E, did not complete inference server initialization on this node. Only the version compressed with CompactifAI technology was able to start and serve requests.

Compared with Llama 3.3 70B Instruct Slim (the other model that is viable only thanks to compression) in the Chatbot scenario, Llama 4 Scout Slim delivers a mean TTFT less than half as high (3,902 ms versus 7,769 ms) and slightly higher total throughput (105 versus 99 tok/s), with an almost identical TPOT (536 ms versus 531 ms). This is consistent with its MoE architecture: although the model has many more total parameters, each token activates only a fraction of them, so the computation cost per generated token remains in the same range as that of a dense model with a similar number of active parameters.

In the Correct scenario, with 2,048 input and output tokens, TPOT jumps to 938 ms, almost twice its value in Chatbot. The pattern is the same as that observed in dense models (the TPOT of the original Llama 3.1 8B Instruct rises from 132 ms in Chatbot to 211 ms in Correct), amplified here by the additional cost of routing between experts over long sequences. With these figures, Llama 4 Scout Slim, like Llama 3.3 70B Instruct Slim, falls outside the range of a low-latency interactive chatbot and is better suited to batch-processing or asynchronous-analysis scenarios, where sustained throughput matters more than perceived latency.

The underlying message is repeated: without CompactifAI compression, this model does not even start on this node; with it, the model is functional within its limits.

1. The vLLM version used was 0.18.1, due to a restriction in the generation of some of the Multiverse-compressed models to which we had access. vLLM 0.18.1 is the latest version that depends on the 4.x branch of Transformers.