Intel-backed SambaNova breathes new life into old Nvidia GPUs

Intel's massive investment in AI chip maker SambaNova appears to be paying off. According to the latest benchmark results announced by the company, their new generation AI acceleration platform demonstrates vastly superior performance compared to competing platforms that rely solely on GPUs. Conducted by experts at Artificial Analysis, these tests reveal that the SN50 series accelerators introduced in February have achieved a remarkable success. In tests conducted with short context lengths, it was determined that SambaNova systems generated 763 tokens per second, leaving their competitors far behind. The company states that its systems can sustain a speed of over 450 tokens per second even at longer context lengths.

This extraordinary speed achieved by SambaNova stems from creating a heterogeneous inference platform by combining Nvidia GPUs with its own Reconfigurable Dataflow Units (RDU) technology. The system heavily optimizes performance by disaggregating the stages of the AI inference process. The 'prefill' phase, which has high computational intensity and processes prompts, is handled by four Nvidia H200 GPUs. The 'decode' phase, which requires memory bandwidth and generates output tokens, is managed by a single SambaNova rack containing 16 SN50 accelerators. This method, which is effective in reducing token costs for long-running AI agents like code assistants, is becoming an increasingly adopted standard in the industry.

The concept of disaggregating inference stages has become popularized in the AI hardware sector, initially with Nvidia's NVL72 rack systems. Nvidia took this disaggregation a step further with its Groq-based LPX racks presented at the GTC event held this spring. Following these developments, all major industry players, from AMD to AWS and Cerebras, have started working on heterogeneous inference platforms utilizing one or more accelerators. In this intensely competitive environment, SambaNova aims to give new life to customers' aging GPU fleets by using its own systems as 'decode' accelerators. The company states that by increasing the efficiency of existing hardware, it aims to both reduce costs and maximize performance.

One of the biggest practical advantages of the system offered by SambaNova is that the devices are air-cooled and can be easily integrated into existing data centers. This feature turns the company into a solution for a deeply rooted infrastructure problem in the industry, as state-of-the-art products like Nvidia's next-generation Rubin GPUs absolutely require liquid cooling. Liquid cooling systems entail extra costs and significant infrastructure changes for most existing data centers. Thanks to its air-cooled systems, SambaNova enables customers to experience performance improvements without making massive infrastructure investments. The company also announced that they plan to prove their highly efficient token production capacity by showcasing much more powerful inference configurations containing 128 and ultimately 256 accelerators in the future.

These successful benchmark results come right after a significant milestone in the partnership between SambaNova and Intel. Last month, the companies announced a collaboration with Vector Core Compute, which will be among the first firms to deploy the combination of GPUs and RDUs, accepting TogetherAI as its first large-scale customer. Although significant capital is needed to accelerate the production of their fifth-generation chips, SambaNova is experiencing no difficulties in this regard. The company has successfully completed the initial phase of a $1 billion Series F funding round led by General Atlantic. This new funding round increased SambaNova's valuation to $11 billion, making it one of the strongest players in the AI chip market and reaffirming its solid position in the industry.