d-Matrix’s Raptor AI accelerator takes a 100 TB/s shot at the memory wall

A vertical answer to AI inference bottlenecks

d-Matrix used Hot Chips 2026 to present Raptor, a generative-AI inference accelerator that stacks a TSMC 4nm logic die directly above a custom DRAM die, connected face-to-face at a 36-micron pitch, and targets 100 TB/s of memory bandwidth from 32 GB per card [1]. The company frames Raptor as a 3D-DRAM accelerator for the phase of large-language-model serving where chips increasingly wait on memory rather than on matrix math [1].

The headline number is striking because it attacks the central weakness of today’s AI servers: moving weights and key-value cache data repeatedly during decode. In that phase, a model emits one token at a time, and the accelerator must repeatedly fetch model weights and cache state; d-Matrix’s argument is that reducing the physical distance between memory and compute can cut both latency and energy [2]. Raptor is therefore less a general-purpose GPU rival than a memory-centric inference bet: put compute on top of DRAM, remove much of the conventional memory-interface overhead, and design the system around the bandwidth-hungry path of token generation [3].

What d-Matrix claims it has built

According to the company’s Hot Chips disclosure as reported this week, one Raptor card combines 32 GB of 3D DRAM with roughly 100 TB/s of bandwidth, while the vertical interface consumes about 0.37 picojoules per transferred bit [1]. TechTimes reported the same 100 TB/s-class bandwidth and 0.37 pJ/bit figure, describing the silicon as working Raptor hardware shown at Hot Chips 2026 [2]. Igor’sLab broke the card down as eight 4 GB chiplets, each delivering about 12.5 TB/s, yielding 32 GB and about 100 TB/s at the card level [3].

The physical architecture is the point. Instead of placing HBM stacks beside a processor on an interposer, Raptor places the logic die above the memory die and connects them through very short vertical links [4]. Digital Citizen lists the implementation as TSMC 4nm logic, face-to-face stacking, a 36-micron interconnect pitch, 32 GB capacity, 100 TB/s bandwidth, and 0.37 pJ/bit measured energy [4]. Tom’s Hardware adds that co-founder and CTO Sudeep Bhoja characterized the 0.37 pJ/bit value as a measured working-silicon number and compared it with roughly 2.4 pJ/bit for moving data into an HBM4 base die [1].

That comparison is the commercial pitch. If a system needs extreme bandwidth for decode, d-Matrix says the conventional HBM path becomes expensive in power because data must traverse a larger physical interface. By bonding memory and compute face-to-face, the company is trying to eliminate much of the PHY burden that makes wide external memory links costly [2]. Digital Citizen notes that even at 0.37 pJ/bit, moving 100 TB/s still implies about 300 W of I/O power, but reports d-Matrix’s estimate that an HBM-based path at around 2.4 pJ/bit would approach 1.92 kW for the same 100 TB/s memory movement [4].

Why 32 GB can be both the catch and the strategy

The obvious caveat is capacity. A 32 GB card is tiny beside the HBM capacity envelopes used in many high-end accelerators, and Tom’s Hardware reports that d-Matrix itself compared Raptor with a 192 GB HBM4 configuration delivering around 18 TB/s [1]. That means the d-Matrix card claims far more bandwidth, but with one-sixth of the memory capacity in that comparison [1].

For inference architects, that tradeoff is not automatically disqualifying. Decode workloads can be split and scheduled across multiple devices, and the relevant question becomes whether many smaller pools of extremely fast memory can outperform fewer large pools of slower memory for particular model-serving patterns [3]. Raptor’s bet is that serving cost and latency are increasingly controlled by bandwidth-per-watt and bandwidth-per-area, not by peak FLOPS alone [2]. If the working set fits, the extra bandwidth can matter immediately; if it does not, software partitioning, interconnect, and orchestration become just as important as the silicon [3].

That is why this announcement matters beyond a single spec. AI infrastructure buyers are no longer evaluating accelerators only by raw compute throughput. They are asking how quickly a rack can generate tokens, how much power it consumes, how it handles long context windows, and how predictable latency remains as users pile onto the service. Raptor is aimed squarely at that serving problem, not at the broader training market where GPUs still dominate through software, scale, and ecosystem depth [1].

The engineering risks are real

Stacking logic directly on DRAM creates hard thermal and reliability problems. Digital Citizen reports that Raptor places the logic die on top so cooling hardware can contact the hotter compute layer directly, while the DRAM is designed for junction temperatures up to 105°C and compensates with a faster refresh interval of about 4 ms [4]. The same report says d-Matrix claims the refresh bandwidth loss is only 1.37%, and that the design uses Reed-Solomon T=2 error correction plus CRC protection to improve reliability [4].

Those details matter because a packaging demonstration is not the same as a production product. Tom’s Hardware reports that d-Matrix did not disclose the manufacturer of the custom DRAM die, even as it named TSMC for the logic process and Alchip as an ASIC design and packaging partner [1]. The same report also cautions that the company gave no firm Hot Chips date for volume availability or pricing, and that performance examples such as token-per-second projections were still company projections built on early silicon [1].

That leaves three open questions for customers. First, can the custom DRAM be manufactured economically and in sufficient volume during a constrained memory market? Second, can yields stay acceptable when compute and memory are bonded in a tight 3D structure? Third, can d-Matrix make the software path simple enough that cloud and enterprise operators can use Raptor without rebuilding their inference stacks around one startup’s architecture?

A credible challenge, not a settled victory

Raptor should be read as an architectural swing, not a completed market upset. The 100 TB/s number is meaningful because it comes from a coherent thesis: inference has become a data-movement problem, and the shortest wire is often the most efficient one. The 0.37 pJ/bit figure strengthens that thesis, especially when set against HBM-style movement across a broader package interface [1].

But the card’s 32 GB capacity, the undisclosed DRAM manufacturing path, and the absence of public pricing or volume timing keep the story unfinished [1]. In practical deployments, d-Matrix must prove not only that Raptor can move data at spectacular speed, but that it can deliver lower cost per token, stable latency, and operational simplicity at scale. The company has shown a plausible way around the memory wall. Now it has to show that the route is manufacturable, programmable, and cheap enough to matter.

Sources from the last 72 hours

1. [1]Hot Chips 2026: d-Matrix stacks AI accelerator directly on custom DRAM for 100 TB/s per card — TSMC 4nm compute die bonded face-to-face at a 36-micron pitch on top of a custom-designed dieAug 26, 2026, 12:00 AM UTC
2. [2]d-Matrix Raptor Delivers 100 TB/s From Stacked DRAM at Fraction of HBM Energy CostAug 24, 2026, 2:47 PM UTC
3. [3]d-Matrix Raptor: 3D-DRAM with 100 TB/s for AI inferenceAug 25, 2026, 4:00 AM UTC
4. [4]d Matrix Raptor 3D DRAM Targets 100 TB/s Bandwidth at 0.37 pJ per Bit for AI InferenceAug 25, 2026, 12:00 AM UTC

AI-generated article based on recent web research, then preserved as a dated editorial snapshot.