Tesla Dojo

Each node (computing core) of the D1 processing chip is a general purpose 64-bit CPU with a superscalar core. It supports internal instruction-level parallelism, and includes simultaneous multithreading (SMT). It does not support virtual memory and uses limited memory protection mechanisms. Dojo software/applications manage chip resources.

Microarchitecture of a node on the D1 chip

The D1 instruction set supports both 64-bit scalar and 64-bit SIMD vector instructions. The integer unit mixes RISC-V and custom instructions, supporting 8, 16, 32, or 64 bit integers. The custom vector math unit is optimized for machine learning kernels and supports multiple data formats, with a mix of precisions and numerical ranges, many of which are compiler composable.[3] Up to 16 vector formats can be used simultaneously.[3]

Each D1 node uses a 32-bit fetch window that holds eight instructions. An eight-wide decoder supports two threads per cycle. This front end feeds a four-wide, four-way SMT scalar scheduler that has two integer units, two address units, and a register file/thread. A two-side vector scheduler has four-way SMT, which feeds a 64-bit wide SIMD unit or four 8×8×4 matrix multiplication units.[3]

The network on-chip (NOC) router links cores into a 2D mesh. It can send one packet in and one packet out in all four directions to/from each neighbor node/clock cycle along with one 64-bit read and one 64-bit write to local SRAM per cycle.[3]

Primitives transfer data across memories and CPUs, and semaphore and barrier constraints. System-wide DDR4 memory works like bulk storage.

Each core has a 1.25 MB SRAM main memory. Load and store speeds reach 400 GB/sec and 270 GB/sec, respectively. The chip has explicit core-to-core data transfer instructions. Each SRAM has a unique list parser that feeds a pair of decoders and a gather engine that feeds the vector register file, which together can directly transfer information across nodes.[3]

In 2021, it was announced the D1 uses a "7-nanometer manufacturing process, with 362 teraflops of processing power".[4] Twelve nodes are grouped into a local block. Nodes (cores) are arranged into an 18×20 array on a single die, of which 354 cores are available for applications.[3] The die runs at 2 GHz and totals 440 MB of SRAM (360 cores × 1.25 MB/core).[3] It reaches 376 teraflops at BF16 or CFloat8, a Tesla proposal for a configurable 8-bit floating point format,[7] and 22 teraflops at FP32.

Each die has 576 bi-directional SerDes channels along the perimeter to link to other dies, and moves 8 TB/sec across all four die edges. The die covers 645 mm² (1.000 sq in).[3] Each D1 chip has a thermal design power of approximately 400 watts.[8]

The water-cooled Training Tile packages 25 D1 chips into a 5×5 array.[3] Each Tile supports 36 TB/sec of aggregate bandwidth via 40 I/O chips - half the bandwidth of the chip mesh. Each Tile supports 10 TB/sec of on-tile bandwidth. Each Tile has 11 GB of SRAM memory (25 D1 chips × 360 cores/D1 × 1.25 MB/core). Each Tile achieves 9 petaflops at BF16/CFloat8 precision (25 D1 chips × 376 TFLOP/D1). Each Tile consumes 15 kilowatts;[3] 288 amperes at 52 volts.[8]

Six Tiles are aggregated into a System Tray, which is integrated with a host interface. Each host interface includes 512 x86 cores, providing a Linux-based user environment.[9] Previously, the Dojo System Tray was known as the Training Matrix, which includes six Training Tiles, 20 Dojo Interface Processor cards across four host servers, and Ethernet-linked adjunct servers. It has 53,100 D1 cores, rated at 1 exaflops at BF16 and CFloat8 formats. It has 1.3 TB of on-tile SRAM memory and 13 TB of dual in-line high bandwidth memory (HBM).

Dojo Interface Processor cards (DIP) sit on the edges of the Tile arrays and are hooked into the mesh. Host systems power the DIPs and perform various system management functions. A DIP memory and I/O co-processor hold 32 GB of shared HBM (either HBM2e or HBM3) – as well as Ethernet interfaces that sidestep the mesh. Each DIP card has 2 I/O processors with 4 memory banks totaling 32 GB with 800 GB/sec of bandwidth.

The DIP plugs into a PCI-Express 4.0 x16 slot that offers 32 GB/sec of bandwidth per card. Five cards per tile edge offer 160 GB/sec of bandwidth to the host servers and 4.5 TB/sec to the tile.

Tesla Transport Protocol (TTP) is a proprietary interconnect over PCI-Express. A 50 GB/sec TTP protocol link runs over Ethernet to access either a single 400 Gb/sec port or a paired set of 200 Gb/sec ports. Crossing the entire 2D mesh might take 30 hops, while TTP over Ethernet takes only four hops (at lower bandwidth), reducing vertical latency.

Dojo stacks tiles vertically in a cabinet to minimize the distance and communications time between them. The Dojo ExaPod system includes 120 tiles, totaling 1,062,000 usable cores, reaching 20 exaflops.