gpu::linalg::kernels
cheatah-gpu-linalg v0.4.4-alpha — Biome Standard 0.6.3-alpha
Functions
CPU GEMM stand-in mirroring linalg.slang's register-tiled gemm result (the tiling is a device optimization; the product is the same): row-major C[M×N] = A[M×K]·B[K×N], dims = {M, N, K}.
The kernel's workgroup grid always covers all of C, so the stand-in iterates the dims directly (the DispatchShape is block-granular for this kernel).
b | The bound buffer pointers, in the kernel's binding order (dims last unless noted). |
n | The binding count (the stand-in is a no-op when fewer than expected). |
O(M·N·K).
none — writes in place through the bound buffers.
none.
gpu:matmulCPU stand-in for gemm_batched: C[z] = A[z]·B[z] per batch layer z of the 3-D grid, dims = {M, N, K, B}.
b | The bound buffer pointers, in the kernel's binding order (dims last unless noted). |
n | The binding count (the stand-in is a no-op when fewer than expected). |
shape | The dispatch thread grid (its height/width/depth bound the loops). |
O(B·M·N·K).
none — writes in place through the bound buffers.
none.
gpu:matmul_batchedCPU stand-in for outer: out[n×m] = a_i·b_j, dims = {n, m}.
b | The bound buffer pointers, in the kernel's binding order (dims last unless noted). |
n | The binding count (the stand-in is a no-op when fewer than expected). |
shape | The dispatch thread grid (its height/width/depth bound the loops). |
O(n·m).
none — writes in place through the bound buffers.
none.
gpu:outerCPU stand-in for transpose: out[c×r] = in[r×c]ᵀ, dims = {r, c} — CONJUGATED for a complex element (the Hermitian adjoint), exactly like the device kernel's CONJ seam.
b | The bound buffer pointers, in the kernel's binding order (dims last unless noted). |
n | The binding count (the stand-in is a no-op when fewer than expected). |
shape | The dispatch thread grid (its height/width/depth bound the loops). |
O(r·c).
none — writes in place through the bound buffers.
none.
gpu:transposeCPU stand-in for the conjugating vdot_partial (complex elements): partial[g] = Σ conj(a[k])·b[k].
b | The bound buffer pointers, in the kernel's binding order (dims last unless noted). |
nb | The binding count (the stand-in is a no-op when fewer than expected). |
width | The 1-D dispatch width (how many partial threads are emulated). |
O(n).
none — writes in place through the bound buffers.
none.
gpu:complexCPU stand-in for kron: K[(ar·br)×(ac·bc)] = A⊗B, dims = {ar, ac, br, bc}.
b | The bound buffer pointers, in the kernel's binding order (dims last unless noted). |
n | The binding count (the stand-in is a no-op when fewer than expected). |
shape | The dispatch thread grid (its height/width/depth bound the loops). |
O(n⁴) in the output area.
none — writes in place through the bound buffers.
none.
gpu:kronCPU stand-in for dot_partial: thread g writes partial[g] = Σ a[k]·b[k], k = g, g+P, …; dims = {n, P}.
Bit-identical to the device kernel — the same strided association order.
b | The bound buffer pointers, in the kernel's binding order (dims last unless noted). |
nb | The binding count (the stand-in is a no-op when fewer than expected). |
width | The 1-D dispatch width (how many partial threads are emulated). |
O(n).
none — writes in place through the bound buffers.
none.
gpu:dotCPU stand-in for trace_partial: partial[g] = Σ a[k·(c+1)], k = g, g+P, …; dims = {m, c, P}.
b | The bound buffer pointers, in the kernel's binding order (dims last unless noted). |
nb | The binding count (the stand-in is a no-op when fewer than expected). |
width | The 1-D dispatch width (how many partial threads are emulated). |
O(m).
none — writes in place through the bound buffers.
none.
gpu:traceCPU stand-in for the ew_* binary family: out[i] = a[i] OP b[i], dims = {n}; the operator is a template parameter mirroring the per-operator kernel compilation.
b | The bound buffer pointers, in the kernel's binding order (dims last unless noted). |
nb | The binding count (the stand-in is a no-op when fewer than expected). |
O(n).
none — writes in place through the bound buffers.
none.
gpu:elementwiseCPU stand-ins for the stream kernels: fill (out[i] = s), copy (bandwidth probe), triad (a = b + s*c, bandwidth probe).
dims = {n} in the last binding, like their GPU forms.
b | The bound buffer pointers, in the kernel's binding order (dims last unless noted). |
nb | The binding count (the stand-in is a no-op when fewer than expected). |
O(n).
none — writes in place through the bound buffers.
none.
gpu:elementwiseCPU stand-in for copy (out[i] = in[i], the bandwidth probe); dims = {n} last.
b | The bound buffer pointers {in, out, dims}. |
nb | The binding count (no-op when fewer than expected). |
O(n).
none — writes in place through the bound buffers.
none.
CPU stand-in for triad (a = x + s·y, the bandwidth probe); dims = {n} last.
b | The bound buffer pointers {a, x, y, s, dims}. |
nb | The binding count (no-op when fewer than expected). |
O(n).
none — writes in place through the bound buffers.
none.
CPU stand-in for the ews_* array⊗scalar family: dims = {n, swap}; swap = 1 computes s OP a.
b | The bound buffer pointers, in the kernel's binding order (dims last unless noted). |
nb | The binding count (the stand-in is a no-op when fewer than expected). |
O(n).
none — writes in place through the bound buffers.
none.
gpu:operatorsCPU stand-in for the ew_* unary family: out[i] = F(a[i]), dims = {n}.
b | The bound buffer pointers, in the kernel's binding order (dims last unless noted). |
nb | The binding count (the stand-in is a no-op when fewer than expected). |
O(n).
none — writes in place through the bound buffers.
none.
gpu:elementwiseCPU stand-in for sum_partial: partial[g] = Σ a[k], k = g, g+P, …; dims = {n, P} — the same strided association order as the device kernel (bit-identical).
b | The bound buffer pointers, in the kernel's binding order (dims last unless noted). |
nb | The binding count (the stand-in is a no-op when fewer than expected). |
width | The 1-D dispatch width (how many partial threads are emulated). |
O(n).
none — writes in place through the bound buffers.
none.
gpu:elementwiseCPU stand-in for the TWO-STAGE dot_partial2/vdot_partial2: reproduces the kernel's exact association order — per-thread strided accumulation, then the fixed groupshared tree.
b | The bound buffer pointers, in the kernel's binding order (dims last unless noted). |
nb | The binding count (the stand-in is a no-op when fewer than expected). |
O(n) accumulation + O(G·256) tree folds.
none — writes in place through the bound buffers.
none.
gpu:dotCPU stand-in for the TWO-STAGE sum_partial2 (same contract over one input).
b | The bound buffer pointers, in the kernel's binding order (dims last unless noted). |
nb | The binding count (the stand-in is a no-op when fewer than expected). |
O(n) accumulation + O(G·256) tree folds.
none — writes in place through the bound buffers.
none.
gpu:elementwiseCPU stand-in for finalize_partials: the same single-group strided + tree fold into out[0].
b | The bound buffer pointers, in the kernel's binding order (dims last unless noted). |
nb | The binding count (the stand-in is a no-op when fewer than expected). |
O(G) + a fixed 256-wide tree.
none — writes in place through the bound buffers.
none.
gpu:dotCPU stand-in for the fused axpy: out[i] = α·x[i] + y[i], dims = {n}.
b | The bound buffer pointers, in the kernel's binding order (dims last unless noted). |
nb | The binding count (the stand-in is a no-op when fewer than expected). |
O(n).
none — writes in place through the bound buffers.
none.
gpu:elementwiseThe conv epilogues' scalar activation (matches the Slang cba_act table exactly): 0 = identity, 1 = relu, 2 = tanh, 3 = sigmoid.
z | The pre-activation value. |
act | The activation code (0–3; validated by the dispatching routine). |
f(z) for the selected activation.
O(1).
none.
none.
gpu:convThe conv epilogues' scalar derivative FROM THE OUTPUT a = f(z) (matches cag_dact): identity' = 1, relu' = [a > 0], tanh' = 1 − a², sigmoid' = a(1 − a).
a | The activation OUTPUT f(z) (not the pre-activation). |
act | The activation code (0–3; validated by the dispatching routine). |
f′(z) expressed in terms of a.
O(1).
none.
none.
gpu:convCPU stand-in for im2col2d: col[((c·KH+kh)·KW+kw)·B·OO + b·OO + oh·OW + ow] = x in-bounds, 0 for padding (FULL overwrite); dims = {B, C, H, W, KH, KW, OH, OW, stride, pad}.
The same per-element gather as the device kernel, iterated over the col linear index.
b | The bound buffer pointers {x, col, dims}. |
nb | The binding count (the stand-in is a no-op when fewer than expected). |
O(B·C·KH·KW·OH·OW).
none — writes in place through the bound buffers.
none.
gpu:convCPU stand-in for col2im2d: dx[i] = Σ over its ≤ KH·KW contributing dcol cells, kh-major — the gather-form adjoint, bit-identical to the reference scatter-add's association order.
Same dims as im2col2d.
b | The bound buffer pointers {dcol, dx, dims}. |
nb | The binding count (the stand-in is a no-op when fewer than expected). |
O(B·C·H·W·KH·KW) bound checks (≤ KH·KW adds per input cell).
none — writes in place through the bound buffers.
none.
gpu:convCPU stand-in for conv_bias_act: a[(b·F+f)·OO + o] = act(yc[f·B·OO + b·OO + o] + bias[f]); dims = {B, F, OO, act}.
b | The bound buffer pointers {yc, bias, a, dims}. |
nb | The binding count (the stand-in is a no-op when fewer than expected). |
O(B·F·OO).
none — writes in place through the bound buffers.
none.
gpu:convCPU stand-in for conv_act_grad: dyc[f·B·OO + b·OO + o] = d[(b·F+f)·OO + o] · act'(a[(b·F+f)·OO + o]) with the derivative FROM THE OUTPUT; dims = {B, F, OO, act}.
b | The bound buffer pointers {d, a, dyc, dims}. |
nb | The binding count (the stand-in is a no-op when fewer than expected). |
O(B·F·OO).
none — writes in place through the bound buffers.
none.
gpu:convRegister every emulated stand-in with the software Metal device, both element types (the device kernels lack a Metal f64 — Metal has no double — but the emulator provides it, which is how cheatah's default double element is exercised on this backend).
Idempotent: registering a name again just overwrites it with the same pointer.
Constants & variables
Threads per workgroup of every 1-D kernel — matches [numthreads(256,1,1)] in linalg.slang.
Threads per workgroup axis of every 2-D kernel — matches [numthreads(16,16,1)].
The partial-sum width cap: a reduction dispatches P = min(n, kMaxReduce) threads, each writing one deterministic partial the routine sums on the host.
The register-tiled GEMM's C-block edge: one 16x16-thread workgroup computes a 64x64 tile (4x4 register microtile per thread) — matches gemm in linalg.slang.
The fast GEMM path's C-block edge (one 16×16-thread workgroup per 128×128 tile, 8×8 microtile per thread) and its K-slab width.
The fast GEMM path's K-slab width (the double-buffered groupshared depth).
The kernel name a routine dispatches for element type T.
Specialized per supported element; the primary is left null so an unsupported element fails loudly at the GpuElement constraint, never as a null-named dispatch. GEMM has TWO paths: gemm_fast (128×128 blocks, double-buffered, vec4 loads, ZERO bounds checks — real elements with M%128==0 && N%128==0 && K%8==0 only) and gemm_edge (64×64, guarded, any shape, all elements). The routine gates on shape.
The f32 entry.
The f64 entry.
The 64-row fast sibling (f32 only): M%64, N%128, K%16 — training-shaped batch-64 rectangles.
The f32 entry.
The guarded 64x64 edge-GEMM kernel name (any shape, every element type).
The f32 entry.
The f64 entry.
The c64 entry.
The c128 entry.
Batched GEMM kernel name per element type (grid z = batch; dims = {M, N, K, B}).
The f32 entry.
The f64 entry.
The c64 entry.
The c128 entry.
Outer product kernel name per element type (see linalg.slang outer).
The f32 entry.
The f64 entry.
The c64 entry.
The c128 entry.
Transpose kernel name per element type (the real-element conj_transpose).
The f32 entry.
The f64 entry.
The c64 entry.
The c128 entry.
Kronecker product kernel name per element type.
The f32 entry.
The f64 entry.
The c64 entry.
The c128 entry.
Dot-product partial-sum kernel name per element type (serves dot, vdot and inner — identical for the real elements this library supports).
The f32 entry.
The f64 entry.
The c64 entry.
The c128 entry.
Conjugating dot (vdot) kernel name — COMPLEX elements only (real vdot is bilinear and shares the dot kernel).
The c64 entry.
The c128 entry.
Trace partial-sum kernel name per element type.
The f32 entry.
The f64 entry.
The c64 entry.
The c128 entry.
Two-stage dot partial kernel names (G workgroups x 256, groupshared tree-reduce — the large-n path; the routine picks stage-1 vs stage-2 by size).
The f32 entry.
The f64 entry.
The c64 entry.
The c128 entry.
Two-stage conjugating vdot partial kernel name — complex elements only.
The c64 entry.
The c128 entry.
Two-stage sum partial kernel name (the large-n reduction path).
The f32 entry.
The f64 entry.
On-device partial finalizer (one 256-thread group tree-sums G partials into out[0]).
The f32 entry.
The f64 entry.
The c64 entry.
The c128 entry.
The two-stage reduction geometry: n above this uses the G-group tree kernels; G is capped.
The cap on tree-reducing workgroups G in the two-stage reduction geometry.
Sum partial-sum kernel name per element type (the deterministic reduction contract).
The f32 entry.
The f64 entry.
Fused axpy (out = α·x + y) kernel name per element type.
The f32 entry.
The f64 entry.
Batched 2-D im2col gather kernel name per element type (real elements — the conv-support tier; dims = {B, C, H, W, KH, KW, OH, OW, stride, pad}).
The f32 entry.
The f64 entry.
Batched 2-D col2im kernel name per element type (the gather-form adjoint; same dims).
The f32 entry.
The f64 entry.
Fused conv forward epilogue (bias + activation + layout transpose) kernel name per element type; dims = {B, F, OO, act} with act 0 = identity, 1 = relu, 2 = tanh, 3 = sigmoid.
The f32 entry.
The f64 entry.
Fused conv backward epilogue (chain rule from the output + layout transpose) kernel name per element type; same dims as conv_bias_act.
The f32 entry.
The f64 entry.
The array⊗array binary kernel-name table for element T, indexed by EwOp.
The f32 array⊗array table (EwOp order).
The f64 array⊗array table (EwOp order).
The f32 entry.
The f64 entry.
The f32 array⊗scalar table (EwOp order).
The f64 array⊗scalar table (EwOp order).
The array⊗scalar binary kernel-name table for element T, indexed by EwOp.
The f32 entry.
The f64 entry.
Constant-fill (out[i] = s) — the factories' device-side zeros/ones/full (no host staging vector, no PCIe upload).
Real elements only, mirroring the ews family.
The f32 entry.
The f64 entry.
The f32 unary table (EwFn order).
The f64 unary table (EwFn order; exp/log dispatch is host-evaluated — see elementwise.hpp).
The unary kernel-name table for element T, indexed by EwFn.
The f32 entry.
The f64 entry.
is_cplx_v<T>: local complex detection for the stand-ins (mirrors the kernels' CONJ seam).
The complex case.
Types
Elementwise binary kernel names per element type, indexed by the shared operator enum: the array⊗array family (ew_*) and the array⊗scalar family (ews_*, with the swap flag for the reversed s−a / s÷a forms).
One table per family keeps every dispatch a straight lookup.
Elementwise unary kernel names per element type, indexed by the shared enum.