Seoul Searching for Silicon: My ICML 2026 Co-Design Highlights
Generated by Nano Banana ProTable of Contents
- Introduction
- Theme 1 - Decompose Your Problem
- Theme 2 - Trade Generalization for Speed
- Theme 3 - When You Can't Engineer the Fix, Change the Math
- Theme 4 - Skip, Loop, Cut, and Sample Your Forward Pass
- Conclusion
Introduction
ICML 2026 kicked off the academic leg of my summer tour in Seoul, South Korea. Think incredible coffee, the suffocating monsoon heat, and Gangnam’s densely packed, ultra-modern skyscrapers. Between stocking up on K-beauty at Olive Young and navigating the Convention and Exhibition Center (COEX) crowds, I spent my week catching up on the latest ML co-design developments.
A few core patterns kept resurfacing, which I’ve broken down below. Each entry focuses strictly on its theme-specific takeaway. So, if your curiosity is piqued, I strongly encourage you to look over the original papers. Feel free to skip around and jump straight into whatever best matches your interests.
Theme 1: Decompose Your Problem
Throwing heavy compute at an entire pipeline is like burning down your kitchen to roast a chestnut. It works, but at an absurd cost. Instead, try looking for a natural split between high- and low-impact sub-problems. Matching compute directly to each piece’s difficulty ensures we never waste resources on low-leverage tasks.
Layer-Wise Context Decomposition by MiMo
The standard attention mechanism scales quadratically with sequence length (source), creating a universal bottleneck for Transformer serving. Existing solutions tackle this from several angles. Sliding Window Attention bounds context locally, Sparse Attention routes connections selectively, and Linear Attention replaces the softmax with linear kernels (paper 1, paper 2, paper 3). The flaw? They all degrade model accuracy in the process.
MiMo bypasses this trade-off with a key insight: routine tokens (like “.” or “and”) only influence local context. By interleaving local and global attention layers, pre-training teaches the network to process filler tokens locally and to reserve global context for high-impact tokens. This reduces KV-cache memory consumption by ~7× and maintains full accuracy on long-context retrieval benchmarks.
Audio Spectrum Decomposition by Darius Petermann et. al
Note. While this work was originally published in 2023, it was featured during Minje Kim’s ICML 2026 Learning to Listen Workshop keynote.
Sampling audio at tens of kilohertz produces high-density time-frequency representations. This expands even brief audio clips into massive 4D tensors across time, frequency, channel, and batch dimensions. Standard models process these tensors in a single, uniform forward pass, making audio inference unusually expensive relative to other ML applications.
The authors leverage a biological reality to bypass all this compute: the human ear requires finer resolution at lower frequencies (source). They apply higher-capacity neural sub-band models only to these low frequencies and run lightweight processing across all other frequencies. This improves perceptual reconstruction quality and bitrate efficiency relative to standard architectures.
Token-Sensitivity Decomposition by Janghwan Lee et al.
Large Reasoning Models rely on long chains-of-thought, where a single flipped digit or operator early in the trace ruins the entire output. Standard 4-bit compression treats every token equally. This forces critical, low-entropy tokens to absorb the exact same degradation as their more noise-tolerant, high-entropy counterparts.
To fix this, the authors introduce Selective Entropy Minimization (SEM) as part of quantization-aware training (QAT). Rather than using hard binary masks, SEM employs a soft weighting function parameterized by an entropy threshold $\tau$ to focus the auxiliary loss on critical low-entropy steps without overly penalizing borderline tokens. This keeps high-confidence tokens sharp in the model’s internal probability distribution and pushes quantization noise into non-critical filler tokens. The result? Under the exact same training budget, the complete 4-bit approach surpasses the baseline by roughly 16% and slightly outperforms full fine-tuning. It also delivers a 3.1x-3.9x speedup on NVIDIA hardware.
Theme 2: Trade Generalization for Speed
Every university student knows that you can only ever pick two out of three: good grades, a social life, or adequate sleep. Machine learning has its own “pick two” trilemma: a model can be universally capable, cheap to serve, or state-of-the-art. Unless your business model hinges on serving massive, generally capable models (think OpenAI), generalizability is the first thing to go.
Hardcode Your System Prompts into Distilled Models by Shopify
Shopify uses an agent to translate natural language into GraphQL API calls so that merchants can query their store data. Because a single syntax error invalidates the entire query, this task demands frontier-level accuracy. At 2,000 requests per minute, running a frontier model becomes prohibitively expensive.
Given the narrow scope of the task, Shopify took a two-pronged specialization approach: (1) distill offline frontier behavior into a much smaller student model, and (2) replace their editable system prompt with a hardcoded, compressed representation. First, an offline critic committee of frontier models converted live query failures into corrected SFT trajectories and GRPO rewards for knowledge distillation. Next, they compressed the static 6,000-token schema down to 1,500 learned Gist tokens. Together, these changes surpassed frontier-model accuracy while decreasing end-to-end latency by 38% and annualized cost by 96%.
Quantize for a Single Task by Amit LeVi et al.
Industry-standard post-training quantization (PTQ) frameworks like GPTQ and AWQ compress LLMs under a sweeping assumption that every transformer layer tolerates precision loss equally. Recent layer-analysis research shows that transformer blocks actually contribute unequally to task execution (source). Because most production endpoints are dominated by a single task (source), uniform compression wastes precious precision on low-impact layers while overcompressing task-critical ones.
Task-Aware Quantization (TAQ) fixes this mismatch by allocating precision based on layer sensitivity to task-specific calibration data. It ranks layer importance by tracking internal information capacity (spectral entropy), task-accuracy drop under per-layer quantization (oracle sensitivity), or by stress-testing how much output predictions shift (KL divergence) under layer noise. It then promotes the top 25% most sensitive blocks to 8-bit precision. The remaining 75% blocks are left at a 4-bit encoding. Compared to their original FP16 baseline, TAQ-quantized models have 35% higher throughput and run 26% faster while retaining near-baseline target-task accuracy. TAQ does this at an average of 5.0 bits per weight (bpw), saving up to 2.0 bpw over standard mixed-precision PTQ methods.
Theme 3: When You Can’t Engineer the Fix, Change the Math
Some everyday problems are easy to engineer your way out of, like using five seconds of hot water and thermal expansion to open a stubborn jar. Other times, “engineering harder” means fighting physics. Leave the fridge open to cool the kitchen, and all you’re doing is feeding the heat sink. Piling software hacks on top of ML problems works the exact same way — except you can actually change the underlying math.
FlashSketch: Sketch-Kernel Co-Design by Rajat Vadiraj Dwaraknath et al.
If you want to trace what training data caused a hallucination or do any other model diagnostic work, you need to compute per-example gradients instead of batch averages. This inflates tensor size far beyond GPU memory limits. Researchers fix this using a linear algebra technique called matrix sketching, which applies random projections to squash a giant tensor into a compact summary matrix.
Aggressive compression maps multiple computations into the same output cell. The result? GPU write collisions. Instead of fighting those collisions with slow global memory locks, FlashSketch redesigns the matrix pattern into neat, predictable blocks. This allows GPU threads to pool their work locally in ultra-fast Shared Memory (SRAM) and then cleanly write their final results back in parallel. This math-first refactoring eliminates global memory write contention and runs up to 3.2x faster than SOTA baselines on data attribution tasks.
Approximate Activation Functions on Edge Devices by Anton Lydike et al.
Non-linear activations gate which features pass between neural network layers, making it possible to model complex representations (source). Modern architectures favor smooth activations like GELU and Swish. The catch? Both depend heavily on exponential math $(e^x)$. Edge microcontrollers lack the specialized math units for exponentiation, which forces the CPU to compute $e^x$ in slow software loops.
Lydike et al. bypass this hardware wall by replacing $e^x$ with a 30-year-old IEEE linear approximation. An IEEE 754 float stores its exponent in its upper bits, meaning its integer bit pattern scales with $\log_2(\text{float})$. Because taking the log of $e^x$ cancels the exponential, computing $y = e^x$ reduces to a direct linear transform:
$$ I_y = \lfloor a \cdot x + b \rfloor $$ $$ y = \text{as_float}(I_y) $$
Where:
- $a = \frac{2^{23}}{\ln(2)} \approx 12102203$ scales $x$ into bit slot 23.
- $b = 127 \cdot 2^{23} = 1065353216$ is the raw integer bit pattern for float $1.0$ (exponent 127 shifted to bit 23), ensuring $x = 0$ evaluates to $e^0 = 1$.
This yields a 15×–22× speedup on edge hardware, as long as input activations remain bounded within the linear fit window.
Theme 4: Skip, Loop, Cut, and Sample Your Forward Pass
A good DJ never plays an eight-minute track from start to finish. They skip weak verses, loop the best hooks, cut the track early when the crowd peaks, and sample long tracks down to a single beat. Standard transformer execution treats your model like a rigid, immutable pipeline. Extreme runtime optimization means thinking like a DJ: skip your redundant layers, loop complex reasoning, exit early, and sample your bloated prompts.
Know When To Stop by Jiawei Gu et al.
In standard Transformer inference, a trivial greeting burns as many FLOPs as a multi-step math proof. That’s because the model pushes every prompt through the exact same static forward pass. Yet representations usually stabilize early. Every step after that point just burns FLOPs. Researchers have targeted this waste for years with early exits. But conventional early exits rely on extra intermediate classification heads, and those additions inflate memory usage and complicate training.
Gu et al. took a simpler, training-free approach: they track representations layer-by-layer and exit the moment they stabilize. The trigger is simple. When update magnitude and directional alignment converge, execution stops. This bypasses all remaining Transformer blocks and sends the hidden state straight to the final LM head. Across question answering and commonsense reasoning benchmarks, this reduces FLOPs by 30% to 35% without any quality loss.
Skip a Layer or Loop It by Ziyue Li et al.
Gu et al. make early exits simple, but early stopping is only half the battle. Truncating final layers saves compute but keeps a forward pass linear. That means standard pipelines still cannot skip redundant intermediate blocks or re-execute reasoning-critical layers.
Program of Layers (PoLar) treats frozen Transformer layers as a flexible function library. Offline, Monte Carlo Tree Search (MCTS) evaluates layer combinations against an accuracy-versus-FLOP reward to map execution paths by prompt difficulty. A lightweight neural predictor then learns these routes for live inference. The result? An up to 22-point increase in accuracy on math reasoning benchmarks with less compute and no model weight changes.
Compress Your System Prompts by Jesse Mu et al.
Every other paper in this post highlights ICML 2026 research, but Shopify’s work drew my attention back to this 2023 paper. Before this work, system prompts occupied memory token-for-token unless teams used lossy text truncation. Jesse Mu et al. changed that paradigm when they proved a model can compress full prompt context into a small set of virtual Gist tokens.
The architecture rests on a simple sequence change. Mu et al. append virtual Gist tokens to the system prompt so the input reads [Prompt] [Gist] [Target]. Next, they modify the causal mask to block target tokens from attending to prompt tokens. Hiding those prompt columns forces the transformer to compress system prompt semantics into Gist vectors during prefill. Once prefill finishes, the engine discards the prompt KV cache entirely. That single optimization reduces prefill memory overhead by up to 26× while preserving downstream accuracy.
Of course, three years is a lifetime in the ML space. Frameworks like DSPy and DeepMind’s OPRO have replaced manual prompt engineering with automated prompt optimization. They also happen to produce massive system contexts. Are semantic compression techniques like Gist tokens at odds with this reality, or secretly complementary to it? If you’re curious to get the gist of these two ideas, please reach out.
Conclusion
All these themes point to the same reality: uniform inference is dead. Treating every token, layer, and operation identically was a convenient crutch for early LLM scaling, but real efficiency demands asymmetric execution. Whether through custom kernels or dynamic forward passes, the future belongs to hardware and software co-designed to spend compute only where it actually pays off.
After ICML, I can’t stop auditing my own assumptions across software, hardware, and modeling. Wherever I go, I’m still soul-searching for the next silicon breakthrough. What hidden defaults are slowing us down? I’ll be sharing my epiphanies as they land. In the meantime, stay curious.
