Together AI has introduced Together Compiler (TKCC), an agent-driven system that keeps re-optimizing inference as production traffic changes. Together Compiler already runs on several of Together’s serverless endpoints. The core claim for developers: an engine tuned once goes stale, so tuning should never stop.
TL;DR
- System: It is an optimization system, tested on Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash and Llama 1B.
- Runs on: NVIDIA B200, B300, GB300, H100, AMD MI355X and AWS Trainium2. Each run used at most 2 nodes per model.
- Performance: Up to 2.6x higher TPS than the best published InferenceX configurations, in under 24 hours per setting.
- Best: 1,111 P90 TPS per user on MiniMax M3.1 Flash (B200, batch size 1), 1.8x over the 633 TPS baseline.
- Worst: GLM 5.3 on MI355X moved only from 463 to 480 TPS (about 4%).
What is Together Compiler?
Together Compiler is a continual compilation loop from Together AI that uses coding agents to rewrite a model’s forward pass. It takes a server launch command, a traffic profile and a hardware configuration. Agents propose faster forward passes, and a persistent engine measures each one. The winning code ships into the serving engine. When traffic drifts, the loop runs again.
Why does inference need continual compilation?
Together AI team reports that its traffic shape never sits still. Prompt lengths in one country ran about 97K tokens versus 70K elsewhere. One customer’s P90 input length grew from 18K to 33K tokens in 3 days.
Kernels tuned on random data can also mislead. Random inputs activated 13 to 38% more MoE experts than real traffic. One agent-written top-k kernel took 20 µs on random data. On real data at 70K context, it took up to 62 ms.
How does the persistent engine work?
Most agent loops restart vLLM, SGLang or TensorRT-LLM for every experiment. The research team measures 761 seconds of cold startup for MiniMax M3.1 Flash on TensorRT-LLM. Together Compiler starts the engine once. Agents submit a candidate forward pass over RPC. The server swaps the forward function pointer and re-captures CUDA graphs.
Two measurement tiers balance speed and fidelity. A fast tier serves 12 layers for quick screening. A slow tier serves 60 layers to confirm gains on real distributions. Two tiers tested 8.1x more candidates than one slow tier (1,354 vs 168). They also found a solution 7.2% faster. Together AI team reports 252x higher agent experiment throughput than an engine-restart baseline.
How does it choose a parallelism layout?
A cost model ranks tensor, expert, data and context parallel layouts before agents start. It mixes spec-sheet numbers, microbenchmarks and measured forward passes. It also uses measured MoE expert imbalance and uneven request lengths. On Kimi K3, it found a production layout with up to 24% higher TPS.
What did agents need from human experts?
An expert-curated knowledge base produced 25 to 31% lower speedup after 12 hours. Naive agents told to make the forward pass “as fast as possible” beat expert strategies. The ensemble scores were 6.2x versus 4.9x after 10 trials. Custom logging structures ate up to 43% of tool calls. Together AI now gives agents raw engine source code, not an intermediate representation.
Benchmarks: how fast is Together Compiler?
On SemiAnalysis InferenceX‘s AgentX workload, Together Compiler beat the best open configurations by up to 2.6x TPS. Each setting ran under 24 hours on at most 2 nodes.
Batch size 1, P90 TPS per user (Together Compiler vs InferenceX best, self-reported):
- MiniMax M3.1 Flash (B200): 1,111 vs 633
- DeepSeek v4.1 Flash (GB300): 1,205 vs 943
- GLM 5.3 (B200): 417 vs 329
- Kimi K3 (GB300): 450 vs 423
- GLM 5.3 (MI355X): 480 vs 463
For MiniMax, the cost model picked TP8 over the baseline’s TP4. That halves each GPU’s weight traffic in memory-bound decode.
On Llama 1B at batch size 1, it beat Hazy Research megakernels, Mirage and CUDA-graph baselines by up to 1.7x. Beyond LLMs, it sped up 8 scientific models by 1.5x to 6.4x with bit-exact outputs. Claude Code with /goal reached 1.2x to 4.4x and failed on AIFS-ENS 2.0.