HOPE: prune experts in MoE models without breaking cooperation

MoE models achieve state-of-the-art performance with hundreds of billions of parameters, but each expert occupies memory. Pruning experts is the most direct way to shrink a model, but existing methods decide the cut for each expert independently, as if the contribution of each one were additive. In practice, experts cooperate: removing one can dismantle a circuit that depended on it.

In this article you will see:

• What HOPE is and why cooperation between experts matters
• How the method identifies the set to prune
• The results disclosed in the paper, including +6.1% on code tasks
• The honest limitations and how to test today

What is HOPE, Amazon's second-order pruning method

HOPE stands for Higher-Order Pruning of Experts. It is a second-order pruning objective that considers pairwise interactions between experts when deciding which to remove. The central idea is that experts do not work alone: co-selection patterns appear early in training and stabilize over it.

A first-order method scores each expert on its own. If expert A seems dispensable, it is removed. The problem appears when expert B, which cooperates frequently with A, is important. Removing A dismantles the cooperative group, and the performance drop goes beyond what the individual score predicted. This effect accumulates at aggressive pruning rates, from 40% to 50%, exactly where memory reduction matters most for production.

The paper shows that REAP, a first-order method considered state of the art, is a special case of HOPE in which the interaction terms are zeroed. In other words, HOPE generalizes REAP by adding the interaction matrix between experts.

How HOPE identifies the set to prune

HOPE formulates pruning as the minimization of an upper bound on the error resulting from removal. The interaction matrix F measures pairwise cooperation: F i,j is the average of g i(x) g j(x) on inputs where both experts are selected, normalized by the norms of their outputs. The larger F i,j, the more the two experts work together.

Instead of ranking experts by an individual score, HOPE searches for the subset whose removal minimizes p transpose F p, the output error that the cut actually causes. This captures the fact that two redundant experts are cheap to remove together, while two complementary ones are not. A ranking by expert cannot distinguish the two cases.

In practice, the method prunes each layer independently and does not consider interactions between layers. This keeps the computational cost viable, but it is a limitation stated in the paper.

The reported results: +6.1% on code with 50% pruning

The authors evaluated HOPE on large scale MoE models, including GLM-4.7, Qwen3-Coder-480B and other variants. The main metric is the average rank across the compared methods, with lower being better.

With 50% of experts removed, HOPE achieved the best average rank among all methods tested. On agentic coding tasks, the gain reaches +6.1% over the second best pruning method. The advantage is more pronounced at high rates, from 40% to 50%. At low rates, from 10% to 20%, first order methods like REAP and MAN are sometimes competitive and HOPE's advantage shrinks.

The explanation is straightforward: the more experts are removed, the more interactions are broken, and the off diagonal terms of F gain weight. At 50%, many cooperative pairs are broken at the same time, and individual scoring cannot predict or prevent that damage.

Gains also appear on tool use and multiple choice tasks, with the advantage increasing as the pruning rate grows.

Honest limitations of HOPE

The paper is clear about what the method does not do. First, HOPE prunes each layer independently and does not consider interactions between layers. Second, the F matrix is estimated from calibration data, so quality depends on the representativeness of that data. Third, the advantage over first order methods is small at low rates, where the simplicity of REAP may be enough.

It is also worth remembering that the paper is from Amazon and the results are self reported, with no independent replication so far. Anyone who wants to apply HOPE should validate on their own set of models and tasks.

How to test HOPE today

HOPE code is available in Amazon's official repository on GitHub, linked to the paper. The implementation works on MoE models hosted on Hugging Face, such as Qwen3-Coder and GLM, and allows reproducing the experiments with the provided calibration data.

For those who want to evaluate expert pruning on their own model, the flow is: choose the MoE model, collect representative calibration data, compute the F matrix per layer and run HOPE with the desired pruning rate. Compare with REAP on the same evaluation set before deciding.

The pruned checkpoints published by Amazon on Hugging Face allow testing the result without running pruning from scratch.