要点
- On 9 November 2026, Amazon researcher Alex M. Tseng published a Hugging Face post on HOPE, Higher-Order Pruning of Experts, pointing to the paper at arXiv:2609.18916 and to code at github.com/awslabs/HOPE.
- First-order methods assign each expert a single importance score and cut the weakest. HOPE records pairwise cooperation in a matrix called F and chooses who to delete by solving a quadratic program, a standard numerical search for the lowest-cost set of deletions. The diagonal of F recovers REAP's rankings with a Spearman rho of 0.988, meaning the two rankings put experts in almost the same order.
- On Qwen3.5-35B-A3B with 50% of experts removed, HOPE scored 42.3 on SWE-Bench Pro, matching the unpruned 41.8. REAP scored 36.2, a drop the post describes as a 13% regression.
- Across three models, two calibration sets, and rates from 10% to 50%, HOPE had the best average rank, 2.07 out of five methods. At 50% pruning that rank improved to 1.58, against 2.42 for REAP.
- The authors say the advantage is largest at 40% to 50% pruning and on agentic coding, meaning multi-step coding work that calls tools. At 10% to 20%, first-order methods are sometimes competitive. HOPE still prunes each layer on its own and does not yet use interactions across layers.
On 9 November 2026, Alex M. Tseng of Amazon posted on Hugging Face about a method called HOPE, short for Higher-Order Pruning of Experts. The post explains a paper, arXiv:2609.18916, written with Prannay Kaul, Luca Zancato, Wei Xia, and Stefano Soatto at AWS Agentic AI, and it points to code released at github.com/awslabs/HOPE. Mixture-of-experts models grew large by splitting the part of each layer that transforms a token, a small piece of text, often part of a word, into many specialist networks, called experts, and sending each token to only a few of them. The unused experts still have to sit in memory. The usual fix is to delete the experts that look least useful. HOPE's claim is that this usual fix scores the wrong thing. Experts that look disposable on their own can be the partners of experts you intended to keep.
[1][2]What HOPE is trying to fix
The Hugging Face post uses Qwen3.5-122B as the memory example. That model has 256 experts in each of 48 mixture-of-experts layers. For any given token, only 8 experts in each layer are used. The post says about 95% of the model's roughly 250 GB parameter memory is those expert networks. The paper gives the same picture with slightly tighter figures. In Qwen3.5-122B-A10B, 116 billion of 122 billion parameters are expert networks, about 240 GB. Cutting 50% of the experts, the paper says, would bring that footprint to about 125 GB and free about 115 GB for longer context, tool calls, and reasoning traces on the same machine.
Loading unused experts from elsewhere at run time is too slow for many serving setups. Quantization shrinks each stored number but still leaves every expert in memory. Pruning deletes experts for good, needs no extra training, and can be combined with quantization. The released package writes a standard Hugging Face checkpoint. The authors say it does not require changes to vLLM or SGLang, two common systems for serving language models.
The hard part is choosing whom to delete.
[1][2]What first-order pruning misses
Existing methods collect statistics on a calibration set, a sample of text used to watch the model before any experts are removed, and then give each expert one number. Frequency keeps the experts that fire most often. EAN and MAN look at the size of each expert's output. REAP, Router-weighted Expert Activation Pruning, weights that output by the router's gate value. The router is the small network that picks which experts handle a token. The paper treats REAP as the leading first-order method for generative tasks.
All of those methods assume that an expert's value can be judged alone. The authors say that is not how these models run. Each token is sent to a small group of experts, often 8, and those experts are added together. Some pairs are chosen together far more often than chance, and those pairings stabilize during training. If you delete expert A because its solo score is low, you may also break expert B, which you kept because its solo score was high.
HOPE tries to price that damage. After one forward pass over calibration data, it builds a matrix F for each layer. The entry at row i and column j is the average, over tokens that selected both experts, of their gate-weighted output sizes multiplied together. The diagonal of F is each expert's individual importance. The off-diagonal entries are the pairwise terms. The authors prove that a simple quantity built from these entries upper-bounds the main part of the error you get by deleting a set of experts, and they turn the search for the least damaging set into a quadratic program. Solving that program takes about 1 to 2 seconds per layer, they say, which is small next to the calibration pass.
The connection to REAP is direct. The diagonal of F recovers REAP's expert rankings almost perfectly, with Spearman rho of 0.988. The paper also states that REAP is the special case of HOPE you get if you ignore the off-diagonal terms. Any gain over REAP, on the authors' account, has to come from those pairing terms.
[1][2]What the numbers say
The authors tested three models: Qwen3.5-122B, Qwen3.5-35B, and GLM-4.5-Air. They used two calibration sets, Evol-CodeAlpaca and SWE-Bench Verified trajectories, and compared HOPE with REAP, MAN, EAN, and Frequency at pruning rates from 10% to 50%. They evaluated on the Tulu3 suite and on SWE-Bench Pro, a test in which a model has to solve real GitHub issues on its own.
Across those conditions, HOPE had the best average rank, 2.07 out of 5 methods. It finished first in 39% of conditions and in the top two in 67%. At 50% pruning, its average rank improved to 1.58, against 2.42 for REAP.
The result the post puts up front is the agentic coding number. On Qwen3.5-35B-A3B with 50% of experts removed, HOPE scored 42.3 on SWE-Bench Pro. The unpruned model scored 41.8. REAP scored 36.2, which the post calls a 13% regression. The paper's matching table reports 42.33 plus or minus 1.23 for HOPE, 41.82 for the unpruned model, and 36.23 plus or minus 0.93 for REAP, after calibration on Evol-CodeAlpaca. The honest reading is that HOPE stayed level with the unpruned model within the reported variation, while REAP did not.
The post says HOPE's edge grows as more experts are removed, and that it is most visible on SWE-Bench Pro, where a model has to compose many kinds of work over many turns. That is the setting where diverse expert combinations are most likely to be used, and where breaking pairs would hurt most. When HOPE and REAP disagree about an expert, the experts HOPE keeps have higher cooperation scores with the other survivors than the experts REAP keeps.
The authors also note that a pruned model can sometimes match or slightly beat the unpruned baseline, probably because dropping irrelevant experts removes noise on the target tasks. They say this is not unique to HOPE.
[1][2]The fair case, and what is missing
The fair case for first-order pruning is straightforward. At 10% to 20% pruning the pairing terms matter less, and the post itself says REAP and MAN are sometimes competitive there. If you only need a modest memory saving, a simpler score may be enough. REAP already has a theoretical justification, and HOPE presents itself as a generalization of that method rather than a rejection of it. The extra cost is small, about 6% to 7% longer calibration than REAP in the paper's timing table, plus a few seconds to solve the program.
The evaluation is still the authors' own. Tseng and colleagues work at Amazon. They compared HOPE with named baselines on public models and published code, which is more checkable than a vendor slide, but it is not an independent laboratory repeating the SWE-Bench Pro number. The 42.3 versus 41.8 comparison should not be sold as HOPE making the model better than it was. The paper's error bar on 42.33 is large enough to include 41.82.
HOPE still decides layer by layer and only models pairs, not groups of three or more. The authors sketch a cross-layer version and call those results preliminary. Calibration data also matter. Different prompt sets produce somewhat different prune-sets, with a Jaccard index around 0.60 between Evol-CodeAlpaca and SWE-Bench Verified, similar to REAP's 0.61. Jaccard index here means the overlap between two sets of deleted experts. An index of 1 would mean the two calibration sets produced the same deletions.
[1][2]Who should care, and what to watch
Teams serving large mixture-of-experts models on memory-limited GPUs should care, especially if they are considering deleting 40% or 50% of experts so that longer context and tool traces fit. People who already use REAP should care, because HOPE says the extra signal is exactly the part REAP throws away.
Over the next six months, the checkable items are reproductions and serving reports. Watch whether another group, outside Amazon, publishes a 50% prune on Qwen3.5 or GLM-4.5 and recovers a similar SWE-Bench Pro gap between HOPE and REAP. Watch whether HOPE-pruned checkpoints show up in public model repositories and whether anyone reports tokens per second and memory use next to an unpruned baseline on the same hardware.
[1][2]Close
The method is a claim about accounting, not a claim that mixture-of-experts models have been solved. If independent runs keep the 50% SWE-Bench Pro gap, then deleting experts by solo scores will look like the wrong cut at the pruning rates that free the most memory. If the gap shrinks under someone else's calibration set or evaluation harness, then HOPE's pairing terms will have been a useful correction on one lab's suite, not a new default. The next half year should decide which of those it is.
[1][2]