CAMBRIDGE, MASSACHUSETTS — Researchers at the Massachusetts Institute of Technology's Computer Science and Artificial Intelligence Laboratory introduced CompreSSM, a method that compresses AI models during training rather than after, achieving up to 1.5 times faster training while maintaining accuracy on image classification benchmarks. The technique targets a family of AI architectures known as state-space models, which power applications ranging from language processing to audio generation and robotics.

CompreSSM uses mathematical tools borrowed from control theory to identify which parts of a model contribute to its behavior and which are unnecessary. The researchers found that the relative importance of components within state-space models stabilizes early during training, allowing them to rank which model dimensions matter after about 10 percent of the training process. Less-important components can then be discarded, and the remaining training proceeds at the speed of a much smaller model.

"It's essentially a technique to make models grow smaller and faster as they are training," MIT CSAIL PhD student Makram Chahine said. "During learning, they're also getting rid of parts that are not useful to their development."

In testing, a compressed model reduced to roughly a quarter of its original state dimension achieved 85.7 percent accuracy on the CIFAR-10 benchmark, compared to 81.8 percent accuracy for a model trained at that smaller size from scratch. On the Mamba state-space architecture, CompreSSM compressed a 128-dimensional model to around 12 dimensions and achieved approximately four times faster training while maintaining performance comparable to the uncompressed model.

"You get the performance of the larger model, because you capture most of the complex dynamics during the warm-up phase, then only keep the most-useful states. The model is still able to perform at a higher level than training a small model from the start," Chahine said. "It gives people control over how much they are willing to pay in terms of performance, rather than having to define an energy threshold."

"What's exciting about this work is that it turns compression from an afterthought into part of the learning process itself. Instead of training a large model and then figuring out how to make it smaller, CompreSSM lets the model discover its own efficient structure as it learns. That's a fundamentally different way to think about building AI systems," said Daniela Rus, professor and director of CSAIL at MIT.

The researchers benchmarked CompreSSM against conventional pruning approaches and knowledge distillation, two established methods for reducing model size. Conventional pruning trains a full model and then strips away parameters afterward, requiring the full computational cost of training the large model. Knowledge distillation requires training a large teacher model to completion and then training a smaller student model on it, with a forward pass through both models at every training step. CompreSSM avoids these costs by making compression decisions during training. Compared to Hankel nuclear norm regularization, CompreSSM was more than 40 times faster and achieved higher accuracy. In CIFAR-10 benchmarks at smaller state dimensions, distilled models experienced accuracy drops while CompreSSM-compressed models maintained accuracy close to the baseline.

The researchers used Hankel singular values to measure how much each internal state contributes to a model's overall behavior and showed empirically that relative rankings of model states remain stable during training. They also proved mathematically, using Weyl's theorem, that the importance of individual model states changes smoothly during training. If a compression step causes an unexpected performance drop, practitioners can revert to a previously saved checkpoint. CompreSSM works best on models that exhibit a strong correlation between internal state dimension and overall performance, and is particularly effective on multi-input, multi-output models, where the relationship between state size and expressivity is strongest.