Intelligence at critical point

Order chaos transitions in a neural network (LSTM) training process. Wei, Wenbo, et al. "Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions." arXiv preprint arXiv:2505.20030 (2025).

My research on AI focuses on the complexity science and dynamics of deep learning models. Drawing from and dynamical systems and spin glasses, my work investigates how neural networks achieve optimal intelligence and generalization at the "edge of chaos" – the critical boundary between ordered and chaotic phases. This principle, inspired by natural systems like the brain, reveals that machine learning models perform best when their asymptotic stability hovers at this transition point, maximizing information processing and adaptability.

In "Optimal Machine Intelligence at the Edge of Chaos" (2020), we develop a general theory for non-linear systems, proving that the edge of chaos, analytically determined by the Jacobian norm, coincides with peak mutual information transfer. Validated on deep vision models like ResNet and DenseNet, training pushes networks toward this edge for superior accuracy.

Figure 1: Three phases of non-linear dynamical systems, from simple quadratic map to neural networks.

Extending this, "Asymptotic Edge of Chaos as Guiding Principle for Neural Network Training" (2023) maps training dynamics to spin glass phase diagrams. We provide a semi-analytical method to set optimal weight decay, ensuring models converge to the edge without manual tuning. This yields highest test accuracy and robustness against noisy labels, as models avoid overfitting shuffled data while fitting clean labels effectively.

Figure 2: controlling training at the edge of chaos prevents over fitting and under fitting, even under noisy labels.

In another work, "Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions" (2026), we observe novel multiple descent cycles in overtrained LSTMs for sentiment analysis. These cycles align with order-chaos transitions, with global optima at the first, widest edge of chaos, enabling better weight exploration.

Fig 3: Multiple test loss descent matches exactly order chaos transitions in LSTM network.

Building upon such principle, we have got into top 10 positions in the below Competition in NeurIPS 2020:

Predicting Deep Learning Model Generalization Using Asymptotic Stability

L Zhang, S Li, L Feng

"Predicting Generalization in Deep Learning" Competition at NeurIPS 2020workshop, ranked 9th place

Theoretically, by treating neural networks as dynamical systems, 'knowledge' learnt in neural networks are possibly the metastable states of the system, and offers avenues to explore explainable AI through such states. Practically, these studies bridge physics and AI, offering principled training strategies for robust, high-performing models.

More details can be found at www.criticality.ai