Release
Phase-HDC paper released
The Phase-HDC paper is available on arXiv, with code and an interactive version in the playground. It trains a hyperdimensional classifier without any optimiser memory.
TLDRA learned classifier stores each parameter as a 6-bit dial. At every update a dial turns one notch against its gradient, but only when the gradient is larger than a fixed write cost, so no optimiser history is kept at all. On a controlled digits benchmark it reaches 92.4% test accuracy against 91.2% for Adam with 6-bit moments, with three times less training state (4.27 KiB against 12.80 KiB). The accuracy gap is within noise, so the result shows that the idea works, not that it beats Adam.
Read on arXiv Code Try it in the playground
Announcement
Mathalyse Research launches
Mathalyse Research is an independent lab working on optimisers, architectures and the theory of learning. This site collects its released work and announcements. Collaborators and students can write through the contact page.
Release
ThermoLion preprint released
The ThermoLion preprint is available on arXiv. It argues that training deep vision networks calls for adjusting the precision of the update dynamically, between the magnitude-based updates of AdamW and the sign-based updates of Lion.
TLDRThermoLion tracks how consistent each parameter’s gradient has been. Consistent gradients get Adam-like proportional steps, and noisy ones get Lion-like fixed-size sign steps plus a little noise that cools over time. The preprint tests this on a single small ConvNet (12 epochs, up to 5,000 training samples per dataset) and reports gains over Adam on the harder image datasets.
Read on arXiv Code Try it in the playground