MathalyseResearch

Playground

Try the methods yourself.

One interactive page for each released paper. Everything runs in your browser, and nothing you draw or change is sent anywhere.

Phase-HDC ThermoLion

Phase-HDC · arXiv:2610.10630

One notch, or none.

In short

Training a model normally needs a second, hidden memory: the optimiser’s record of past gradients. Phase-HDC throws that record away. Every parameter is a knob with 64 click positions. At each step a knob turns one click against its gradient, but only if the gradient is bigger than a fixed write cost. Below, 5,824 such knobs learn to read handwritten digits.

Read the paperCodeOriginal interactive companion
Step 1 · The rule

Turn one click, or stay.

Think of a knob with a click at every position. Training pushes on each knob: the gradient g says which way, and how hard, turning it would help the model.

Turning a knob also has a price, a fixed write cost τ. At each step the knob has three choices: one click back, stay, or one click forward. It takes the cheapest. That always comes down to the same rule: turn one click against the push if the push is bigger than τ, otherwise stay.

With τ = 0 this is plain sign descent. The rule justifies a single step. It does not by itself guarantee that training converges.

one click = 22.5° (the model uses 64 clicks)
Turn back
Stay0
Turn forward

Cost of each choice, lower is better. Turning costs the loss it would add or remove (to first order) plus τ.

Step 2 · Watch it learn

5,824 knobs, learning digits.

Each small square is one knob, coloured by its angle. The model has three tables: P holds one angle for every pixel position, V one for every pixel darkness (0 to 16), and ψ one for every digit class. Press start, then watch which knobs turn. A bright square turned in the latest step. A dark dot marks a knob that jumped more than 3 clicks.

Switch to Adam to see the competitor on the same knobs. Adam decides when to move from two extra arrays per knob, which is where the extra memory goes.

colour = knob anglebright = turned just nowdot = jumped > 3 clicks
Step
0 / 180
epoch 0 / 20
Knobs turned
—
knobs turned in the latest step
Training loss
—
latest minibatch
Validation accuracy
—
360 held-out digits, each epoch
Test accuracy
—
360 test digits, after step 180
Memory kept
4.27 KiB
the dials are all it keeps
Your runs against the paper

Test accuracy after 180 steps. Filled dots are runs from this page, each with its own random start. Hollow dots use settings other than the paper’s, or fall off the scale. The grey tick is the mean of the paper’s five controlled runs.

Step 3 · Read a digit

How the knobs vote.

For every pixel the model takes two angles, one for where the pixel is and one for how dark it is, and adds them. That gives one arrow per pixel. It lays the 64 arrows head to tail, in 64 different directions at once. Arrows that agree add up to a long sum, and scattered ones cancel.

Then it compares those 64 sums with each digit’s own row of knobs, and the digit that lines up best wins. Draw on the grid or load a test digit to see it.

Step 4 · What history was for

Adam needs its memory to decide when to move.

Adam keeps a running average of past gradients and of their squares, for every knob. On a grid of whole clicks, a step smaller than half a click rounds to nothing. So Adam’s memory ends up deciding whether a knob moves at all. Phase-HDC makes the same decision from the current gradient alone, so it keeps nothing.

Try the button under Step 2. At a small learning rate, Adam’s steps all round to zero and no knob ever turns. The price of keeping no memory is that τ must be tuned per table, per dataset and per bit width, because it is measured in gradient units.

Compressing Adam’s memory to 6 bits has its own failure. A rare pixel value, such as one dark shade, has such small second moments that they fall below the first quantisation step. Adam then divides by almost zero and those knobs jump many clicks at once.

State kept (controlled test)
3× less
4.27 KiB against 12.80 KiB for Adam with 6-bit moments
Test accuracy
92.4 vs 91.2%
five seeds. The gap is within noise: the paired 95% interval is [−0.44, 2.89] points
Eleven datasets
16–23× less
state than float32 Adam, at an average cost of 5.3 accuracy points
What these numbers do not show
  • Memory figures count ideal packed arrays, not measured device memory, speed or energy.
  • The controlled comparison is one small dataset with five seeds. It supports parity, not superiority.
  • The eleven-dataset results are exploratory: tuning reused the reported evaluation data.

Data: scikit-learn digits (UCI Optical Recognition of Handwritten Digits, CC BY 4.0), with the paper’s split of 1,077 training, 360 validation and 360 test digits. The trainer here is a JavaScript port of the NumPy reference in the code repository (MIT licence). Runs on this page use their own random starts, so their accuracies vary around the paper’s.

ThermoLion · arXiv:2512.01881

Sign steps or proportional steps.

In short

Adam scales each step by how large the gradient is. Lion ignores the size and moves a fixed distance in the gradient’s direction. ThermoLion decides per parameter which to use. It measures how consistent the gradient has been, and gives consistent gradients Adam-like proportional steps and noisy ones Lion-like sign steps, plus a little noise that cools over time.

Read the paper
Step 1 · The gate

How consistent is the gradient?

One parameter receives a new gradient at every step. Part of it is a steady signal that keeps pointing the same way, and part is random noise. ThermoLion keeps a running average of the gradient (the purple line) and compares its size with the typical size of the gradient.

That ratio, passed through a smooth squashing function, gives the gate λ between 0 and 1. High λ means the signal dominates, and the step is proportional to the average gradient, like Adam. Low λ means noise dominates, and the step is a fixed-size sign step, like Lion. Move the sliders to see the gate respond.

Gate: λ = tanh(|m| / (√v + ε)), where m is the running average of the gradient and v the running average of its square (Eqs. 1 to 4).

Typical step length, in units of the learning rate
Adam
Lion
ThermoLion
Step 2 · The race

Three optimisers on a curved valley.

The valley is a standard test surface with its minimum at the ring. All three optimisers start at the white dot and receive the same noisy gradients. Click the surface to choose another start. Raise the noise to see how each update rule copes, and change ThermoLion’s settings to see what they do.

This is a two-parameter toy, not the paper’s vision benchmarks, and the order of the optimisers changes with the settings. The paper does not fix the temperature schedule, so the toy cools the noise by 0.99 per step and measures it in units of one step.

Step