Why handwriting autocomplete?
Autocomplete is familiar when you’re typing or coding, but it’s rarely available when you write by hand. A chat sidebar makes you look away from your work, type out a question, and wait for a response. I’m building the assistant so you can stay focused on the page and get back to writing as soon as a suggestion helps you express what you had in mind.
How suggestions get onto the page
I self-host the LLM with vLLM to keep inference latency down and stream pen strokes to the server over a WebSocket as the user writes. The recognizer turns the ink into LaTeX, and the LLM can call a math solver to improve the accuracy of its suggestions. The handwriting decoder renders the suggested LaTeX as pen strokes.
I initially explored 2D rotary positional embeddings (RoPE) and having the model output coordinates for each suggestion. I switched to simple placement rules. The LLM returns an instruction like right, next line, or under, and the app places the suggestion relative to the previous block of handwriting.
Model design and preliminary results
- Synthesis. The flow-matching transformer learns to write a line of handwriting from LaTeX using public handwriting datasets. I’ve experimented with CFG-RL and synthetic handwriting generated from fonts to improve decoder training. The current decoder replaced a Mamba2 prototype distilled from a pretrained LSTM.
- Recognition. A Conv1d + transformer model reads ink and outputs LaTeX. I train it with an auxiliary CTC loss.
- Style. I trained a writer-style encoder with supervised contrastive losses and experimented with a VAE-style objective with adaptive KL weighting.
- Representation. I fit pen strokes with least-squares cubic B-splines to shorten the sequences the models process.
What I want to build next
I’d like to add a tutoring mode that flags mistakes as you work, and a widget mode that suggests things like a graph of the equation you just wrote or a generated illustration for an idea you want to sketch out.