← Back to the universe 02 / 13

02 / MACHINE LEARNING · PERSONAL PROJECT

Online Handwriting Assistant

Autocomplete for handwritten math. The assistant suggests a next step on the whiteboard in your own handwriting. The project is still in development, and evaluation is ongoing.

Code on GitHub ↗
Handwritten proof demo
Handwritten calculus computation demo

Why handwriting autocomplete?

Autocomplete is familiar when you’re typing or coding, but it’s rarely available when you write by hand. A chat sidebar makes you look away from your work, type out a question, and wait for a response. I’m building the assistant so you can stay focused on the page and get back to writing as soon as a suggestion helps you express what you had in mind.

PyTorchFlow matchingTransformersCTCContrastive learning

How suggestions get onto the page

Pen strokes→Recognizer→LLM next step→Handwriting decoder→Ink on the board

I self-host the LLM with vLLM to keep inference latency down and stream pen strokes to the server over a WebSocket as the user writes. The recognizer turns the ink into LaTeX, and the LLM can call a math solver to improve the accuracy of its suggestions. The handwriting decoder renders the suggested LaTeX as pen strokes.

I initially explored 2D rotary positional embeddings (RoPE) and having the model output coordinates for each suggestion. I switched to simple placement rules. The LLM returns an instruction like right, next line, or under, and the app places the suggestion relative to the previous block of handwriting.

Model design and preliminary results

5.0%token error, ink-to-LaTeX recognizer
80%unseen-writer ID vs. 9.8% chance
3.3×shorter stroke sequences with B-splines

What I want to build next

I’d like to add a tutoring mode that flags mistakes as you work, and a widget mode that suggests things like a graph of the equation you just wrote or a generated illustration for an idea you want to sketch out.