I’m vegetarian and allergic to peanuts. I built a tool that takes a restaurant name and optional dietary restrictions, searches for the menu, and extracts dishes, descriptions, and prices from the source.
The visualizer ran locally. Keeping a GPU server online was too expensive for me to host a permanent public demo.
Menu visualizer
The local visualizer lets me switch models and prompts. The example shows the output for Nori in Austin, rendered from the JSON returned by Gemma served through vLLM.
System design
Agent tools
I built two tools for Gemma. web_search(query) takes a search string and returns candidate URLs and snippets. scrape_url(url, mode="direct") takes a URL and an optional fetch mode. direct starts with a plain HTTP request, while browser renders the page in headless Chromium. My runner sends the tool definitions to Gemma through vLLM, parses the call and arguments, then appends the tool response to the conversation. Gemma uses that response to decide whether to search again, fetch a result, or return menu JSON. It gets up to eight tool calls per request.
web_search(query)scrape_url(url, mode)GET /res/v1/web/searchRead-through cache
Both tools check cache.sqlite before reaching the network. The key hashes the tool namespace, normalized query or URL, cache version, and scrape mode when applicable. Search misses call Brave’s web search endpoint. Scrape misses run the HTTP or Chromium backend and store the cleaned page text. The runner caps the amount of text at read time, so I can change the model’s input budget without fetching pages again. Cached empty results can be reused, while fetch errors can be retried in live mode. Evaluation uses a frozen cache so each model works from the same web responses.
Collecting and cleaning data
I collected restaurant names and cities from OpenStreetMap and Google Places, then ran Qwen3-235B with the same tools to generate 2,233 training traces. In corpus.sqlite, the restaurants table records the split per restaurant, and the traces table links each dietary variant by restaurant ID. Each trace stores the full conversation, final JSON, and review results. I checked schema validity, source quality, and how many extracted dishes appeared in the scraped text. The review tool exposes those checks and lets me compare variants from the same restaurant. After filtering and formatting, 2,074 examples went into fine-tuning.
I used thread pools to warm the scrape cache and collect teacher traces in parallel. Each worker reused its own Chromium instance and opened a fresh browser context for each scrape. A wedged page could block Playwright during auto-scroll or content extraction even after navigation completed, so I added a 180-second watchdog around each render. If it fired, it killed the stuck Playwright driver, marked the scrape as failed, and let the worker launch a fresh browser on its next render.
corpus.sqlite · Schema and dataset outputsrestaurantsrestaurant_id (primary key), name, city, splittracestrace_id (primary key), restaurant_id (foreign key)Dietary restrictions, tool history, menu JSON, review verdict
Fine-tuning
I started by running Gemma 4 E4B with the detailed teacher prompt, which spelled out when to retry a search, how to verify the restaurant, and which sources to prefer. For context distillation, I took stronger Qwen3-235B traces and rebuilt them with the shorter student prompt. I trained Gemma on the teacher’s tool calls and final JSON, with the scraped pages supplied as context. The student kept the tools, schema, and dietary restrictions in its prompt and learned the search and scraping strategy from examples.
I used LoRA to train small updates to Gemma’s attention and feed-forward layers while freezing the base weights. The final supervised fine-tuning (SFT) adapter used rank 16 and updated about 35 million parameters. I tried GRPO to improve menu completeness and explored Q8 quantization for faster serving. GRPO failed to improve recall, and the quantized build stayed experimental. The SFT checkpoint remained the final model.
Evaluation harness
I built an evaluation harness that runs the model checkpoints on the same seeded plan of 500 held-out episodes. Forty percent include a dietary restriction. Every run uses the same frozen search and scrape cache, so a site changing between runs can’t change the evidence available to a model.
The harness pairs each student response with the teacher trace for the same restaurant and dietary restrictions. It validates the JSON schema, checks whether the model got the menu’s found status right, and fuzzy-matches normalized dish names one-to-one to calculate precision, recall, and F1. I track false finds and missed menus separately so I can tell whether a model got better at finding evidence or simply got more willing to claim it found a menu. Item scores measure agreement with the teacher’s extraction.
SFT cut false menu finds from 29 to 10. Item precision reached 82.7% and recall reached 57.3%, measured against the teacher’s traces. The GRPO grounding reward likely favored short lists of well-supported items, leaving the recall gap open.

