Case study
ASO Audit Agent
Audits an App Store listing for App Store Optimization (ASO): a scored report, quoted evidence, and a prioritized list of fixes.
- Role
- Solo - architecture and implementation
- Year
- 2026
- Status
- Live
At a glance
- listing areas scored
- 10
- recommendation tiers
- 3
- model and data providers
- 3
- automated evals
- 2
Overview
ASO Audit Agent reviews an app's App Store listing and tells you what is working, what is weak, and what to change first. You paste an App Store link, confirm it matched the right app, and get back a score out of 100, a fix list ordered by impact, rewritten text you can paste in, and a comparison against similar apps.
Every score quotes the listing text it came from, so a grade can be checked rather than taken on trust. When a dimension cannot be assessed from public data, the report drops it from the total and says so, instead of scoring it zero. Built on Mastra: one conversational agent owns the chat, one deterministic workflow owns the audit.
Why I Built It
Advice about App Store listings is easy to produce and hard to check. Ask a model to review one and it returns ten confident paragraphs that trace back to nothing on the page.
The interesting problem is the inverse of generation: constraining a model until its output can be checked. A score must attach to a quote. A dimension with no data must leave the total. The pipeline must run the same way on an app it has never seen. Apple makes it harder - the keyword field is private and promotional surfaces are not exposed - so deciding the honest ceiling, and reporting it instead of filling it, is most of the work.
The Approach
Conversation and audit are different problems, so they are built separately. The chat side is free-form: it takes a link, confirms the app, streams progress, and answers follow-up questions. The audit side is a fixed sequence that runs the same way on every app.
A Mastra agent owns the chat - URL parsing, the confirmation card, progress, follow-ups. A deterministic workflow owns the audit: scrape the listing, fan out in parallel across reviews, competitors and one vision call per screenshot, make a single structured scoring call, then assemble the report in a pure step. Named step events stream into the chat, so progress is a real signal.
Scope boundaries, chosen before the code and stated in the report:
- Keyword field is Apple-private - inferred from title, subtitle and description, and flagged as inferred.
- Preview-video presence is detected; its content is not evaluated - that needs decoding the asset for a 5%-weight dimension.
- Competitors come from a keyword search - framed as directional, not exact.
- No paid ASO data service for one dimension in ten.
Architecture
A single Next.js app on the Node runtime with Mastra's chat route mounted as a handler - one process, one dev command. All network code lives in tools and nothing else does I/O; the rubric is a versioned prompt imported only by the scoring step, so swapping it never touches the agent.
Three providers, chosen per task: Gemini Flash-Lite for chat and vision, Groq's 70B llama for structured scoring, Firecrawl for the listing fields Apple's APIs do not expose. Model ids are env-only and the agent provider is selectable, so a swap is config, not infrastructure.
Run state is two keys in thread metadata - the confirmed app and the last report - written deterministically; follow-ups answer from conversation history. Testing is two evals rather than a unit suite: evidence faithfulness (every quote a literal substring of the collected data) and schema-validity variance across repeated runs. They run locally, since they hit live providers.
Decisions & Tradeoffs
A deterministic workflow for the audit; an agent only for the conversation
Tradeoff accepted
Two orchestration primitives instead of one. In return the audit runs identically on an unseen app, the fan-out gets concurrency for free, and progress events exist without inventing a protocol. Suspend and resume is wrong for chat - the user's next message already is the resume event.
Alternatives considered (2)
- One agent calling the data tools in sequence, then a scoring tool
- A workflow at the top, using suspend and resume for the confirmation gate
Exclude dimensions with no usable signal and renormalize; never score a missing source as zero
Tradeoff accepted
The overall score is not comparable across audits that evaluated different dimensions, so every report carries a data-quality block and a "scored on N of 10" line. A missing source posing as a zero is the one failure that makes the number worthless; the rule is enforced after parsing, in code, because a model that agrees with it in the prompt still breaks it.
Alternatives considered (3)
- Score an unavailable dimension zero
- Assign a neutral default
- Drop the dimension without saying so
Score with a dedicated structured-output model, not with the conversational agent
Tradeoff accepted
Two providers means two rate-limit regimes. It buys isolation - the agent cannot drift from numbers it did not produce, and emits no prose summary on success - and lets each side be chosen on merit; the vision model was swapped after an A/B on the real pipeline.
Alternatives considered (2)
- Let the chat agent produce the scores
- One frontier model for chat, vision and scoring
Keep run state in thread metadata, not the framework's working memory
Tradeoff accepted
Framework working memory cannot be enabled without folding its JSON into context and exposing a model-facing mutate tool. Thread metadata keeps the same two-key deterministic contract with no model-visible blob. The cost: state is in-process and does not survive a restart.
Alternatives considered (2)
- Mastra working memory
- A database for thread state
Gate the audit behind an explicit confirmation of the matched app
Tradeoff accepted
One extra round trip and a small input state machine. It stops a nine-call, multi-provider pipeline from running against the wrong app - the expensive failure, and the one the user notices last.
Alternatives considered (2)
- Run the audit immediately on a valid URL
- Confirm only when the URL is ambiguous
Engineering Challenges
Getting a model to return data in exactly the shape the rest of the code expects.
How it was solved
Scoring is one call returning all ten dimensions in a Zod shape. A parse failure retries once with the errors appended; a second failure emits a partial report and lists the rest under data quality. A post-parse guard catches the model excluding a dimension it may not, so nothing disappears silently - the workflow always reaches assembly.
A provider rate limit that turned out to be a design constraint, not an ops detail.
How it was solved
Groq reserves the declared output budget against its per-minute window, so an oversized cap trips "request too large" before a token is produced. That one fact set the output cap, shrank the review corpus, and shaped the retry: 429s and the matched rate-limit error back off; timeouts and 400s fail fast.
A wrong assumption that shipped, then hid - every app quietly skipped the same check.
How it was solved
An early spike judged preview-video presence undetectable, so the code read it from an Apple field that does not exist - and every app's video dimension excluded itself while working exactly as written. The fix counts player-widget markers in the scraped HTML, keyed on the CSS class not the localized label; absence is still never asserted.
Features
- Ten-dimension scorecard with one overall score, renormalized over the dimensions actually evaluated
- Verbatim cited evidence behind every score and recommendation
- Recommendations grouped into quick wins, high-impact and strategic
- Confirmation gate before any audit runs
- Live progress streamed from the workflow into the chat
- Inferred and excluded dimensions reported explicitly, never silently
Tech Stack
- Next.js 16 (App Router, Node runtime)
- React 19
- TypeScript
- Tailwind CSS 4
- Mastra (agent + workflow)
- Vercel AI SDK
- Google Gemini (chat + vision)
- Groq llama-3.3-70b (scoring)
- Firecrawl
- Apple iTunes Lookup / Search / RSS
- Zod
- Vercel
Lessons Learned
- The cheapest guard against hallucinated evidence is a substring check - a mechanical eval caught more than any prompt tightening.
- Where correctness matters, enforce in code what the prompt asks for; a model that agrees with a rule still breaks it.
- A degraded report is a product; an exception is not. Every failure mode got a defined output before it got a retry.
Screenshots
Contact
If you need an engineer who can own it end to end, from the interface to the infrastructure, let's talk.
Email me