AK
Ania Kowalska
11 hours ago
Groq replied before my Flutter spinner could mount
Groq
Ultra-fast LLM inference on custom LPU chips
Switched our Flutter chat stub from a generic OpenAI-compatible endpoint to Groq (llama-3.3-70b) on Friday.
First reply landed in ~280ms. Cool. Also broke the loading UI — the CircularProgressIndicator never painted because the stream started before setState finished. Testers said the app "felt broken" even though it was faster.
Anyone else redesigning skeletons just because inference got too quick?
5 comments
Join the discussion
Log in to comment.
same energy on the web side. we had a 600ms min-delay on the skeleton "so it feels intentional" — with Groq that delay is now longer than the actual answer. users see a flash of empty then text. i deleted the artificial wait and suddenly support tickets about "blank chat" dropped.
fast models make bad UX assumptions louder.
yeah the 600ms "intentional" delay aged badly overnight. we had the same pattern in a Next chat widget — brand said skeleton must show. after Groq, it looked like the product was stalling on purpose.
killed the min-delay, kept a 120ms fade-in for the first token so it doesn't pop. support tickets about blank chat basically vanished same day.
do you still gate on skeleton duration anywhere, or is it all token-driven now?
lol yes. on Expo I had an ActivityIndicator that never showed for the same reason — stream callback beat the next paint.
hack that worked for me: keep the spinner up until at least 1 token AND 200ms have passed. feels dumb but testers stopped complaining. also watch cold starts on free tier, first call after idle is still kinda slow for me.
the 1 token + 200ms rule is good. on RN/Expo I went one step further: two states —
pending(nothing yet) andstreaming(at least one token).spinners only in
pending. once streaming starts, swap to a thin progress shimmer under the first line so people know more is coming. empty flash was killing trust in demos even when latency was fine.also watch AnimatedOpacity — if you fade the spinner out after the reply mounts, you get a one-frame blank on slow devices. ask me how I know.
hit this on Flutter 3.24 + llama-3.3-70b via Groq too. CircularProgressIndicator never got a frame because the first SSE chunk arrived in the same event loop turn as setState.
what fixed it for me: schedule the spinner with WidgetsBinding.instance.addPostFrameCallback, then cancel it when token 1 lands. if the reply is already there, skip the spinner entirely. felt weird writing "optimistic empty" but the QA spreadsheet stopped saying "feels broken".
also — if you're on free tier, measure cold start separately. my p50 is ~250ms, p99 still spikes past 1.2s after idle.