Flutter LLM On Device with kiln: Chat, Decision and Embedding Models on Android

A Flutter LLM on device runs today on arm64 Android phones through kiln_flutter, with no server behind it. In kiln's October 2026 benchmarks, Qwen3.5 0.8B generated 37 tokens per second on a cool Pixel 8 and 26 once the phone was hot. The same package runs decision models, embedding models, speech to text and text to speech. The numbers below come from kiln's own benchmark runs on a Pixel 8, a Galaxy S26 Ultra and an Apple M4 Pro. Each one names the device and the run it came from.

Flutter LLM on device: what kiln runs on Android

kiln is an open-source (MIT) runtime for AI models on phones and desktops, built at CortexTech. It runs on llama.cpp, the widely used C++ inference engine. It loads standard GGUF files, the single-file model format llama.cpp uses, and takes each model's chat template from the file itself. The core is a C API, and Flutter, Kotlin, React Native and Rust bindings wrap it. As of kiln_flutter 0.5.2 on pub.dev (October 2026), the Flutter package targets arm64 Android only, with minSdk 29.

kiln runs five kinds of on-device AI model. The table shows what each kind returns and which models kiln's README lists as verified end to end.

A developer who needs more than chat can use one package and one model format for all five jobs.

Run an LLM on Android with kiln_flutter

To run an LLM on Android with kiln, add the package with flutter pub add kiln_flutter and set minSdk = 29. Test on an arm64 phone. The model file is a GGUF the app downloads or bundles. No call leaves the phone, so the same setup works as an offline LLM on Android. Each job is one call on a loaded context, shown here as the pub.dev README writes them:

final refund = await ctx.decide(KilnQuestion.yesNo('Is the customer asking for a refund?'), state: mail);
final query = await ctx.embed('how do I bake bread', kind: KilnEmbedKind.query);
final photo = await ctx.embedMedia([KilnMedia.file('/path/to/holiday.jpg')]);
final t = ctx.transcribe(KilnMedia.file('/path/to/voice_note.wav'));
final s = ctx.speak('Your parcel arrives on Thursday.', voice: KilnMedia.file('/path/to/me.wav'));

Chat goes through ctx.chat, which streams the reply in gen.pieces, reports gen.stats at the end, and stops on gen.cancel(). The stats report speed in tokens per second, how many prompt tokens came from cache, and why generation stopped. An app that logs those stats from real users learns how its model performs on hardware the developer never tested.

Chat speed on Android: cool phone vs hot phone

Chat speed on a phone falls within the first minute of use, because the chip heats up and lowers its clock. kiln-bench measures this by running the same 65-token prompt and 256-token reply five times back to back. Run 1 starts on a cool phone, and run 5 starts after 40 to 60 seconds of continuous generation. The table gives tokens per second for run 1 and run 5.

On the Pixel 8, Qwen3.5 lost about 30% of its speed as the fast cores dropped from 2,476 MHz to 1,607 MHz. The S26 Ultra series started on a warm phone, at 38.5 °C skin temperature, and still lost 15 to 35%. At 26 tokens per second, a 256-token answer takes about 10 seconds on a hot Pixel 8. kiln's benchmark notes recommend quoting run-5 figures for chat. A single short test reports the cool-phone speed, which users get only at the start of a session.

Memory sets the other limit. Qwen3.5 0.8B used more than twice its file size in memory on the Pixel 8, at 1,218 MB. Peak memory is roughly the model file plus the conversation cache plus compute buffers, and kiln-bench found no leaks across runs. A product that targets phones with 4 GB of RAM should start from LFM2.5 350M or Gemma 3 270M, which stayed under 700 MB.

How do you run a local LLM on an Android phone at a usable speed?

Running a local LLM on an Android phone at a usable speed depends as much on avoiding repeated work as on the model. kiln does three things here, and each one has a measured effect.

The first is thread placement. Phone chips mix fast cores with slow, power-saving ones, and the Pixel 8's Tensor G3 has one Cortex-X3, four A715s and four A510s. llama.cpp's compute library makes every thread wait at a barrier after each step, so one thread on a slow A510 holds back the others. kiln reads each core's maximum clock and pins one worker thread per fast core, leaving the A510s idle. On a warm Pixel 8, kiln measured 21 tokens per second pinned and 3 unpinned.

The second is reusing the prompt. kiln keeps the processed prompt in a cache between turns and reuses whatever matches. Hybrid models such as Qwen3.5 and LFM2 keep a running summary instead of one entry per token, so the cache cannot be cut back to any point. kiln therefore saves up to three checkpoints: after the system prompt, after the latest turn, and one turn back. In one extraction prompt of 167 tokens, 146 were fixed instructions. With the system-prompt checkpoint, the Pixel 8 read the remaining 21 tokens in 0.8 seconds, against 2.9 seconds for all 167.

The third is a budget for thinking. Qwen3.5 0.8B can reason before it answers, and on a phone that reasoning can run for minutes. kiln's max_reasoning_tokens closes the reasoning block when the budget runs out and sets a flag the app can show. With a 256-token budget, Qwen3.5 finished 43 of 48 runs. Without a limit, it finished 1 of 20 within 1,500 tokens, which is about a minute at 26 tokens per second.

Decision models on Android: one pass, no text

A decision model answers a question about a text with a probability for every option, without writing any words. kiln supports three question kinds: yes or no, one choice from a listed set, and a score on a scale of 2 to 10 levels. Because nothing is sampled, the same question always returns the same numbers. Liquid AI's d1-omni 600M runs this way, and its Hugging Face page shows it served through llama.cpp's llama-server at a /v1/systemone endpoint. kiln ports that path into a library call, so an app gets the same kind of answer without running a server on the phone.

The benchmark asked 40 yes/no, choice and score questions about one support message. The table gives milliseconds per question for run 1 and run 5.

On a hot Pixel 8, ten questions about one message take about 4 seconds. Two limits matter before building on it. kiln passes d1-omni text only, although the model itself accepts images or audio. d1-omni ships under Liquid AI's LFM Open License v1.0, not MIT, so a commercial product needs that license reviewed. We tested how a hosted decision model behaves inside a control loop in our PID controller vs AI benchmark.

Embedding models: search by meaning on the phone

An embedding model turns a text, an image or a sound into a list of numbers, so that inputs with similar meaning get similar numbers. That is the step behind search by meaning over a user's own notes and photos. EmbeddingGemma 2 places text, images and audio in one space, so a typed query can find a photo or a voice note. The table gives kiln's times per input for run 1 and run 5.

Images cost over 100 times more than notes. kiln's README reports a single photo at 2.8 seconds on a Pixel 8. The five-run series ran with the fast cores near 1,100 MHz and took 6 seconds per image. At that rate, a 1,000-photo library takes over an hour and a half on a hot Pixel 8. An app should index photos in the background, ideally while the phone charges, and index notes on demand.

Embeddings fail quietly in one way that kiln's docs call out. Many models expect a prefix such as search_query: or search_document:, and kiln adds it from the kind argument. Without the prefix, the vectors still come out, but search gets worse and nothing reports an error. Queries and documents must also come from the same model. kiln computes vectors and leaves storing and searching them to the app.

Speech to text and text to speech on Android

Speech to text on a Pixel 8 is fast enough for voice notes with Parakeet. A 3-second clip took 0.28 seconds on the CPU, and later bursts took 0.8 to 1.5 seconds once the chip reached 82 °C. Whisper base took 3.5 seconds for the same clip, because it always encodes a 30-second window. Short clips should go to Parakeet.

Text to speech on a Pixel 8 runs slower than real time. Pocket TTS took 3.4 to 4.1 seconds to produce a 2.7-second sentence. kiln delivers audio in pieces of about a second, so playback that starts at the first piece will run out of audio. An app should generate a sentence ahead of playback or accept a short pause.

kiln on Android: what is not covered yet

kiln's tested range is narrower than "Android" suggests. iOS is in progress and is not in the pub.dev package. kiln has not yet been tested on MediaTek chips, older or mid-range Snapdragon chips, phones with 4 GB of RAM, or x86 emulators. The October 2026 changelog lists, as unreleased, a fallback for phones whose CPU lacks the dot-product instruction, which used to crash. A product with a wide device range should test its lowest-end target phone before committing to a model size.

CortexTech's perspective: the thread fix that put an encoder on one core

Over at CortexTech, we built kiln and hit a problem that came from our own speed fix. Pinning worker threads to the fast cores had taken chat on a warm Pixel 8 from 3 to 21 tokens per second. Then we added image and audio embedding, and a small image took 14.3 seconds on the Pixel 8. A 3-second audio clip took 9.6 seconds. Text embedding was unaffected, which pointed away from the model and toward how the encoders used the CPU.

The cause was thread inheritance. The image and audio encoders start their own threads. On Linux, a new thread inherits a copy of its creator's CPU affinity mask, per the pthread_create manual. The encoder's threads were created from a calling thread that was pinned to one core, so all of them shared that one core. Turning pinning off would have spread them out, but it would also have brought chat back to 3 tokens per second.

The fix kept pinning and stopped the encoders from inheriting a single core. The 0.3.0 fix cut the small image to about 2.5 seconds and the audio clip to about 0.5 seconds on the Pixel 8. In 0.4.0, every engine thread inherits the calling thread's full set of fast cores, because calls run inside an affinity scope. With that rule, pinned Parakeet encoding took a steady 0.3 seconds, against 0.4 to 7 seconds unpinned. The technique we reuse: when a library pins its own threads, time every input type separately. Any thread pool started from a pinned thread inherits the pin.

Our past work includes a non-technical founder's connected home-entry device with video calling, on-device AI, and a companion app.

If your app or device needs models running on the hardware itself, measured on what you will ship, we build that at https://cortextech.dev.

Sources

• kiln_flutter on pub.dev (version 0.5.2, API examples, minSdk, license): https://pub.dev/packages/kiln_flutter
• kiln repository, read October 2026: https://github.com/RaihanulHaque/kiln
• kiln benchmark notes: https://github.com/RaihanulHaque/kiln/blob/main/docs/bench/README.md
• kiln concepts (threads, cache, decision and embedding behavior): https://github.com/RaihanulHaque/kiln/blob/main/docs/CONCEPTS.md
• kiln changelog: https://github.com/RaihanulHaque/kiln/blob/main/CHANGELOG.md
• d1-omni-600M GGUF model card (llama-server usage, /v1/systemone, license): https://huggingface.co/LiquidAI/d1-omni-600M-GGUF
• pthread_create(3) Linux manual page (inherited CPU affinity mask): https://man7.org/linux/man-pages/man3/pthread_create.3.html