Gloopy: Training a Tiny Language Model from Scratch

Gloopy is a small alien character model that answers short questions about its moon Luma, its spaceship, snacks, and discoveries on Earth. The experiment brings dataset generation, tokenizer preparation, training, export, and browser inference into one inspectable project.

The model has 8,726,016 parameters and a context window of 128 tokens. It is trained for a narrow character format. Its short, sometimes odd replies are part of the practical result of working at that scale; it has no current-events knowledge or memory of earlier messages.

Open Gloopy in your browser to try the small language model, or view the project preview. Start with one short question about Luma, then one about an everyday Earth object. Compare how the character handles familiar topics with a prompt outside its dataset's theme. That contrast makes the training scope easier to see than a model-size label alone.

Starting With a Character Dataset

The prepared dataset contains 120,000 generated single-turn conversations. The split is 108,000 training rows, 6,000 validation rows, and 6,000 test rows. An authored topic pack supplies the material from which examples are generated.

The repository includes generation, preparation, and audit tools, plus a manifest and preview. Keeping these alongside the checkpoint makes the training material visible instead of presenting the final model as an unexplained artifact.

The conversational scope is deliberately small. Consistent character examples can teach a compact model a recognizable style, but they cannot provide the breadth of a general assistant's knowledge. Gloopy's dataset is designed around short alien responses rather than open-ended expertise.

How a Conversation Becomes a Training Example

The dataset loader tokenizes each stored conversation into IDs, then builds two sequences offset by one position. If the input contains tokens A, B, and C, the corresponding targets begin B, C, and D. The task is to predict the next token at each position.

Short examples are padded when batched together, and the loss ignores padding ID zero. Longer examples are truncated to the configured limit. These details determine which parts of an authored conversation actually become training signal.

This training objective also explains a common misunderstanding: the model learns patterns in the provided token sequences. It does not acquire a separate database of verified facts about the world. The character format, vocabulary, and topic distribution all influence what replies are likely to follow a prompt.

What Fits Inside 8.7 Million Parameters

The transformer uses six layers, six attention heads, a hidden size of 384, and a vocabulary of 4,096 tokens. The training tools use PyTorch, and the repository includes a Windows launcher for configuring and recording runs.

The original run processed 648,000 examples across 10,125 steps. The recorded best validation loss is 0.4785. Those numbers describe this training run and dataset; they are not a measure of general reasoning ability.

Training, validation, and test examples are separate, but they share template families. A low validation loss can therefore indicate that the model learned this format well without establishing how it behaves on unrelated questions.

That limitation belongs beside the result. A character model can be entertaining and technically useful to study while still being poor at tasks outside its training distribution.

The Transformer in Plain Terms

The implementation combines token embeddings with learned position embeddings. Each block applies layer normalization, multi-head attention, and a ReLU feed-forward network with residual connections. The output projection shares weights with the token embedding matrix, keeping the input and output vocabulary representations tied together.

Attention uses a causal mask: a position can attend to earlier tokens and itself, while later tokens are hidden. Otherwise, next-token training could leak the answer into the input. Cross-entropy compares predicted token scores with the shifted targets.

During generation, the model repeatedly predicts one additional token from the available context. Its Python generator applies temperature and top-k sampling, then stops at the end token or the generation limit. Sampling allows different replies to the same prompt; it also means a playful answer should not be treated as a deterministic factual lookup.

Training from scratch here means learning a fresh checkpoint from the prepared dataset rather than adapting a supplied pretrained language-model checkpoint. The code and architecture still have an upstream origin, which is recorded below.

From Checkpoint to Browser Chat

The browser version exports the model to ONNX and uses ONNX Runtime Web for inference. A build script embeds the exported model, tokenizer, and logo into a single HTML file. The resulting page is approximately 15.6 MB.

Embedding the model makes distribution straightforward: the page contains the material needed for the character. It also means the model is downloadable by anyone who receives the public page. There is no hidden server-side model behind the chat interface.

The first load fetches the inference runtime from a CDN. Replies are then generated on the visitor's device, without a chat backend receiving the conversation. This is browser inference, with an initial network dependency for runtime assets.

The site builder checks that the exported model, checkpoint, and tokenizer match. That consistency matters because a tokenizer mismatch can make an otherwise valid model file produce meaningless results.

The Browser Is Part of the Inference Pipeline

The runtime first needs a compatible exported graph, the correct tokenizer, and a ready inference session. The interface validates tokenizer behavior before allowing unreliable replies, sanitizes input, and checks the prompt's token budget. It limits generation to the remaining context space rather than letting a long prompt silently crowd out the answer.

The template also updates the answer as generated text becomes available and provides retry actions for failures. A repeated-token guard limits degenerate output. These behaviors turn model execution into an interaction a visitor can follow instead of a blank wait followed by a mysterious string.

The ONNX Runtime Web documentation explains browser inference and execution-provider options. Gloopy's export and page builder form the project-specific bridge between the PyTorch checkpoint and that browser runtime.

There are separate startup and response costs. Downloading the embedded model and initializing the runtime happen before useful inference can begin; generating tokens costs work after each prompt. Measuring both would reveal whether a slow first interaction is mostly asset loading or model execution.

A Short Exchange, Not a Persistent Conversation

Gloopy processes single-turn exchanges. The interface can display earlier messages, but the model does not remember them as conversational context. A follow-up question that depends on a previous answer may therefore be confusing to it.

The 128-token context also limits how much information can be supplied at once. Keeping prompts and replies short fits the experiment better than treating the chat surface as a promise of unrestricted conversation.

These constraints make the model easier to understand. The complete pipeline can be inspected locally, and the browser page exposes the result without requiring a hosted inference service.

Giving a Small Model an Approachable Interface

The character and suggested prompts provide an entry point for people who do not know what to ask. The page includes clickable prompt suggestions and a Say that again action. Those controls support experimentation within the model's scope, while the input-budget check gives a specific explanation when a message is too long.

The important design boundary is expectation. The visible transcript is useful for reading the exchange, but it is not evidence that earlier turns are supplied to the model. A person can still enjoy a sequence of short character responses if that limitation is clear.

I would evaluate this experience by asking visitors to describe what they believe Gloopy remembers, then observing whether they can start a prompt, understand loading, and recover from an error. Response latency, abandoned first loads, and repeated-prompt behavior would be useful measurements. No user-study or retention result is claimed here.

For a small experiment, ask the same self-contained question a few times. Compare character consistency, wording variety, repetition, and whether the reply stays on topic. Then try a follow-up that relies on an earlier message. The difference illustrates the gap between a visible chat history and the context actually supplied to inference.

Provenance and the Project Archive

Gloopy's model and training code derive from GuppyLM. The repository records that provenance separately from its generated character dataset and supplied visual assets. Training a fresh checkpoint does not erase the origins of the architecture and tooling.

For another example of running models on a visitor's device, read the Pyvi toolkit article. Its transcription and translation tools use downloaded pretrained models for specific tasks, while Gloopy explores the full path from a generated dataset to a newly trained character checkpoint.

The Gloopy project page includes its desktop chat preview and implementation summary. The live Gloopy website is hosted on GitHub Pages, with model inference running in the visitor's browser.

Gloopy makes a small model concrete: a prepared dataset becomes a trained checkpoint, the checkpoint becomes an exported graph, and that graph becomes a character answering from inside a browser tab.