I trained my own Vision-Language Model under 1B parameters

Introduction

What is it actually like to build and train your own Vision-Language Model nowadays? 👀

To find out, I took a base language model, plugged in a vision backbone and trained it for a few days on a single GPU rented on RunPod, until it could describe images.

Small, good vision encoders and language models are easy to download; what is usually missing is the recipe to glue them together. A few works showed the way:

• SmolVLM-500M^[1] from Hugging Face showed what it takes to build a fully reproducible, small and competitive vision-language model, and the Train Your Own Encoder-Free VLM in $100^[2] blog post is a complete single-GPU recipe, budget included
• nanoVLM^[3] is a minimal, educational library: you choose the pieces and the framework does the rest
• LFM2.5-VL-450M^[4] showed what a vision-language family can do at this size

This post redoes something that works, entirely "in house" and with open-source models, references and techniques only. It is a recipe you can follow and an honest log of everything I got wrong along the way. Code and model weights are open on GitHub and Hugging Face 🤗

AI Assistant Note: Claude has been used throughout the development, especially for brainstorming and project setup, evaluation and wrapping up results. The relevant files and the decisions we made together are in the codebase.

Motivation: a model I liked that could not see

The idea came from Baguettotron^[5], a text-only model, and as much from the way Pleias presented it as from the model itself: data, weights, training story and design choices, all out in the open. That is rarer than it should be, and it is the kind of release I wanted to build on.

Baguettotron is a 321M-parameter Small Reasoning Model, unusual in two ways. Its shape: 80 layers at hidden size 576, which makes it, in its authors' words, the deepest SLM in its size range (the "baguette" of the name). And its data: 200 billion tokens of SYNTH^[6], a fully synthetic generalist dataset seeded from 50,000 vital Wikipedia articles, with no scraped web data. In other words, you can, in principle, read this model's entire pretraining corpus, a different category from models whose only disclosure is "trained on a large corpus of internet text".

The one thing it could not do was see. So I gave it eyes 👀 Here is a real exchange with the finished model, greedy decoding, running on a laptop:

user: Describe the image.

assistant: The image is a picture of two cats sitting on the couch. The cat has its head down and legs are raised.

Two cats, a couch and a posture, from a 628M-parameter model that cost about $200 to train and answers in about a second on a laptop 🎉 It is not going to write your alt text at scale, but it is looking at the picture, and every step of how it got there is open.

Architecture

Image (448×448)
  → InternViT-300M-448px-V2.5 (304M, frozen) → 1024 tokens × 1024d
  → Pixel unshuffle (factor=2)                → 256 tokens × 4096d
  → MLP projector (2-layer, ~2.7M)            → 256 tokens × 576d
  → interleave with text tokens
  → Baguettotron (321M, 80 layers, h=576)

Total: ~628M parameters

I used the "standard" ViT–MLP–LLM paradigm as formulated by InternVL^[7]: a frozen vision encoder, a small randomly initialised MLP that projects visual features into the language model's embedding space, and the LLM itself.

Encoder and Projector 🔭

I picked InternViT-300M-448px-V2.5^[8] over a general-purpose CLIP or SigLIP encoder, because of a warning its own authors put on the model card: "In our experience, the InternViT V2.5 series is better suited for building MLLMs than traditional computer vision tasks."

The encoder turns a 448×448 image into 1024 patch tokens: too many for an 80-layer language model at this budget, and enough to make a short question a rounding error. A pixel unshuffle with factor 2 folds each 2×2 neighbourhood into one token with 4× the channels, so 1024 tokens × 1024d become 256 tokens × 4096d: no parameters, just a reshape that trades spatial resolution for channel depth. The projector (two linear layers and a GELU, ~2.7M parameters trained from random init) then maps 4096 → 576, Baguettotron's hidden size.

Nothing original here: SmolVLM^[1:1], the closest published model to mine, inherits this compression from Idefics3^[9], and InternVL^[8:1] applies it for the same reason: "we applied a pixel unshuffle operation, reducing the number of visual tokens to one-quarter of the original". Everything except the projector is a pretrained component someone else published, which is the only reason this fits in the budget.

Training

Three stages, all on one H100 SXM rented from RunPod, ~$200 all in.

Stage 1 — teaching visual tokens to align with the language model's space

The projector starts from random weights, so at first it feeds the language model noise. Stage 1 freezes both the ViT and the LLM and trains only the projector's 2.7M parameters on 595K image–caption pairs from LLaVA-CC3M, following the two-stage recipe LLaVA established^[10]. The point is alignment, not capability: teaching one small module to emit vectors the language model already knows how to read.

Stage 2 — teach the model to answer

Now the projector and the full language model train together on The Cauldron^[11] (47 subsets of visual instruction data), mixed with 10% text-only SYNTH to avoid catastrophic forgetting. Two decisions mattered:

Keeping the ViT frozen. With 324M trainable parameters and 60 hours, any budget spent on the vision encoder would have meant less of everything else. The encoder was already good; the mapping into my LLM was the bottleneck.

Pre-filling </think>. Baguettotron natively wants to emit a reasoning trace, and The Cauldron contains none. Rather than fight the backbone, every assistant turn in training begins with a closing </think> token, telling the model, in its own vocabulary, "no thinking this time, just answer". The processor does the same at inference, so the two match. My first run skipped it and did not work at all, most likely because the model drifted too far from the distribution Baguettotron was trained on. Ouch ❤️‍🩹

Stage 3 — teach the model to think about what it sees

To bring back the reasoning that The Cauldron had suppressed, I ran SFT on ~150K correct-only samples from R1-Vision-Reasoning-Instructions, which carry DeepSeek-R1-style^[12] <think> traces about images. The model would have needed far more data and training time to actually learn that behaviour: as the results below show, the Stage 3 model hallucinated most, if not all, of the time while reasoning 😅

Evaluation and results

I wanted a fast, qualitative check of whether the model at least understands what an image depicts. I also pushed it a little further with some "complex" questions and reasoning, and failed miserably: anything of real quality beyond plain description needs more training, which was out of scope here.

For Stages 1 and 2 I judged a few COCO^[13] images by eye; for Stage 2 vs Stage 3 I used RealWorldQA^[14] with a vision model as judge (details below).

Stage 1 vs Stage 2

Four photos, same prompt for both checkpoints:

	alignment (stage 1)	vision-vqa (stage 2)
	a cat is sleeping on the couch	The image is a picture of two cats sitting on the couch. The cat has its head down and legs are raised.
	the bear is a good friend.	The image shows a bear sitting on the grass.
	a sign for a stop	A sign that says stop.
	the bus is a red double - decoration	The red bus is on the road.

Same images, same prompt, greedy decoding. Verbatim output from both checkpoints.

Stage 1 gives short, barely meaningful captions (the bear is "a good friend" 🐻); Stage 2 writes longer, more factual ones. That is what the two stages are for. Stage 1 only trained 2.7M parameters, so it learns what is in the picture and almost nothing about finishing a sentence; Stage 2 unfreezes the language model, and grammar comes back with it.

Push Stage 2 past a short answer, though, and it starts inventing:

user: Describe the image in detail.

assistant: In this image, two cats are sitting on a white couch. They have their ears covered and the other cat is wearing a light flashlight.

The couch is pink, nobody's ears are covered, and the "light flashlight" is a TV remote lying next to the cats: the model sees that something elongated is there, cannot resolve what, and makes up an object and a relationship to go with it. The sentence is grammatical, well structured and about the right objects, just wrong in the details, which is the hardest kind of wrong to catch automatically 🫠

Stage 2: more "complex" questions

What happens if we actually ask it something? Three questions that work and three that do not, roughly the hit rate I would expect from it:

Bear. is the model doing exactly its job. STOP. surprised me, though I cannot tell whether it read the letters or recognised a red octagon, and the bus route number (real OCR on small text) comes back as Yes., which suggests the latter.

The failures are more interesting. No. about a photo of two sleeping cats is not a near miss, and What kind of animal is this?, which works perfectly on the bear, returns Yes. on the cats. So it is not that the model cannot see cats: its answer depends alarmingly on how you phrase the question 🙃

Stage 2 vs Stage 3

Six held-out RealWorldQA images, four question types and four decoding presets per stage mode, each answer judged 1–5 by Claude Opus 5: 244 of 288 cells before I stopped the run. Small enough that everything below is directional, not conclusive.

Roughly three standard errors apart: the reasoning SFT made the model more fluent and less correct. At 628M the traces are undertrained for images, so they mostly add confident, well-structured hallucination. Fluency up 📈, grounding down 📉. The decoding presets made no real difference: greedy, conservative, creative and a Qwen-like configuration scored 2.17 / 1.88 / 1.88 / 1.88, with a standard error around 0.25–0.30. That is noise, so I ship greedy because it is deterministic.

Given these results, I released only the Stage 2 model. Stage 3's code is in the repo and fully runnable, and I can share its weights if anyone is interested, but that model is mostly unusable as of today.

Conclusions 🎁

You can build a "working" Vision-Language Model on your own for about the price of a nice dinner for four. A few years ago this was a research-lab project. Now that I know what I would not do again, I could probably halve the price.

My best takeaways:

• Keep the well-trodden architecture. ViT–MLP–LLM is documented and it works. Spend your originality budget on the data and the evaluation, not on the wiring.
• Rebalance the vision side. I would use a smaller vision backbone and a higher input resolution. My guess is that training would be easier and faster, and the model would use more of the information actually present in the images.
• Skip the reasoning stage at this budget. Recovering that behaviour needs far more than I spent: either the LLM is trained for it from the start or, as in my case, you stop at Stage 2. It was nice to give it a try either way.

Good descriptions are a few hours and a few dollars away, while a genuinely competitive model, even at this size, needs considerably more data and resources than a few days on a rented GPU. Still, it was fun to take a language-only model through the whole process and get coherent image descriptions out of it.

Update 4 November 2026

Two days ago Takehi, Teoh, Zhang and Khattab published Sparks of Weight-Space Programming^[15]: instead of training a network, you change it by reading its activations and writing its weights directly, with no gradients and no training loop. Inspired by it, I tried the same on this model with Claude Opus 5.5 at max effort.

As their own control experiment warns, "Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked." Most of this model's failures (counting, reading text, multiple choice) are missing abilities, and no edit can add them. A couple of things could be fixed, touching 0.0014% of the weights:

• The stop token. My training loss masked <|im_end|>, so the model never learned to emit it and spelled it out instead. Swapping 4 rows of the output layer makes the model's own .< and <| land on the real stop tokens, so plain generate() now stops by itself.
• Object presence and loops. The activations already carry whether an object is in the picture, but the Yes/No answer ignored it. Two neurons in the last layer now pass that evidence on, at a small cost on other yes/no questions, and two more end long answers once they start repeating themselves.

Measured on images and questions never used to build the edits (fp32):

The GQA row is the price: the presence gate also opens on attribute questions like "Is the car red?", where there is no evidence to read, and pushes those answers towards "No".

On this post's own examples, the model gives more or less the same answers as before, so its overall behaviour did not change. The one exception is the question I cared about: asked Is there a cat in the picture? about the two sleeping cats, it now answers Yes 🎉

The edited weights are on Hugging Face as v1.1; the original release stays available as revision="v1.0". A possible follow-up: narrow the task to something really specific, with a dedicated dataset, and use Claude Fable for it.

References