Get live statistics and analysis of Andi Marafioti's profile on X / Twitter

🚀We just dropped SmolDocling: a 256M open-source vision LM for complete document OCR!📄✨ It's lightning fast, process a page in 0.35 sec on consumer GPU using < 500MB VRAM⚡ SOTA in document conversion, beating every competing model we tested up to 27x larger🤯 But how? 🧶⬇️

📢 A new open-source OCR model is breaking the internet: Nanonets-OCR-s! Nanonets understands context and semantic structures, transforming documents into clean, structured markdown. It has an Apache 2.0 license, and the authors compare it to Mistral-OCR 🧵 Let's look closer:

Fuck it, today we're open-sourcing the codebase used to train SmolVLM from scratch on 256 H100s🔥 Inspired by our team's effort to open-source DeepSeek's R1 training, we are releasing the training and evaluation code on top of the weights 🫡 Now you can train any of our SmolVLMs—or create your own custom VLMs!

Introducing the smollest VLMs yet! 🤏 SmolVLM (256M & 500M) runs on <1GB GPU memory. Fine-tune it on your laptop and run it on your toaster. 🚀 Even the 256M model outperforms our Idefics 80B (Aug '23). How small can we go? 👀

Fuck it. Today, we open source FineVision: the finest curation of datasets for VLMs, over 200 sources! > 20% improvement across 10 benchmarks > 17M unique images > 10B answer tokens > New capabilities: GUI navigation, pointing, counting FineVision 10x’s open-source VLMs.

Everyone hypes new OCR models, but olmOCR quietly updates every few months, stays SOTA, and costs $178 per 1M pages. Don’t skip it—it even beats DeepSeek-OCR

Funny enough, a few years ago I was shopping for data labeling companies with a 100k/year budget and did a POC with Scale and a few others vendors. They were by far the worst. I thought our contract just wasn’t big enough for them to care, but maybe that was just their standard.

Mixing vision and robotics is incredibly hard, but when it finally works it feels like magic.

Just read the Qwen2.5-Omni technical report from the Qwen team, it's super interesting. Here are my notes. Qwen2.5-Omni is a unified end-to-end model that can perceive text, images, audio, and video — and generate both text and natural speech responses in a streaming fashion. At its core is the Thinker-Talker architecture: Thinker: a large language model that processes multimodal inputs and generates text. Talker: an autoregressive speech decoder that turns Thinker's hidden states into speech tokens. They're trained together, end-to-end. Handling audio: audio is converted to 128-channel mel-spectrograms (16kHz, 25ms window, 10ms hop). Encoded via a modified Whisper model. Audio is processed in 2s blocks with streaming-compatible attention to reduce latency. Handling video: uses a ViT-based encoder with dynamic frame sampling. Each frame is treated like an image. To sync with audio, they introduce TMRoPE — Time-aligned Multimodal RoPE — a novel positional embedding that aligns video and audio in time. TMRoPE splits positional encoding into temporal, height, and width axes, letting Qwen2.5-Omni represent image/video/audio/text all on the same timeline. Interleaving of audio and visual tokens every 2 seconds enables synchronized fusion. Streaming audio generation: audio tokens from Talker are decoded using a sliding-window DiT model + modified BigVGAN. The receptive field includes 2 lookback blocks and 1 lookahead to allow context-aware streaming audio generation. Pretraining involved locking the LLM and training the audio/vision encoders first. Later stages unfreeze everything and train on a massive mix of audio-text, video-text, image-text, and long-sequence (32k tokens) data. Post-training includes reinforcement learning for Talker to reduce hallucinations and improve pronunciation/timing. Plus, multi-speaker fine-tuning for better prosody and naturalness. Qwen2.5-Omni achieves SOTA on OmniBench, AV-Odyssey, and strong results across text, image, audio, and video tasks. End-to-end speech instruction following is nearly on par with text-based inputs. That's rare. Overall: a super ambitious and well-integrated multimodal model. The Thinker-Talker separation is elegant. TMRoPE is a clever solution to a tricky problem. That said, I wish the paper had included more ablation studies or experiments justifying some of the architectural decisions. Many claims are reasonable but would benefit from more empirical evidence. Still, major kudos to the team. Qwen2.5-Omni is a big step toward real-time, unified multimodal assistants.

Let's go! We are releasing SmolVLM, a smol 2B VLM built for on-device inference that outperforms all models at similar GPU RAM usage and tokens throughputs. SmolVLM can be fine-tuned on a Google collab and be run on a laptop! Or process millions of documents with a consumer GPU!

🚨 New paper out! “FineVision: Open Data Is All You Need” 🥳 We unified 200+ data sources into 24M samples. That’s 17.3M images and 9.5B answer tokens, the largest open VLM dataset ever released. All fully documented, reproducible, and available for everyone. And there's more! 🎢

🚀 We're thrilled to launch four new OCR datasets with 20M images: DoclingMatix, SynthFormulaNet, SynthCodeNet, and SynthChartNet. We used them train SmolDocling, our ultra‑compact (256M) full-page document conversion VLM with performance rivaling models up to 27× larger.

A warm welcome to Moonshine, a new family of speech-to-text models! Moonshine claims to be as fast and accurate as whisper-base, while being up to 5x faster! 🤯 They achieve this by removing whisper's constraint on 30-second length audios. Instead, Moonshine processes audios of any length. 🧠 I will be trying this with our speech-to-speech pipeline and reporting back the results. So far, I'm sad not to see an MLX native implementation.

Today, we share the tech report for SmolVLM: Redefining small and efficient multimodal models. 🔥 Explaining how to design a tiny 256M VLM that uses less than 1GB of RAM and outperforms our 80B models from 18 months ago! Here are the coolest insights from our experiments: ✨ Longer context = Big wins: Increasing the context length from 2K to 16K gave our tiny VLMs a 60% performance boost! ✨ Smaller is smarter with SigLIP: Surprise! Smaller LLMs didn't benefit from the usual large SigLIP (400M). Instead, we use the 80M base SigLIP that performs equally well at just 20% of the original size! ✨ Pixel shuffling magic: Aggressively pixel shuffling helped our compact VLMs "see" better, achieving the same performance with sequences 16x shorter! ✨ Learned positional tokens FTW: For compact models, learned positional tokens significantly outperform raw text tokens, enhancing efficiency and accuracy. ✨ System prompts and special tokens are key: Introducing system prompts and dedicated media intro/outro tokens significantly boosted our compact VLM’s performance—especially for video tasks. ✨ Less CoT, more efficiency: Turns out, too much Chain-of-Thought (CoT) data actually hurts performance in small models. They dumb ✨ Longer videos, better results: Increasing video length during training enhanced performance on both video and image tasks. 🌟 State-of-the-Art Performance, SmolVLM comes in three powerful yet compact sizes—256M, 500M, and 2.2B parameters—each setting new SOTA benchmarks for their hardware constraints in image and video understanding. 📱 Real-world Efficiency: We've created an app using SmolVLM on an iPhone 15 and got real-time inference directly from its camera! 🌐 Browser-based Inference? Yep! We get lightning-fast inference speeds of 40-80 tokens per second directly in a web browser. No tricks, just compact, efficient models! If you’re into efficient multimodal models, you’ll love this one.

Sneak peek of Reachy Mini's conversation capabilities! He can chat fluidly and change languages naturally. You can interrupt him and he can see and react to his environment. They start shipping soon, I'm excited to see what people start building with him!

🚀We just dropped SmolDocling: a 256M open-source vision LM for complete document OCR!📄✨ It's lightning fast, process a page in 0.35 sec on consumer GPU using < 500MB VRAM⚡ SOTA in document conversion, beating every competing model we tested up to 27x larger🤯 But how? 🧶⬇️

Mixing vision and robotics is incredibly hard, but when it finally works it feels like magic.

Sneak peek of Reachy Mini's conversation capabilities! He can chat fluidly and change languages naturally. You can interrupt him and he can see and react to his environment. They start shipping soon, I'm excited to see what people start building with him!

Fuck it, today we're open-sourcing the codebase used to train SmolVLM from scratch on 256 H100s🔥 Inspired by our team's effort to open-source DeepSeek's R1 training, we are releasing the training and evaluation code on top of the weights 🫡 Now you can train any of our SmolVLMs—or create your own custom VLMs!

I heard X likes ML interview questions. So here's one: what's wrong with this PyTorch DataLoader worker seed?

Funny enough, a few years ago I was shopping for data labeling companies with a 100k/year budget and did a POC with Scale and a few others vendors. They were by far the worst. I thought our contract just wasn’t big enough for them to care, but maybe that was just their standard.

Fuck it. Today, we open source FineVision: the finest curation of datasets for VLMs, over 200 sources! > 20% improvement across 10 benchmarks > 17M unique images > 10B answer tokens > New capabilities: GUI navigation, pointing, counting FineVision 10x’s open-source VLMs.

📢 A new open-source OCR model is breaking the internet: Nanonets-OCR-s! Nanonets understands context and semantic structures, transforming documents into clean, structured markdown. It has an Apache 2.0 license, and the authors compare it to Mistral-OCR 🧵 Let's look closer:

Let's go! We are releasing SmolVLM, a smol 2B VLM built for on-device inference that outperforms all models at similar GPU RAM usage and tokens throughputs. SmolVLM can be fine-tuned on a Google collab and be run on a laptop! Or process millions of documents with a consumer GPU!

I came up with a technique for dynamic token selection in Vision-Language Models. Instead of wasting compute on every part of an image, this method adapts the number of tokens based on the complexity of each region. Here’s an example of how it works: 👇

Everyone hypes new OCR models, but olmOCR quietly updates every few months, stays SOTA, and costs $178 per 1M pages. Don’t skip it—it even beats DeepSeek-OCR

Introducing the smollest VLMs yet! 🤏 SmolVLM (256M & 500M) runs on <1GB GPU memory. Fine-tune it on your laptop and run it on your toaster. 🚀 Even the 256M model outperforms our Idefics 80B (Aug '23). How small can we go? 👀

What would you like the Hugging Face multimodal team to release for the community? Is there anything you feel is missing to train truly great multimodal models?

You can now train SOTA models without any storage!🌩️ We completely revamped the Hub’s backend to enable streaming at scale. We streamed TBs of data to 100s of H100s to train SOTA VLMs and saw serious speed-ups. But how?

New ML interview question: I have a training job that fails 1/5 times when I launch with 64 parallel jobs. This is the error. What is happening? (Feel free to check the files in nanoVLM, this is not a drill)