Get live statistics and analysis of Sebastian Raschka's profile on X / Twitter

When I started LLMs-from-scratch I just hoped it might help a few people learn. Just saw the GitHub the repo has now been forked 10k times! More than the stars, the best part is seeing thousands of people actually use and build on the code ☺️

Kimi K2 is basically DeepSeek V3 but with fewer heads and more experts:

Maybe a hot take, but what about the following advice to the next gen: Don't get an AI degree; the curriculum will be outdated before you graduate. Instead, study math, stats, or physics as your foundation, and stay current with AI through code-focused books, blogs, and papers.

I’ve been working on something new: 📚 Build a Reasoning Model (From Scratch). The first chapters just went live! (The book will cover topics from inference-time scaling to reinforcement learning)

Updated & turned my Big LLM Architecture Comparison article into a narrated video lecture. The 11 LLM architectures covered in this video: 1. DeepSeek V3/R1 2. OLMo 2 3. Gemma 3 4. Mistral Small 3.1 5. Llama 4 6. Qwen3 7. SmolLM3 8. Kimi 2 9. GPT-OSS 10. Grok 2.5 11. GLM-4.5

Just shared a new article on "The State of Reinforcement Learning for LLM Reasoning"! If you are new to reinforcement learning, this article has a pretty generous intro section (PPO, GRPO, etc) I addition, I cover 15 recent articles focused on RL and Reasoning Models,

So, I did some coding this week... - Qwen3 Coder Flash (30B-A3B) - Mixture-of-Experts setup with 128 experts, 8 active per token - In pure PyTorch (optimized for human readability) - in a standalone Jupyter notebook - Runs on a single A100

Hey everyone! I just wanted to say how incredible this journey here has been over the past 12 years, when I started sharing and creating Python, ML, and AI content. The positive feedback I’ve received from you all has been such a powerful motivator. I had some exciting plans lined up for the coming weeks, including a comprehensive yearly review and outlook of AI research. Unfortunately, I suffered a serious injury that’s made it impossible for me to use a computer for the last few weeks. Unfortunately, I’ll need to take a break from content creation while I focus on recovery. But no worries, I’m optimistic about making a full recovery and can’t wait to get back to creating and sharing more AI research content in the future. Thank you for your understanding and support over the years!

Usually, I post AI content and not personal life updates, but it somehow helps to get this off my chest. If I haven't been responding to messages, emails, or PRs, it's not because I don't care or don't want to. It's just physically hard at the moment, even with aids like speech-to-text. Life can be brutal sometimes. I was just getting back on my feet in March after a back injury that took 3 months to heal. I was so excited to be back coding and writing. And I had so many plans for this year. AI is a fast-moving field! And most importantly, I was hoping for this to be my last ER and hospital visit in life (or at least the next few decades), but no, life had other plans, and I have been dealing with a bad neck injury for the past 3 weeks. I have been seeing multiple doctors to get multiple opinions, and unfortunately, it may take 1–6 months until I can expect an improvement and start healing phase. So many talks and projects to cancel. And the worst feeling is the fear of missing out. Missing out on life and doing normal things like going for a hike on a beautiful spring day. And missing out on all the exciting AI developments and the missed project opportunities. I am working hard to get back on my feet as soon as possible or at least do some things even if it means I can do less and have to do things more slowly. I have done it before and can do it again!

Maybe a hot take, but what about the following advice to the next gen: Don't get an AI degree; the curriculum will be outdated before you graduate. Instead, study math, stats, or physics as your foundation, and stay current with AI through code-focused books, blogs, and papers.

I’ve been working on something new: 📚 Build a Reasoning Model (From Scratch). The first chapters just went live! (The book will cover topics from inference-time scaling to reinforcement learning)

Here’s the 2025 LLM roadmap 😊 1. Code and train your own LLM to really understand the fundamentals 2. Train models more conveniently using production-ready libraries 3. Learn about the big-picture considerations for real-world LLM/AI apps

Kimi K2 is basically DeepSeek V3 but with fewer heads and more experts:

DeepSeek finally released a new model and paper. And because this DeepSeek-OCR release is a bit different from what everyone expected, and DeepSeek releases are generally a big deal, I wanted to do a brief explainer of what it is all about. In short, they explore how vision encoders can improve the efficiency of LLMs in processing and compressing textual information. And the takeaway is that rendering text as images and feeding that to the model results in more efficient compression than working with text directly. My first intuition was that this sounds very inefficient and shouldn't work as well as using text tokenizers (or alternatives like Byte Latent Transformer) to prepare the input. It actually reminded me of the line of research I saw years ago, where researchers represented 3D molecules as 3D inputs or 2D images for ConvNets instead of using graph neural nets. This shouldn't work well and should be prone to overfitting. In the case of DeepSeek-OCR, why even try such an approach? I imagine it started as a curiosity, but then it may have turned into an interesting idea for long-context scaling in LLMs and how to make it cheaper by using vision tokens and representations. (An image can say more than a thousand words, but who would have thought that an image of text can say 1000 words more efficiently!) In any case, this DeepSeek-OCR approach turns out to be surprisingly efficient. In particular, they found that at a fixed  precision of 97% for long-context decoding (i.e., how well the model can compress information into a latent representation and reconstruct it), the OCR version needed 10 times fewer visual tokens than text tokens. In other words, the OCRed version can compress information 10x better than the text version. How is it different compared to other VLLM architectures? - They don't use a monolithic ViT as encoder, instead they fuse local and global vision features through a clever 16x convolutional compressor (this can handle high-resolution inputs efficiently in terms of memory and token counts). - They are (to the best of my knowledge) those who use an MoE as a decoder. I think it's an interesting, refreshing approach, and the twist here is that it works surprisingly well. However, I don't think that visual representations of text will solve the limitations of LLMs. Also, while it is popular to dislike text tokenizers like BPE, image representations are messy as well (one has to deal with aspect ratios, resolutions, croppings, color intensity variations, brightness levels, etc.). Still, it’s an interesting idea. Also, if this approach is more efficient than regular black&white text, I am curious to see compression ratios when we add syntax color to code. Regarding code, this may be an interesting alternative for storing contextual information, as spacing and subword tokenization remain challenges in traditional tokenizers. (Especially when working with code that uses many custom variable names that may not be represented in vocabularies and that have to be broken down into many individual subword tokens.) Overall, it’s still such an esoteric concept to encode text in images that I am (still) surprised it could do well (and may it would only make sense for very long documents or special domains like OCR or code, not general language modeling). (PS: Personally, I expected the DeepSeek team to follow up with a V4 model using the sparse attention mechanism they tried in V3.2 recently, but maybe that's still forthcoming. Now, after reading this paper, V4 is perhaps going to be a VLLM.)

Can there be too much of a good thing? The 7-day view so far: - Grok 3 was released - Claude 3.7 was released - DeepSeek shipped some awesome stuff in their open-source week - DeepSeek announced to release R2 soon - OpenAI made Deep Research free for Plus users - Google is made their Gemini Code Assist free - OpenAI rolled out GPT-4.5

From the Hierarchical Reasoning Model (HRM) to a new Tiny Recursive Model (TRM). A few months ago, the HRM made big waves in the AI research community as it showed really good performance on the ARC challenge despite its small 27M size. (That's about 22x smaller than the smallest Qwen3 0.6B model.) Now, the new "Less is More: Recursive Reasoning with Tiny Networks" paper proposes Tiny Recursive Model (TRM), which a simpler and even smaller model (7M, 4× smaller than HRM) that performs even better on the ARC challenge. 🔹 What does recursion mean here? TRM refines its answer in two steps: 1. It updates a latent (reasoning) state from the current question and answer. 2. Then it updates the answer based on that latent state. Training runs for up to 16 refinement steps per batch. Each step does several no-grad loops to improve the answer, followed by one gradient loop that learns from the full reasoning process. By the way, the question and the answer are grids of discrete tokens, not text. (E.g., 9×9 Sudoku and up to 30×30 ARC and Maze.) 🔹 And how does it differ from HRM? In short, HRM recurses multiple times through two small neural nets with 4 transformer blocks each (high and low frequency). TRM is much smaller (i.e., 4x) and only a single network with 2 transformer blocks. TRM backpropagates through the full recursion once per step, whereas HRM only backpropagates through the final few steps. And TRM also removes HRM's extra forward pass for halting and instead uses a simple binary cross-entropy loss to learn when to stop iterating. 🔹 Surprising tidbits 1. The author found that adding layers decreased generalization due to overfitting. And going from 4 to 2 layers improved the model from 79.5% to 87.4% on Sudoku. 2. Replacing the self-attention layer with an MLP layer also improved accuracy (74.7% -> 87.4% on Sudoku); however, note that this only make sense here since we have a fixed-length, small context to work with. 🔹 Bigger picture My personal caveat: comparing this method (or HRMs) to LLMs feels a bit unfair since HRMs/TRM are specialized models trained for specific tasks (here: ARC, Sudoku, and Maze pathfinding) while LLMs are generalists. It’s like comparing a pocket calculator to a laptop. Both serve a purpose, just in different contexts. That said, HRMs and the recursive model proposed here are fascinating proof‑of‑concepts that show what’s possible with relatively small and efficient architectures. I'm still curious what the real‑world use case will look like. Maybe they could serve as reasoning or planning modules within a larger tool‑calling system. In practice, we often start by throwing LLMs at a problem, which makes sense for quick prototyping and establishing a baseline. But I can see a point where someone sits down afterward and trains a focused model like this to solve the same task more efficiently.