Get live statistics and analysis of Ross Taylor's profile on X / Twitter

I am the first author of the Galactica paper and have been quiet about it for a year. Maybe I will write a blog post talking about what actually happened, but if you want the TLDR: 1. Galactica was a base model trained on scientific literature and modalities. 2. We approached it with a number of hypotheses about data quality, reasoning, scientific modalities, LLM training, that hadn’t been covered in the literature - you can read about these in the paper. 3. For its time, it was a good model for its domain; outperforming PaLM and Chinchilla with 10x and 2x less compute. 4. We did this with a 8 person team which is an order of magnitude fewer people than other LLM teams at the time. 5. We were overstretched and lost situational awareness at launch by releasing demo of a *base model* without checks. We were aware of what potential criticisms would be, but we lost sight of the obvious in the workload we were under. 6. One of the considerations for a demo was we wanted to understand the distribution of scientific queries that people would use for LLMs (useful for instruction tuning and RLHF). Obviously this was a free goal we gave to journalists who instead queried it outside its domain. But yes we should have known better. 7. We had a “good faith” assumption that we’d share the base model, warts and all, with four disclaimers about hallucinations on the demo - so people could see what it could do (openness). Again, obviously this didn’t work. 8. A mistake on our part that didn’t help was people treated the site like a *product*. We put our vision etc on the site, which misled about expectations. We definitely did not view it as a product! It was a base model demo. 9. Pretty much every LLM researcher I’ve talked to (including at ICML recently) was complimentary about the strength of the research, which was sadly overshadowed by the demo drama - yes this was our fault for allowing this to happen. 10. Fortunately most of the lessons and work went into LLaMA 2; the RLHF research you see in that paper is from the Galactica team. Further research coming soon that should be interesting. It’s a bit of a riddle because on the one hand the demo drama could have been avoided by us, but at the same time the “fake science” fears were very ridiculous and despite being on HuggingFace for a year, the model hasn’t caused any damage. To reiterate: the anti-Galactica commentary was really stupid, however we should not have allowed that to even happen if we had launched it better. I stick by the research completely - and even the demo decision, which was unprecedented openness for a big company with an LLM at the time, wasn’t inherently bad - but it was just misguided given the attack vectors it opened for us. Despite all the above, I would do it all again in a heartbeat. Better to do something and regret, then not do anything at all. Still hurts though! 🙂

Supplementary information for the new DeepSeek R1 Nature paper is very interesting! Details on training data, hyperparameters, base model importance, and more.

No one is saying RL didn’t work for reasoning. The argument is about internal reasoning emergence, not absolute performance boosts with RL. Quite the opposite in fact - we had PPO on Llama 2 base models 2 years ago with verifiable rewards and had 90%+ on GSM8k. We already knew that RL worked. Hell we even did this with the bronze-age Galactica model and it also worked (see my ICML talk 3 years ago). What didn’t “work” was that we didn’t see emergence of longer traces with backtracking, checking, error-correction and other branching-like behaviour. So when people say “the base model wasn’t strong enough”, the contention is that there wasn’t enough knowledge in the base model, subsequent RL compute and context window length for the model to develop long CoT, internal reasoning on its own. That’s the argument, not whether “RL worked”. It always did, but the magic seems to have emerged relatively recently.

It’s funny that people on this site think major LLM efforts are talent-bound rather than org-bound. The talent differential has never been big between major orgs. Most of the difference in outcomes is due to organisational factors - like allocating compute to the right bets, and letting good research and engineering triumph over destructive politics. This makes for a less sexy story though. People prefer to believe that breakthroughs are made by lone geniuses - instead of the cumulative effort of many nameless, social media averse people — supported by an org that allows the best ideas to win and manages big egos. If you don’t believe me - then consider how some researchers suddenly gain or lose impact and productivity when they switch orgs. Was it because they gained or lost IQ points? 🙂 (Sorry, this is super obvious to anyone who’s actually worked in these labs - but you wouldn’t believe it based on the X feed right now!)

Why are LLMs bad at reasoning? One theory says this is due to weaknesses in maximum likelihood, where the probability mass “overgeneralises” to low quality solutions. Because our pretraining objective (likelihood) doesn’t transfer to our evaluation objective (accuracy), the theory goes that we need reinforce high quality solutions to fix the problem. But this theory is probably incorrect for reasoning in academic subjects. The internet is heavily biased towards examples of correct solutions - textbooks, incentive-aligned sites like StackExchange, etc. So poor performance is unlikely to be explained by the prevalence of incorrect solutions. Instead the problem is that reasoning is a task which requires high precision. So it is harder to generalise from solutions of seen problems to solutions for unseen problems. And once you make a mistake, you are conditioning on an unlikely sequence of tokens (dissimilar to what appears in training) so errors compound. That means we need an order of magnitude more compute to get reasoning to the level of precision required to perform well compared to other tasks. This is also why reasoning has been the “last to scale” of the classical LLM tasks, and why MATH has been the hardest benchmark to excel at. But high precision tasks remain: as we move into more agentic settings and task horizon increases, LLMs will need to reason over much longer time periods - where similar problems will apply. It seems unlikely that simply more training FLOPs will solve the problem. Eventually we’ll need to lean on search again as a way to find better outputs and achieve high precision.

Maybe OpenAI had a point with “high taste testers”. I didn’t like the phrase initially because it felt a little elitist. But maybe I can reconcile with it by treating “high taste” as folks who care more about the outputs they are getting, and scrutinise them more carefully. In other words: optimise models for the users who care the most / who spend more glucose scrutinising your outputs.

RL is not enough. It only reaches its potential when combined with other ideas. The most famous example is AlphaZero. RL was combined with self-play which created an implicit task curriculum that evolved through training. This is very different from many RL datasets for LLMs which have a fixed set of tasks. Even where the task set is fixed, RL still needs to be combined with other ideas to show signs of life. Thinking models only come to life through RL when we remove length penalisation and have enough prior knowledge in the training data. Looking ahead, long horizon tasks will require much more exploration and “going off-piste”. But current RL methods induce policy entropy collapse. Diverse mid-training before RL could help, but fundamentally current RL objectives don’t reward “interestingness” and deviating from the “current (high reward) thing”. And yet, discovery is all about deviating from the current thing - and the best ideas come from the wilderness. Deep learning is no exception. It was interesting long before it reached its true potential. We didn’t wait to introduce backprop and SGD until compute and data came online :). This is a long way of saying: crude RL maximalism is overhyped. The magic comes from the interplay of RL with other things, and the angels are in the details.

R1 paper and work truly excellent. Questions about the reward shaping + training data: 1. The paper says the reward “mainly consists of accuracy and format rewards”. Is this it or do they have any other rewards to incentivise longer traces? 2. How many unique (verifiable) questions do they need for things to kick into gear in the RL stage? But I am very glad they have helped put to bed PRMs, MCTS and all that excess complexity! 🪦

The aha moment is indeed overrated. The internal monologue style of R1 appears to come from SFT. The length scaling from R1-Zero appears to come from learning to verify after answering. These are two separate things! This leads to a very different character of reasoning trace. R1-Zero looks like a final answer; R1-Zero looks like internal thought. Example 1: R1-Zero on the left; R1 on the right. R1-Zero looks like a typical CoT trace...R1 looks like o1-style branching ("wait, no"). So if R1-Zero looks like a regular CoT trace, then what is different? 🧵...

🎉 Excited to release General Reasoning: a new community resource for building open reasoning models. We’re looking to make personal, open reasoners a reality. Starting with a small step in that direction today! Read the thread in the quote tweet for details, or my personal analysis below!

Distillation outperforms RL recipes at smaller scales. One simple way to think about this is a large model is a better way to discover good data than a smaller one. A point made in original DeepSeek-R1 paper, but missed by many in the wake of that release in the GRPO wave. That being said, there’s still an opportunity to do effective RL for smaller models. But hinges on getting details right to make training more efficient, including filtering the difficulty of the RL corpus and several algorithmic tweaks (eg overlong filtering). I also think the community is overoptimising benchmarks like AIME, MATH500 and GPQA. It’s not very interesting to report a model doing well on just these benchmarks anymore. What matters is if the recipe generalises to more benchmarks that don’t have heavy optimisation pressure applied to them.

Sorry, but “we have internal models that are better” public statements are very lame - especially when Microsoft doesn’t have a track record of delivering LLMs that are anywhere near SoTA.

Most of the folks who were early on AI cared about AI safety from the outset. Most of the anti AI safety takes I see are from folks who only got involved after ChatGPT - once the technology had been de-risked by others. So revision: “Those who take AI seriously care about AI safety; those who only care about short-term profits don’t”

I am the first author of the Galactica paper and have been quiet about it for a year. Maybe I will write a blog post talking about what actually happened, but if you want the TLDR: 1. Galactica was a base model trained on scientific literature and modalities. 2. We approached it with a number of hypotheses about data quality, reasoning, scientific modalities, LLM training, that hadn’t been covered in the literature - you can read about these in the paper. 3. For its time, it was a good model for its domain; outperforming PaLM and Chinchilla with 10x and 2x less compute. 4. We did this with a 8 person team which is an order of magnitude fewer people than other LLM teams at the time. 5. We were overstretched and lost situational awareness at launch by releasing demo of a *base model* without checks. We were aware of what potential criticisms would be, but we lost sight of the obvious in the workload we were under. 6. One of the considerations for a demo was we wanted to understand the distribution of scientific queries that people would use for LLMs (useful for instruction tuning and RLHF). Obviously this was a free goal we gave to journalists who instead queried it outside its domain. But yes we should have known better. 7. We had a “good faith” assumption that we’d share the base model, warts and all, with four disclaimers about hallucinations on the demo - so people could see what it could do (openness). Again, obviously this didn’t work. 8. A mistake on our part that didn’t help was people treated the site like a *product*. We put our vision etc on the site, which misled about expectations. We definitely did not view it as a product! It was a base model demo. 9. Pretty much every LLM researcher I’ve talked to (including at ICML recently) was complimentary about the strength of the research, which was sadly overshadowed by the demo drama - yes this was our fault for allowing this to happen. 10. Fortunately most of the lessons and work went into LLaMA 2; the RLHF research you see in that paper is from the Galactica team. Further research coming soon that should be interesting. It’s a bit of a riddle because on the one hand the demo drama could have been avoided by us, but at the same time the “fake science” fears were very ridiculous and despite being on HuggingFace for a year, the model hasn’t caused any damage. To reiterate: the anti-Galactica commentary was really stupid, however we should not have allowed that to even happen if we had launched it better. I stick by the research completely - and even the demo decision, which was unprecedented openness for a big company with an LLM at the time, wasn’t inherently bad - but it was just misguided given the attack vectors it opened for us. Despite all the above, I would do it all again in a heartbeat. Better to do something and regret, then not do anything at all. Still hurts though! 🙂

Why are LLMs bad at reasoning? One theory says this is due to weaknesses in maximum likelihood, where the probability mass “overgeneralises” to low quality solutions. Because our pretraining objective (likelihood) doesn’t transfer to our evaluation objective (accuracy), the theory goes that we need reinforce high quality solutions to fix the problem. But this theory is probably incorrect for reasoning in academic subjects. The internet is heavily biased towards examples of correct solutions - textbooks, incentive-aligned sites like StackExchange, etc. So poor performance is unlikely to be explained by the prevalence of incorrect solutions. Instead the problem is that reasoning is a task which requires high precision. So it is harder to generalise from solutions of seen problems to solutions for unseen problems. And once you make a mistake, you are conditioning on an unlikely sequence of tokens (dissimilar to what appears in training) so errors compound. That means we need an order of magnitude more compute to get reasoning to the level of precision required to perform well compared to other tasks. This is also why reasoning has been the “last to scale” of the classical LLM tasks, and why MATH has been the hardest benchmark to excel at. But high precision tasks remain: as we move into more agentic settings and task horizon increases, LLMs will need to reason over much longer time periods - where similar problems will apply. It seems unlikely that simply more training FLOPs will solve the problem. Eventually we’ll need to lean on search again as a way to find better outputs and achieve high precision.

No one is saying RL didn’t work for reasoning. The argument is about internal reasoning emergence, not absolute performance boosts with RL. Quite the opposite in fact - we had PPO on Llama 2 base models 2 years ago with verifiable rewards and had 90%+ on GSM8k. We already knew that RL worked. Hell we even did this with the bronze-age Galactica model and it also worked (see my ICML talk 3 years ago). What didn’t “work” was that we didn’t see emergence of longer traces with backtracking, checking, error-correction and other branching-like behaviour. So when people say “the base model wasn’t strong enough”, the contention is that there wasn’t enough knowledge in the base model, subsequent RL compute and context window length for the model to develop long CoT, internal reasoning on its own. That’s the argument, not whether “RL worked”. It always did, but the magic seems to have emerged relatively recently.

It’s funny that people on this site think major LLM efforts are talent-bound rather than org-bound. The talent differential has never been big between major orgs. Most of the difference in outcomes is due to organisational factors - like allocating compute to the right bets, and letting good research and engineering triumph over destructive politics. This makes for a less sexy story though. People prefer to believe that breakthroughs are made by lone geniuses - instead of the cumulative effort of many nameless, social media averse people — supported by an org that allows the best ideas to win and manages big egos. If you don’t believe me - then consider how some researchers suddenly gain or lose impact and productivity when they switch orgs. Was it because they gained or lost IQ points? 🙂 (Sorry, this is super obvious to anyone who’s actually worked in these labs - but you wouldn’t believe it based on the X feed right now!)

More evidence for the base model hypothesis: RL more effective with a better base model. This is important as it shows why similar efforts a few years ago didn’t reach escape velocity. “If I have seen further it is only by standing on the shoulders of a better base model”.

Maybe OpenAI had a point with “high taste testers”. I didn’t like the phrase initially because it felt a little elitist. But maybe I can reconcile with it by treating “high taste” as folks who care more about the outputs they are getting, and scrutinise them more carefully. In other words: optimise models for the users who care the most / who spend more glucose scrutinising your outputs.

The aha moment is indeed overrated. The internal monologue style of R1 appears to come from SFT. The length scaling from R1-Zero appears to come from learning to verify after answering. These are two separate things! This leads to a very different character of reasoning trace. R1-Zero looks like a final answer; R1-Zero looks like internal thought. Example 1: R1-Zero on the left; R1 on the right. R1-Zero looks like a typical CoT trace...R1 looks like o1-style branching ("wait, no"). So if R1-Zero looks like a regular CoT trace, then what is different? 🧵...

Sorry, but “we have internal models that are better” public statements are very lame - especially when Microsoft doesn’t have a track record of delivering LLMs that are anywhere near SoTA.

Supplementary information for the new DeepSeek R1 Nature paper is very interesting! Details on training data, hyperparameters, base model importance, and more.

R1 paper and work truly excellent. Questions about the reward shaping + training data: 1. The paper says the reward “mainly consists of accuracy and format rewards”. Is this it or do they have any other rewards to incentivise longer traces? 2. How many unique (verifiable) questions do they need for things to kick into gear in the RL stage? But I am very glad they have helped put to bed PRMs, MCTS and all that excess complexity! 🪦

RL is not enough. It only reaches its potential when combined with other ideas. The most famous example is AlphaZero. RL was combined with self-play which created an implicit task curriculum that evolved through training. This is very different from many RL datasets for LLMs which have a fixed set of tasks. Even where the task set is fixed, RL still needs to be combined with other ideas to show signs of life. Thinking models only come to life through RL when we remove length penalisation and have enough prior knowledge in the training data. Looking ahead, long horizon tasks will require much more exploration and “going off-piste”. But current RL methods induce policy entropy collapse. Diverse mid-training before RL could help, but fundamentally current RL objectives don’t reward “interestingness” and deviating from the “current (high reward) thing”. And yet, discovery is all about deviating from the current thing - and the best ideas come from the wilderness. Deep learning is no exception. It was interesting long before it reached its true potential. We didn’t wait to introduce backprop and SGD until compute and data came online :). This is a long way of saying: crude RL maximalism is overhyped. The magic comes from the interplay of RL with other things, and the angels are in the details.

Most of the folks who were early on AI cared about AI safety from the outset. Most of the anti AI safety takes I see are from folks who only got involved after ChatGPT - once the technology had been de-risked by others. So revision: “Those who take AI seriously care about AI safety; those who only care about short-term profits don’t”

Distillation outperforms RL recipes at smaller scales. One simple way to think about this is a large model is a better way to discover good data than a smaller one. A point made in original DeepSeek-R1 paper, but missed by many in the wake of that release in the GRPO wave. That being said, there’s still an opportunity to do effective RL for smaller models. But hinges on getting details right to make training more efficient, including filtering the difficulty of the RL corpus and several algorithmic tweaks (eg overlong filtering). I also think the community is overoptimising benchmarks like AIME, MATH500 and GPQA. It’s not very interesting to report a model doing well on just these benchmarks anymore. What matters is if the recipe generalises to more benchmarks that don’t have heavy optimisation pressure applied to them.