Get live statistics and analysis of david rein's profile on X / Twitter

When I was 13 I played a game of hide and seek with some friends, and me and my crush went off and hid together. While we were looking around for a place to hide, she said “haha this would be such a great make-out spot, right?”. I failed to understand this as anything but a simple observation, so I said “oh haha yeah definitely” or something. I don’t know if she made any other moves after that, but she was kind of cold to me later and I was confused and sad. What I do know is that this is definitely where it all went wrong for me

Claude 3 gets ~60% accuracy on GPQA. It's hard for me to understate how hard these questions are—literal PhDs (in different domains from the questions) with access to the internet get 34%. PhDs *in the same domain* (also with internet access!) get 65% - 75% accuracy.

“the purpose of a a system is what it does” is one of the dumbest phrases ever. how do people believe this

Honestly, major props to OpenAI for publishing (and tweeting/highlighting!) results where Anthropic models are better than theirs.

Imagine seeing BERT to GPT-2 to GPT-3 to chatGPT to GPT-4 and thinking we’re at a plateau because “it’s been a year”. Absolutely insane.

I’m excited to share details about HCAST (Human-Calibrated Autonomy Software Tasks), a benchmark we’ve been developing at METR for the past year to measure the abilities of frontier AI systems to complete diverse software tasks autonomously.

It’s incredible to me how the person who tweeted this didn’t even read through the titles of these papers. Almost all of these are *about* chatGPT. Read your own tweet next time please! And shame on anyone liking/retweeting this without taking <10s to read the content.

I’ve moved to SF, probably until the singularity or whatever!

I’m really sad to announce that exercising regularly, and keeping to a healthy routine have not cured my depression

Hate it when I ask someone at a party what they work on and they say “pretraining”. Kills the conversation quicker than if they said they work in intelligence

if you're not extremely paranoid about your swe/agentic evals being silently broken, your swe/agentic evals are silently broken

When making AI benchmarks, people often think you need to make the instructions as precise as possible, otherwise its unfair to the AI system. This causes us to overestimate AI capabilities, because the real world isn't precise.

I realized recently that I actually resent Nick Cammarata’s whole “anyone can experience pure 10/10 happiness and equanimity” thing because I feel like it gave me an unattainable goal that I now compare to my life against and obviously come up so short.

When I was 13 I played a game of hide and seek with some friends, and me and my crush went off and hid together. While we were looking around for a place to hide, she said “haha this would be such a great make-out spot, right?”. I failed to understand this as anything but a simple observation, so I said “oh haha yeah definitely” or something. I don’t know if she made any other moves after that, but she was kind of cold to me later and I was confused and sad. What I do know is that this is definitely where it all went wrong for me

“the purpose of a a system is what it does” is one of the dumbest phrases ever. how do people believe this

I’m really sad to announce that exercising regularly, and keeping to a healthy routine have not cured my depression

I’ve moved to SF, probably until the singularity or whatever!

Claude 3 gets ~60% accuracy on GPQA. It's hard for me to understate how hard these questions are—literal PhDs (in different domains from the questions) with access to the internet get 34%. PhDs *in the same domain* (also with internet access!) get 65% - 75% accuracy.

Imagine seeing BERT to GPT-2 to GPT-3 to chatGPT to GPT-4 and thinking we’re at a plateau because “it’s been a year”. Absolutely insane.

It’s incredible to me how the person who tweeted this didn’t even read through the titles of these papers. Almost all of these are *about* chatGPT. Read your own tweet next time please! And shame on anyone liking/retweeting this without taking <10s to read the content.

When making AI benchmarks, people often think you need to make the instructions as precise as possible, otherwise its unfair to the AI system. This causes us to overestimate AI capabilities, because the real world isn't precise.

I realized recently that I actually resent Nick Cammarata’s whole “anyone can experience pure 10/10 happiness and equanimity” thing because I feel like it gave me an unattainable goal that I now compare to my life against and obviously come up so short.

I’m excited to share details about HCAST (Human-Calibrated Autonomy Software Tasks), a benchmark we’ve been developing at METR for the past year to measure the abilities of frontier AI systems to complete diverse software tasks autonomously.

Honestly, major props to OpenAI for publishing (and tweeting/highlighting!) results where Anthropic models are better than theirs.

Hate it when I ask someone at a party what they work on and they say “pretraining”. Kills the conversation quicker than if they said they work in intelligence

Almost all arguments about x-risk from AI hinge on whether someone believes superintelligence could be developed within a decently short period of time (eg 50 years or something). I basically don’t get into discussions about AI x-risk without clarifying where we stand on this point at the start