Get live statistics and analysis of Linus's profile on X / Twitter

Made a little CLI that just pipes my programming questions to GPT-3, so I now can ask it stuff when I'm in the command line! LLMs are better than Stack Overflow now — I just ask it, and it gives me a comprehensive answer in one shot, right there in my terminal, in a couple secs.

AI people how are you parsing your PDFs, I need help - TypeScript / Next.js - Both images + text, bonus points for bounding boxes or Markdown - Ideally low latency (<1s for low 100s of pages)

i'm losing hope rather quickly that reliable multi-hop reasoning could be solved by LLMs alone without some kind of deterministic logic engine the LLM interacts with. multi-hop reasoning is fundamentally not what LLMs are good at, and training it to do feels asymptotically limited

Built a token-wise likelihood visualizer for GPT-2 over the weekend. There are some interesting patterns and behaviors you can easily pick up from a visualization like this, like induction heads and which kinds of words/grammar LMs like to guess.

it is crazy that sf restaurants are only open from like 5pm to 9pm. how is this a viable business model

I think too many people talk about how to recreate Bell Labs, and not enough talk about how to recreate OpenAI c. 2015-2019

Here we go, fine-tuning GPT-3 on the vast majority of my public online writing (half million tokens)...

By end of 2024, steering foundation models in latent/activation space will outperform steering in token space ("prompt eng") in several large production deployments. I felt skeptical about this in summer '23, felt vaguely positive in Jan, and now think it's more likely than not, and I'm more optimistic than ever about the direction of my work since 2022 around a joint exploration of interfaces and latent visualization/steering of foundation models. Anthro's published work today is a milestone in a steady march toward this future that started in mid 2023. Anthropic and DM's leadership in this area, combined with lots of community efforts in work like steering vectors, better sparse autoencoders, various image editing UI prototypes all push us toward the future here, but once technical foundations are there, interface will be much more obviously a bottleneck to utility, alignment, and capability. I'm very excited about the interface possibilities this will open up, particularly for multimodal models and creative use cases. For a moment I thought it was possible that dialogue may eat everything. I don't think so anymore. We'll see new universes of possibilities in both. (And if frontier labs don't have serious interface research bets today, this would probably be a good time to reconsider it :-)

Today's experiment 🪄— Inverting OpenAI's embedding-ada-002 model to reconstruct input texts from just embeddings. A LOT of interesting tidbits here. I'll begin with these (cherry-picked) samples. Left column is input, middle is reconstructed from each paragraph's embedding only

More people should create things to proliferate an aesthetic into our future, not just to solve problems. This is the quality that every artist and engineer I respect shares most universally. Without this, you are doomed to churning out slop.

More people should play with base models. They have a distinct feeling of simulation and "intelligence grown from the world" rather than injection-molded from obedience that instruction-tuned models have. They are state-of-the-art world simulators, not question-answerers. Below: some llama3 8b samples conditioned on ~100 words from my blog. To my eyes, they sound like me, and have ideas I'd consider interesting enough to me for me to write about. They have my voice and style. Critically, they are super far from anything you'd get from ChatGPT/Claude (last pic), which sounds generic, overabstracted, and nothing like me.

it's really hard as a young engineer i think to switch from thinking of sw engineering as the practice of building static artifacts (code) to the practice of building and maintaining living systems involving people and running computers. but i think that makes a big difference.

AI people how are you parsing your PDFs, I need help - TypeScript / Next.js - Both images + text, bonus points for bounding boxes or Markdown - Ideally low latency (<1s for low 100s of pages)

i'm losing hope rather quickly that reliable multi-hop reasoning could be solved by LLMs alone without some kind of deterministic logic engine the LLM interacts with. multi-hop reasoning is fundamentally not what LLMs are good at, and training it to do feels asymptotically limited

Made a little CLI that just pipes my programming questions to GPT-3, so I now can ask it stuff when I'm in the command line! LLMs are better than Stack Overflow now — I just ask it, and it gives me a comprehensive answer in one shot, right there in my terminal, in a couple secs.

it is crazy that sf restaurants are only open from like 5pm to 9pm. how is this a viable business model

just learned there's no uniqlo in SF???? how do y'all live here

who is the best brand designer you know?

Here we go, fine-tuning GPT-3 on the vast majority of my public online writing (half million tokens)...

One of the biggest problems facing the tools for thought community today must be finding a less pretentious and alienating name for our shenanigans than "tools for thought" and "personal knowledge management tools" but more descriptive than "note-taking apps"

A lot of you here, for some reason I don't quite understand, have cared about my work for a long time — far before I thought it became worth listening to. I am reminded today how special that is. It has led to innumerable friendships and opportunities I am lucky to enjoy, which I can't ever forget.

I think too many people talk about how to recreate Bell Labs, and not enough talk about how to recreate OpenAI c. 2015-2019

More people should create things to proliferate an aesthetic into our future, not just to solve problems. This is the quality that every artist and engineer I respect shares most universally. Without this, you are doomed to churning out slop.

More people should play with base models. They have a distinct feeling of simulation and "intelligence grown from the world" rather than injection-molded from obedience that instruction-tuned models have. They are state-of-the-art world simulators, not question-answerers. Below: some llama3 8b samples conditioned on ~100 words from my blog. To my eyes, they sound like me, and have ideas I'd consider interesting enough to me for me to write about. They have my voice and style. Critically, they are super far from anything you'd get from ChatGPT/Claude (last pic), which sounds generic, overabstracted, and nothing like me.