When reasoning generative AI models hit the market I did expect a certain number of posts that would try and sell the abilities of the new models. I’d suspect such posts would normally come from the AI companies themselves, affiliated influencers and journalists.
However, what surprised me is that even after the initial dust is long settled from the companies, completely random people have continued to post their “creations” online - particularly on X.
Why is it surprising?
By this time, it is already common knowledge what generative AI with reasoning mode can do. Very few people doubt the actual capabilities of such inference from these models within its narrow scope of things it can do.
You start with some very high level prompt of some idea of yours, and most of the time, you get back a working application that you can then tweak by giving more instructions.
The cost of such an enterprise is just the ability to buy a subscription for one of these models and then type in your prompts in english.
This is such a trivial activity that the software generated is no longer impressive. Anyone can do this.
Is showcasing projects bad?
The reason such projects used to garner praise for themselves is because that praise was targetted at people behind the project. It was clear from the output that it involved real effort. And that is praiseworthy.
LLM generated software is not praiseworthy. There is very little effort involved in making any such standalone software. Generally these kinds of software are developed for a very short period of time - sometimes for hobby/recreational reasons and then dropped.
There is nothing wrong in the activity of developing such software, but advertising every single such software is childlish.
So what would impress you?
That is subjective. But I’ll attempt an answer.
Let us keep aside the obvious legal, ethical, regulation, economic and environmental problems associated with the act of training and using generative AI. Because if we consider them, then only those kinds of results that really improve mankind as a whole in an immediate tangible way - such as a curing cancer - would impress me.
Outside of that, the only kinds of software that I would be interested in, are those that push the frontier in some way. Since I’m a programmer myself, I can give an example of what that would look like.
We saw the development of Rust as a programming language whose unique selling point has been the automatic management of memory via borrow checking instead of garbage collection. Additionally, it also has lifetime tracking and together, these two features help write code that avoids a class of bugs and at the same time push a lot of invariant checking to compile time. This is a novel idea to be supported natively in a language (as C++ does support it via smart pointers features).
Now Rust is very mature and a large number of software applications are running it in productions. This is software that people depend on with their money.
About 10 years back, few years after Rust was first announced, there was another language called Zig that had explicit memory management. You could write code in what is called “Group Thinking” paradigm where the lifetimes of a group of data is handled together (as opposed to “individual thinking” in Rust). This can be done in Rust as well via Arena allocation, but this concept of explicit memory allocation at the language level is novel. You know which code allocates memory on the heap just by reading it. Zig pushed the frontier of languages.
Ghostty is a terminal emulator built on Zig whose creator Mitchell Hashimoto says that he uses AI to speed up the development. He prides himself in the fact that he reads AI generated code before commiting to it. This terminal emulator runs on the GPU and he is working on constantly adding frontier pushing features to this terminal.
The mother of all terminals (subjective opinion) is definitely the “Kitty” terminal whose developer Kovid Goyal has literally pushed the frontier long before AI. From image display to rendering multiwidth and multi size fonts. This also runs on the GPU and Ghostty actually constantly implements the spec defined by Kovid. If Kovid comes up and says tomorrow that he has been using AI to speed up development, I would know that he has been using it to do some real work on the terminal and will make the ecosystem a better place.
If these are too grand, consider even smaller projects such as improving collision detection in video games. Mike Acton recently implemented such a paper and used AI to speed it up by 100x. In the comments he clearly calls out that this matched his expectation of the kind of gains he would have gotten had he done it himself, and also said he could squeeze out a little more performance if he handcoded. Finally, he clearly called out that this took 125M tokens. (That would be around $1500 in current subsidized token pricing and around $10K without the current subsidy). This is a good showcase of an AI use that also provides the right expectations for the readers.
Contrast this with…
the kind of project advertisements made using AI.
Generally half-baked, and often just strings together multiple solutions for different hard problems together into a single coherent application that works for the time-being for one particular use-case. Often with some outrageous caption - something that places the LLM on a pedestal and derides those who do not use AI in that field.
Why then do they want to show that they used AI to generate a software? The implications of this are not good on the posters. You had access to the latest generative AI tech and all you could do was just this? Do you understand that this is the equivalent of a rich person’s vanity project - writing a book using a ghost writer, or getting a building named after you using your influence.
The “rage-bait” caption posts are the most frustrating lot. When people point out that this is nowwhere close to something that people currently pay money for, they fallback to their defence - “shut up doomer - you are in denial of LLM capabilities, they have improved a lot since 2022”.
They are missing the point.
A travelling circus puts up a show of a dancing bear that can do a lot of dance tricks. When people point out that the dance isn’t that great, or that this is nothing new they tell us to not scrutinize the bear’s tricks, but be surprised that a bear dances at all.
Yes, we all were surprised that the bear could dance last year. But we are not going to pay for your circus if you are not giving us anything valuable.
So why do people do it?
My original intention of “why” here was to point out the ridiculousness of this activity.
But the actual “why” might be philosophical. Maybe people who haven’t produced anything so far suddenly find themselves in possession of a tool that can make any software they want. Maybe it gives such people a sense of satisfaction to claim such generated software as their own.
I wouldn’t go to a restarant, order a pizza, and when it comes tell my family sitting across the table that “I made this Pizza”.