hn-sota
Mining the Most Popular Coding Models from Hacker news#
It’s easy to feel left behind by the speed of progress that is being made with AI-assisted coding. HN-SOTA tries to mitigate this, by keeping tabs on the most recent developments in that space, so you can quickly get up to speed and get an overview over what others are using.
It mines recommendation from Hacker News commenters, one of the most tech-savy demographics that I’ve encountered. For a public forum on the internet, discussions are surprisingly level-headed, and the same can be said for the moderation. And despite AI flooding many social platforms like Reddit or YouTube, HN has shown itself to be more resilient than others to the slop.
Considerations#
When using AI for data mining, observability and grounding are very important. I didn’t just want to trust an AI to accurately extract model names and sentiment from the comments. So I searched for a list of existing models that the AI gets to choose from, which OpenRouter thankfully provides (although there was some missing data for recently released models). This list, after some filtering, gets passed inside the prompt, to provide anchoring to the model when scanning the comments.
During processing, I pass each comment ID to the model and ask it to reference specific comment IDs for sentiment ratings, so I can always trace where the model identified certain models and how it rated the sentiment. This came in handy while refining the prompt, as I could check the results manually and see how well the outputs matched my intentions.
All the data gets logged to a public Google Sheet, so users can trace each data point for accuracy, and analyze the data themselves to come to their own conclusions.
Discussion#
I’ve outlined the process a little more on the page itself. You can also have a look at the HN thread that I posted for further discussions.