Something I have always missed is a resource that is both comprehensive, up to date, and easy to use/reason about. Something like internet directories, where you can find every single active domain on the internet, so in a sense you have the entire internet laid out in front of you. Of course domains like x.com and reddit.com are like an internet on their own but at least you know that any website you can visit on the internet will belong to a domain in the directory.
A directory like this will incidentally serve a lot of niche use cases. Looking for every SaaS out there? It's in this set. Every ecommerce website? Here. Every blog? Also here. Every website related to a very specific niche or location? Also here.
Of course the problem would be categorization and keeping the whole thing up to date. But I feel like it's a worthwhile effort even if it's not 100% comprehensive or up to date or 100% categorized accurately. It's still very useful and can be improved and curated with time.
Or maybe instead of an internet directory you want to view a list of every single product currently listed for sale online anywhere on earth. This is a much harder set to curate and keep up to date. But it's possible to do a pretty good job there and it would be immensely useful to have this data even if it's a bit stale or not 100% comprehensive.
Or maybe you want a list of all laptops ever released. I don't know of a resource that provides this. You can ask AI to compile a list or for approaches for making a comprehensive list, but the resource itself doesn't exist or isn't easily accessible as far as I know.
Yes, there's wikipedia, random github repos where someone collects a surprisingly large and useful amount of data, datasets on kaggle and huggingface, and of course entire businesses whose purpose is to collect that data as timely, comprehensively, and accurately as possible, whether their business model is to sell this data for a premium, or they collect it internally for their own benefit.
But all of that is either expensive, not comprehensive, not up-to-date, not accurate, or a mix of all that. It feels like there should be more effort to collect, curate, categorize, and publish this data. Just like there are open data initiatives like Our World In Data, there should be efforts to collect, curate, and organize all sorts of public data and knowledge.
And I don't think this kind of data has to be crowd-sourced or open to edit by anyone like wikipedia. Because adding the option for anyone to add things quickly spirals into an issue of spam, deduplication, monitoring, and verification of what's being submitted. And it isn't necessary if you're comprehensively scanning all of the internet, your goal is to include everything yourself anyway.
This will make it easier and more manageable to tackle all the complexity in the world, it can be a source for AI agents instead of having them search in multiple places for the most up to date and comprehensive knowledge. And, more importantly, it's a source of truth and mental map for humans to tackle and handle all that complexity that's there and increasing every day.
As long as it's accurately categorized, which is a separate problem: How many categories should exist? Overlaps definitely would exist, so tags are a better way to think about it. How to tag as accurately as possible though?
For example: How many database projects exist? There's a difference between database projects on Github -whether they're popular or not- and database companies. And even within companies, some are actively making and developing a database, while others help customers set up a popular database like Supabase and Postgres.
Another big problem of course, is trust. How can you trust that this data is comprehensive, up to date, and accurate? How can you be sure you don't need to look elsewhere? This ties in with issues like data lineage and provenance. Where did this data come from? How do we know it's accurate, comprehensive, and up-to-date? Data sources and methodology help alleviate that, but it's a hard problem to solve.
Github awesome-lists exist, but they quickly become out of date due to the maintenance burden and often aren't comprehensive because they depend on people bothering to add pull requests instead of comprehensively scanning for all approaches on the internet.
Common Crawl exists, but it's very large and unwieldy (something I sympathize with because the amount of data is large and their mission is to collect it not organize it). Wikipedia exists, but its goal isn't to list every individual thing. It has the laptop page and the list of laptop brands and manufacturers page as well as pages for each brand, but not every single laptop ever released. And even if that page existed, it assumes people have done the work of submitting everything and ensuring it's comprehensive and accurate.
There's historical precedent for this, of course: Yahoo! Directory launched in 1994 as "Jerry and David's Guide to the World Wide Web" and ran for twenty years before being shut down in 2014. In its later years it charged a $299 non-refundable review fee for submissions -pay to be listed- so they fought spam with ads.
DMOZ, the Open Directory Project, ran from 1998 to 2017, at its peak listing millions of sites maintained by tens of thousands of volunteer editors, and it was part of the inspiration for Nupedia, out of which Wikipedia grew. They closed because it's hard to curate all this manually -even with volunteers-, most people went to search engines instead of directories, and directories got hit with a lot of spam.
Google Directory itself ran on DMOZ data until Google shut it down in 2011 - the search side eating the directory side. It also costs money to host and is hard to monetize.
DMOZ's successor, Curlie, carries the hierarchy on today, but it inherited the same model that killed DMOZ.
So where did internet directories go? Search ate them, curation didn't scale, and directories needed lots of money. But that has changed: search became filled with spam itself, so a high-quality directory can rank websites in a better way, agents can categorize websites fairly accurately, and compute became faster per dollar compared to before.
Agents are also a new source of demand that could benefit from directories. Search engines provide pages, not domains or comprehensive lists of laptops. - I know I mentioned laptops a lot, I really like them :) -
I'm starting by collecting all browsable domains on the internet. I don't know exactly what that'll look like or how it'll operate, but I'm excited for it!