When I search for anything on Google or DuckDuckGo, more than half of the results are useless AI generated articles.
Those articles are generated to get in the first results of requests, since the search engine use algorithms to index websites and pages.
If we manually curate “good” websites (newspapers, forums, encyclopedias, anything that can be considered a good source) and only index their contents, would it be possible to create a good ol’fashioned search engine? Does it already exist?
That’s how Yahoo originally worked. They gave up on the “directory” because they utterly failed to keep up with the expansion of the world wide web. Google with its automatic crawlers did a much better job of listing new websites.
Yahoo could go back to the directory approach and advertise as having less SEO garbage and AI slop.
Skipping a few generations there. Manual indexing wasn’t feasible long before Google existed. Engines like the eponymous “webcrawler” would follow the links between sites and allowed searching the text of them.
Google came around later with enhancements to how the pages are ranked based on the relationship of links between pages.
There’s space in the market for a better search engine. It could be built using an AI-assisted tool for semi-automatic curation by a team of a hundred librarians that could speedily vet the AI’s suggestions for quality.
I’m not sure the web is even that big any more, due to all the big web 2.0 social apps like Facebook and Instagram sucking everything into their uncrawlable platforms.
Yep. Web indexing (or rather internet indexing) for us started with a notebook (paper and pen version) in the computer room. Later Yahoo and Altavista joined in (we still used the notebook for the good sites, browser bookmarks did not yet exist). But Google, when it still was good, wiped this all off the face of the earth.