100 sats \ 1 reply \ @apxhard 6 Dec 2022 \ on: Building a new Web Search Engine. Just for you, Stackers! bitcoin
Is it possible to build this as a totally decentralized project running on LN rather than as a single centralized site?
Lightning nodes could all host and expose 'indices', which could link to each other.
You could imagine people competing to run the best index on various topics (i.e. a jazz index, a beer index, a scifi index, etc), and then various 'metaindices' competing to store indices of indices.
So when you did a search, your own local node would consult its index and the index of everyone you were connected to, etc. People could add or drop nodes that gave results they did, or didn't like. Censorship would become basically impossible, and the search engine would learn people's preferences over time, locally, on your own machine. Personalization without surveillance.
API requests could contain tiny lightning invoices for payment, and API operates could decide what sort of payment they requested.
That's a great question, I've thought about it, and I don't really have the answer (
There exists a distributed search engine, it's called YaCy. Here is their public demo web UI (no need to setup the client): https://yacy.searchlab.eu/
As far as I can see, a decade of work on this thing, and it's search results are still really bad. Even my toy project seems to do better. Why is that? I'm not sure.
My guess is that the most important thing with search is fighting abuse (spam, etc). With many servers operated by independent entities, each having their own view on quality or shilling different sites, getting consistently good results might be a really big problem.
I'd say we should start with our goals. Why decentralize it?
- To avoid censorship - that's a very broad idea, must dig very deep into what, who, how etc.
- To get personalization without surveillance - that can be done with a local app (like a browser extension), that will do query-rewriting and post-filtering of the non-personalized search result on the client, without decentralizing the engine.
I agree this is an attractive direction to think about, but at this point I'm not ready to invest in experiments there :)
reply