Search and Dat - smart ways to go about it?

I’ve been thinking about distributed search (search-index library) and looking for something-leveldown to solve my case, replicating the actual index. But replicating the data to index will also do and that actually makes things within reach.

So, what is the best and what is the simplest way of going about this?

Franz you mentioned kappa style json database on top of hyperdrive that feeds the search engine. How does that work, and what does it solve?

2 Likes

In Beaker, I use sqlite to store search indexes. My indexer uses a pattern that’s similar to https://github.com/beakerbrowser/dat-archive-map-reduce to find and process data.

Eventually I’d like to try moving the indexes into a dat data-structure, perhaps using https://github.com/mafintosh/hypertrie, so that users could share their indexes on the network. Then you’d be able to do distributed search by running your query on multiple users’ indexes.

2 Likes

Nice! Distributing the search query on multiple user indexes opens a lot of interesting use cases. I’m not super techie, so I think I’ll start slow with just distributing JSON and se what I can figure out from there.

1 Like

Trying to figure out how to do stuff. I want a web application (so, using the web-part of the SDK) that is basically an HTML file + CSS + JavaScript. The ideal goal is to not be dependent on any server at all, but I want to reach as many users as possible.

Thinking a webserver is only there to serve those static files. User A chooses to add data to her search engine (some JSON from somewhere. I.e. a local file). That data is stored using hypertrie (?) and indexed (so it’s stored as searchable data in IndexedDB). What do I need to communicate to user B (in addition to the static files) so that that user B gets a copy of the JSON data. Just the public key and discovery key?

EDIT: I’ll test out the SDK Hypertrie example. That seems to do what I want =)