It’s possible that if you’re trying to build a Twitter substitute on top of the protocol, the issue looks like a mere Sicilian dwarf elephant, since Twitter doesn’t have great discoverability either (or need it, really), and synchronization hiccups matter less for that kind of service. It’s when you’re trying to build a Reddit/Usenet substitute that things fall down.

But yeah, it really is a gap in the design.

A quick skim of the protocol documents suggests you’re right: Activity Vocabulary doesn’t even mention the “discovery” use case, and no allowance seems to have been made for it. I can’t fathom what they were thinking.

The short-term workaround would be scraping the Web portal “Communities > Local” page of each instance periodically for a list (and cache the info centrally? Not sure.) Hopefully the raw HTML is parsable enough that you wouldn’t have to involve Selenium (which I’ve used in my day job—it’s awful) or its ilk.

The correct fix is, of course, to add the “discovery” case and messages supporting it to the protocol, either as an extension or as core for the next version. This might take years.

It might be worth asking browse.feddit.de how they get their information.

Upon further consideration, the smart way to do this would be to have the servers exchange community lists, with post subjects and other metadata. That’s enough information to find and evaluate a community. Post bodies and comments can then be fetched on request.

Putting all the data in the client strikes me as more bandwidth-inefficient, but whether that matters these days is questionable. Usenet clients don’t download subject lines from the server until after you subscribe to a group, but that’s a relic of the days of modems whose speed was measured in baud. And of course it’s always easier to start your own project than try to fork the existing server code. Not to mention that I haven’t bothered to look at the API yet, so my thinking may be way off-base.

Plus, I get the impression that the existing clients are half-baked, so the more the merrier. ;)

Usenet propagates all messages to every server (unless the receiving server specifically blocks a newsgroup or hierarchy). Messages originate at a specific server, but they’re copied everywhere.

The Fediverse, in its current state, stores each message on only one server. It will provide a pointer to a given message to another server only if the other server asks for it.

In theory, you could copy all communities and messages from every server yours is federated with onto your home server for a more Usenet-like effect. If you did that, you would be able to view the whole set of communities and messages from your home server even if no one there had subscribed to them yet. In practice no one does that. Yet.

I suspect the best way of achieving what the original poster wants is to copy the community list and message text content, but leave any embedded media on the originating server—text is low-bandwidth, so you could probably fit the entire Fediverse’s daily production of same into the bandwidth required for a couple of hours of 4K Netflix.

The Usenet model does seem a better fit for Lemmy than the current setup, because community discovery here is kind of painful, really.