Any idea on running a (very) small silent disco system?

For the last few summers I've tried (and failed) to get a silent disco system working for a small group of friends. The requirements are

1. Anyone should be able to join (locally) with a phone and a pair of bluetooth headphones. With the absence of headphone jacks I've found most people rely on bluetooth headphones.
2. Low enough latency.
3. Decent enough audio fidelity.
4. No weird monetized apps you have to sign in to.

In a post covid age where we all had low latency video calls, it seems crazy there isn't an obvious way to have <10 people connected to one 128kbps audio stream. Here are the shortcomings

• Most silent disco systems (for events) use FM to broadcast to FM receivers. Broadcasting without a license is technically illegal, but easy enough to do. The lack of wired headphones means most phones no longer support receiving FM frequencies, as they used the headphone wire as an antenna. It's not ideal checking up on everyone's phone models to see whether or not they support FM ahead of time.

• Throughout covid we used Discord to listen to music together many miles apart. The trouble is bluetooth does not have enough bandwidth for speakers and a microphone. So - those with wired headphones it worked perfectly, but with bluetooth headphones the audio drops to landline phone quality, far below what's listenable. Discord supports 'Stage' calls where some participants are talking and others are only listening. Unfortunately this doesn't disable the microphone for the audience, and so the audio is still poor.

• Lastly is streaming. This solves everything above but the latency is too high. Using software called 'Stream What You Hear' allowed us to create a webpage with a stream running, but each person could be many seconds ahead or behind depending on when they loaded the stream. Attempts to sync everyone up would fail if someone accidentally locked their phone.

I'm wondering if the solution is going to have to be a bit more technically minded, which I'm open to investigating, but wondered if anyone here had any ideas to bounce.

Thanks in advance!

EDIT: I tried SnapCast as recommendation by @arch and it seems to do exactly what I was setting out to achieve, and FOSS software too! Thanks to everyone for their suggestions and help, I'm really excited to trial it.