Opened 2 months ago

Last modified 3 weeks ago

#33700 assigned project

audio- and video-conferencing considerations

Reported by: anarcat Owned by: gaba
Priority: High Milestone:
Component: Internal Services/Services Admin Team Version:
Severity: Major Keywords:
Cc: mcs, gaba, ggus Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description (last modified by anarcat)

With the rise of the coronavirus, even Tor, which generally works remotely, is affected because we were still having physical meetings from time to time, and we'll have to find other ways to deal with this.

This ticket aims at establishing the problem space ("what we're trying to solve") and evaluate possible solutions ("what could fix it").

The requirements and alternatives evaluation are being done in https://help.torproject.org/tsa/howto/conference/

Child Tickets

Change History (39)

comment:1 Changed 2 months ago by anarcat

my inventory of possible solutions is detailed in this blog post:

https://anarc.at/blog/2020-03-15-remote-tools/

the TL;DR: is basically:

  • BBB (Big Blue Button)
  • Jitsi
  • Mumble
  • Nextcloud talk, for small meetings

some technical considerations regarding install methods:

mumble

there are two different puppet modules to setup mumble:

still need to be evaluated, but i'd be tempted to use the voxpupuli module because they tend to be better tested and it's more recent

jitsi

ansible roles: https://code.immerda.ch/o/ansible-jitsi-meet/ https://github.com/UdelaRInterior/ansible-role-jitsi-meet

there's also a docker container and (messy) debian packages

prometheus exporter: https://github.com/systemli/prometheus-jitsi-meet-exporter

Nextcloud

systemli is using this ansible role to install coturn: https://github.com/systemli/ansible-role-coturn

BBB

no known install procedure, might be messy.

comment:2 Changed 2 months ago by hiro

Must have:

  • video/audio communication for a group of people of approx 2-10 people
  • chat fallback
  • have a mobile app

Nice to have:

  • Reliable video support. Video chat is nice, but most video chat systems usually require all participants to have video off otherwise the communication is sensibly lagged.
  • allow people to call in

comment:3 in reply to:  1 Changed 2 months ago by hiro

Replying to anarcat:

Nextcloud

systemli is using this ansible role to install coturn: https://github.com/systemli/ansible-role-coturn

Is this something our nextcloud could support? Or we would have to host it and setup a way for it to talk to our nc?

comment:4 Changed 2 months ago by anarcat

Is this something our nextcloud could support? Or we would have to host it and setup a way for it to talk to our nc?

i don't know. NC Talk doesn't work so well, IMHO, with lots of people.

comment:5 Changed 2 months ago by gaba

Must have:

  • allow people to call in
  • video/audio communication for a group of people of approx 2-10 people
  • mobile app
  • chat
  • a way for one person to mute themselves

My experience with nextcloud talk is that does not even work well for 2 people. Signal is much better for 2 people talking.

Are you all thinking on one server to host any video-conferencing software? How much $ would that cost?

comment:6 Changed 2 months ago by anarcat

allow people to call in

you mean phone?

Are you all thinking on one server to host any video-conferencing software? How much $ would that cost?

depends on the service we pick. biggest cost center is staff, so it depends on how hard it would be to setup. would evaluate once we know what we do. :)

comment:7 Changed 2 months ago by hiro

I have some doubts regarding hosting our video-conferencing system. I understand there are some nice aspect of hosting our own tools, but at the same time there is a reason big organization spend a lot of resources in video conferencing systems and smaller organization decide to buy in some solution. VC isn't that easy and it doesn't really scale that well in terms of resources.

So if we want a solution so that 5 people can have a meeting at the same time that's ok, we can host it. But I am not sure we can have a solution where we can have 3 meeting of 5 people at the same time and it would work as well...

This is just some food for thoughts about making estimates and deciding what we want.

comment:8 Changed 2 months ago by mcs

Cc: mcs added

comment:9 in reply to:  7 ; Changed 2 months ago by gaba

Replying to hiro:

I have some doubts regarding hosting our video-conferencing system. I understand there are some nice aspect of hosting our own tools, but at the same time there is a reason big organization spend a lot of resources in video conferencing systems and smaller organization decide to buy in some solution. VC isn't that easy and it doesn't really scale that well in terms of resources.

So if we want a solution so that 5 people can have a meeting at the same time that's ok, we can host it. But I am not sure we can have a solution where we can have 3 meeting of 5 people at the same time and it would work as well...

This is just some food for thoughts about making estimates and deciding what we want.

Yes. We do not have to do this. We could just support one of the collectives that are running jitsi and use their infra instead. It is valid to bring up this point that may not be something we have the resources or time to do right now.

comment:10 in reply to:  6 Changed 2 months ago by gaba

Replying to anarcat:

allow people to call in

you mean phone?

yes. Some people use a phone to call in into audio conferences.

Are you all thinking on one server to host any video-conferencing software? How much $ would that cost?

depends on the service we pick. biggest cost center is staff, so it depends on how hard it would be to setup. would evaluate once we know what we do. :)

comment:11 in reply to:  9 Changed 2 months ago by hiro

Replying to gaba:

Replying to hiro:

I have some doubts regarding hosting our video-conferencing system. I understand there are some nice aspect of hosting our own tools, but at the same time there is a reason big organization spend a lot of resources in video conferencing systems and smaller organization decide to buy in some solution. VC isn't that easy and it doesn't really scale that well in terms of resources.

So if we want a solution so that 5 people can have a meeting at the same time that's ok, we can host it. But I am not sure we can have a solution where we can have 3 meeting of 5 people at the same time and it would work as well...

This is just some food for thoughts about making estimates and deciding what we want.

Yes. We do not have to do this. We could just support one of the collectives that are running jitsi and use their infra instead. It is valid to bring up this point that may not be something we have the resources or time to do right now.

That is exactly what I meant. As you mentioned there are different collectives that are running solutions and we could partner with them if we want something long term for all our meetings. If instead we want something small for a few meetings per day and only a few people than I think we could host something pretty quickly.
Or maybe we decide this is something core for us and we prefer to invest resources in a communication solution.

Last edited 2 months ago by hiro (previous) (diff)

comment:12 Changed 2 months ago by anarcat

some notes about jitsi optimizations:

  • disabling google's servers: https://mastodon.infra.de/@galaxis/103861095334491566
  • limit the number of concurrent speakers:
    10:43:49 <+ahf>     // Default value for the channel "last N" attribute. -1 for unlimited.
    10:43:49 <+ahf>     // channelLastN: -1,
    10:43:49 <+ahf>     channelLastN: 10,
    
  • change the JVM heap allocation in /usr/share/jitsi-videobridge/lib/videobridge.rc
  • raise logging level to ERROR to keep logs from flooding the disk in /etc/jitsi/videobridge/logging.properties
  • open up tcp/4443 and udp/10000 to the video bridge

comment:13 Changed 2 months ago by anarcat

Description: modified (diff)

i've edited the summary to summarize the discussion of Goals so far along with the known possible solutions.

i'm susprised to read that "reliable video support" is a "nice to have have". i am not surprised, however, to see that everything *else* ends up in the "must have" section: this is often what happens in those kind of exercises, as we want everything.

i would still encourage folks to consider what is out of scope here. do we want whiteboarding for example? or is that a nice to have? how about moderation or capacity beyond 10-20 people (say to host a tor meeting)?

i think the next step here is to do a short inventory of external alternatives we could consider, and evaluate costs.

and if i may make an editorial comment...

if this was just my call, my gut feeling would be to just setup a mumble server:

  1. a lot of collectives run it
  2. it's much *much* simpler to setup than jitsi (there's a debian package *and* a puppet module)
  3. many of us are already familiar with it, and those who aren't should be able to cross over
  4. it scales

But that said I understand if people would prefer to have video and are worried about the usability aspects of Mumble (which are real).

I would also like gaba to clarify the scope of the project: is this for vegas meetings? for team meetings? for workshops? for the tor meeting? for meeting with funders? please file those use cases in the Goals so we have a better idea of the feature set required.

comment:14 Changed 2 months ago by anarcat

Description: modified (diff)
Status: newneeds_revision

after a quick chat with gaba, i clarified the goals to include:

  • must have: internal team meetings
  • nice to have: free software, privacy, tor meeting replacement (which involves much more people and whiteboarding)

comment:15 Changed 2 months ago by anarcat

VC isn't that easy and it doesn't really scale that well in terms of resources.

This is why, by the way, I'm thinking we could just host a mumble instance instead. This is much, *much* simpler to deploy in terms of staff skills, but it also uses much less resources, both on the client and the server side.

But if we have "video" as a "must-have", i guess that kicks out mumble... :/ (Dial-in, by the way, is a possibility with mumsi although that project is not as well packaged and maintained as mumble itself.)

comment:16 Changed 2 months ago by anarcat

Description: modified (diff)

document BBB features and installation better

comment:17 Changed 2 months ago by anarcat

Description: modified (diff)

comment:18 Changed 2 months ago by anarcat

the approach with mumble would require something else for screensharing.

also with jitsi, it would be nice to have some sort of whiteboarding... here are a few solutions:

http://deadsimplewhiteboard.herokuapp.com/
https://awwapp.com/
https://www.webwhiteboard.com/

comment:19 Changed 2 months ago by arma

Maybe I don't understand the goal of this ticket, but: it seems to me that our answer for audio / video meetings is: "use one of these external jitsi servers if there aren't very many people, use zoom if there are, and if the group doing the meeting wants to use something different they can do that."

That is, we've got no business setting up and tweaking our own jitsi (or whatever) server when there are already folks in the broader community who are doing it.

comment:20 Changed 2 months ago by anarcat

Maybe I don't understand the goal of this ticket, but: it seems to me that our answer for audio / video meetings is: "use one of these external jitsi servers if there aren't very many people, use zoom if there are, and if the group doing the meeting wants to use something different they can do that."

Well, for the "Zoom" part, there's certainly problems:

So there is a case to be made that we want to have some alternatives to Zoom, specifically...

That is, we've got no business setting up and tweaking our own jitsi (or whatever) server when there are already folks in the broader community who are doing it.

... and "use someone else's Jitsi instance" can certainly be an answer. But one of the things I have heard from people running those instances is that they need help. The call first came from the Framasoft people, and they basically said that organisations should run their own Jitsi instances instead of all relying on a central one.

I'll also note that the last Jitsi call we made wasn't made on some random part of the internet: it was made on ahf's instance, who set it up in his free time.

The other thing is that Jitsi does take a surprising amount of bandwidth when we're streaming video. It's logical, when you think about it, but video scales up quite fast, and it's not unheard of to have those saturate gigabit links. If all of TPI starts using Jitsi more intensively, we will monopolize that instance and deteriorate the service for smaller, one-time users that do not have our resources.

But sure, we could use someone else's instance. I'll note we need to be very careful about this. One of the things Jitsi does, out of the box, is that it lists the conferences currently running on the server. If you don't set a password, anyone can join in and start listening on the conversation, which I find rather strange. If we setup our own instance, we could make a better policy. (You can set a password when you're the first to join, but I suspect I'm the only one who really does this, and it's really easy to forget.)

Finally, I'll note that I have tried to establish a process by which we make this decision here. This ticket is not called "let's install an A/V server". The ticket is "audio- and video-conferencing considerations". It's explicitly aimed at documenting what are our requirements, the possible solutions and how the rate against our requirements.

Hosting externally has always been an option. I'm just trying to formalize the process. If you have a "must have" that is "do not host ourselves" then, by all means add it if you think that should be a hard requirement. But so far I've gone with "long term maintenance costs covered" instead, which seems more in line with what, as an organization, we might be worried about.

comment:21 Changed 2 months ago by anarcat

regarding the scalability issues with Jitsi, they seem to be claiming to be able to handle hundreds of streams in parallel, server side:

https://jitsi.org/jitsi-videobridge-performance-evaluation/

so maybe the experience we had with things breaking up at 10-12 people is just on the client side, and could be fixed with the channelLastN tweak.

also, arma: if i remember correctly, the reason gaba asked me for this is exactly because they had reliability issues in the session they hosted on another server... so that's also something to keep in mind: if someone else is hosting the server, I can't fix it.

comment:22 Changed 2 months ago by anarcat

Description: modified (diff)

added details on mumble - there's a web interface!

comment:23 Changed 8 weeks ago by anarcat

Description: modified (diff)

add puppet module from the SMASH group

comment:25 Changed 8 weeks ago by gaba

Cc: gaba added

comment:26 Changed 7 weeks ago by anarcat

Description: modified (diff)

i wrote this about mumble which outlines limitations and possible improvements to the project

https://anarc.at/blog/2020-04-09-mumble-dreams/

also added mumsi to the mumble section

comment:27 Changed 6 weeks ago by ggus

Cc: ggus added

comment:28 Changed 5 weeks ago by anarcat

another interesting review

https://tech.firstlook.media/how-to-pick-a-video-conferencing-platform

of the free platforms we considered, they only reviewed jitsi, the others are Duo, Facetime, Hangouts, Jami, Keybase, Signal, Vidyo, Webex, and Zoom.

comment:29 Changed 5 weeks ago by anarcat

also, on linux jitsi eats CPU like crazy, apparently because it doesn't do hardware acceleration. this reddit thread has some more information:

https://old.reddit.com/r/linux/comments/fxcyyw/jitsi_meet_features_update_april_2020/

important quotes:

I've noticed high cpu usage on Linux side due to no hardware acceleration. Haven't noticed it in the windows side

Chrome and Firefox don't have hardware acceleration in Linux unless you are running a patched version of chromium with va-api added.

It was supposed to be enabled in Firefox 75 when using Wayland.

To see if things would be hardware accelerated you could play a YouTube video 4k and see how the CPU is spking vs gpu

that's correct but slightly misleading. In order to get WebRTC VAAPI decoding support on Linux the upstream WebRTC project code would need to support it.

The VAAPI work on Chromium and Firefox regarding videos is separate. AFAIK, no upstream support looks like it will be added soon.

Firefox particularly has problems working with Jitsi and WebRTC because of missing features.

They (Jitsi) did improve their support for non-chrome browser, apparently:

https://github.com/jitsi/jitsi-meet/issues/4758#issuecomment-614063335

Jitsi are also working on E2E encryption but that requires "insertable streams":

https://bugzilla.mozilla.org/show_bug.cgi?id=1631263

In general, here are the issues tagged with jitsi in the firefox bugtracker:

https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=substring&status_whiteboard=jitsi-meet

comment:30 Changed 5 weeks ago by anarcat

Description: modified (diff)

note that BBB requires Ubuntu and will probably fail to install in our debian environments.

comment:31 Changed 5 weeks ago by anarcat

Description: modified (diff)

anadahz has an ansible role for BBB!

comment:32 Changed 5 weeks ago by anarcat

Description: modified (diff)

i'm looking at a budget for hosting a BBB instance and it seems bigbluebutton.org references 5 commercial service providers:

Hopefully that will give us an idea of the commercial side of things.

Also, correction: anadahz setup BBB *using* the Ansible role, they didn't write it from scratch but provided patches to customize it.

There is no recent Puppet module for BBB: https://forge.puppet.com/modules?q=big%20blue%20button I found an old Puppet manifest for BBB 0.8 here: https://github.com/flyapen/puppet-bigbluebutton

The numbers I have right now in terms of labour and hardware requirements are:

  • 2 vCPU and 4GB ram
  • "1 day of work"

That does not include VoIP/dialin configuration and long term support. For example, before next year, Ubuntu 16.04 will be EOL and the entire platform will be upgraded, which could be a significant challenge in terms of work and reliability.

We're having a meeting on monday to clarify the requirements on this project.

What I want to see in the meeting:

  1. discuss process
  2. finalize requirements (e.g. self-hosted? number of participants? scope)
  3. establish timeline

comment:33 in reply to:  32 Changed 5 weeks ago by anarcat

Replying to anarcat:

The numbers I have right now in terms of labour and hardware requirements are:

  • 2 vCPU and 4GB ram
  • "1 day of work"

For the record, I got other feedback from people saying that this was severely under-estimated. The setup, in particular, can take many days. Looking at the setup instructions and Ansible manifest, I can confirm we would definitely have... challenges in making it work in our infrastructure, if only because of Ubuntu and let's encrypt...

comment:34 Changed 4 weeks ago by anarcat

Notes from the jitsi meeting this morning

We started by clearing the air: process wasn't ideal last week, because the budget request didn't come with any context or explanation. So it was difficult to respond. It's important to bring more context for those requests to come through.

As for the background: budget request comes from a 3 year project in africa and latin america. It's part of the 4th phase, to support for partners online

Tor has been doing training in about 11 countries, but has been trying to transition into partners on the ground, for them to do the training. Then the pandemic started and orgs are moving online for training. We reached out to partners to see how they're doing it. Physical meetings are not going to happen. We have a year to figure out what to do with the funder and partners. Two weeks ago gus talked with trainers in brazil, tried jitsi which works well but facing problems for trainings (cannot mute people, cannot share presentations). They tried BBB and it's definitely better than Jitsi for training as it's more like an online classroom.

The requirements we discussed:

  • host partner organizations in a private area in our infrastructure
  • number of participants: 14 organisations with one training per month, max 4 per month, about 14-15 people per session, less than 20
  • make a new proposal for the next phase (starting in june) in two weeks
  • self-hosted seems like an important requirement as the idea is this provides a trusted platform

Other alternatives are not completely eliminated: we should also provide estimates on jitsi and other platforms we (TPA) are considering. And if we strongly oppose hosting a platform, this should be said as well.

Regarding the timeline, we're talking about deploying this within the next year, starting in June, with a plan set within two weeks.

Last edited 4 weeks ago by anarcat (previous) (diff)

comment:35 Changed 4 weeks ago by anarcat

feedback from one of the commercial providers (mconf), paraphrased:

  1. they don't provide support for BBB directly, but instead on https://elos.vc/
  2. they provide a free trial with a special 15-user limit because of covid-19 (was previously 4)
  3. their premium plan is charged by access per month, with 4$ per person per month, with a minimum of 60$ per month (15 people) and an upper limit of 100 simultaneous users

comment:36 Changed 4 weeks ago by anarcat

Description: modified (diff)

move requirements, goals and alternatives to the wiki

comment:37 Changed 4 weeks ago by anarcat

Owner: set to hiro
Status: needs_revisionassigned

i made cost estimates in:

https://help.torproject.org/tsa/howto/conference/#Cost

i asked hiro to take a look, we'll come back with more definite numbers hopefully tomorrow or thursday.

comment:38 Changed 4 weeks ago by anarcat

Owner: changed from hiro to gaba

gaba, the budget is ready in https://help.torproject.org/tsa/howto/conference/#Cost

do not send to funders without a final review on my part please.

comment:39 Changed 3 weeks ago by gaba

Thanks! We will discuss it in the next week.

Note: See TracTickets for help on using tickets.