[ops] ops meetups team meeting 2020-5-5
Hey all, The openstack ops meetups team has been fairly dormant recently for obvious reasons. However with the realisation that in-person meetups, conferences, summits etc are simply not coming back any time soon, we've started to regroup to discuss next steps. We are also seeing some requests for virtual meetups from some operators via the OpsMeetup twitter account. Thus we had a quick IRC meeting today, and also a trial run at an open source based video conference meeting using jitsi via an instance running on infra provided by Erik McCormick. This seems to be promising. We'll look into trialling some ops related events leveraging this. We also agreed to try and have an operators session during the upcoming virtual PTG (replacing the Vancouver event now cancelled). More info will be shared as and when we can. I hope you're all doing as well as can be expected. Chris - on behalf of the ops meetups team Meeting minutes: Minutes: http://eavesdrop.openstack.org/meetings/ops_meetup_team/2020/ops_meetup_team... 12:01 PM Minutes (text): http://eavesdrop.openstack.org/meetings/ops_meetup_team/2020/ops_meetup_team... 12:01 PM Log: http://eavesdrop.openstack.org/meetings/ops_meetup_team/2020/ops_meetup_team... -- Chris Morgan <mihalis68@gmail.com>
On 2020-05-05 12:06:42 -0400 (-0400), Chris Morgan wrote: [...]
we had a quick IRC meeting today, and also a trial run at an open source based video conference meeting using jitsi via an instance running on infra provided by Erik McCormick. This seems to be promising. We'll look into trialling some ops related events leveraging this. [...]
It's probably been flying under the radar a bit so far, but the OpenDev community has put together an Etherpad-integrated Jitsi-Meet service at https://meetpad.opendev.org/ which you're free to try out as well. We'd love feedback and help tuning it. Also if you want to reuse anything we've done to set it up, the Ansible playbook we use is here: https://opendev.org/opendev/system-config/src/branch/master/playbooks/servic... It utilizes this role to install and configure jitsi-meet containers with docker-compose (mostly official docker.io/jitsi images, though we build our own jitsi-meet-web published under docker.io/opendevorg which applies our Etherpad integration patch): https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/... We're not making any stability or reusability guarantees on the Ansible orchestration (or our custom image which will hopefully disappear once https://github.com/jitsi/jitsi-meet/pull/5270 is accepted upstream), but like everything we run in OpenDev we publish it for the sake of transparency, in case anyone else wants to help us or take some ideas for their own efforts. -- Jeremy Stanley
this is super cool, thanks Jeremy, will give it a go! Chris On Tue, May 5, 2020 at 12:57 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2020-05-05 12:06:42 -0400 (-0400), Chris Morgan wrote: [...]
we had a quick IRC meeting today, and also a trial run at an open source based video conference meeting using jitsi via an instance running on infra provided by Erik McCormick. This seems to be promising. We'll look into trialling some ops related events leveraging this. [...]
It's probably been flying under the radar a bit so far, but the OpenDev community has put together an Etherpad-integrated Jitsi-Meet service at https://meetpad.opendev.org/ which you're free to try out as well. We'd love feedback and help tuning it. Also if you want to reuse anything we've done to set it up, the Ansible playbook we use is here:
https://opendev.org/opendev/system-config/src/branch/master/playbooks/servic...
It utilizes this role to install and configure jitsi-meet containers with docker-compose (mostly official docker.io/jitsi images, though we build our own jitsi-meet-web published under docker.io/opendevorg which applies our Etherpad integration patch):
https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/...
We're not making any stability or reusability guarantees on the Ansible orchestration (or our custom image which will hopefully disappear once https://github.com/jitsi/jitsi-meet/pull/5270 is accepted upstream), but like everything we run in OpenDev we publish it for the sake of transparency, in case anyone else wants to help us or take some ideas for their own efforts. -- Jeremy Stanley
-- Chris Morgan <mihalis68@gmail.com>
We on the ops meetups team had a trial meeting on meetpad.opendev.org this morning and for the most part it worked very well (detailed feedback below). Speaking personally, I am very happy to see an open source solution for video conferencing being adopted by the foundation to some extent. I had and continue to have reservations about Zoom, but at the end of the day no matter how well they respond to the security and privacy concerns it will still be a proprietary solution and no more true to the openstack tenets than slack is as a replacement for irc. feedback - etherpad integration is cool but several of us found the window seemed to disappear inexplicably - colored highlighting of fragments on the etherpad showed up overlapped for some but not all meeting members, obscuring some text - background blurring seemed very heavy for some participants computers and this possibly lead to some sessions locking up - it's not clear what named meetings persistence is, for example a named meeting from yesterday is still shown today, but doesn't have the password I applied yesterday My guess is some (all?) of this is just how Jitsi is right now. The team (openstack ops meetups) is now talking about possibly hosting a global ops meetup on this platform. How can we determine when and if the infra for this is ready for it, and how many participants is a reasonable cap? What about streaming, can we stream the whole thing continuously to youtube? We are thinking that each topic would have a small number of presenters/participants in the video conference itself, but allow a larger group to see it on youtube and contribute via the etherpad. Is this a reasonable plan? Thanks for doing this! Chris On Tue, May 5, 2020 at 12:57 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2020-05-05 12:06:42 -0400 (-0400), Chris Morgan wrote: [...]
we had a quick IRC meeting today, and also a trial run at an open source based video conference meeting using jitsi via an instance running on infra provided by Erik McCormick. This seems to be promising. We'll look into trialling some ops related events leveraging this. [...]
It's probably been flying under the radar a bit so far, but the OpenDev community has put together an Etherpad-integrated Jitsi-Meet service at https://meetpad.opendev.org/ which you're free to try out as well. We'd love feedback and help tuning it. Also if you want to reuse anything we've done to set it up, the Ansible playbook we use is here:
https://opendev.org/opendev/system-config/src/branch/master/playbooks/servic...
It utilizes this role to install and configure jitsi-meet containers with docker-compose (mostly official docker.io/jitsi images, though we build our own jitsi-meet-web published under docker.io/opendevorg which applies our Etherpad integration patch):
https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/...
We're not making any stability or reusability guarantees on the Ansible orchestration (or our custom image which will hopefully disappear once https://github.com/jitsi/jitsi-meet/pull/5270 is accepted upstream), but like everything we run in OpenDev we publish it for the sake of transparency, in case anyone else wants to help us or take some ideas for their own efforts. -- Jeremy Stanley
-- Chris Morgan <mihalis68@gmail.com>
On Thu, May 14, 2020, 1:44 PM Chris Morgan <mihalis68@gmail.com> wrote:
We on the ops meetups team had a trial meeting on meetpad.opendev.org this morning and for the most part it worked very well (detailed feedback below). Speaking personally, I am very happy to see an open source solution for video conferencing being adopted by the foundation to some extent. I had and continue to have reservations about Zoom, but at the end of the day no matter how well they respond to the security and privacy concerns it will still be a proprietary solution and no more true to the openstack tenets than slack is as a replacement for irc.
feedback
- etherpad integration is cool but several of us found the window seemed to disappear inexplicably - colored highlighting of fragments on the etherpad showed up overlapped for some but not all meeting members, obscuring some text - background blurring seemed very heavy for some participants computers and this possibly lead to some sessions locking up - it's not clear what named meetings persistence is, for example a named meeting from yesterday is still shown today, but doesn't have the password I applied yesterday
I can answer this one. The list on the landing page is merely a list of your history, and yours alone. It has no bearing on the persistence of a room. Rooms are normally ephemeral and vanish from the server side when the last person leaves it.
My guess is some (all?) of this is just how Jitsi is right now.
The team (openstack ops meetups) is now talking about possibly hosting a global ops meetup on this platform. How can we determine when and if the infra for this is ready for it, and how many participants is a reasonable cap? What about streaming, can we stream the whole thing continuously to youtube? We are thinking that each topic would have a small number of presenters/participants in the video conference itself, but allow a larger group to see it on youtube and contribute via the etherpad. Is this a reasonable plan?
Thanks for doing this!
Chris
On Tue, May 5, 2020 at 12:57 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2020-05-05 12:06:42 -0400 (-0400), Chris Morgan wrote: [...]
we had a quick IRC meeting today, and also a trial run at an open source based video conference meeting using jitsi via an instance running on infra provided by Erik McCormick. This seems to be promising. We'll look into trialling some ops related events leveraging this. [...]
It's probably been flying under the radar a bit so far, but the OpenDev community has put together an Etherpad-integrated Jitsi-Meet service at https://meetpad.opendev.org/ which you're free to try out as well. We'd love feedback and help tuning it. Also if you want to reuse anything we've done to set it up, the Ansible playbook we use is here:
https://opendev.org/opendev/system-config/src/branch/master/playbooks/servic...
It utilizes this role to install and configure jitsi-meet containers with docker-compose (mostly official docker.io/jitsi images, though we build our own jitsi-meet-web published under docker.io/opendevorg which applies our Etherpad integration patch):
https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/...
We're not making any stability or reusability guarantees on the Ansible orchestration (or our custom image which will hopefully disappear once https://github.com/jitsi/jitsi-meet/pull/5270 is accepted upstream), but like everything we run in OpenDev we publish it for the sake of transparency, in case anyone else wants to help us or take some ideas for their own efforts. -- Jeremy Stanley
-- Chris Morgan <mihalis68@gmail.com>
-Erik
On Thu, 2020-05-14 at 13:59 -0400, Erik McCormick wrote:
- it's not clear what named meetings persistence is, for example a named meeting from yesterday is still shown today, but doesn't have the password I applied yesterday
just a a genereral pratice we proably should not be using passwords for our meetings. we pratice open development and design across openstack and opendev projects ingeneral so unless jitsi require a password to have a meeting we shoudl proably not set one. if we do set one we shoudl list it publicaly is the correstpoding etherpad , wiki meeting email or where ever the meeting is annoched/tracked so people can join freely.
On Thu, May 14, 2020, at 10:42 AM, Chris Morgan wrote:
We on the ops meetups team had a trial meeting on meetpad.opendev.org this morning and for the most part it worked very well (detailed feedback below). Speaking personally, I am very happy to see an open source solution for video conferencing being adopted by the foundation to some extent. I had and continue to have reservations about Zoom, but at the end of the day no matter how well they respond to the security and privacy concerns it will still be a proprietary solution and no more true to the openstack tenets than slack is as a replacement for irc.
feedback
- etherpad integration is cool but several of us found the window seemed to disappear inexplicably
It seems that when a call starts the etherpad starts "pinned" but then certain events can cause jitsi to go back to its normal "focus on the person talking" mode of operation. In the bottom right is a "3 dot" menu and from there you can open and close the "shared document" this allows you to toggle the etherpad manually. Even after toggling it directly it seems that jitsi wants to keep focus on speakers in some cases. I've been fiddling with it to try and understand that better and it seems like some of these things may help (though I can't say for sure): * Click in the document directly * Collapse the right hand column of mini webcams using the > in the bottom right
- colored highlighting of fragments on the etherpad showed up overlapped for some but not all meeting members, obscuring some text
I've noticed this too. Toggling authorship colors in etherpad's settings menu seems to correct this.
- background blurring seemed very heavy for some participants computers and this possibly lead to some sessions locking up
I believe this is specifically listed as an "experimental" feature in the menu. I expect this is why. Should probably avoid using it.
- it's not clear what named meetings persistence is, for example a named meeting from yesterday is still shown today, but doesn't have the password I applied yesterday
My guess is some (all?) of this is just how Jitsi is right now.
Few other things we've noticed: * Chrome/Chromium seem more reliable than Firefox * The jitsi logo watermark in the overlay can clip text in the etherpad. * If your webcam is operating in a resolution that your browser and/or jitsi don't like then you'll get a message about the webcam not satisfying required constraints and it will refuse to work. At least one person had this happen because their webcam resolution was tiny (100x100 ish pixels). I expect these issues will improve over time. If people know how to address these problems we're more than happy to help incorporate fixes into this deployment.
The team (openstack ops meetups) is now talking about possibly hosting a global ops meetup on this platform. How can we determine when and if the infra for this is ready for it, and how many participants is a reasonable cap? What about streaming, can we stream the whole thing continuously to youtube? We are thinking that each topic would have a small number of presenters/participants in the video conference itself, but allow a larger group to see it on youtube and contribute via the etherpad. Is this a reasonable plan?
I think we are ready for people to use it understanding its a new service we are trying to run and there may be hiccups. In particular I don't think we've been able to do any serious scale testing yet. If you are using it and find those limits we'd love to hear about it. Also feedback like what you posted above is great too. Jitsi does have a component, jibri, which enables streaming (and recording). I don't think we've deployed any of that toolchain yet. It appears that a jibri process per recording is required and distributing jibri instances across servers is recommended for performance reasons. If there is interest in pushing patches to add a jibri instances I expect you'll find plenty of help. Zuul acts as our continuous deployment management system and all of the configuration for that is in the open at https://opendev.org/opendev/system-config. playbooks/roles/jitsi-meet/ is probably a good place to start looking. One drawback to the "few presenters, many participants" model for collaborative discussions is that you're preselecting who can get their voice heard. It might be best to find the limits of the existing tool before we design solutions for potential bottlenecks? I expect there will be limitations, we just don't know what they are yet so designing solutions is difficult.
Thanks for doing this!
Chris
On Tue, May 5, 2020 at 12:57 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2020-05-05 12:06:42 -0400 (-0400), Chris Morgan wrote: [...]
we had a quick IRC meeting today, and also a trial run at an open source based video conference meeting using jitsi via an instance running on infra provided by Erik McCormick. This seems to be promising. We'll look into trialling some ops related events leveraging this. [...]
It's probably been flying under the radar a bit so far, but the OpenDev community has put together an Etherpad-integrated Jitsi-Meet service at https://meetpad.opendev.org/ which you're free to try out as well. We'd love feedback and help tuning it. Also if you want to reuse anything we've done to set it up, the Ansible playbook we use is here:
https://opendev.org/opendev/system-config/src/branch/master/playbooks/servic...
It utilizes this role to install and configure jitsi-meet containers with docker-compose (mostly official docker.io/jitsi images, though we build our own jitsi-meet-web published under docker.io/opendevorg which applies our Etherpad integration patch):
https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/...
We're not making any stability or reusability guarantees on the Ansible orchestration (or our custom image which will hopefully disappear once https://github.com/jitsi/jitsi-meet/pull/5270 is accepted upstream), but like everything we run in OpenDev we publish it for the sake of transparency, in case anyone else wants to help us or take some ideas for their own efforts. -- Jeremy Stanley
-- Chris Morgan <mihalis68@gmail.com>
On 2020-05-14 13:17:53 -0700 (-0700), Clark Boylan wrote:
On Thu, May 14, 2020, at 10:42 AM, Chris Morgan wrote: [...]
- etherpad integration is cool but several of us found the window seemed to disappear inexplicably
It seems that when a call starts the etherpad starts "pinned" but then certain events can cause jitsi to go back to its normal "focus on the person talking" mode of operation. In the bottom right is a "3 dot" menu and from there you can open and close the "shared document" this allows you to toggle the etherpad manually.
Even after toggling it directly it seems that jitsi wants to keep focus on speakers in some cases. I've been fiddling with it to try and understand that better and it seems like some of these things may help (though I can't say for sure):
* Click in the document directly
* Collapse the right hand column of mini webcams using the > in the bottom right
Even doing these, I found that at times it would switch back to an active speaker faster than I could click in the pad after redisplaying.
- colored highlighting of fragments on the etherpad showed up overlapped for some but not all meeting members, obscuring some text
I've noticed this too. Toggling authorship colors in etherpad's settings menu seems to correct this. [...]
Temporarily at least, though it does seem to recur. Also this is a local setting, it does not alter how other users see the pad (it's only changing the author colors setting for your view of the pad). Leaving author colors off may work around it, though at a loss of that feature of course.
* Chrome/Chromium seem more reliable than Firefox
At least that's how it was for me when I tried it. I normally use FF (76 at present) and so have it configured for incognito mode always, all sorts of possible snooping avenues disabled in preferences, and several security/privacy-oriented extensions for blocking unwanted content and access. Things worked well for me in Chromium (81), but that may be because it's on default settings with no extensions. It's quite possible FireFox works just fine for this, as long as it's not configured by a paranoid nutcase like me. FF kept showing my camera footage to me, and seemed to be picking up my microphone and outputting to my speakers, but did not send or receive any video or audio streams over the network whatsoever. I'll likely stick with Chromium for meetpad use (and use it only for that), as it provides me a nice privacy buffer from all my other browsing by being an entirely separate application.
I think we are ready for people to use it understanding its a new service we are trying to run and there may be hiccups. In particular I don't think we've been able to do any serious scale testing yet. If you are using it and find those limits we'd love to hear about it. Also feedback like what you posted above is great too. [...]
I think Kendall Nelson is planning to use it as part of the community release celebration later today, so hopefully we'll get a bit more scale testing that way. -- Jeremy Stanley
On 2020-05-14 20:47:37 +0000 (+0000), Jeremy Stanley wrote: [...]
I think Kendall Nelson is planning to use it as part of the community release celebration later today, so hopefully we'll get a bit more scale testing that way.
I guess that's actually tomorrow (Friday) at 20:00 UTC. I've lost all track of time. -- Jeremy Stanley
On Thu, May 14, 2020, at 1:17 PM, Clark Boylan wrote:
On Thu, May 14, 2020, at 10:42 AM, Chris Morgan wrote:
We on the ops meetups team had a trial meeting on meetpad.opendev.org this morning and for the most part it worked very well (detailed feedback below). Speaking personally, I am very happy to see an open source solution for video conferencing being adopted by the foundation to some extent. I had and continue to have reservations about Zoom, but at the end of the day no matter how well they respond to the security and privacy concerns it will still be a proprietary solution and no more true to the openstack tenets than slack is as a replacement for irc.
feedback
- etherpad integration is cool but several of us found the window seemed to disappear inexplicably
It seems that when a call starts the etherpad starts "pinned" but then certain events can cause jitsi to go back to its normal "focus on the person talking" mode of operation. In the bottom right is a "3 dot" menu and from there you can open and close the "shared document" this allows you to toggle the etherpad manually.
Even after toggling it directly it seems that jitsi wants to keep focus on speakers in some cases. I've been fiddling with it to try and understand that better and it seems like some of these things may help (though I can't say for sure):
* Click in the document directly * Collapse the right hand column of mini webcams using the > in the bottom right
- colored highlighting of fragments on the etherpad showed up overlapped for some but not all meeting members, obscuring some text
I've noticed this too. Toggling authorship colors in etherpad's settings menu seems to correct this.
- background blurring seemed very heavy for some participants computers and this possibly lead to some sessions locking up
I believe this is specifically listed as an "experimental" feature in the menu. I expect this is why. Should probably avoid using it.
- it's not clear what named meetings persistence is, for example a named meeting from yesterday is still shown today, but doesn't have the password I applied yesterday
My guess is some (all?) of this is just how Jitsi is right now.
Few other things we've noticed:
* Chrome/Chromium seem more reliable than Firefox
Apparently there is some technical reason for this around how chrom* and firefox handle webrtc video? I don't have the details but apparently this is expected. Additionally, if you've found that the web browser version is giving you trouble, the mobile (iOS and Android) jitsi meet apps apparently work quite well. Since we're hosting our own server you need to update the settings to point to our server and not the default server, but otherwise people have reported this works well. On Android you can change the server by opening the in app menu, going to settings, and changing the server URL to https://meetpad.opendev.org. I assume its similar on iOS but don't have an iOS device to confirm. Hope this helps, Clark
participants (5)
-
Chris Morgan
-
Clark Boylan
-
Erik McCormick
-
Jeremy Stanley
-
Sean Mooney