[sig] Forming a Large scale SIG
Hi everyone, In Shanghai we held a forum session to gauge interest in a new SIG to specifically address cluster scaling issues. In the past we had several groups ("Large deployments", "Performance", LCOO...) but those efforts were arguably a bit too wide and those groups are now abandoned. My main goal here is to get large users directly involved in a domain where their expertise can best translate into improvements in the software. It's easy for such a group to go nowhere while trying to boil the ocean. To maximize its chances of success and make it sustainable, the group should have a narrow focus, and reasonable objectives. My personal idea for the group focus was to specifically address scaling issues within a single cluster: basically identify and address issues that prevent scaling a single cluster (or cell) past a number of nodes. By sharing analysis and experience, the group could identify common pain points that, once solved, would help raising that number. There was a lot of interest in that session[1], and it predictably exploded in lots of different directions, including some that are definitely past a single cluster (like making Neutron better support cells). I think it's fine: my initial proposal was more of a strawman. Active members of the group should really define what they collectively want to work on. And the SIG name should be picked to match that. I'd like to help getting that group off the ground and to a place where it can fly by itself, without needing external coordination. The first step would be to identify interested members and discuss group scope and objectives. Given the nature of the group (with interested members in Japan, Europe, Australia and the US) it will be hard to come up with a synchronous meeting time that will work for everyone, so let's try to hold that discussion over email. So to kick this off: if you are interested in that group, please reply to this email, introduce yourself and tell us what you would like the group scope and objectives to be, and what you can contribute to the group. Thanks! [1] https://etherpad.openstack.org/p/PVG-large-scale-SIG -- Thierry Carrez (ttx)
Hi Thierry & all - Thanks for your mail. I’m interested in joining this SIG. Among others, I’m interested in participating in discussions around these common problems: - golden signals for scaling bottlenecks (and what to do about them) - using Ansible at scale - strategies for simplifying OpenStack functionality in order to scale Cheers, Stig
On 13 Nov 2019, at 11:18, Thierry Carrez <thierry@openstack.org> wrote:
Hi everyone,
In Shanghai we held a forum session to gauge interest in a new SIG to specifically address cluster scaling issues. In the past we had several groups ("Large deployments", "Performance", LCOO...) but those efforts were arguably a bit too wide and those groups are now abandoned.
My main goal here is to get large users directly involved in a domain where their expertise can best translate into improvements in the software. It's easy for such a group to go nowhere while trying to boil the ocean. To maximize its chances of success and make it sustainable, the group should have a narrow focus, and reasonable objectives.
My personal idea for the group focus was to specifically address scaling issues within a single cluster: basically identify and address issues that prevent scaling a single cluster (or cell) past a number of nodes. By sharing analysis and experience, the group could identify common pain points that, once solved, would help raising that number.
There was a lot of interest in that session[1], and it predictably exploded in lots of different directions, including some that are definitely past a single cluster (like making Neutron better support cells). I think it's fine: my initial proposal was more of a strawman. Active members of the group should really define what they collectively want to work on. And the SIG name should be picked to match that.
I'd like to help getting that group off the ground and to a place where it can fly by itself, without needing external coordination. The first step would be to identify interested members and discuss group scope and objectives. Given the nature of the group (with interested members in Japan, Europe, Australia and the US) it will be hard to come up with a synchronous meeting time that will work for everyone, so let's try to hold that discussion over email.
So to kick this off: if you are interested in that group, please reply to this email, introduce yourself and tell us what you would like the group scope and objectives to be, and what you can contribute to the group.
Thanks!
[1] https://etherpad.openstack.org/p/PVG-large-scale-SIG
-- Thierry Carrez (ttx)
Hi all, +1 for me and my employer (OVH). We are mostly interested in sharing good practices when deploying a region at scale, and operating it. For the deployment part, my main pain point is about the configuration parameters I should use on different software (e.g. nova behind wsgi). The current doc is designed to deploy a small pod, but when we are going large, usually some of those params needs tuning. I'd like to identify them and eventually tag them to help other being aware that they are useful at large scale. About operating, I am pretty sure we can share some good advices as well. E.g., avoid restarting neutron agents in a single shot. So definitely interested in that group. Thanks for bringing that up. Cheers. Le mer. 13 nov. 2019 à 19:00, Stig Telfer <stig.openstack@telfer.org> a écrit :
Hi Thierry & all -
Thanks for your mail. I’m interested in joining this SIG. Among others, I’m interested in participating in discussions around these common problems:
- golden signals for scaling bottlenecks (and what to do about them) - using Ansible at scale - strategies for simplifying OpenStack functionality in order to scale
Cheers, Stig
On 13 Nov 2019, at 11:18, Thierry Carrez <thierry@openstack.org> wrote:
Hi everyone,
In Shanghai we held a forum session to gauge interest in a new SIG to specifically address cluster scaling issues. In the past we had several groups ("Large deployments", "Performance", LCOO...) but those efforts were arguably a bit too wide and those groups are now abandoned.
My main goal here is to get large users directly involved in a domain where their expertise can best translate into improvements in the software. It's easy for such a group to go nowhere while trying to boil the ocean. To maximize its chances of success and make it sustainable, the group should have a narrow focus, and reasonable objectives.
My personal idea for the group focus was to specifically address scaling issues within a single cluster: basically identify and address issues that prevent scaling a single cluster (or cell) past a number of nodes. By sharing analysis and experience, the group could identify common pain points that, once solved, would help raising that number.
There was a lot of interest in that session[1], and it predictably exploded in lots of different directions, including some that are definitely past a single cluster (like making Neutron better support cells). I think it's fine: my initial proposal was more of a strawman. Active members of the group should really define what they collectively want to work on. And the SIG name should be picked to match that.
I'd like to help getting that group off the ground and to a place where it can fly by itself, without needing external coordination. The first step would be to identify interested members and discuss group scope and objectives. Given the nature of the group (with interested members in Japan, Europe, Australia and the US) it will be hard to come up with a synchronous meeting time that will work for everyone, so let's try to hold that discussion over email.
So to kick this off: if you are interested in that group, please reply to this email, introduce yourself and tell us what you would like the group scope and objectives to be, and what you can contribute to the group.
Thanks!
[1] https://etherpad.openstack.org/p/PVG-large-scale-SIG
-- Thierry Carrez (ttx)
Hi, first of all thanks to Thierry for driving this SIG creation. Having a SIG to discuss how to deploy/operate a large deployment will be incredibly useful. In my opinion we shouldn't restrict ourselves to a specific project or deployment size (or number of cells) but discuss the limits of each project architecture, the projects dependencies, limitations at scale (functionality vs simplicity), operational difficulties... Sharing experiences and understand the different challenges and actions that we are using to mitigate them will be extremely valuable. I think that we already have a lot of examples of companies/organizations that are deploying OpenStack at large scale. Compiling all this information (Summit presentations, blogs, superuser articles, ...) will be a good starting point for all operators and discussions. Every deployment is different. I also would like this SIG to be the bridge between the operators of large deployments and developers. Bringing specific pain points to discussion with developers. cheers, Belmiro CERN On Thu, Nov 14, 2019 at 8:25 AM Arnaud MORIN <arnaud.morin@gmail.com> wrote:
Hi all,
+1 for me and my employer (OVH). We are mostly interested in sharing good practices when deploying a region at scale, and operating it.
For the deployment part, my main pain point is about the configuration parameters I should use on different software (e.g. nova behind wsgi). The current doc is designed to deploy a small pod, but when we are going large, usually some of those params needs tuning. I'd like to identify them and eventually tag them to help other being aware that they are useful at large scale.
About operating, I am pretty sure we can share some good advices as well. E.g., avoid restarting neutron agents in a single shot.
So definitely interested in that group. Thanks for bringing that up.
Cheers.
Le mer. 13 nov. 2019 à 19:00, Stig Telfer <stig.openstack@telfer.org> a écrit :
Hi Thierry & all -
Thanks for your mail. I’m interested in joining this SIG. Among others, I’m interested in participating in discussions around these common problems:
- golden signals for scaling bottlenecks (and what to do about them) - using Ansible at scale - strategies for simplifying OpenStack functionality in order to scale
Cheers, Stig
On 13 Nov 2019, at 11:18, Thierry Carrez <thierry@openstack.org> wrote:
Hi everyone,
In Shanghai we held a forum session to gauge interest in a new SIG to specifically address cluster scaling issues. In the past we had several groups ("Large deployments", "Performance", LCOO...) but those efforts were arguably a bit too wide and those groups are now abandoned.
My main goal here is to get large users directly involved in a domain where their expertise can best translate into improvements in the software. It's easy for such a group to go nowhere while trying to boil the ocean. To maximize its chances of success and make it sustainable, the group should have a narrow focus, and reasonable objectives.
My personal idea for the group focus was to specifically address scaling issues within a single cluster: basically identify and address issues that prevent scaling a single cluster (or cell) past a number of nodes. By sharing analysis and experience, the group could identify common pain points that, once solved, would help raising that number.
There was a lot of interest in that session[1], and it predictably exploded in lots of different directions, including some that are definitely past a single cluster (like making Neutron better support cells). I think it's fine: my initial proposal was more of a strawman. Active members of the group should really define what they collectively want to work on. And the SIG name should be picked to match that.
I'd like to help getting that group off the ground and to a place where it can fly by itself, without needing external coordination. The first step would be to identify interested members and discuss group scope and objectives. Given the nature of the group (with interested members in Japan, Europe, Australia and the US) it will be hard to come up with a synchronous meeting time that will work for everyone, so let's try to hold that discussion over email.
So to kick this off: if you are interested in that group, please reply to this email, introduce yourself and tell us what you would like the group scope and objectives to be, and what you can contribute to the group.
Thanks!
[1] https://etherpad.openstack.org/p/PVG-large-scale-SIG
-- Thierry Carrez (ttx)
Hi, Belmiro's point of linking operators and developers is hugely important because the developers have the tough job of catering for both large and small deployments. What can look like a safety net for small systems (e.g. per-container file descriptor limits) turns into a huge pitfall when deploying at scale. I'm really interested to be involved in this SIG. Cheers, Dave -- ** Dave Holland ** Systems Support -- Informatics Systems Group ** ** 01223 496923 ** Wellcome Sanger Institute, Hinxton, UK ** On Thu, Nov 14, 2019 at 10:31:10AM +0100, Belmiro Moreira wrote:
Hi, first of all thanks to Thierry for driving this SIG creation. Having a SIG to discuss how to deploy/operate a large deployment will be incredibly useful. In my opinion we shouldn't restrict ourselves to a specific project or deployment size (or number of cells) but discuss the limits of each project architecture, the projects dependencies, limitations at scale (functionality vs simplicity), operational difficulties... Sharing experiences and understand the different challenges and actions that we are using to mitigate them will be extremely valuable. I think that we already have a lot of examples of companies/organizations that are deploying OpenStack at large scale. Compiling all this information (Summit presentations, blogs, superuser articles, ...) will be a good starting point for all operators and discussions. Every deployment is different. I also would like this SIG to be the bridge between the operators of large deployments and developers. Bringing specific pain points to discussion with developers. cheers, Belmiro CERN
On Thu, Nov 14, 2019 at 8:25 AM Arnaud MORIN <[1]arnaud.morin@gmail.com> wrote:
Hi all, +1 for me and my employer (OVH). We are mostly interested in sharing good practices when deploying a region at scale, and operating it. For the deployment part, my main pain point is about the configuration parameters I should use on different software (e.g. nova behind wsgi). The current doc is designed to deploy a small pod, but when we are going large, usually some of those params needs tuning. I'd like to identify them and eventually tag them to help other being aware that they are useful at large scale. About operating, I am pretty sure we can share some good advices as well. E.g., avoid restarting neutron agents in a single shot. So definitely interested in that group. Thanks for bringing that up. Cheers.
Le mer. 13 nov. 2019 à 19:00, Stig Telfer <[2]stig.openstack@telfer.org> a écrit :
On 13 Nov 2019, at 11:18, Thierry Carrez <[3]thierry@openstack.org> wrote:
Hi everyone,
In Shanghai we held a forum session to gauge interest in a new SIG to specifically address cluster scaling issues. In the past we had several groups ("Large deployments", "Performance", LCOO...) but those efforts were arguably a bit too wide and those groups are now abandoned.
My main goal here is to get large users directly involved in a domain where their expertise can best translate into improvements in the software. It's easy for such a group to go nowhere while
Hi Thierry & all - Thanks for your mail. I’m interested in joining this SIG. Among others, I’m interested in participating in discussions around these common problems: - golden signals for scaling bottlenecks (and what to do about them) - using Ansible at scale - strategies for simplifying OpenStack functionality in order to scale Cheers, Stig trying to boil the ocean. To maximize its chances of success and make it sustainable, the group should have a narrow focus, and reasonable objectives.
My personal idea for the group focus was to specifically address
scaling issues within a single cluster: basically identify and address issues that prevent scaling a single cluster (or cell) past a number of nodes. By sharing analysis and experience, the group could identify common pain points that, once solved, would help raising that number.
There was a lot of interest in that session[1], and it
predictably exploded in lots of different directions, including some that are definitely past a single cluster (like making Neutron better support cells). I think it's fine: my initial proposal was more of a strawman. Active members of the group should really define what they collectively want to work on. And the SIG name should be picked to match that.
I'd like to help getting that group off the ground and to a
place where it can fly by itself, without needing external coordination. The first step would be to identify interested members and discuss group scope and objectives. Given the nature of the group (with interested members in Japan, Europe, Australia and the US) it will be hard to come up with a synchronous meeting time that will work for everyone, so let's try to hold that discussion over email.
So to kick this off: if you are interested in that group, please
reply to this email, introduce yourself and tell us what you would like the group scope and objectives to be, and what you can contribute to the group.
Thanks!
[etherpad.openstack.org]
-- Thierry Carrez (ttx)
References
1. mailto:arnaud.morin@gmail.com 2. mailto:stig.openstack@telfer.org 3. mailto:thierry@openstack.org 4. https://urldefense.proofpoint.com/v2/url?u=https-3A__etherpad.openstack.org_p_PVG-2Dlarge-2Dscale-2DSIG&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=64bKjxgut4Pa0xs5b84yPg&m=DdEhOLy_myry74y3z2LhDWbl3ztokcSVufGIqfDSCaM&s=L7GyQqoSsD_56ROhOkKxfMtbER6jrPjcNSZrjNsQrMg&e=
-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
On 11/14/2019 1:10 AM, Arnaud MORIN wrote:
The current doc is designed to deploy a small pod, but when we are going large, usually some of those params needs tuning. I'd like to identify them and eventually tag them to help other being aware that they are useful at large scale.
For anything nova specific you could dump it into [1] with a comment. That's a bug tracking stuff like this that should eventually be documented in nova for large scale performance considerations. [1] https://bugs.launchpad.net/nova/+bug/1838819 -- Thanks, Matt
Hi all, Thank you to Thierry for providing this SIG! +1 for me and my employer (Yahoo! JAPAN). We are interested in building over 1K compute-node scale cluster with an upstreamed code. From YJ, we can share our way of deploying/operating clusters, because we've already deployed/operated many clusters that have several hundred compute. It also may be possible to share the actual performance data of our clusters or to try revised configs to our deployed cluster from this SIG. Cheers, Tatsumi. ________________________________ �����: Arnaud MORIN <arnaud.morin@gmail.com> �����Օr: 2019��11��14�� 16:10 ����: Stig Telfer <stig.openstack@telfer.org> CC: openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org> ����: Re: [sig] Forming a Large scale SIG Hi all, +1 for me and my employer (OVH). We are mostly interested in sharing good practices when deploying a region at scale, and operating it. For the deployment part, my main pain point is about the configuration parameters I should use on different software (e.g. nova behind wsgi). The current doc is designed to deploy a small pod, but when we are going large, usually some of those params needs tuning. I'd like to identify them and eventually tag them to help other being aware that they are useful at large scale. About operating, I am pretty sure we can share some good advices as well. E.g., avoid restarting neutron agents in a single shot. So definitely interested in that group. Thanks for bringing that up. Cheers. Le mer. 13 nov. 2019 �� 19:00, Stig Telfer <stig.openstack@telfer.org<mailto:stig.openstack@telfer.org>> a ��crit : Hi Thierry & all - Thanks for your mail. I��m interested in joining this SIG. Among others, I��m interested in participating in discussions around these common problems: - golden signals for scaling bottlenecks (and what to do about them) - using Ansible at scale - strategies for simplifying OpenStack functionality in order to scale Cheers, Stig
On 13 Nov 2019, at 11:18, Thierry Carrez <thierry@openstack.org<mailto:thierry@openstack.org>> wrote:
Hi everyone,
In Shanghai we held a forum session to gauge interest in a new SIG to specifically address cluster scaling issues. In the past we had several groups ("Large deployments", "Performance", LCOO...) but those efforts were arguably a bit too wide and those groups are now abandoned.
My main goal here is to get large users directly involved in a domain where their expertise can best translate into improvements in the software. It's easy for such a group to go nowhere while trying to boil the ocean. To maximize its chances of success and make it sustainable, the group should have a narrow focus, and reasonable objectives.
My personal idea for the group focus was to specifically address scaling issues within a single cluster: basically identify and address issues that prevent scaling a single cluster (or cell) past a number of nodes. By sharing analysis and experience, the group could identify common pain points that, once solved, would help raising that number.
There was a lot of interest in that session[1], and it predictably exploded in lots of different directions, including some that are definitely past a single cluster (like making Neutron better support cells). I think it's fine: my initial proposal was more of a strawman. Active members of the group should really define what they collectively want to work on. And the SIG name should be picked to match that.
I'd like to help getting that group off the ground and to a place where it can fly by itself, without needing external coordination. The first step would be to identify interested members and discuss group scope and objectives. Given the nature of the group (with interested members in Japan, Europe, Australia and the US) it will be hard to come up with a synchronous meeting time that will work for everyone, so let's try to hold that discussion over email.
So to kick this off: if you are interested in that group, please reply to this email, introduce yourself and tell us what you would like the group scope and objectives to be, and what you can contribute to the group.
Thanks!
[1] https://etherpad.openstack.org/p/PVG-large-scale-SIG
-- Thierry Carrez (ttx)
On Wed, Nov 13, 2019 at 6:22 AM Thierry Carrez <thierry@openstack.org> wrote:
Hi everyone,
In Shanghai we held a forum session to gauge interest in a new SIG to specifically address cluster scaling issues. In the past we had several groups ("Large deployments", "Performance", LCOO...) but those efforts were arguably a bit too wide and those groups are now abandoned.
My main goal here is to get large users directly involved in a domain where their expertise can best translate into improvements in the software. It's easy for such a group to go nowhere while trying to boil the ocean. To maximize its chances of success and make it sustainable, the group should have a narrow focus, and reasonable objectives.
My personal idea for the group focus was to specifically address scaling issues within a single cluster: basically identify and address issues that prevent scaling a single cluster (or cell) past a number of nodes. By sharing analysis and experience, the group could identify common pain points that, once solved, would help raising that number.
There was a lot of interest in that session[1], and it predictably exploded in lots of different directions, including some that are definitely past a single cluster (like making Neutron better support cells). I think it's fine: my initial proposal was more of a strawman. Active members of the group should really define what they collectively want to work on. And the SIG name should be picked to match that.
I'd like to help getting that group off the ground and to a place where it can fly by itself, without needing external coordination. The first step would be to identify interested members and discuss group scope and objectives. Given the nature of the group (with interested members in Japan, Europe, Australia and the US) it will be hard to come up with a synchronous meeting time that will work for everyone, so let's try to hold that discussion over email.
So to kick this off: if you are interested in that group, please reply to this email, introduce yourself and tell us what you would like the group scope and objectives to be, and what you can contribute to the group.
Count me in, I'll be watching from the sidelines and chiming in when I see things happen and come up.
Thanks!
[1] https://etherpad.openstack.org/p/PVG-large-scale-SIG
-- Thierry Carrez (ttx)
-- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser@vexxhost.com W. https://vexxhost.com
Hi Thierry and All, Sorry for the late reply. Thank you for these efforts. +1 from me and my employer LINE. Regards, Dinesh Bhor On Wed, 13 Nov 2019, 8:19 pm Thierry Carrez, <thierry@openstack.org> wrote:
Hi everyone,
In Shanghai we held a forum session to gauge interest in a new SIG to specifically address cluster scaling issues. In the past we had several groups ("Large deployments", "Performance", LCOO...) but those efforts were arguably a bit too wide and those groups are now abandoned.
My main goal here is to get large users directly involved in a domain where their expertise can best translate into improvements in the software. It's easy for such a group to go nowhere while trying to boil the ocean. To maximize its chances of success and make it sustainable, the group should have a narrow focus, and reasonable objectives.
My personal idea for the group focus was to specifically address scaling issues within a single cluster: basically identify and address issues that prevent scaling a single cluster (or cell) past a number of nodes. By sharing analysis and experience, the group could identify common pain points that, once solved, would help raising that number.
There was a lot of interest in that session[1], and it predictably exploded in lots of different directions, including some that are definitely past a single cluster (like making Neutron better support cells). I think it's fine: my initial proposal was more of a strawman. Active members of the group should really define what they collectively want to work on. And the SIG name should be picked to match that.
I'd like to help getting that group off the ground and to a place where it can fly by itself, without needing external coordination. The first step would be to identify interested members and discuss group scope and objectives. Given the nature of the group (with interested members in Japan, Europe, Australia and the US) it will be hard to come up with a synchronous meeting time that will work for everyone, so let's try to hold that discussion over email.
So to kick this off: if you are interested in that group, please reply to this email, introduce yourself and tell us what you would like the group scope and objectives to be, and what you can contribute to the group.
Thanks!
[1] https://etherpad.openstack.org/p/PVG-large-scale-SIG
-- Thierry Carrez (ttx)
Thanks everyone for volunteering to participate to this group! We'll have our first meeting next week, on IRC (#openstack-meeting), on Wednesday, November 27, 09:00 UTC: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20191127T09 The agenda will be the following: - Agree on SIG name - Volunteers for SIG chairing - Need for synchronous meetings - If meetings are needed, meeting frequency and tooling - If any time is left, discuss initial SIG objectives Notes at https://etherpad.openstack.org/p/large-scale-sig-meeting Hoping to see you there, -- Thierry Carrez (ttx)
participants (9)
-
Arnaud MORIN
-
Belmiro Moreira
-
Dave Holland
-
Dinesh Bhor
-
Matt Riedemann
-
Mohammed Naser
-
Stig Telfer
-
Thierry Carrez
-
Yusuke Tatsumi