[puppet] Configuring Debian's uwsgi for each service
Hi, In Debian all service APIs are by default using uwsgi. This means that there's no reason to setup Apache for the API, as uwsgi comes by default. However, there are a number of things that operators may want to configure, like for example, the number of workers. This is why I started this PR: https://review.opendev.org/725065 Since Tobias asked for more details about what I was doing, I thought it was a good idea to start a thread in the list, rather than just discussing in the patch review. So, what I did for Neutron is only the beginning. If it works as I expect, I intend to generalize this on all OpenStack services supported by Debian. This means at least: - nova - keystone - glance - cinder - barbican - neutron - aodh - panko - gnocchi - etc. The first step is having this neutron_uwsgi_config provider up and running, then I'll see how to integrate with the rest of puppet-openstack and have things like number of workers integrated with <SERVICE>::api stuff... Your thoughts? Cheers, Thomas Goirand (zigo)
Hello Thomas, Thanks for starting this thread! I've been thinking about this as well, but due to other reason, how we can continue to abstract the service layer used in the Puppet modules. The most deployed way right now is the <module>::wsgi::apache (using puppetlabs-apache). Do you think there is a way we could move uWSGI into it's own <module>::wsgi::uwsgi layer and then use an upstream uWSGI module to do the actual configuration? Since uWSGI isn't in the OpenStack scope, same as Apache configuration hence the usage of puppetlabs-apache. I've been thinking about a similar concept with other service layers, like Docker with Kolla containers etc even though it's kind of out of scope for the Puppet OpenStack project, just thoughts. Best regards Tobias ________________________________________ From: Thomas Goirand <zigo@debian.org> Sent: Sunday, May 3, 2020 9:16 PM To: openstack-discuss Subject: [puppet] Configuring Debian's uwsgi for each service Hi, In Debian all service APIs are by default using uwsgi. This means that there's no reason to setup Apache for the API, as uwsgi comes by default. However, there are a number of things that operators may want to configure, like for example, the number of workers. This is why I started this PR: https://review.opendev.org/725065 Since Tobias asked for more details about what I was doing, I thought it was a good idea to start a thread in the list, rather than just discussing in the patch review. So, what I did for Neutron is only the beginning. If it works as I expect, I intend to generalize this on all OpenStack services supported by Debian. This means at least: - nova - keystone - glance - cinder - barbican - neutron - aodh - panko - gnocchi - etc. The first step is having this neutron_uwsgi_config provider up and running, then I'll see how to integrate with the rest of puppet-openstack and have things like number of workers integrated with <SERVICE>::api stuff... Your thoughts? Cheers, Thomas Goirand (zigo)
On 5/3/20 10:17 PM, Tobias Urdin wrote:
Hello Thomas, Thanks for starting this thread!
I've been thinking about this as well, but due to other reason, how we can continue to abstract the service layer used in the Puppet modules.
The most deployed way right now is the <module>::wsgi::apache (using puppetlabs-apache).
Do you think there is a way we could move uWSGI into it's own <module>::wsgi::uwsgi layer and then use an upstream uWSGI module to do the actual configuration? Since uWSGI isn't in the OpenStack scope, same as Apache configuration hence the usage of puppetlabs-apache.
I've been thinking about a similar concept with other service layers, like Docker with Kolla containers etc even though it's kind of out of scope for the Puppet OpenStack project, just thoughts.
Best regards Tobias
Hi Tobias, What you must know, is that uwsgi is not in main in Ubuntu (it is in "universe", meaning it's just imported from Debian Sid when they release the LTS). This means that Canonical don't provide support for it at all. If one day, there's a security issue on it, Canonical will claim it is "community maintained" and therefore, they will put no effort on upgrading it. The consequence is that it's not reasonable to use uwsgi in Ubuntu, unless the situation with this package changes. We could still support it, but I wouldn't recommend it for security and maintainability reasons. In Debian, the situation is very different. All OpenStack services are using uwsgi directly, and don't need to be configured in any way to use uwsgi. You just install <FOO>-api and it runs over UWSGI. So, in terms of puppet, for Debian, the only thing we need to do is to use something like this: service{ 'foo-api':} which looks like what we used to do with Eventlet. Now, I don't know what the situation is on CentOS/Red Hat. If you think it's valuable to support uwsgi there, then why not. But I'm not interested in contributing it. What I care right now, is being able to configure the uwsgi thing for the API. Namely, being able to configure the "processes" directive is the most important bit. Just having the puppet provider could be enough, though integrating it with the API's <service>::api "workers" parameter would be nice. Your thoughts? Cheers, Thomas Goirand (zigo)
From a RHEL/CentOS perspective, uWSGI is also not packaged in the main repositories meaning it is effectively unsupported as well. It is in the Extra Packages for Enterprise Linux (EPEL) repository but we do not support mixing our RDO (OpenStack) packages with EPEL as it will almost always guarantee issues with dependencies. In the TripleO community, because of
Hey folks, this, we have had to stick with using Apache's WSGI module for serving the OpenStack API endpoints. I do think the proposal sounds great and if anyone wants to work on adding the Puppet bits to support uWSGI then I do not believe anyone will stop you. The real-world testing and usage of it by other distributions may be minimal due to these packaging and support restraints. Alternatively you may also want to look into OpenStack-Ansible as they heavily use uWSGI <https://github.com/openstack/ansible-role-uwsgi> with Ubuntu based containers managed by Ansible. Sincerely, Luke Short On Mon, May 4, 2020 at 6:35 AM Thomas Goirand <zigo@debian.org> wrote:
Hello Thomas, Thanks for starting this thread!
I've been thinking about this as well, but due to other reason, how we can continue to abstract the service layer used in the Puppet modules.
The most deployed way right now is the <module>::wsgi::apache (using
On 5/3/20 10:17 PM, Tobias Urdin wrote: puppetlabs-apache).
Do you think there is a way we could move uWSGI into it's own
upstream uWSGI module to do the actual configuration? Since uWSGI isn't in the OpenStack scope, same as Apache configuration hence the usage of puppetlabs-apache.
I've been thinking about a similar concept with other service layers,
though it's kind of out of scope for the Puppet OpenStack project, just
<module>::wsgi::uwsgi layer and then use an like Docker with Kolla containers etc even thoughts.
Best regards Tobias
Hi Tobias,
What you must know, is that uwsgi is not in main in Ubuntu (it is in "universe", meaning it's just imported from Debian Sid when they release the LTS). This means that Canonical don't provide support for it at all. If one day, there's a security issue on it, Canonical will claim it is "community maintained" and therefore, they will put no effort on upgrading it.
The consequence is that it's not reasonable to use uwsgi in Ubuntu, unless the situation with this package changes. We could still support it, but I wouldn't recommend it for security and maintainability reasons.
In Debian, the situation is very different. All OpenStack services are using uwsgi directly, and don't need to be configured in any way to use uwsgi. You just install <FOO>-api and it runs over UWSGI. So, in terms of puppet, for Debian, the only thing we need to do is to use something like this:
service{ 'foo-api':}
which looks like what we used to do with Eventlet.
Now, I don't know what the situation is on CentOS/Red Hat. If you think it's valuable to support uwsgi there, then why not. But I'm not interested in contributing it.
What I care right now, is being able to configure the uwsgi thing for the API. Namely, being able to configure the "processes" directive is the most important bit. Just having the puppet provider could be enough, though integrating it with the API's <service>::api "workers" parameter would be nice.
Your thoughts?
Cheers,
Thomas Goirand (zigo)
On 5/4/20 3:37 PM, Luke Short wrote:
Hey folks,
From a RHEL/CentOS perspective, uWSGI is also not packaged in the main repositories meaning it is effectively unsupported as well. It is in the Extra Packages for Enterprise Linux (EPEL) repository but we do not support mixing our RDO (OpenStack) packages with EPEL as it will almost always guarantee issues with dependencies. In the TripleO community, because of this, we have had to stick with using Apache's WSGI module for serving the OpenStack API endpoints.
I do think the proposal sounds great and if anyone wants to work on adding the Puppet bits to support uWSGI then I do not believe anyone will stop you. The real-world testing and usage of it by other distributions may be minimal due to these packaging and support restraints. Alternatively you may also want to look into OpenStack-Ansible as they heavily use uWSGI <https://github.com/openstack/ansible-role-uwsgi> with Ubuntu based containers managed by Ansible.
Sincerely, Luke Short
Hi, Ok, so we may as well forget having support for Red Hat, unless the situation changes there too. IMO, it's a shame that uwsgi doesn't have enough support in downstream distros. :( Anyway, let's move forward, since it looks like everybody is happy about my proposal. One of the problems is that I've added some unit tests, but they can't be launch because currently there's no Debian testing activated. Hopefully, I'll be able to work again on having Debian to gate on the OpenDev CI again, and have a colleague to co-maintain it. Does anyone have an idea on how to fix the situation before this happens? Though I've tested locally this patch: https://review.opendev.org/725065 tweaking the amount of processes with this code: neutron_uwsgi_config { 'uwsgi/processes': value => 12; } and it worked as expected (ie: the number of processes switches from 8 to 12, which is what I wanted). The only problem is that I would expect the neutron-api service to be restarted if one of the neutron_uwsgi_config resource is changed, and this didn't happen. How can I fix this? Also, how can I make sure resources are defined only once, like with neutron_config? Cheersm Thomas Goirand (zigo)
On 2020-05-04 23:00:11 +0200 (+0200), Thomas Goirand wrote: [...]
One of the problems is that I've added some unit tests, but they can't be launch because currently there's no Debian testing activated. Hopefully, I'll be able to work again on having Debian to gate on the OpenDev CI again, and have a colleague to co-maintain it. Does anyone have an idea on how to fix the situation before this happens? [...]
Assistance keeping the image builds working as things change in relevant distros is of course highly appreciated, and even necessary. That said, we have debian-buster and debian-buster-arm64 images currently you can run jobs on, the image updates seems to be in working order. We can probably get some debian-bullseye or debian-sid images implemented too if there's sufficient demand. -- Jeremy Stanley
On 5/5/20 12:15 AM, Jeremy Stanley wrote:
On 2020-05-04 23:00:11 +0200 (+0200), Thomas Goirand wrote: [...]
One of the problems is that I've added some unit tests, but they can't be launch because currently there's no Debian testing activated. Hopefully, I'll be able to work again on having Debian to gate on the OpenDev CI again, and have a colleague to co-maintain it. Does anyone have an idea on how to fix the situation before this happens? [...]
Assistance keeping the image builds working as things change in relevant distros is of course highly appreciated, and even necessary. That said, we have debian-buster and debian-buster-arm64 images currently you can run jobs on, the image updates seems to be in working order. We can probably get some debian-bullseye or debian-sid images implemented too if there's sufficient demand.
Hi Jeremy, Thanks for always being welcoming and nice, though the issue here is not really having a Buster image (or Sid, or Bullseye) in the infra. I know it's there. The problem is making puppet-openstack CI itself to work with Debian. This means that I must fully upstream my patches, including (and mainly) in the puppet-openstack-integration repository where tests are actually happening. That's a lot of work, which I currently don't have enough time for. So I keep postponing it, and just test why my local environment. The other thing which I should work on is having OCI integrated with the OpenDev CI. I believe we talked about it briefly during fossdem, not sure if you remember. Currently, testing half-manually takes a lot of my time, and I'd love to automate it fully. Though I'm having a hard time figuring out how I could do. In my current environment, I have a big machine with 256 GB of RAM, where I install 20 VMs that I can install OpenStack on. It wouldn't be hard to scale this down by half, but I hardly see how I could work with 8GB of RAM, or how I could make all of this tested. My tool includes bare-metal provisioning, where VMs are PXE-booting and receive some IPMI commands (I use ipmi_sim...). I don't know how that would work in a multi-VM thing. Cheers, Thomas Goirand (zigo)
participants (4)
-
Jeremy Stanley
-
Luke Short
-
Thomas Goirand
-
Tobias Urdin