' tag:

Nova Cells¶

...

So to fix this, we'll have to update the openstackdocstheme to handle these changes. I can try to take a look at this next week but I really wouldn't mind if someone beat me to it. Stephen [1] https://docs.openstack.org/nova/latest/configuration/extra-specs.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From tkajinam at redhat.com Fri May 14 11:46:35 2021 From: tkajinam at redhat.com (Takashi Kajinami) Date: Fri, 14 May 2021 20:46:35 +0900 Subject: [puppet][tripleo] Inviting tripleo CI cores to maintain tripleo jobs ? In-Reply-To: References: Message-ID: Hi Marios, On Fri, May 14, 2021 at 8:10 PM Marios Andreou wrote: > On Fri, May 14, 2021 at 8:40 AM Takashi Kajinami > wrote: > > > > Hi team, > > > > Hi Takashi > > > > As you know, we currently have TripleO jobs in some of the puppet repos > > to ensure a change in puppet side doesn't break TripleO which consumes > > some of the modules. > > in case it isn't clear and for anyone else reading, you are referring > to things like [1]. > This is a nitfixing but puppet-pacemaker is a repo under the TripleO project. I intend a job like https://zuul.opendev.org/t/openstack/builds?job_name=puppet-nova-tripleo-standalone&project=openstack/puppet-nova which is maintained under puppet repos. > > > > > Because these jobs hugely depend on the job definitions in TripleO repos, > > I'm wondering whether we can invite a few cores from the TripleO CI team > > to the puppet-openstack core group to maintain these jobs. > > I expect the scope here is very limited to tripleo job definitions and > doesn't > > expect any +2 for other parts. > > > > I'd be nice if I can hear any thoughts on this topic. > > Main question is what kind of maintenance do you have in mind? Is it > that these jobs are breaking often and they need fixes in the > puppet-repos themselves so we need more cores there? (though... I > would expect the fixes to be needed in tripleo-ci where the job > definitions are, unless the repos are overriding those definitions)? > We define our own base tripleo-puppet-ci-centos-8-standalone job[4] and each puppet module defines their own tripleo job[5] by overriding the base job, so that we can define some basic items like irellevant files or voting status for all puppet modules in a single place. [4] https://github.com/openstack/puppet-openstack-integration/blob/master/zuul.d/tripleo.yaml [5] https://github.com/openstack/puppet-nova/blob/master/.zuul.yaml > Or is it that you don't have enough folks to get fixes merged so this > is mostly about growing the pool of reviewers? > Yes. My main intention is to have more reviewers so that we can fix our CI jobs timely. Actually the proposal came to my mind when I was implementing the following changes to solve very frequent job timeouts which we currently observe in puppet-nova wallaby. IMO these changes need more attention from TripleO's perspective rather than puppet's perspective. https://review.opendev.org/q/topic:%22tripleo-tempest%22+(status:open) In the past when we introduced content provider jobs, we ended up with a bunch of patches submitted to both tripleo jobs and puppet jobs. Having some people from TripleO team would help moving forward such a transition more smoothly. In the past we have had three people (Alex, Emilien and I) involved in both TripleO and puppet but since Emilien has shifted this focus, we have now 2 activities left. Additional one or two people would help us move patches forward more efficiently. (Since I can't approve my own patch.) I think limiting the scope to just the contents of zuul.d/ or > .zuul.yaml can work; we already have a trust based system in TripleO > with some cores only expected to exercise their voting rights in > particular repos even though they have full voting rights across all > tripleo repos). > > Are you able to join our next tripleo-ci community call? It is on > Tuesday 1330 UTC @ [2] and we use [3] for the agenda. If you can join, > perhaps we can work something out depending on what you need. > Otherwise no problem let's continue to discuss here > Sure. I can join and bring up this topic. I'll keep this thread to hear some opinions from the puppet side as well. > > regards, marios > > [1] > https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario004-standalone&project=openstack/puppet-pacemaker > [2] https://meet.google.com/bqx-xwht-wky > [3] https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both > > > > > > > Thank you, > > Takashi > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marios at redhat.com Fri May 14 11:57:12 2021 From: marios at redhat.com (Marios Andreou) Date: Fri, 14 May 2021 14:57:12 +0300 Subject: [puppet][tripleo] Inviting tripleo CI cores to maintain tripleo jobs ? In-Reply-To: References: Message-ID: On Fri, May 14, 2021 at 2:46 PM Takashi Kajinami wrote: > > Hi Marios, > > On Fri, May 14, 2021 at 8:10 PM Marios Andreou wrote: >> >> On Fri, May 14, 2021 at 8:40 AM Takashi Kajinami wrote: >> > >> > Hi team, >> > >> >> Hi Takashi >> >> >> > As you know, we currently have TripleO jobs in some of the puppet repos >> > to ensure a change in puppet side doesn't break TripleO which consumes >> > some of the modules. >> >> in case it isn't clear and for anyone else reading, you are referring >> to things like [1]. > > This is a nitfixing but puppet-pacemaker is a repo under the TripleO project. > I intend a job like > https://zuul.opendev.org/t/openstack/builds?job_name=puppet-nova-tripleo-standalone&project=openstack/puppet-nova > which is maintained under puppet repos. > ack thanks for the clarification ;) makes more sense now >> >> >> > >> > Because these jobs hugely depend on the job definitions in TripleO repos, >> > I'm wondering whether we can invite a few cores from the TripleO CI team >> > to the puppet-openstack core group to maintain these jobs. >> > I expect the scope here is very limited to tripleo job definitions and doesn't >> > expect any +2 for other parts. >> > >> > I'd be nice if I can hear any thoughts on this topic. >> >> Main question is what kind of maintenance do you have in mind? Is it >> that these jobs are breaking often and they need fixes in the >> puppet-repos themselves so we need more cores there? (though... I >> would expect the fixes to be needed in tripleo-ci where the job >> definitions are, unless the repos are overriding those definitions)? > > > We define our own base tripleo-puppet-ci-centos-8-standalone job[4] and > each puppet module defines their own tripleo job[5] by overriding the base job, > so that we can define some basic items like irellevant files or voting status > for all puppet modules in a single place. > > [4] https://github.com/openstack/puppet-openstack-integration/blob/master/zuul.d/tripleo.yaml > [5] https://github.com/openstack/puppet-nova/blob/master/.zuul.yaml > > >> >> Or is it that you don't have enough folks to get fixes merged so this >> is mostly about growing the pool of reviewers? > > > Yes. My main intention is to have more reviewers so that we can fix our CI jobs timely. > > Actually the proposal came to my mind when I was implementing the following changes > to solve very frequent job timeouts which we currently observe in puppet-nova wallaby. > IMO these changes need more attention from TripleO's perspective rather than puppet's > perspective. > https://review.opendev.org/q/topic:%22tripleo-tempest%22+(status:open) > > In the past when we introduced content provider jobs, we ended up with a bunch of patches > submitted to both tripleo jobs and puppet jobs. Having some people from TripleO team > would help moving forward such a transition more smoothly. > > In the past we have had three people (Alex, Emilien and I) involved in both TripleO and puppet > but since Emilien has shifted this focus, we have now 2 activities left. > Additional one or two people would help us move patches forward more efficiently. > (Since I can't approve my own patch.) > >> I think limiting the scope to just the contents of zuul.d/ or >> .zuul.yaml can work; we already have a trust based system in TripleO >> with some cores only expected to exercise their voting rights in >> particular repos even though they have full voting rights across all >> tripleo repos). >> >> Are you able to join our next tripleo-ci community call? It is on >> Tuesday 1330 UTC @ [2] and we use [3] for the agenda. If you can join, >> perhaps we can work something out depending on what you need. >> Otherwise no problem let's continue to discuss here > > > Sure. I can join and bring up this topic. > I'll keep this thread to hear some opinions from the puppet side as well. > > ok thanks look forward to discussing on Tuesday then, regards, marios >> >> >> regards, marios >> >> [1] https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario004-standalone&project=openstack/puppet-pacemaker >> [2] https://meet.google.com/bqx-xwht-wky >> [3] https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both >> >> >> >> > >> > Thank you, >> > Takashi >> > >> From tobias.urdin at binero.com Fri May 14 12:54:08 2021 From: tobias.urdin at binero.com (Tobias Urdin) Date: Fri, 14 May 2021 12:54:08 +0000 Subject: [puppet][tripleo] Inviting tripleo CI cores to maintain tripleo jobs ? In-Reply-To: References: Message-ID: <987B8C72-C316-4D8B-A388-5EC5EE23E97E@binero.com> Hello Takashi, >From an Puppet OpenStack core perspective being outside of RedHat I have no problem with it since the mentioned trust from the TripleO side is to only approve patches directly related to it. Hopefully the person becomes interested and starts working on all the Puppet parts as well :) Best regards Tobias > On 14 May 2021, at 07:38, Takashi Kajinami wrote: > > Hi team, > > > As you know, we currently have TripleO jobs in some of the puppet repos > to ensure a change in puppet side doesn't break TripleO which consumes > some of the modules. > > Because these jobs hugely depend on the job definitions in TripleO repos, > I'm wondering whether we can invite a few cores from the TripleO CI team > to the puppet-openstack core group to maintain these jobs. > I expect the scope here is very limited to tripleo job definitions and doesn't > expect any +2 for other parts. > > I'd be nice if I can hear any thoughts on this topic. > > Thank you, > Takashi > From manchandavishal143 at gmail.com Fri May 14 13:26:52 2021 From: manchandavishal143 at gmail.com (vishal manchanda) Date: Fri, 14 May 2021 18:56:52 +0530 Subject: [horizon][stable] [release] Proposal to make stable/ocata and stable/pike as EOL Message-ID: Hi all, As discussed in the last horizon weekly meeting[1], [2] on 2021-05-05, I would like to announce that the horizon team decided to make stable/ocata and stable/pike branches as EOL. Consider this mail as an official announcement for that. I would like to know if anyone still would like to keep those branches to be open otherwise, I will purpose an EOL patch for these branches. Thanks & Regards, Vishal Manchanda [1] http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-alt/%23openstack-meeting-alt.2021-05-05.log.html#t2021-05-05T15:55:59 [2] http://eavesdrop.openstack.org/irclogs/%23openstack-horizon/%23openstack-horizon.2021-05-05.log.html#t2021-05-05T16:03:28 -------------- next part -------------- An HTML attachment was scrubbed... URL: From james.slagle at gmail.com Fri May 14 13:28:21 2021 From: james.slagle at gmail.com (James Slagle) Date: Fri, 14 May 2021 09:28:21 -0400 Subject: [TripleO] Opting out of global-requirements.txt In-Reply-To: References:

Message-ID: On Fri, May 14, 2021 at 6:41 AM Marios Andreou wrote: > On Thu, May 13, 2021 at 9:41 PM James Slagle > wrote: > > > > I'd like to propose that TripleO opt out of dependency management by > removing tripleo-common from global-requirements.txt. I do not feel that > the global dependency management brings any advantages or anything needed > for TripleO. I can't think of any reason to enforce the ability to be > globally pip installable with the rest of OpenStack. > > To add a bit more context as to how this discussion came about, we > tried to remove tripleo-common from global-requirements and the > check-requirements jobs in tripleoclient caused [1] as a result. > > So we need to decide whether to continue to be part of that > requirements contract [2] or if we will do what is proposed here and > remove ourselves altogether. > If we decide _not_ to implement this proposal then we will also have > to add the requirements-check jobs in tripleo-ansible [3] and > tripleo-validations [4] as they are currently missing. > > > > > Two of our most critical projects, tripleoclient and tripleo-common do > not even put many of their data files in the right place where our code > expects them when they are pip installed. So, I feel fairly confident that > no one is pip installing TripleO and relying on global requirements > enforcement. > > I don't think this is _just_ about pip installs. It is generally about > the contents of each project requirements.txt. As part of the > requirements contract, it means that those repos with which we are > participating (the ones in projects.txt [5]) are protected against > other projects making any breaking changes in _their_ > requirements.txt. Don't the contents of requirements.txt also end up > in the .spec file from which we are building rpm e.g. [6] for tht? In > which case if we remove this and just stop catching any breaking > changes in the check/gate check-requirements jobs, I suspect we will > just move the problem to the rpm build and it will fail there. > I don't see that in the spec file. Unless there is some other automation somewhere that regenerates all of the BuildRequires/Requires and modifies them to match requirements.txt/test-requirements.txt? > > > > One potential advantage of not being in global-requirements.txt is that > our unit tests and functional tests could actually test the same code. As > things stand today, our unit tests in projects that depend on > tripleo-common are pinned to the version in global-requirements.txt, while > our functional tests currently run with tripleo-common from master (or > included depends-on). > > I don't think one in particular is a very valid point though - as > things currently stand in global-requirements we aren't 'pinning' all > we have there is "tripleo-common!=11.3.0 # Apache-2.0" [7] to avoid > (I assume) some bad release we made. > tripleo-common is pinned to the latest release when it's pip installed in the venv, instead of using latest git (and including depends-on). You're right that it's probably what we want to keep doing, and this is probably not related to opting out of g-r. Especially since we don't want to require latest git of a dependency when running unit tests locally. However it is worth noting that our unit tests (tox) and functional tests (tripleo-ci) use different code for the dependencies. That was not obvious to me and others on the surface. Perhaps we could add additional tox jobs that do require the latest tripleo-common from git to also cover that scenario. Here's an example: https://review.opendev.org/c/openstack/python-tripleoclient/+/787907 https://review.opendev.org/c/openstack/tripleo-common/+/787906 The tripleo-common patch fails unit tests as expected, the tripleoclient which depends-on the tripleo-common patch passes unit tests, but fails functional. I'd rather see that failure caught by a unit test as well. -- -- James Slagle -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From mihalis68 at gmail.com Fri May 14 15:05:55 2021 From: mihalis68 at gmail.com (Chris Morgan) Date: Fri, 14 May 2021 11:05:55 -0400 Subject: threat to freenode (where openstack irc hangs out) Message-ID: https://twitter.com/dmsimard/status/1393203159770804225?s=20 https://p.haavard.me/407 I have no independent validation of this. Chris -- Chris Morgan -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Fri May 14 15:14:25 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 14 May 2021 15:14:25 +0000 Subject: threat to freenode (where openstack irc hangs out) In-Reply-To: References: Message-ID: <20210514151425.zxgeym4lnhmbdrt3@yuggoth.org> On 2021-05-14 11:05:55 -0400 (-0400), Chris Morgan wrote: > https://twitter.com/dmsimard/status/1393203159770804225?s=20 > https://p.haavard.me/407 > > I have no independent validation of this. It seems like that may not be the whole story. Regardless, the infra team have registered a small foothold of copies of our critical channels on OFTC years ago, and have always considered that a reasonable place to relocate in the event something happens to Freenode which makes it no longer suitable. Letting people know to switch the IRC server name in their clients and updating lots of documentation mentioning Freenode would be the hardest part, honestly. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From luke.camilleri at zylacomputing.com Fri May 14 15:15:11 2021 From: luke.camilleri at zylacomputing.com (Luke Camilleri) Date: Fri, 14 May 2021 17:15:11 +0200 Subject: [Octavia][Victoria] No service listening on port 9443 in the amphora instance In-Reply-To: References: <038c74c9-1365-0c08-3b5b-93b4d175dcb3@zylacomputing.com> <326471ef-287b-d937-a174-0b1ccbbd6273@zylacomputing.com>

<8ce76cbd-ca2c-033a-5406-5a1557d84302@zylacomputing.com> Message-ID: Hi Michael, thanks as always for the below, I have watched the video and configured all the requirements as shown in those guides. Working great right now. I have noticed the following points and would like to know if you can give me some feedback please: 1. In the Octavia project at the networks screen --> lb-mgmt-net --> Ports --> octavia-health-manager-listen-port (the IP bound to the health-manager service) has its status Down. It does not create any sort of issue but was wondering if this was normal behavior? 2. Similarly to the above point, in the tenant networks screen, every port that is "Attached Device - Octavia" has its status reported as "Down" ( these are the VIP addresses assigned to the amphora). Just need to confirm that this is normal behaviour 3. Creating a health monitor of type ping fails to get the operating status of the nodes and the nodes are in error (horizon) and the amphora reports that there are no backends and hence it is not working (I am using the same backend nodes with another loadbalancer but with an HTTP check and it is working fine. a security group is setup to allow ping from 0.0.0.0/0 and from the amphora-haproxy network namespace on the amphora instance I can ping both nodes without issues ). Below the amphora's haproxy.log May 14 15:00:50 amphora-9658d9ec-3bf1-407f-a134-86304899c015 haproxy[1984]: Server c0092bf4-d2a2-431f-8b7f-9dc3ace52933:e268db93-2d20-4395-bd6f-f6d835bce769/f04824b7-6fdf-46dc-bc83-b98b3b9f5be0 is DOWN, reason: Socket error, info: "Resource temporarily unavailable", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. May 14 15:00:50 amphora-9658d9ec-3bf1-407f-a134-86304899c015 haproxy[1984]: Server c0092bf4-d2a2-431f-8b7f-9dc3ace52933:e268db93-2d20-4395-bd6f-f6d835bce769/f04824b7-6fdf-46dc-bc83-b98b3b9f5be0 is DOWN, reason: Socket error, info: "Resource temporarily unavailable", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. May 14 15:00:50 amphora-9658d9ec-3bf1-407f-a134-86304899c015 haproxy[1984]: backend c0092bf4-d2a2-431f-8b7f-9dc3ace52933:e268db93-2d20-4395-bd6f-f6d835bce769 has no server available! May 14 15:00:50 amphora-9658d9ec-3bf1-407f-a134-86304899c015 haproxy[1984]: backend c0092bf4-d2a2-431f-8b7f-9dc3ace52933:e268db93-2d20-4395-bd6f-f6d835bce769 has no server available! Thanks in advance On 13/05/2021 18:33, Michael Johnson wrote: > You are correct that two IPs are being allocated for the VIP, one is a > secondary IP which neutron implements as an "allowed address pairs" > port. We do this to allow failover of the amphora instance should nova > fail the service VM. We hold the VIP IP in a special port so the IP is > not lost while we rebuild the service VM. > If you are using the active/standby topology (or an Octavia flavor > with active/standby enabled), this failover is accelerated with nearly > no visible impact to the flows through the load balancer. > Active/Standby has been an Octavia feature since the Mitaka release. I > gave a demo of it at the Tokoyo summit here: > https://youtu.be/8n7FGhtOiXk?t=1420 > > You can enable active/standby as the default by setting the > "loadbalancer_topology" setting in the configuration file > (https://docs.openstack.org/octavia/latest/configuration/configref.html#controller_worker.loadbalancer_topology) > or by creating an Octavia flavor that creates the load balancer with > an active/standby topology > (https://docs.openstack.org/octavia/latest/admin/flavors.html). > > Michael > > On Thu, May 13, 2021 at 4:23 AM Luke Camilleri > wrote: >> HI Michael, thanks a lot for the below information it is very helpful. I >> ended up setting the o-hm0 interface statically in the >> octavia-interface.sh script which is called by the service and also >> added a delay to make sure that the bridges are up before trying to >> create a veth pair and connect the endpoints. >> >> Also I edited the unit section of the health-manager service and at the >> after option I added octavia-interface.service or else on startup the >> health manager will not bind to the lb-mgmt-net since it would not be up yet >> >> The floating IPs part was a bit tricky until I understood what was >> really going on with the VIP concept and how better and more flexible it >> is to set the VIP on the tenant network and then associate with public >> ip to the VIP. >> >> With this being said I noticed that 2 IPs are being assigned to the >> amphora instance and that the actual port assigned to the instance has >> an allowed pair with the VIP port. I checked online and it seems that >> there is an active/standby project going on with VRRP/keepalived and in >> fact the keepalived daemon is running in the amphora instance. >> >> Am I on the right track with the active/standby feature and if so do you >> have any installation/project links to share please so that I can test it? >> >> Regards >> >> On 12/05/2021 08:37, Michael Johnson wrote: >>> Answers inline below. >>> >>> Michael >>> >>> On Mon, May 10, 2021 at 5:15 PM Luke Camilleri >>> wrote: >>>> Hi Michael and thanks a lot for the detailed answer below. >>>> >>>> I believe I have got most of this sorted out apart from some small issues below: >>>> >>>> If the o-hm0 interface gets the IP information from the DHCP server setup by neutron for the lb-mgmt-net, then the management node will always have 2 default gateways and this will bring along issues, the same DHCP settings when deployed to the amphora do not have the same issue since the amphora only has 1 IP assigned on the lb-mgmt-net. Can you please confirm this? >>> The amphorae do not have issues with DHCP and gateways as we control >>> the DHCP client configuration inside the amphora. It does only have >>> one IP on the lb-mgmt-net, it will honor gateways provided by neutron >>> for the lb-mgmt-net traffic, but a gateway is not required on the >>> lb-mgmt-network unless you are routing the lb-mgmt-net traffic across >>> subnets. >>> >>>> How does the amphora know where to locate the worker and housekeeping processes or does the traffic originate from the services instead? Maybe the addresses are "injected" from the config file? >>> The worker and housekeeping processes only create connections to the >>> amphora, they do not receive connections from them. The amphora send a >>> heartbeat packet to the health manager endpoints every ten seconds by >>> default. The list of valid health manager endpoints is included in the >>> amphora agent configuration file that is injected into the service VM >>> at boot time. It can be updated using the Octavia admin API for >>> refreshing the amphora agent configuration. >>> >>>> Can you please confirm if the same floating IP concept runs from public (external) IP to the private (tenant) and from private to lb-mgmt-net please? >>> Octavia does not use floating IPs. Users can create and assign >>> floating IPs via neutron if they would like, but they are not >>> necessary. Octavia VIPs can be created directly on neutron "external" >>> networks, avoiding the NAT overhead of floating IPs. >>> There is no practical reason to assign a floating IP to a port on the >>> lb-mgmt-net as tenant traffic is never on or accessible from that >>> network. >>> >>>> Thanks in advance for any feedback >>>> >>>> On 06/05/2021 22:46, Michael Johnson wrote: >>>> >>>> Hi Luke, >>>> >>>> 1. I agree that DHCP is technically unnecessary for the o-hm0 >>>> interface if you can manage your address allocation on the network you >>>> are using for the lb-mgmt-net. >>>> I don't have detailed information about the Ubuntu install >>>> instructions, but I suspect it was done to simplify the IPAM to be >>>> managed by whatever is providing DHCP on the lb-mgmt-net provided (be >>>> it neutron or some other resource on a provider network). >>>> The lb-mgmt-net is simply a neutron network that the amphora >>>> management address is on. It is routable and does not require external >>>> access. The only tricky part to it is the worker, health manager, and >>>> housekeeping processes need to be reachable from the amphora, and the >>>> controllers need to reach the amphora over the network(s). There are >>>> many ways to accomplish this. >>>> >>>> 2. See my above answer. Fundamentally the lb-mgmt-net is just a >>>> neutron network that nova can use to attach an interface to the >>>> amphora instances for command and control traffic. As long as the >>>> controllers can reach TCP 9433 on the amphora, and the amphora can >>>> send UDP 5555 back to the health manager endpoints, it will work fine. >>>> >>>> 3. Octavia, with the amphora driver, does not require any special >>>> configuration in Neutron (beyond the advanced services RBAC policy >>>> being available for the neutron service account used in your octavia >>>> configuration file). The neutron_lbaas.conf and services_lbaas.conf >>>> are legacy configuration files/settings that were used for >>>> neutron-lbaas which is now end of life. See the wiki page for >>>> information on the deprecation of neutron-lbaas: >>>> https://wiki.openstack.org/wiki/Neutron/LBaaS/Deprecation. >>>> >>>> Michael >>>> >>>> On Thu, May 6, 2021 at 12:30 PM Luke Camilleri >>>> wrote: >>>> >>>> Hi Michael and thanks a lot for your help on this, after following your >>>> steps the agent got deployed successfully in the amphora-image. >>>> >>>> I have some other queries that I would like to ask mainly related to the >>>> health-manager/load-balancer network setup and IP assignment. First of >>>> all let me point out that I am using a manual installation process, and >>>> it might help others to understand the underlying infrastructure >>>> required to make this component work as expected. >>>> >>>> 1- The installation procedure contains this step: >>>> >>>> $ sudo cp octavia/etc/dhcp/dhclient.conf /etc/dhcp/octavia >>>> >>>> which is later on called to assign the IP to the o-hm0 interface which >>>> is connected to the lb-management network as shown below: >>>> >>>> $ sudo dhclient -v o-hm0 -cf /etc/dhcp/octavia >>>> >>>> Apart from having a dhcp config for a single IP seems a bit of an >>>> overkill, using these steps is injecting an additional routing table >>>> into the default namespace as shown below in my case: >>>> >>>> # route -n >>>> Kernel IP routing table >>>> Destination Gateway Genmask Flags Metric Ref Use >>>> Iface >>>> 0.0.0.0 172.16.0.1 0.0.0.0 UG 0 0 0 o-hm0 >>>> 0.0.0.0 10.X.X.1 0.0.0.0 UG 100 0 0 ensX >>>> 10.X.X.0 0.0.0.0 255.255.255.0 U 100 0 0 ensX >>>> 169.254.169.254 172.16.0.100 255.255.255.255 UGH 0 0 0 o-hm0 >>>> 172.16.0.0 0.0.0.0 255.240.0.0 U 0 0 0 o-hm0 >>>> >>>> Since the load-balancer management network does not need any external >>>> connectivity (but only communication between health-manager service and >>>> amphora-agent), why is a gateway required and why isn't the IP address >>>> allocated as part of the interface creation script which is called when >>>> the service is started or stopped (example below)? >>>> >>>> --- >>>> >>>> #!/bin/bash >>>> >>>> set -ex >>>> >>>> MAC=$MGMT_PORT_MAC >>>> BRNAME=$BRNAME >>>> >>>> if [ "$1" == "start" ]; then >>>> ip link add o-hm0 type veth peer name o-bhm0 >>>> brctl addif $BRNAME o-bhm0 >>>> ip link set o-bhm0 up >>>> ip link set dev o-hm0 address $MAC >>>> *** ip addr add 172.16.0.2/12 dev o-hm0 >>>> ***ip link set o-hm0 mtu 1500 >>>> ip link set o-hm0 up >>>> iptables -I INPUT -i o-hm0 -p udp --dport 5555 -j ACCEPT >>>> elif [ "$1" == "stop" ]; then >>>> ip link del o-hm0 >>>> else >>>> brctl show $BRNAME >>>> ip a s dev o-hm0 >>>> fi >>>> >>>> --- >>>> >>>> 2- Is there a possibility to specify a fixed vlan outside of tenant >>>> range for the load balancer management network? >>>> >>>> 3- Are the configuration changes required only in neutron.conf or also >>>> in additional config files like neutron_lbaas.conf and >>>> services_lbaas.conf, similar to the vpnaas configuration? >>>> >>>> Thanks in advance for any assistance, but its like putting together a >>>> puzzle of information :-) >>>> >>>> On 05/05/2021 20:25, Michael Johnson wrote: >>>> >>>> Hi Luke. >>>> >>>> Yes, the amphora-agent will listen on 9443 in the amphorae instances. >>>> It uses TLS mutual authentication, so you can get a TLS response, but >>>> it will not let you into the API without a valid certificate. A simple >>>> "openssl s_client" is usually enough to prove that it is listening and >>>> requesting the client certificate. >>>> >>>> I can't talk to the "openstack-octavia-diskimage-create" package you >>>> found in centos, but I can discuss how to build an amphora image using >>>> the OpenStack tools. >>>> >>>> If you get Octavia from git or via a release tarball, we provide a >>>> script to build the amphora image. This is how we build our images for >>>> the testing gates, etc. and is the recommended way (at least from the >>>> OpenStack Octavia community) to create amphora images. >>>> >>>> https://opendev.org/openstack/octavia/src/branch/master/diskimage-create >>>> >>>> For CentOS 8, the command would be: >>>> >>>> diskimage-create.sh -g stable/victoria -i centos-minimal -d 8 -s 3 (3 >>>> is the minimum disk size for centos images, you may want more if you >>>> are not offloading logs) >>>> >>>> I just did a run on a fresh centos 8 instance: >>>> git clone https://opendev.org/openstack/octavia >>>> python3 -m venv dib >>>> source dib/bin/activate >>>> pip3 install diskimage-builder PyYAML six >>>> sudo dnf install yum-utils >>>> ./diskimage-create.sh -g stable/victoria -i centos-minimal -d 8 -s 3 >>>> >>>> This built an image. >>>> >>>> Off and on we have had issues building CentOS images due to issues in >>>> the tools we rely on. If you run into issues with this image, drop us >>>> a note back. >>>> >>>> Michael >>>> >>>> On Wed, May 5, 2021 at 9:37 AM Luke Camilleri >>>> wrote: >>>> >>>> Hi there, i am trying to get Octavia running on a Victoria deployment on >>>> CentOS 8. It was a bit rough getting to the point to launch an instance >>>> mainly due to the load-balancer management network and the lack of >>>> documentation >>>> (https://docs.openstack.org/octavia/victoria/install/install.html) to >>>> deploy this oN CentOS. I will try to fix this once I have my deployment >>>> up and running to help others on the way installing and configuring this :-) >>>> >>>> At this point a LB can be launched by the tenant and the instance is >>>> spawned in the Octavia project and I can ping and SSH into the amphora >>>> instance from the Octavia node where the octavia-health-manager service >>>> is running using the IP within the same subnet of the amphoras >>>> (172.16.0.0/12). >>>> >>>> Unfortunately I keep on getting these errors in the log file of the >>>> worker log (/var/log/octavia/worker.log): >>>> >>>> 2021-05-05 01:54:49.368 14521 WARNING >>>> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect >>>> to instance. Retrying.: requests.exceptions.ConnectionError: >>>> HTTPSConnectionPool(host='172.16.4.46', p >>>> ort=9443): Max retries exceeded with url: // (Caused by >>>> NewConnectionError('>>> at 0x7f83e0181550>: Failed to establish a new connection: [Errno 111] >>>> Connection ref >>>> used',)) >>>> >>>> 2021-05-05 01:54:54.374 14521 ERROR >>>> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Connection retries >>>> (currently set to 120) exhausted. The amphora is unavailable. Reason: >>>> HTTPSConnectionPool(host='172.16 >>>> .4.46', port=9443): Max retries exceeded with url: // (Caused by >>>> NewConnectionError('>>> at 0x7f83e0181550>: Failed to establish a new connection: [Errno 111] Conne >>>> ction refused',)) >>>> >>>> 2021-05-05 01:54:54.374 14521 ERROR >>>> octavia.controller.worker.v1.tasks.amphora_driver_tasks [-] Amphora >>>> compute instance failed to become reachable. This either means the >>>> compute driver failed to fully boot the >>>> instance inside the timeout interval or the instance is not reachable >>>> via the lb-mgmt-net.: >>>> octavia.amphorae.driver_exceptions.exceptions.TimeOutException: >>>> contacting the amphora timed out >>>> >>>> obviously the instance is deleted then and the task fails from the >>>> tenant's perspective. >>>> >>>> The main issue here is that there is no service running on port 9443 on >>>> the amphora instance. I am assuming that this is in fact the >>>> amphora-agent service that is running on the instance which should be >>>> listening on this port 9443 but the service does not seem to be up or >>>> not installed at all. >>>> >>>> To create the image I have installed the CentOS package >>>> "openstack-octavia-diskimage-create" which provides the utility >>>> disk-image-create but from what I can conclude the amphora-agent is not >>>> being installed (thought this was done automatically by default :-( ) >>>> >>>> Can anyone let me know if the amphora-agent is what gets queried on port >>>> 9443 ? >>>> >>>> If the agent is not installed/injected by default when building the >>>> amphora image? >>>> >>>> The command to inject the amphora-agent into the amphora image when >>>> using the disk-image-create command? >>>> >>>> Thanks in advance for any assistance >>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From marios at redhat.com Fri May 14 15:15:23 2021 From: marios at redhat.com (Marios Andreou) Date: Fri, 14 May 2021 18:15:23 +0300 Subject: [TripleO] Opting out of global-requirements.txt In-Reply-To: References:

Message-ID: On Fri, May 14, 2021 at 4:28 PM James Slagle wrote: > > > > On Fri, May 14, 2021 at 6:41 AM Marios Andreou wrote: >> >> On Thu, May 13, 2021 at 9:41 PM James Slagle wrote: >> > >> > I'd like to propose that TripleO opt out of dependency management by removing tripleo-common from global-requirements.txt. I do not feel that the global dependency management brings any advantages or anything needed for TripleO. I can't think of any reason to enforce the ability to be globally pip installable with the rest of OpenStack. >> >> To add a bit more context as to how this discussion came about, we >> tried to remove tripleo-common from global-requirements and the >> check-requirements jobs in tripleoclient caused [1] as a result. >> >> So we need to decide whether to continue to be part of that >> requirements contract [2] or if we will do what is proposed here and >> remove ourselves altogether. >> If we decide _not_ to implement this proposal then we will also have >> to add the requirements-check jobs in tripleo-ansible [3] and >> tripleo-validations [4] as they are currently missing. >> >> > >> > Two of our most critical projects, tripleoclient and tripleo-common do not even put many of their data files in the right place where our code expects them when they are pip installed. So, I feel fairly confident that no one is pip installing TripleO and relying on global requirements enforcement. >> >> I don't think this is _just_ about pip installs. It is generally about >> the contents of each project requirements.txt. As part of the >> requirements contract, it means that those repos with which we are >> participating (the ones in projects.txt [5]) are protected against >> other projects making any breaking changes in _their_ >> requirements.txt. Don't the contents of requirements.txt also end up >> in the .spec file from which we are building rpm e.g. [6] for tht? In >> which case if we remove this and just stop catching any breaking >> changes in the check/gate check-requirements jobs, I suspect we will >> just move the problem to the rpm build and it will fail there. > > > I don't see that in the spec file. Unless there is some other automation somewhere that regenerates all of the BuildRequires/Requires and modifies them to match requirements.txt/test-requirements.txt? > I was looking at that in particular: "tripleo-common>=7.1.0 # Apache-2.0" at https://opendev.org/openstack/tripleo-heat-templates/src/commit/fe2373225f039d795970b70fe9b2f28e0e7cd6a4/requirements.txt#L8 and in the specfile: " 42 Requires: openstack-tripleo-common >= 7.1.0" at https://review.rdoproject.org/r/gitweb?p=openstack/tripleo-heat-templates-distgit.git;a=blob;f=openstack-tripleo-heat-templates.spec;h=36ee20fb8042bbdd254b4eab6ab1f3272bd8a6c5;hb=HEAD#l42 I assumed there was correlation there with the value originally coming from the project requirements.txt - but perhaps that is wrong; I realise now that the specfile is referencing 'openstack-tripleo-common' the built rpm, not 'tripleo-common'. Well OK, that was/is my biggest concern; i.e. we are getting value from having this requirements compatibility check, and if we remove it we will just end up having those compatibility problems further down the chain. On the flip side, it just doesn't seem to me to be that big a cost for us to stay inside this check. We aren't running check-requirements job on every posted change it is only ever triggered if you submit a change to requirements.txt https://opendev.org/openstack/requirements/src/commit/ce19462764940a4ce99dae4ac2ec7a004c68e9a4/.zuul.d/project-template.yaml#L16 . > >> >> > >> > One potential advantage of not being in global-requirements.txt is that our unit tests and functional tests could actually test the same code. As things stand today, our unit tests in projects that depend on tripleo-common are pinned to the version in global-requirements.txt, while our functional tests currently run with tripleo-common from master (or included depends-on). >> >> I don't think one in particular is a very valid point though - as >> things currently stand in global-requirements we aren't 'pinning' all >> we have there is "tripleo-common!=11.3.0 # Apache-2.0" [7] to avoid >> (I assume) some bad release we made. > > > tripleo-common is pinned to the latest release when it's pip installed in the venv, instead of using latest git (and including depends-on). You're right that it's probably what we want to keep doing, and this is probably not related to opting out of g-r. Especially since we don't want to require latest git of a dependency when running unit tests locally. However it is worth noting that our unit tests (tox) and functional tests (tripleo-ci) use different code for the dependencies. That was not obvious to me and others on the surface. Perhaps we could add additional tox jobs that do require the latest tripleo-common from git to also cover that scenario. ack I see what you mean now about pip install => tagged release vs dlrn current => git master/latest commit and also agree that is not directly related to this discussion about the check-requirements jobs. regards, marios > > Here's an example: > https://review.opendev.org/c/openstack/python-tripleoclient/+/787907 > https://review.opendev.org/c/openstack/tripleo-common/+/787906 > > The tripleo-common patch fails unit tests as expected, the tripleoclient which depends-on the tripleo-common patch passes unit tests, but fails functional. I'd rather see that failure caught by a unit test as well. > > -- > -- James Slagle > -- From jpena at redhat.com Fri May 14 15:33:25 2021 From: jpena at redhat.com (Javier Pena) Date: Fri, 14 May 2021 11:33:25 -0400 (EDT) Subject: [TripleO] Opting out of global-requirements.txt In-Reply-To: References:

Message-ID: <124843068.26462828.1621006405455.JavaMail.zimbra@redhat.com> > On Fri, May 14, 2021 at 6:41 AM Marios Andreou < marios at redhat.com > wrote: > > On Thu, May 13, 2021 at 9:41 PM James Slagle < james.slagle at gmail.com > > > wrote: > > > > > > > > I'd like to propose that TripleO opt out of dependency management by > > > removing tripleo-common from global-requirements.txt. I do not feel that > > > the global dependency management brings any advantages or anything needed > > > for TripleO. I can't think of any reason to enforce the ability to be > > > globally pip installable with the rest of OpenStack. > > > To add a bit more context as to how this discussion came about, we > > > tried to remove tripleo-common from global-requirements and the > > > check-requirements jobs in tripleoclient caused [1] as a result. > > > So we need to decide whether to continue to be part of that > > > requirements contract [2] or if we will do what is proposed here and > > > remove ourselves altogether. > > > If we decide _not_ to implement this proposal then we will also have > > > to add the requirements-check jobs in tripleo-ansible [3] and > > > tripleo-validations [4] as they are currently missing. > > > > > > > > Two of our most critical projects, tripleoclient and tripleo-common do > > > not > > > even put many of their data files in the right place where our code > > > expects them when they are pip installed. So, I feel fairly confident > > > that > > > no one is pip installing TripleO and relying on global requirements > > > enforcement. > > > I don't think this is _just_ about pip installs. It is generally about > > > the contents of each project requirements.txt. As part of the > > > requirements contract, it means that those repos with which we are > > > participating (the ones in projects.txt [5]) are protected against > > > other projects making any breaking changes in _their_ > > > requirements.txt. Don't the contents of requirements.txt also end up > > > in the .spec file from which we are building rpm e.g. [6] for tht? In > > > which case if we remove this and just stop catching any breaking > > > changes in the check/gate check-requirements jobs, I suspect we will > > > just move the problem to the rpm build and it will fail there. > > I don't see that in the spec file. Unless there is some other automation > somewhere that regenerates all of the BuildRequires/Requires and modifies > them to match requirements.txt/test-requirements.txt? We run periodic syncs once every cycle, and try our best to make spec requirements match requirements.txt/test-requirements.txt for the project. See https://review.rdoproject.org/r/c/openstack/tripleoclient-distgit/+/33367 for a recent example on tripleoclient. I don't have a special opinion on keeping tripleo-common inside global-requirements.txt or not. However, all TripleO projects still need to be co-installable with other OpenStack projects, otherwise we will not be able to build packages for them due to all the dependency issues that could arise. I'm not sure if that was implied in the original post. Regards, Javier > > > > > > > One potential advantage of not being in global-requirements.txt is that > > > our > > > unit tests and functional tests could actually test the same code. As > > > things stand today, our unit tests in projects that depend on > > > tripleo-common are pinned to the version in global-requirements.txt, > > > while > > > our functional tests currently run with tripleo-common from master (or > > > included depends-on). > > > I don't think one in particular is a very valid point though - as > > > things currently stand in global-requirements we aren't 'pinning' all > > > we have there is "tripleo-common!=11.3.0 # Apache-2.0" [7] to avoid > > > (I assume) some bad release we made. > > tripleo-common is pinned to the latest release when it's pip installed in the > venv, instead of using latest git (and including depends-on). You're right > that it's probably what we want to keep doing, and this is probably not > related to opting out of g-r. Especially since we don't want to require > latest git of a dependency when running unit tests locally. However it is > worth noting that our unit tests (tox) and functional tests (tripleo-ci) use > different code for the dependencies. That was not obvious to me and others > on the surface. Perhaps we could add additional tox jobs that do require the > latest tripleo-common from git to also cover that scenario. > Here's an example: > https://review.opendev.org/c/openstack/python-tripleoclient/+/787907 > https://review.opendev.org/c/openstack/tripleo-common/+/787906 > The tripleo-common patch fails unit tests as expected, the tripleoclient > which depends-on the tripleo-common patch passes unit tests, but fails > functional. I'd rather see that failure caught by a unit test as well. > -- > -- James Slagle > -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Fri May 14 05:03:55 2021 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Fri, 14 May 2021 07:03:55 +0200 Subject: [stein][neutron] gratuitous arp In-Reply-To: <35985fecc7b7658d70446aa816d8ed612f942115.camel@redhat.com> References:

<9ac105e8b7176ecc085f57ec84d891afa927c637.camel@redhat.com> <7de015a7292674b4ed5aa4926f01de760d133de9.camel@redhat.com> <4fa3e29a7e654e74bc96ac67db0e755c@binero.com> <95ccfc366d4b497c8af232f38d07559f@binero.com> <35985fecc7b7658d70446aa816d8ed612f942115.camel@redhat.com> Message-ID: Hello, I am trying to apply suggested patches but after applying some python error codes do not allow neutron services to start. Since my python skill is poor, I wonder if that patches are for python3. I am on centos 7 and (probably???) patches are for centos 8. I also wonder if it possible to upgrade a centos 7 train to centos 8 train without reinstalling all nodes. This could be important for next release upgrade (ussuri, victoria and so on). Ignazio Il Ven 12 Mar 2021, 23:44 Sean Mooney ha scritto: > On Fri, 2021-03-12 at 08:13 +0000, Tobias Urdin wrote: > > Hello, > > > > If it's the same as us, then yes, the issue occurs on Train and is not > completely solved yet. > there is a downstream bug trackker for this > > https://bugzilla.redhat.com/show_bug.cgi?id=1917675 > > its fixed by a combination of 3 enturon patches and i think 1 nova one > > https://review.opendev.org/c/openstack/neutron/+/766277/ > https://review.opendev.org/c/openstack/neutron/+/753314/ > https://review.opendev.org/c/openstack/neutron/+/640258/ > > and > https://review.opendev.org/c/openstack/nova/+/770745 > > the first tree neutron patches would fix the evauate case but break live > migration > the nova patch means live migration will work too although to fully fix > the related > live migration packet loss issues you need > > https://review.opendev.org/c/openstack/nova/+/747454/4 > https://review.opendev.org/c/openstack/nova/+/742180/12 > to fix live migration with network abckend that dont suppor tmultiple port > binding > and > https://review.opendev.org/c/openstack/nova/+/602432 (the only one not > merged yet.) > for live migrateon with ovs and hybridg plug=false (e.g. ovs firewall > driver, noop or ovn instead of ml2/ovs. > > multiple port binding was not actully the reason for this there was a race > in neutorn itslef that would have haapend > even without multiple port binding between the dhcp agent and l2 agent. > > some of those patches have been backported already and all shoudl > eventually make ti to train the could be brought to stine potentially > if peopel are open to backport/review them. > > > > > > > > Best regards > > > > ________________________________ > > From: Ignazio Cassano > > Sent: Friday, March 12, 2021 7:43:22 AM > > To: Tobias Urdin > > Cc: openstack-discuss > > Subject: Re: [stein][neutron] gratuitous arp > > > > Hello Tobias, the result is the same as your. > > I do not know what happens in depth to evaluate if the behavior is the > same. > > I solved on stein with patch suggested by Sean : force_legacy_port_bind > workaround. > > So I am asking if the problem exists also on train. > > Ignazio > > > > Il Gio 11 Mar 2021, 19:27 Tobias Urdin tobias.urdin at binero.com>> ha scritto: > > > > Hello, > > > > > > Not sure if you are having the same issue as us, but we are following > https://bugs.launchpad.net/neutron/+bug/1901707 but > > > > are patching it with something similar to > https://review.opendev.org/c/openstack/nova/+/741529 to workaround the > issue until it's completely solved. > > > > > > Best regards > > > > ________________________________ > > From: Ignazio Cassano ignaziocassano at gmail.com>> > > Sent: Wednesday, March 10, 2021 7:57:21 AM > > To: Sean Mooney > > Cc: openstack-discuss; Slawek Kaplonski > > Subject: Re: [stein][neutron] gratuitous arp > > > > Hello All, > > please, are there news about bug 1815989 ? > > On stein I modified code as suggested in the patches. > > I am worried when I will upgrade to train: wil this bug persist ? > > On which openstack version this bug is resolved ? > > Ignazio > > > > > > > > Il giorno mer 18 nov 2020 alle ore 07:16 Ignazio Cassano < > ignaziocassano at gmail.com> ha scritto: > > Hello, I tried to update to last stein packages on yum and seems this > bug still exists. > > Before the yum update I patched some files as suggested and and ping to > vm worked fine. > > After yum update the issue returns. > > Please, let me know If I must patch files by hand or some new parameters > in configuration can solve and/or the issue is solved in newer openstack > versions. > > Thanks > > Ignazio > > > > > > Il Mer 29 Apr 2020, 19:49 Sean Mooney smooney at redhat.com>> ha scritto: > > On Wed, 2020-04-29 at 17:10 +0200, Ignazio Cassano wrote: > > > Many thanks. > > > Please keep in touch. > > here are the two patches. > > the first https://review.opendev.org/#/c/724386/ is the actual change > to add the new config opition > > this needs a release note and some tests but it shoudl be functional > hence the [WIP] > > i have not enable the workaround in any job in this patch so the ci run > will assert this does not break > > anything in the default case > > > > the second patch is https://review.opendev.org/#/c/724387/ which > enables the workaround in the multi node ci jobs > > and is testing that live migration exctra works when the workaround is > enabled. > > > > this should work as it is what we expect to happen if you are using a > moderne nova with an old neutron. > > its is marked [DNM] as i dont intend that patch to merge but if the > workaround is useful we migth consider enableing > > it for one of the jobs to get ci coverage but not all of the jobs. > > > > i have not had time to deploy a 2 node env today but ill try and test > this locally tomorow. > > > > > > > > > Ignazio > > > > > > Il giorno mer 29 apr 2020 alle ore 16:55 Sean Mooney < > smooney at redhat.com> > > > ha scritto: > > > > > > > so bing pragmatic i think the simplest path forward given my other > patches > > > > have not laned > > > > in almost 2 years is to quickly add a workaround config option to > disable > > > > mulitple port bindign > > > > which we can backport and then we can try and work on the actual fix > after. > > > > acording to https://bugs.launchpad.net/neutron/+bug/1815989 that > shoudl > > > > serve as a workaround > > > > for thos that hav this issue but its a regression in functionality. > > > > > > > > i can create a patch that will do that in an hour or so and submit a > > > > followup DNM patch to enabel the > > > > workaound in one of the gate jobs that tests live migration. > > > > i have a meeting in 10 mins and need to finish the pacht im > currently > > > > updating but ill submit a poc once that is done. > > > > > > > > im not sure if i will be able to spend time on the actul fix which i > > > > proposed last year but ill see what i can do. > > > > > > > > > > > > On Wed, 2020-04-29 at 16:37 +0200, Ignazio Cassano wrote: > > > > > PS > > > > > I have testing environment on queens,rocky and stein and I can > make test > > > > > as you need. > > > > > Ignazio > > > > > > > > > > Il giorno mer 29 apr 2020 alle ore 16:19 Ignazio Cassano < > > > > > ignaziocassano at gmail.com> ha > scritto: > > > > > > > > > > > Hello Sean, > > > > > > the following is the configuration on my compute nodes: > > > > > > [root at podiscsivc-kvm01 network-scripts]# rpm -qa|grep libvirt > > > > > > libvirt-daemon-driver-storage-iscsi-4.5.0-33.el7.x86_64 > > > > > > libvirt-daemon-kvm-4.5.0-33.el7.x86_64 > > > > > > libvirt-libs-4.5.0-33.el7.x86_64 > > > > > > libvirt-daemon-driver-network-4.5.0-33.el7.x86_64 > > > > > > libvirt-daemon-driver-nodedev-4.5.0-33.el7.x86_64 > > > > > > libvirt-daemon-driver-storage-gluster-4.5.0-33.el7.x86_64 > > > > > > libvirt-client-4.5.0-33.el7.x86_64 > > > > > > libvirt-daemon-driver-storage-core-4.5.0-33.el7.x86_64 > > > > > > libvirt-daemon-driver-storage-logical-4.5.0-33.el7.x86_64 > > > > > > libvirt-daemon-driver-secret-4.5.0-33.el7.x86_64 > > > > > > libvirt-daemon-4.5.0-33.el7.x86_64 > > > > > > libvirt-daemon-driver-nwfilter-4.5.0-33.el7.x86_64 > > > > > > libvirt-daemon-driver-storage-scsi-4.5.0-33.el7.x86_64 > > > > > > libvirt-daemon-driver-storage-rbd-4.5.0-33.el7.x86_64 > > > > > > libvirt-daemon-config-nwfilter-4.5.0-33.el7.x86_64 > > > > > > libvirt-daemon-driver-storage-disk-4.5.0-33.el7.x86_64 > > > > > > libvirt-bash-completion-4.5.0-33.el7.x86_64 > > > > > > libvirt-daemon-driver-qemu-4.5.0-33.el7.x86_64 > > > > > > libvirt-daemon-driver-storage-4.5.0-33.el7.x86_64 > > > > > > libvirt-python-4.5.0-1.el7.x86_64 > > > > > > libvirt-daemon-driver-interface-4.5.0-33.el7.x86_64 > > > > > > libvirt-daemon-driver-storage-mpath-4.5.0-33.el7.x86_64 > > > > > > [root at podiscsivc-kvm01 network-scripts]# rpm -qa|grep qemu > > > > > > qemu-kvm-common-ev-2.12.0-44.1.el7_8.1.x86_64 > > > > > > qemu-kvm-ev-2.12.0-44.1.el7_8.1.x86_64 > > > > > > libvirt-daemon-driver-qemu-4.5.0-33.el7.x86_64 > > > > > > centos-release-qemu-ev-1.0-4.el7.centos.noarch > > > > > > ipxe-roms-qemu-20180825-2.git133f4c.el7.noarch > > > > > > qemu-img-ev-2.12.0-44.1.el7_8.1.x86_64 > > > > > > > > > > > > > > > > > > As far as firewall driver > > > > > > > > /etc/neutron/plugins/ml2/openvswitch_agent.ini: > > > > > > > > > > > > firewall_driver = iptables_hybrid > > > > > > > > > > > > I have same libvirt/qemu version on queens, on rocky and on stein > > > > > > > > testing > > > > > > environment and the > > > > > > same firewall driver. > > > > > > Live migration on provider network on queens works fine. > > > > > > It does not work fine on rocky and stein (vm lost connection > after it > > > > > > > > is > > > > > > migrated and start to respond only when the vm send a network > packet , > > > > > > > > for > > > > > > example when chrony pools the time server). > > > > > > > > > > > > Ignazio > > > > > > > > > > > > > > > > > > > > > > > > Il giorno mer 29 apr 2020 alle ore 14:36 Sean Mooney < > > > > > > > > smooney at redhat.com> > > > > > > ha scritto: > > > > > > > > > > > > > On Wed, 2020-04-29 at 10:39 +0200, Ignazio Cassano wrote: > > > > > > > > Hello, some updated about this issue. > > > > > > > > I read someone has got same issue as reported here: > > > > > > > > > > > > > > > > https://bugs.launchpad.net/neutron/+bug/1866139 > > > > > > > > > > > > > > > > If you read the discussion, someone tells that the garp must > be > > > > > > > > sent by > > > > > > > > qemu during live miration. > > > > > > > > If this is true, this means on rocky/stein the qemu/libvirt > are > > > > > > > > bugged. > > > > > > > > > > > > > > it is not correct. > > > > > > > qemu/libvir thas alsway used RARP which predates GARP to serve > as > > > > > > > > its mac > > > > > > > learning frames > > > > > > > instead > > > > > > > > https://en.wikipedia.org/wiki/Reverse_Address_Resolution_Protocol > > > > > > > > https://lists.gnu.org/archive/html/qemu-devel/2009-10/msg01457.html > > > > > > > however it looks like this was broken in 2016 in qemu 2.6.0 > > > > > > > > https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg04645.html > > > > > > > but was fixed by > > > > > > > > > > > > > > > > https://github.com/qemu/qemu/commit/ca1ee3d6b546e841a1b9db413eb8fa09f13a061b > > > > > > > can you confirm you are not using the broken 2.6.0 release and > are > > > > > > > > using > > > > > > > 2.7 or newer or 2.4 and older. > > > > > > > > > > > > > > > > > > > > > > So I tried to use stein and rocky with the same version of > > > > > > > > libvirt/qemu > > > > > > > > packages I installed on queens (I updated compute and > controllers > > > > > > > > node > > > > > > > > > > > > > > on > > > > > > > > queens for obtaining same libvirt/qemu version deployed on > rocky > > > > > > > > and > > > > > > > > > > > > > > stein). > > > > > > > > > > > > > > > > On queens live migration on provider network continues to > work > > > > > > > > fine. > > > > > > > > On rocky and stein not, so I think the issue is related to > > > > > > > > openstack > > > > > > > > components . > > > > > > > > > > > > > > on queens we have only a singel prot binding and nova blindly > assumes > > > > > > > that the port binding details wont > > > > > > > change when it does a live migration and does not update the > xml for > > > > > > > > the > > > > > > > netwrok interfaces. > > > > > > > > > > > > > > the port binding is updated after the migration is complete in > > > > > > > post_livemigration > > > > > > > in rocky+ neutron optionally uses the multiple port bindings > flow to > > > > > > > prebind the port to the destiatnion > > > > > > > so it can update the xml if needed and if post copy live > migration is > > > > > > > enable it will asyconsly activate teh dest port > > > > > > > binding before post_livemigration shortenting the downtime. > > > > > > > > > > > > > > if you are using the iptables firewall os-vif will have > precreated > > > > > > > > the > > > > > > > ovs port and intermediate linux bridge before the > > > > > > > migration started which will allow neutron to wire it up (put > it on > > > > > > > > the > > > > > > > correct vlan and install security groups) before > > > > > > > the vm completes the migraton. > > > > > > > > > > > > > > if you are using the ovs firewall os-vif still precreates teh > ovs > > > > > > > > port > > > > > > > but libvirt deletes it and recreats it too. > > > > > > > as a result there is a race when using openvswitch firewall > that can > > > > > > > result in the RARP packets being lost. > > > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > > > Ignazio Cassano > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Il giorno lun 27 apr 2020 alle ore 19:50 Sean Mooney < > > > > > > > > > > > > > > smooney at redhat.com> > > > > > > > > ha scritto: > > > > > > > > > > > > > > > > > On Mon, 2020-04-27 at 18:19 +0200, Ignazio Cassano wrote: > > > > > > > > > > Hello, I have this problem with rocky or newer with > > > > > > > > iptables_hybrid > > > > > > > > > > firewall. > > > > > > > > > > So, can I solve using post copy live migration ??? > > > > > > > > > > > > > > > > > > so this behavior has always been how nova worked but rocky > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/neutron-new-port-binding-api.html > > > > > > > > > spec intoduced teh ablity to shorten the outage by pre > biding the > > > > > > > > > > > > > > port and > > > > > > > > > activating it when > > > > > > > > > the vm is resumed on the destiation host before we get to > pos > > > > > > > > live > > > > > > > > > > > > > > migrate. > > > > > > > > > > > > > > > > > > this reduces the outage time although i cant be fully > elimiated > > > > > > > > as > > > > > > > > > > > > > > some > > > > > > > > > level of packet loss is > > > > > > > > > always expected when you live migrate. > > > > > > > > > > > > > > > > > > so yes enabliy post copy live migration should help but be > aware > > > > > > > > that > > > > > > > > > > > > > > if a > > > > > > > > > network partion happens > > > > > > > > > during a post copy live migration the vm will crash and > need to > > > > > > > > be > > > > > > > > > restarted. > > > > > > > > > it is generally safe to use and will imporve the migration > > > > > > > > performace > > > > > > > > > > > > > > but > > > > > > > > > unlike pre copy migration if > > > > > > > > > the guess resumes on the dest and the mempry page has not > been > > > > > > > > copied > > > > > > > > > > > > > > yet > > > > > > > > > then it must wait for it to be copied > > > > > > > > > and retrive it form the souce host. if the connection too > the > > > > > > > > souce > > > > > > > > > > > > > > host > > > > > > > > > is intrupted then the vm cant > > > > > > > > > do that and the migration will fail and the instance will > crash. > > > > > > > > if > > > > > > > > > > > > > > you > > > > > > > > > are using precopy migration > > > > > > > > > if there is a network partaion during the migration the > > > > > > > > migration will > > > > > > > > > fail but the instance will continue > > > > > > > > > to run on the source host. > > > > > > > > > > > > > > > > > > so while i would still recommend using it, i it just good > to be > > > > > > > > aware > > > > > > > > > > > > > > of > > > > > > > > > that behavior change. > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > Ignazio > > > > > > > > > > > > > > > > > > > > Il Lun 27 Apr 2020, 17:57 Sean Mooney < > smooney at redhat.com> ha > > > > > > > > > > > > > > scritto: > > > > > > > > > > > > > > > > > > > > > On Mon, 2020-04-27 at 17:06 +0200, Ignazio Cassano > wrote: > > > > > > > > > > > > Hello, I have a problem on stein neutron. When a vm > migrate > > > > > > > > > > > > > > from one > > > > > > > > > > > > > > > > > > node > > > > > > > > > > > > to another I cannot ping it for several minutes. If > in the > > > > > > > > vm I > > > > > > > > > > > > > > put a > > > > > > > > > > > > script that ping the gateway continously, the live > > > > > > > > migration > > > > > > > > > > > > > > works > > > > > > > > > > > > > > > > > > fine > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > I can ping it. Why this happens ? I read something > about > > > > > > > > > > > > > > gratuitous > > > > > > > > > > > > > > > > > > arp. > > > > > > > > > > > > > > > > > > > > > > qemu does not use gratuitous arp but instead uses an > older > > > > > > > > > > > > > > protocal > > > > > > > > > > > > > > > > > > called > > > > > > > > > > > RARP > > > > > > > > > > > to do mac address learning. > > > > > > > > > > > > > > > > > > > > > > what release of openstack are you using. and are you > using > > > > > > > > > > > > > > iptables > > > > > > > > > > > firewall of openvswitch firewall. > > > > > > > > > > > > > > > > > > > > > > if you are using openvswtich there is is nothing we > can do > > > > > > > > until > > > > > > > > > > > > > > we > > > > > > > > > > > finally delegate vif pluging to os-vif. > > > > > > > > > > > currently libvirt handels interface plugging for > kernel ovs > > > > > > > > when > > > > > > > > > > > > > > using > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > openvswitch firewall driver > > > > > > > > > > > https://review.opendev.org/#/c/602432/ would adress > that > > > > > > > > but it > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > neutron patch are > > > > > > > > > > > https://review.opendev.org/#/c/640258 rather out > dated. > > > > > > > > while > > > > > > > > > > > > > > libvirt > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > pluging the vif there will always be > > > > > > > > > > > a race condition where the RARP packets sent by qemu > and > > > > > > > > then mac > > > > > > > > > > > > > > > > > > learning > > > > > > > > > > > packets will be lost. > > > > > > > > > > > > > > > > > > > > > > if you are using the iptables firewall and you have > opnestack > > > > > > > > > > > > > > rock or > > > > > > > > > > > later then if you enable post copy live migration > > > > > > > > > > > it should reduce the downtime. in this conficution we > do not > > > > > > > > have > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > race > > > > > > > > > > > betwen neutron and libvirt so the rarp > > > > > > > > > > > packets should not be lost. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please, help me ? > > > > > > > > > > > > Any workaround , please ? > > > > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > > > > > > > Ignazio > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From SSelf at performair.com Fri May 14 17:29:52 2021 From: SSelf at performair.com (SSelf at performair.com) Date: Fri, 14 May 2021 17:29:52 +0000 Subject: [victoria] Windows Server 2008 SP2 - No bootble device Message-ID: All; I'm having some trouble getting a Windows Server 2008 SP2 image to boot. On instance start I am greeted with the following: Booting from Hard Disk... Boot failed: not a bootable disk No bootable device. I've followed the instructions, referenced below, with a minor exception; the "Red Hat VirtIO SCSI controller" driver was installed after the OS had been installed as the Pre-installation Environment had troubles recognizing them (yay 2008?). Any recommendations would be welcome. Reference: https://superuser.openstack.org/articles/how-to-deploy-windows-on-openstack/ Thank you, Stephen Self IT Manager Perform Air International Inc. sself at performair.com www.performair.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnsomor at gmail.com Fri May 14 18:12:23 2021 From: johnsomor at gmail.com (Michael Johnson) Date: Fri, 14 May 2021 11:12:23 -0700 Subject: [Octavia][Victoria] No service listening on port 9443 in the amphora instance In-Reply-To: References: <038c74c9-1365-0c08-3b5b-93b4d175dcb3@zylacomputing.com> <326471ef-287b-d937-a174-0b1ccbbd6273@zylacomputing.com>

<8ce76cbd-ca2c-033a-5406-5a1557d84302@zylacomputing.com>

Message-ID: Hi Luke, 1. The "octavia-health-manager listent-port"(s) should be "ACTIVE" in neutron. Something may have gone wrong in the deployment tooling or in neutron for those ports. 2. As for the VIP ports on the amphora, the base port should be "ACTIVE", but the VIP port we use to store the VIP IP address should be "DOWN". The "ACTIVE" base ports will list the VIP IP as it's "allowed-address-pairs" port/ip. 3. On the issue with the health monitor of type PING, it's rarely used outside of some of our tests as it's a poor gauge of the health of an endpoint (https://docs.openstack.org/octavia/latest/user/guides/basic-cookbook.html#other-health-monitors). That said, it should be working. It's an external test in the amphora provider that runs "ping/ping6", so it's interesting that you can ping from the netns successfully. Inside the amphora network namespace, can you try running the ping script directly? export HAPROXY_SERVER_ADDR= /var/lib/octavia/ping-wrapper.sh echo $? An answer of 0 means the ping was successful, 1 a failure. Michael On Fri, May 14, 2021 at 8:15 AM Luke Camilleri wrote: > > Hi Michael, thanks as always for the below, I have watched the video and configured all the requirements as shown in those guides. Working great right now. > > I have noticed the following points and would like to know if you can give me some feedback please: > > In the Octavia project at the networks screen --> lb-mgmt-net --> Ports --> octavia-health-manager-listen-port (the IP bound to the health-manager service) has its status Down. It does not create any sort of issue but was wondering if this was normal behavior? > Similarly to the above point, in the tenant networks screen, every port that is "Attached Device - Octavia" has its status reported as "Down" ( these are the VIP addresses assigned to the amphora). Just need to confirm that this is normal behaviour > Creating a health monitor of type ping fails to get the operating status of the nodes and the nodes are in error (horizon) and the amphora reports that there are no backends and hence it is not working (I am using the same backend nodes with another loadbalancer but with an HTTP check and it is working fine. a security group is setup to allow ping from 0.0.0.0/0 and from the amphora-haproxy network namespace on the amphora instance I can ping both nodes without issues ). Below the amphora's haproxy.log > > May 14 15:00:50 amphora-9658d9ec-3bf1-407f-a134-86304899c015 haproxy[1984]: Server c0092bf4-d2a2-431f-8b7f-9dc3ace52933:e268db93-2d20-4395-bd6f-f6d835bce769/f04824b7-6fdf-46dc-bc83-b98b3b9f5be0 is DOWN, reason: Socket error, info: "Resource temporarily unavailable", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. > > May 14 15:00:50 amphora-9658d9ec-3bf1-407f-a134-86304899c015 haproxy[1984]: Server c0092bf4-d2a2-431f-8b7f-9dc3ace52933:e268db93-2d20-4395-bd6f-f6d835bce769/f04824b7-6fdf-46dc-bc83-b98b3b9f5be0 is DOWN, reason: Socket error, info: "Resource temporarily unavailable", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. > > May 14 15:00:50 amphora-9658d9ec-3bf1-407f-a134-86304899c015 haproxy[1984]: backend c0092bf4-d2a2-431f-8b7f-9dc3ace52933:e268db93-2d20-4395-bd6f-f6d835bce769 has no server available! > > May 14 15:00:50 amphora-9658d9ec-3bf1-407f-a134-86304899c015 haproxy[1984]: backend c0092bf4-d2a2-431f-8b7f-9dc3ace52933:e268db93-2d20-4395-bd6f-f6d835bce769 has no server available! > > Thanks in advance > > On 13/05/2021 18:33, Michael Johnson wrote: > > You are correct that two IPs are being allocated for the VIP, one is a > secondary IP which neutron implements as an "allowed address pairs" > port. We do this to allow failover of the amphora instance should nova > fail the service VM. We hold the VIP IP in a special port so the IP is > not lost while we rebuild the service VM. > If you are using the active/standby topology (or an Octavia flavor > with active/standby enabled), this failover is accelerated with nearly > no visible impact to the flows through the load balancer. > Active/Standby has been an Octavia feature since the Mitaka release. I > gave a demo of it at the Tokoyo summit here: > https://youtu.be/8n7FGhtOiXk?t=1420 > > You can enable active/standby as the default by setting the > "loadbalancer_topology" setting in the configuration file > (https://docs.openstack.org/octavia/latest/configuration/configref.html#controller_worker.loadbalancer_topology) > or by creating an Octavia flavor that creates the load balancer with > an active/standby topology > (https://docs.openstack.org/octavia/latest/admin/flavors.html). > > Michael > > On Thu, May 13, 2021 at 4:23 AM Luke Camilleri > wrote: > > HI Michael, thanks a lot for the below information it is very helpful. I > ended up setting the o-hm0 interface statically in the > octavia-interface.sh script which is called by the service and also > added a delay to make sure that the bridges are up before trying to > create a veth pair and connect the endpoints. > > Also I edited the unit section of the health-manager service and at the > after option I added octavia-interface.service or else on startup the > health manager will not bind to the lb-mgmt-net since it would not be up yet > > The floating IPs part was a bit tricky until I understood what was > really going on with the VIP concept and how better and more flexible it > is to set the VIP on the tenant network and then associate with public > ip to the VIP. > > With this being said I noticed that 2 IPs are being assigned to the > amphora instance and that the actual port assigned to the instance has > an allowed pair with the VIP port. I checked online and it seems that > there is an active/standby project going on with VRRP/keepalived and in > fact the keepalived daemon is running in the amphora instance. > > Am I on the right track with the active/standby feature and if so do you > have any installation/project links to share please so that I can test it? > > Regards > > On 12/05/2021 08:37, Michael Johnson wrote: > > Answers inline below. > > Michael > > On Mon, May 10, 2021 at 5:15 PM Luke Camilleri > wrote: > > Hi Michael and thanks a lot for the detailed answer below. > > I believe I have got most of this sorted out apart from some small issues below: > > If the o-hm0 interface gets the IP information from the DHCP server setup by neutron for the lb-mgmt-net, then the management node will always have 2 default gateways and this will bring along issues, the same DHCP settings when deployed to the amphora do not have the same issue since the amphora only has 1 IP assigned on the lb-mgmt-net. Can you please confirm this? > > The amphorae do not have issues with DHCP and gateways as we control > the DHCP client configuration inside the amphora. It does only have > one IP on the lb-mgmt-net, it will honor gateways provided by neutron > for the lb-mgmt-net traffic, but a gateway is not required on the > lb-mgmt-network unless you are routing the lb-mgmt-net traffic across > subnets. > > How does the amphora know where to locate the worker and housekeeping processes or does the traffic originate from the services instead? Maybe the addresses are "injected" from the config file? > > The worker and housekeeping processes only create connections to the > amphora, they do not receive connections from them. The amphora send a > heartbeat packet to the health manager endpoints every ten seconds by > default. The list of valid health manager endpoints is included in the > amphora agent configuration file that is injected into the service VM > at boot time. It can be updated using the Octavia admin API for > refreshing the amphora agent configuration. > > Can you please confirm if the same floating IP concept runs from public (external) IP to the private (tenant) and from private to lb-mgmt-net please? > > Octavia does not use floating IPs. Users can create and assign > floating IPs via neutron if they would like, but they are not > necessary. Octavia VIPs can be created directly on neutron "external" > networks, avoiding the NAT overhead of floating IPs. > There is no practical reason to assign a floating IP to a port on the > lb-mgmt-net as tenant traffic is never on or accessible from that > network. > > Thanks in advance for any feedback > > On 06/05/2021 22:46, Michael Johnson wrote: > > Hi Luke, > > 1. I agree that DHCP is technically unnecessary for the o-hm0 > interface if you can manage your address allocation on the network you > are using for the lb-mgmt-net. > I don't have detailed information about the Ubuntu install > instructions, but I suspect it was done to simplify the IPAM to be > managed by whatever is providing DHCP on the lb-mgmt-net provided (be > it neutron or some other resource on a provider network). > The lb-mgmt-net is simply a neutron network that the amphora > management address is on. It is routable and does not require external > access. The only tricky part to it is the worker, health manager, and > housekeeping processes need to be reachable from the amphora, and the > controllers need to reach the amphora over the network(s). There are > many ways to accomplish this. > > 2. See my above answer. Fundamentally the lb-mgmt-net is just a > neutron network that nova can use to attach an interface to the > amphora instances for command and control traffic. As long as the > controllers can reach TCP 9433 on the amphora, and the amphora can > send UDP 5555 back to the health manager endpoints, it will work fine. > > 3. Octavia, with the amphora driver, does not require any special > configuration in Neutron (beyond the advanced services RBAC policy > being available for the neutron service account used in your octavia > configuration file). The neutron_lbaas.conf and services_lbaas.conf > are legacy configuration files/settings that were used for > neutron-lbaas which is now end of life. See the wiki page for > information on the deprecation of neutron-lbaas: > https://wiki.openstack.org/wiki/Neutron/LBaaS/Deprecation. > > Michael > > On Thu, May 6, 2021 at 12:30 PM Luke Camilleri > wrote: > > Hi Michael and thanks a lot for your help on this, after following your > steps the agent got deployed successfully in the amphora-image. > > I have some other queries that I would like to ask mainly related to the > health-manager/load-balancer network setup and IP assignment. First of > all let me point out that I am using a manual installation process, and > it might help others to understand the underlying infrastructure > required to make this component work as expected. > > 1- The installation procedure contains this step: > > $ sudo cp octavia/etc/dhcp/dhclient.conf /etc/dhcp/octavia > > which is later on called to assign the IP to the o-hm0 interface which > is connected to the lb-management network as shown below: > > $ sudo dhclient -v o-hm0 -cf /etc/dhcp/octavia > > Apart from having a dhcp config for a single IP seems a bit of an > overkill, using these steps is injecting an additional routing table > into the default namespace as shown below in my case: > > # route -n > Kernel IP routing table > Destination Gateway Genmask Flags Metric Ref Use > Iface > 0.0.0.0 172.16.0.1 0.0.0.0 UG 0 0 0 o-hm0 > 0.0.0.0 10.X.X.1 0.0.0.0 UG 100 0 0 ensX > 10.X.X.0 0.0.0.0 255.255.255.0 U 100 0 0 ensX > 169.254.169.254 172.16.0.100 255.255.255.255 UGH 0 0 0 o-hm0 > 172.16.0.0 0.0.0.0 255.240.0.0 U 0 0 0 o-hm0 > > Since the load-balancer management network does not need any external > connectivity (but only communication between health-manager service and > amphora-agent), why is a gateway required and why isn't the IP address > allocated as part of the interface creation script which is called when > the service is started or stopped (example below)? > > --- > > #!/bin/bash > > set -ex > > MAC=$MGMT_PORT_MAC > BRNAME=$BRNAME > > if [ "$1" == "start" ]; then > ip link add o-hm0 type veth peer name o-bhm0 > brctl addif $BRNAME o-bhm0 > ip link set o-bhm0 up > ip link set dev o-hm0 address $MAC > *** ip addr add 172.16.0.2/12 dev o-hm0 > ***ip link set o-hm0 mtu 1500 > ip link set o-hm0 up > iptables -I INPUT -i o-hm0 -p udp --dport 5555 -j ACCEPT > elif [ "$1" == "stop" ]; then > ip link del o-hm0 > else > brctl show $BRNAME > ip a s dev o-hm0 > fi > > --- > > 2- Is there a possibility to specify a fixed vlan outside of tenant > range for the load balancer management network? > > 3- Are the configuration changes required only in neutron.conf or also > in additional config files like neutron_lbaas.conf and > services_lbaas.conf, similar to the vpnaas configuration? > > Thanks in advance for any assistance, but its like putting together a > puzzle of information :-) > > On 05/05/2021 20:25, Michael Johnson wrote: > > Hi Luke. > > Yes, the amphora-agent will listen on 9443 in the amphorae instances. > It uses TLS mutual authentication, so you can get a TLS response, but > it will not let you into the API without a valid certificate. A simple > "openssl s_client" is usually enough to prove that it is listening and > requesting the client certificate. > > I can't talk to the "openstack-octavia-diskimage-create" package you > found in centos, but I can discuss how to build an amphora image using > the OpenStack tools. > > If you get Octavia from git or via a release tarball, we provide a > script to build the amphora image. This is how we build our images for > the testing gates, etc. and is the recommended way (at least from the > OpenStack Octavia community) to create amphora images. > > https://opendev.org/openstack/octavia/src/branch/master/diskimage-create > > For CentOS 8, the command would be: > > diskimage-create.sh -g stable/victoria -i centos-minimal -d 8 -s 3 (3 > is the minimum disk size for centos images, you may want more if you > are not offloading logs) > > I just did a run on a fresh centos 8 instance: > git clone https://opendev.org/openstack/octavia > python3 -m venv dib > source dib/bin/activate > pip3 install diskimage-builder PyYAML six > sudo dnf install yum-utils > ./diskimage-create.sh -g stable/victoria -i centos-minimal -d 8 -s 3 > > This built an image. > > Off and on we have had issues building CentOS images due to issues in > the tools we rely on. If you run into issues with this image, drop us > a note back. > > Michael > > On Wed, May 5, 2021 at 9:37 AM Luke Camilleri > wrote: > > Hi there, i am trying to get Octavia running on a Victoria deployment on > CentOS 8. It was a bit rough getting to the point to launch an instance > mainly due to the load-balancer management network and the lack of > documentation > (https://docs.openstack.org/octavia/victoria/install/install.html) to > deploy this oN CentOS. I will try to fix this once I have my deployment > up and running to help others on the way installing and configuring this :-) > > At this point a LB can be launched by the tenant and the instance is > spawned in the Octavia project and I can ping and SSH into the amphora > instance from the Octavia node where the octavia-health-manager service > is running using the IP within the same subnet of the amphoras > (172.16.0.0/12). > > Unfortunately I keep on getting these errors in the log file of the > worker log (/var/log/octavia/worker.log): > > 2021-05-05 01:54:49.368 14521 WARNING > octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect > to instance. Retrying.: requests.exceptions.ConnectionError: > HTTPSConnectionPool(host='172.16.4.46', p > ort=9443): Max retries exceeded with url: // (Caused by > NewConnectionError(' at 0x7f83e0181550>: Failed to establish a new connection: [Errno 111] > Connection ref > used',)) > > 2021-05-05 01:54:54.374 14521 ERROR > octavia.amphorae.drivers.haproxy.rest_api_driver [-] Connection retries > (currently set to 120) exhausted. The amphora is unavailable. Reason: > HTTPSConnectionPool(host='172.16 > .4.46', port=9443): Max retries exceeded with url: // (Caused by > NewConnectionError(' at 0x7f83e0181550>: Failed to establish a new connection: [Errno 111] Conne > ction refused',)) > > 2021-05-05 01:54:54.374 14521 ERROR > octavia.controller.worker.v1.tasks.amphora_driver_tasks [-] Amphora > compute instance failed to become reachable. This either means the > compute driver failed to fully boot the > instance inside the timeout interval or the instance is not reachable > via the lb-mgmt-net.: > octavia.amphorae.driver_exceptions.exceptions.TimeOutException: > contacting the amphora timed out > > obviously the instance is deleted then and the task fails from the > tenant's perspective. > > The main issue here is that there is no service running on port 9443 on > the amphora instance. I am assuming that this is in fact the > amphora-agent service that is running on the instance which should be > listening on this port 9443 but the service does not seem to be up or > not installed at all. > > To create the image I have installed the CentOS package > "openstack-octavia-diskimage-create" which provides the utility > disk-image-create but from what I can conclude the amphora-agent is not > being installed (thought this was done automatically by default :-( ) > > Can anyone let me know if the amphora-agent is what gets queried on port > 9443 ? > > If the agent is not installed/injected by default when building the > amphora image? > > The command to inject the amphora-agent into the amphora image when > using the disk-image-create command? > > Thanks in advance for any assistance > > From elod.illes at est.tech Fri May 14 20:09:17 2021 From: elod.illes at est.tech (=?UTF-8?B?RWzFkWQgSWxsw6lz?=) Date: Fri, 14 May 2021 22:09:17 +0200 Subject: [all][stable] Delete $series-eol tagged branches Message-ID: <96e1cb5a-99c5-7a05-01c0-1635743d9c1d@est.tech> Hi, As I wrote previously [1] the long-waited deletion of $series-eol tagged branches started with the removal of ocata-eol tagged ones first (for the list of deleted branches, see: [2]) Then I also sent out a warning [3] about the next step, to delete pike-eol tagged branches, which finally happened today (for the list of deleted branches, see: [4]). So now I'm sending out a *warning* again, that as a 3rd step, the deletion of already tagged but still open branches will continue in 1 or 2 weeks time frame. If everything works as expected, then branches with *queens-eol*, *rocky-eol* and *stein-eol* tags can be processed in one batch. Also I would like to ask the teams who have $series-eol tagged branches to abandon all open patches on those branches, otherwise the branch cannot be deleted. Thanks, Előd [1] http://lists.openstack.org/pipermail/openstack-discuss/2021-April/021949.html [2] http://paste.openstack.org/show/804953/ [3] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022173.html [4] http://paste.openstack.org/show/805404/ From elod.illes at est.tech Fri May 14 20:18:18 2021 From: elod.illes at est.tech (=?UTF-8?B?RWzFkWQgSWxsw6lz?=) Date: Fri, 14 May 2021 22:18:18 +0200 Subject: [neutron][stadium][stable] Proposal to make stable/ocata and stable/pike branches EOL In-Reply-To: <6767170.WaEBY35tY3@p1> References: <15209060.0YdeOJI3E6@p1> <55ff12c8-1e9c-16b5-578b-834d1ccf2563@est.tech> <6767170.WaEBY35tY3@p1> Message-ID: <5d31de6f-162c-56c8-eee1-efeb212df05d@est.tech> Hi, The patch was merged, so the ocata-eol tags were created for neutron projects. After the successful tagging I have executed the branch deletion, which has the following result: Branch stable/ocata successfully deleted from openstack/networking-ovn! Branch stable/ocata successfully deleted from openstack/neutron-dynamic-routing! Branch stable/ocata successfully deleted from openstack/neutron-fwaas! Branch stable/ocata successfully deleted from openstack/neutron-lbaas! Branch stable/ocata successfully deleted from openstack/neutron-lib! Branch stable/ocata successfully deleted from openstack/neutron! Thanks, Előd On 2021. 05. 12. 9:22, Slawek Kaplonski wrote: > > Hi, > > > Dnia środa, 5 maja 2021 20:35:48 CEST Előd Illés pisze: > > > Hi, > > > > > > Ocata is unfortunately unmaintained for a long time as some general test > > > jobs are broken there, so as a stable-maint-core member I support to tag > > > neutron's stable/ocata as End of Life. After the branch is tagged, > > > please ping me and I can arrange the deletion of the branch. > > > > > > For Pike, I volunteered at the PTG in 2020 to help with reviews there, I > > > still keep that offer, however I am clearly not enough to keep it > > > maintained, besides backports are not arriving for stable/pike in > > > neutron. Anyway, if the gate is functional there, then I say we could > > > keep it open (but as far as I see how gate situation is worsen now, as > > > more and more things go wrong, I don't expect that will take long). If > > > not, then I only ask that let's do the EOL'ing first with Ocata and when > > > it is done, then continue with neutron's stable/pike. > > > > > > For the process please follow the steps here: > > > > https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-life > > > (with the only exception, that in the last step, instead of infra team, > > > please turn to me/release team - patch for the documentation change is > > > on the way: > > > https://review.opendev.org/c/openstack/project-team-guide/+/789932 ) > > > Thx. I just proposed patch > https://review.opendev.org/c/openstack/releases/+/790904 > to make > ocata-eol in all neutron projects. > > > > > > > Thanks, > > > > > > Előd > > > > > > On 2021. 05. 05. 16:13, Slawek Kaplonski wrote: > > > > Hi, > > > > > > > > > > > > I checked today that stable/ocata and stable/pike branches in both > > > > Neutron and neutron stadium projects are pretty inactive since > long time. > > > > > > > > * according to [1], last patch merged patch in Neutron for stable/pike > > > > was in July 2020 and in ocata October 2019, > > > > > > > > * for stadium projects, according to [2] it was September 2020. > > > > > > > > > > > > According to [3] and [4] there are no opened patches for any of those > > > > branches for Neutron and any stadium project except neutron-lbaas. > > > > > > > > > > > > So based on that info I want to propose that we will close both those > > > > branches are EOL now and before doing that, I would like to know if > > > > anyone would like to keep those branches to be open still. > > > > > > > > > > > > [1] > > > > > https://review.opendev.org/q/project:%255Eopenstack/neutron+(branch:stable/ocata+OR+branch:stable/pike)+status:merged > > > > > > > > > > > > > [2] > > > > > https://review.opendev.org/q/(project:openstack/ovsdbapp+OR+project:openstack/os-ken+OR+project:%255Eopenstack/neutron-.*+OR+project:%255Eopenstack/networki > > > > ng-.*)+(branch:stable/ocata+OR+branch:stable/pike)+status:merged > > > > > > > > king-.*)+(branch:stable/ocata+OR+branch:stable/pike)+status:merged> > > > > > > > > [3] > > > > > https://review.opendev.org/q/project:%255Eopenstack/neutron+(branch:stable/ocata+OR+branch:stable/pike)+status:open > > > > > > > > > > > > > [4] > > > > > https://review.opendev.org/q/(project:openstack/ovsdbapp+OR+project:openstack/os-ken+OR+project:%255Eopenstack/neutron-.*+OR+project:%255Eopenstack/networki > > > > ng-.*)+(branch:stable/ocata+OR+branch:stable/pike)+status:open > > > > > > > > king-.*)+(branch:stable/ocata+OR+branch:stable/pike)+status:open> > > > > > > > > > > > > -- > > > > > > > > Slawek Kaplonski > > > > > > > > Principal Software Engineer > > > > > > > > Red Hat > > > > -- > > Slawek Kaplonski > > Principal Software Engineer > > Red Hat > -------------- next part -------------- An HTML attachment was scrubbed... URL: From klemen at psi-net.si Fri May 14 20:26:31 2021 From: klemen at psi-net.si (Klemen Pogacnik) Date: Fri, 14 May 2021 22:26:31 +0200 Subject: [kolla] Reorganization of kolla-ansible documentation Message-ID: Hello! I promised to prepare my view as a user of kolla-ansible on its documentation. In my opinion the division between admin guides and user guides is artificial, as the user of kolla-ansible is actually the cloud administrator. Maybe it would be good to think about reorganizing the structure of documentation. Many good chapters are already written, they only have to be positioned in the right place to be found more easily. So here is my proposal of kolla-ansible doc's structure: 1. Introduction 1.1. mission 1.2. benefits 1.3. support matrix 2. Architecture 2.1. basic architecture 2.2. HA architecture 2.3. network architecture 2.4. storage architecture 3. Workflows 3.1. preparing the surroundings (networking, docker registry, ...) 3.2. preparing servers (packages installation) 3.3. configuration (of kolla-ansible and description of basic logic for configuration of Openstack modules) 3.4. 1st day procedures (bootstrap, deploy, destroy) 3.5. 2nd day procedures (reconfigure, upgrade, add, remove nodes ...) 3.6. multiple regions 3.7. multiple cloud 3.8. security 3.9. troubleshooting (how to check, if cloud works, what to do, if it doesn't) 4. Use Cases 4.1. all-in-one 4.2. basic vm multinode 4.3. some production use cases 5. Reference guide Mostly the same structure as already is. Except it would be desirable that description of each module has: - purpose of the module - configuration of the module - how to use it with links to module docs - basic troubleshooting 6. Contributor guide The documentation also needs figures, pictures, diagrams to be more understandable. So at least in the first chapters some of them shall be added. I'm also thinking about convergence of documentation of kayobe, kolla and kolla-ansible projects. It's true that there's no strict connection between kayobe and other two and kolla containers can be used without kolla-ansible playbooks. But the real benefit the user can get is to use all three projects together. But let's leave that for the second phase. So please comment on this proposal. Do you think it's going in the right direction? If yes, I can refine it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From elod.illes at est.tech Fri May 14 20:39:30 2021 From: elod.illes at est.tech (=?UTF-8?B?RWzFkWQgSWxsw6lz?=) Date: Fri, 14 May 2021 22:39:30 +0200 Subject: [octavia][tripleo][kolla][stable][release] $series-eol delete problem Message-ID: Hi teams in $SUBJECT, during the deletion of $series-eol tagged branches it turned out that the below listed branches / repositories contains merged patches on top of $series-eol tag. The issue is with this that whenever the branch is deleted only the $series-eol (and other) tags can be checked out, so the changes that were merged after the eol tags, will be *lost*. There are two options now: 1. Create another tag (something like: "$series-eol-extra"), so that the extra patches will not be lost completely, because they can be checked out with the newly created tags 2. Delete the branch anyway and don't care about the lost patch(es) Here are the list of such branches, please consider which option is good for the team and reply to this mail: openstack/octavia * stable/stein has patches on top of the stein-eol tag * stable/queens has patches on top of the queens-eol tag openstack/kolla * stable/pike has patches on top of the pike-eol tag * stable/ocata has patches on top of the ocata-eol tag openstack/tripleo-common * stable/rocky has patches on top of the rocky-eol tag openstack/os-apply-config * stable/pike has patches on top of the pike-eol tag * stable/ocata has patches on top of the ocata-eol tag openstack/os-cloud-config stable/ocata has patches on top of the ocata-eol tag Thanks, Előd From sebastian.luna.valero at gmail.com Sat May 15 08:08:24 2021 From: sebastian.luna.valero at gmail.com (Sebastian Luna Valero) Date: Sat, 15 May 2021 10:08:24 +0200 Subject: Restart cinder-volume with Ceph rdb In-Reply-To: <771F27B8-6C13-4F04-85D3-331E2AF7D89F@binero.com> References: <20210511203053.Horde.nJ-7FFjvzdcxuyQKn9UmErJ@webmail.nde.ag> <08C8BC3F-3930-4803-B007-3E3C6BD1F411@iaa.es> <20210512094943.nfttmyxoss3zut2n@localhost> <90542A09-3A7D-4FE2-83FD-10D46CCEF5A2@iaa.es> <20210512213446.7c222mlcdwxiosly@localhost> <0292176B-BE6D-448E-8948-EE10300E2520@iaa.es> <20210513073722.x3z3qkcpvg5am6ia@localhost> <771F27B8-6C13-4F04-85D3-331E2AF7D89F@binero.com> Message-ID: Hi All, Thanks for your inputs so far. I am also trying to help Manu with this issue. The "cinder-volume" service was working properly with the existing configuration. However, after a power outage the service is no longer reported as "up". Looking at the source code, the service status is reported as "down" by "cinder-scheduler" in here: https://github.com/openstack/cinder/blob/stable/train/cinder/scheduler/host_manager.py#L618 With message: "WARNING cinder.scheduler.host_manager [req-<>- default default] volume service is down. (host: rbd:volumes at ceph-rbd)" I printed out the "service" tuple https://github.com/openstack/cinder/blob/stable/train/cinder/scheduler/host_manager.py#L615 and we get: "2021-05-15 09:57:24.918 7 WARNING cinder.scheduler.host_manager [<> - default default] Service(active_backend_id=None,availability_zone='nova',binary='cinder-volume',cluster=,cluster_name=None,created_at=2020-06-12T07:53:42Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,frozen=False,host='rbd:volumes at ceph-rbd ',id=12,modified_at=None,object_current_version='1.38',replication_status='disabled',report_count=8067424,rpc_current_version='3.16',topic='cinder-volume',updated_at=2021-05-12T15:37:52Z,uuid='604668e8-c2e7-46ed-a2b8-086e588079ac')" Cinder is configured with a Ceph RBD backend, as explained in https://github.com/openstack/kolla-ansible/blob/stable/train/doc/source/reference/storage/external-ceph-guide.rst#cinder That's where the "backend_host=rbd:volumes" configuration is coming from. We are using 3 controller nodes for OpenStack and 3 monitor nodes for Ceph. The Ceph cluster doesn't report any error. The "cinder-volume" containers don't report any error. Moreover, when we go inside the "cinder-volume" container we are able to list existing volumes with: rbd -p cinder.volumes --id cinder -k /etc/ceph/ceph.client.cinder.keyring ls So the connection to the Ceph cluster works. Why is "cinder-scheduler" reporting the that the backend Ceph cluster is down? Many thanks, Sebastian On Thu, 13 May 2021 at 13:12, Tobias Urdin wrote: > Hello, > > I just saw that you are running Ceph Octopus with Train release and wanted > to let you know that we saw issues with the os-brick version shipped with > Train not supporting client version of Ceph Octopus. > > So for our Ceph cluster running Octopus we had to keep the client version > on Nautilus until upgrading to Victoria which included a newer version of > os-brick. > > Maybe this is unrelated to your issue but just wanted to put it out there. > > Best regards > Tobias > > > On 13 May 2021, at 12:55, ManuParra wrote: > > > > Hello Gorka, not yet, let me update cinder configuration, add the > option, restart cinder and I’ll update the status. > > Do you recommend other things to try for this cycle? > > Regards. > > > >> On 13 May 2021, at 09:37, Gorka Eguileor wrote: > >> > >>> On 13/05, ManuParra wrote: > >>> Hi Gorka again, yes, the first thing is to know why you can't connect > to that host (Ceph is actually set up for HA) so that's the way to do it. I > tell you this because previously from the beginning of the setup of our > setup it has always been like that, with that hostname and there has been > no problem. > >>> > >>> As for the errors, the strangest thing is that in Monasca I have not > found any error log, only warning on “volume service is down. (host: > rbd:volumes at ceph-rbd)" and info, which is even stranger. > >> > >> Have you tried the configuration change I recommended? > >> > >> > >>> > >>> Regards. > >>> > >>>> On 12 May 2021, at 23:34, Gorka Eguileor wrote: > >>>> > >>>> On 12/05, ManuParra wrote: > >>>>> Hi Gorka, let me show the cinder config: > >>>>> > >>>>> [ceph-rbd] > >>>>> rbd_ceph_conf = /etc/ceph/ceph.conf > >>>>> rbd_user = cinder > >>>>> backend_host = rbd:volumes > >>>>> rbd_pool = cinder.volumes > >>>>> volume_backend_name = ceph-rbd > >>>>> volume_driver = cinder.volume.drivers.rbd.RBDDriver > >>>>> … > >>>>> > >>>>> So, using rbd_exclusive_cinder_pool=True it will be used just for > volumes? but the log is saying no connection to the backend_host. > >>>> > >>>> Hi, > >>>> > >>>> Your backend_host doesn't have a valid hostname, please set a proper > >>>> hostname in that configuration option. > >>>> > >>>> Then the next thing you need to have is the cinder-volume service > >>>> running correctly before making any requests. > >>>> > >>>> I would try adding rbd_exclusive_cinder_pool=true then tailing the > >>>> volume logs, and restarting the service. > >>>> > >>>> See if the logs show any ERROR level entries. > >>>> > >>>> I would also check the service-list output right after the service is > >>>> restarted, if it's up then I would check it again after 2 minutes. > >>>> > >>>> Cheers, > >>>> Gorka. > >>>> > >>>> > >>>>> > >>>>> Regards. > >>>>> > >>>>> > >>>>>> On 12 May 2021, at 11:49, Gorka Eguileor > wrote: > >>>>>> > >>>>>> On 12/05, ManuParra wrote: > >>>>>>> Thanks, I have restarted the service and I see that after a few > minutes then cinder-volume service goes down again when I check it with the > command openstack volume service list. > >>>>>>> The host/service that contains the cinder-volumes is > rbd:volumes at ceph-rbd that is RDB in Ceph, so the problem does not come > from Cinder, rather from Ceph or from the RDB (Ceph) pools that stores the > volumes. I have checked Ceph and the status of everything is correct, no > errors or warnings. > >>>>>>> The error I have is that cinder can’t connect to > rbd:volumes at ceph-rbd. Any further suggestions? Thanks in advance. > >>>>>>> Kind regards. > >>>>>>> > >>>>>> > >>>>>> Hi, > >>>>>> > >>>>>> You are most likely using an older release, have a high number of > cinder > >>>>>> RBD volumes, and have not changed configuration option > >>>>>> "rbd_exclusive_cinder_pool" from its default "false" value. > >>>>>> > >>>>>> Please add to your driver's section in cinder.conf the following: > >>>>>> > >>>>>> rbd_exclusive_cinder_pool = true > >>>>>> > >>>>>> > >>>>>> And restart the service. > >>>>>> > >>>>>> Cheers, > >>>>>> Gorka. > >>>>>> > >>>>>>>> On 11 May 2021, at 22:30, Eugen Block wrote: > >>>>>>>> > >>>>>>>> Hi, > >>>>>>>> > >>>>>>>> so restart the volume service;-) > >>>>>>>> > >>>>>>>> systemctl restart openstack-cinder-volume.service > >>>>>>>> > >>>>>>>> > >>>>>>>> Zitat von ManuParra : > >>>>>>>> > >>>>>>>>> Dear OpenStack community, > >>>>>>>>> > >>>>>>>>> I have encountered a problem a few days ago and that is that > when creating new volumes with: > >>>>>>>>> > >>>>>>>>> "openstack volume create --size 20 testmv" > >>>>>>>>> > >>>>>>>>> the volume creation status shows an error. If I go to the error > log detail it indicates: > >>>>>>>>> > >>>>>>>>> "Schedule allocate volume: Could not find any available weighted > backend". > >>>>>>>>> > >>>>>>>>> Indeed then I go to the cinder log and it indicates: > >>>>>>>>> > >>>>>>>>> "volume service is down - host: rbd:volumes at ceph-rbd”. > >>>>>>>>> > >>>>>>>>> I check with: > >>>>>>>>> > >>>>>>>>> "openstack volume service list” in which state are the services > and I see that indeed this happens: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> | cinder-volume | rbd:volumes at ceph-rbd | nova | enabled | down > | 2021-04-29T09:48:42.000000 | > >>>>>>>>> > >>>>>>>>> And stopped since 2021-04-29 ! > >>>>>>>>> > >>>>>>>>> I have checked Ceph (monitors,managers, osds. etc) and there are > no problems with the Ceph BackEnd, everything is apparently working. > >>>>>>>>> > >>>>>>>>> This happened after an uncontrolled outage.So my question is how > do I restart only cinder-volumes (I also have cinder-backup, > cinder-scheduler but they are ok). > >>>>>>>>> > >>>>>>>>> Thank you very much in advance. Regards. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > >> > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From laurentfdumont at gmail.com Sat May 15 17:39:57 2021 From: laurentfdumont at gmail.com (Laurent Dumont) Date: Sat, 15 May 2021 13:39:57 -0400 Subject: Restart cinder-volume with Ceph rdb In-Reply-To: References: <20210511203053.Horde.nJ-7FFjvzdcxuyQKn9UmErJ@webmail.nde.ag> <08C8BC3F-3930-4803-B007-3E3C6BD1F411@iaa.es> <20210512094943.nfttmyxoss3zut2n@localhost> <90542A09-3A7D-4FE2-83FD-10D46CCEF5A2@iaa.es> <20210512213446.7c222mlcdwxiosly@localhost> <0292176B-BE6D-448E-8948-EE10300E2520@iaa.es> <20210513073722.x3z3qkcpvg5am6ia@localhost> <771F27B8-6C13-4F04-85D3-331E2AF7D89F@binero.com> Message-ID: That is a bit strange. I don't use the Ceph backend so I don't know any magic tricks. - I'm surprised that the Debug logging level doesn't add anything else. Is there any other lines besides the "connecting" one? - Can we narrow down the port/IP destination for the Ceph RBD traffic? - Can we failover the cinder-volume service to another controller and check the status of the volume service? - Did the power outage impact the Ceph cluster + network gear + all the controllers? - Does the content of /etc/ceph/ceph.conf appear to be valid inside the container? Looking at the code - https://github.com/openstack/cinder/blob/stable/train/cinder/volume/drivers/rbd.py#L432 It should raise an exception if there is a timeout when the connection client is built. except self.rados.Error: msg = _("Error connecting to ceph cluster.") LOG.exception(msg) client.shutdown() raise exception.VolumeBackendAPIException(data=msg) On Sat, May 15, 2021 at 4:16 AM Sebastian Luna Valero < sebastian.luna.valero at gmail.com> wrote: > > Hi All, > > Thanks for your inputs so far. I am also trying to help Manu with this > issue. > > The "cinder-volume" service was working properly with the existing > configuration. However, after a power outage the service is no longer > reported as "up". > > Looking at the source code, the service status is reported as "down" by > "cinder-scheduler" in here: > > > https://github.com/openstack/cinder/blob/stable/train/cinder/scheduler/host_manager.py#L618 > > With message: "WARNING cinder.scheduler.host_manager [req-<>- default > default] volume service is down. (host: rbd:volumes at ceph-rbd)" > > I printed out the "service" tuple > https://github.com/openstack/cinder/blob/stable/train/cinder/scheduler/host_manager.py#L615 > and we get: > > "2021-05-15 09:57:24.918 7 WARNING cinder.scheduler.host_manager [<> - > default default] > Service(active_backend_id=None,availability_zone='nova',binary='cinder-volume',cluster=,cluster_name=None,created_at=2020-06-12T07:53:42Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,frozen=False,host='rbd:volumes at ceph-rbd > ',id=12,modified_at=None,object_current_version='1.38',replication_status='disabled',report_count=8067424,rpc_current_version='3.16',topic='cinder-volume',updated_at=2021-05-12T15:37:52Z,uuid='604668e8-c2e7-46ed-a2b8-086e588079ac')" > > Cinder is configured with a Ceph RBD backend, as explained in > https://github.com/openstack/kolla-ansible/blob/stable/train/doc/source/reference/storage/external-ceph-guide.rst#cinder > > That's where the "backend_host=rbd:volumes" configuration is coming from. > > We are using 3 controller nodes for OpenStack and 3 monitor nodes for Ceph. > > The Ceph cluster doesn't report any error. The "cinder-volume" containers > don't report any error. Moreover, when we go inside the "cinder-volume" > container we are able to list existing volumes with: > > rbd -p cinder.volumes --id cinder -k /etc/ceph/ceph.client.cinder.keyring > ls > > So the connection to the Ceph cluster works. > > Why is "cinder-scheduler" reporting the that the backend Ceph cluster is > down? > > Many thanks, > Sebastian > > > On Thu, 13 May 2021 at 13:12, Tobias Urdin > wrote: > >> Hello, >> >> I just saw that you are running Ceph Octopus with Train release and >> wanted to let you know that we saw issues with the os-brick version shipped >> with Train not supporting client version of Ceph Octopus. >> >> So for our Ceph cluster running Octopus we had to keep the client version >> on Nautilus until upgrading to Victoria which included a newer version of >> os-brick. >> >> Maybe this is unrelated to your issue but just wanted to put it out there. >> >> Best regards >> Tobias >> >> > On 13 May 2021, at 12:55, ManuParra wrote: >> > >> > Hello Gorka, not yet, let me update cinder configuration, add the >> option, restart cinder and I’ll update the status. >> > Do you recommend other things to try for this cycle? >> > Regards. >> > >> >> On 13 May 2021, at 09:37, Gorka Eguileor wrote: >> >> >> >>> On 13/05, ManuParra wrote: >> >>> Hi Gorka again, yes, the first thing is to know why you can't connect >> to that host (Ceph is actually set up for HA) so that's the way to do it. I >> tell you this because previously from the beginning of the setup of our >> setup it has always been like that, with that hostname and there has been >> no problem. >> >>> >> >>> As for the errors, the strangest thing is that in Monasca I have not >> found any error log, only warning on “volume service is down. (host: >> rbd:volumes at ceph-rbd)" and info, which is even stranger. >> >> >> >> Have you tried the configuration change I recommended? >> >> >> >> >> >>> >> >>> Regards. >> >>> >> >>>> On 12 May 2021, at 23:34, Gorka Eguileor >> wrote: >> >>>> >> >>>> On 12/05, ManuParra wrote: >> >>>>> Hi Gorka, let me show the cinder config: >> >>>>> >> >>>>> [ceph-rbd] >> >>>>> rbd_ceph_conf = /etc/ceph/ceph.conf >> >>>>> rbd_user = cinder >> >>>>> backend_host = rbd:volumes >> >>>>> rbd_pool = cinder.volumes >> >>>>> volume_backend_name = ceph-rbd >> >>>>> volume_driver = cinder.volume.drivers.rbd.RBDDriver >> >>>>> … >> >>>>> >> >>>>> So, using rbd_exclusive_cinder_pool=True it will be used just for >> volumes? but the log is saying no connection to the backend_host. >> >>>> >> >>>> Hi, >> >>>> >> >>>> Your backend_host doesn't have a valid hostname, please set a proper >> >>>> hostname in that configuration option. >> >>>> >> >>>> Then the next thing you need to have is the cinder-volume service >> >>>> running correctly before making any requests. >> >>>> >> >>>> I would try adding rbd_exclusive_cinder_pool=true then tailing the >> >>>> volume logs, and restarting the service. >> >>>> >> >>>> See if the logs show any ERROR level entries. >> >>>> >> >>>> I would also check the service-list output right after the service is >> >>>> restarted, if it's up then I would check it again after 2 minutes. >> >>>> >> >>>> Cheers, >> >>>> Gorka. >> >>>> >> >>>> >> >>>>> >> >>>>> Regards. >> >>>>> >> >>>>> >> >>>>>> On 12 May 2021, at 11:49, Gorka Eguileor >> wrote: >> >>>>>> >> >>>>>> On 12/05, ManuParra wrote: >> >>>>>>> Thanks, I have restarted the service and I see that after a few >> minutes then cinder-volume service goes down again when I check it with the >> command openstack volume service list. >> >>>>>>> The host/service that contains the cinder-volumes is >> rbd:volumes at ceph-rbd that is RDB in Ceph, so the problem does not come >> from Cinder, rather from Ceph or from the RDB (Ceph) pools that stores the >> volumes. I have checked Ceph and the status of everything is correct, no >> errors or warnings. >> >>>>>>> The error I have is that cinder can’t connect to >> rbd:volumes at ceph-rbd. Any further suggestions? Thanks in advance. >> >>>>>>> Kind regards. >> >>>>>>> >> >>>>>> >> >>>>>> Hi, >> >>>>>> >> >>>>>> You are most likely using an older release, have a high number of >> cinder >> >>>>>> RBD volumes, and have not changed configuration option >> >>>>>> "rbd_exclusive_cinder_pool" from its default "false" value. >> >>>>>> >> >>>>>> Please add to your driver's section in cinder.conf the following: >> >>>>>> >> >>>>>> rbd_exclusive_cinder_pool = true >> >>>>>> >> >>>>>> >> >>>>>> And restart the service. >> >>>>>> >> >>>>>> Cheers, >> >>>>>> Gorka. >> >>>>>> >> >>>>>>>> On 11 May 2021, at 22:30, Eugen Block wrote: >> >>>>>>>> >> >>>>>>>> Hi, >> >>>>>>>> >> >>>>>>>> so restart the volume service;-) >> >>>>>>>> >> >>>>>>>> systemctl restart openstack-cinder-volume.service >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> Zitat von ManuParra : >> >>>>>>>> >> >>>>>>>>> Dear OpenStack community, >> >>>>>>>>> >> >>>>>>>>> I have encountered a problem a few days ago and that is that >> when creating new volumes with: >> >>>>>>>>> >> >>>>>>>>> "openstack volume create --size 20 testmv" >> >>>>>>>>> >> >>>>>>>>> the volume creation status shows an error. If I go to the >> error log detail it indicates: >> >>>>>>>>> >> >>>>>>>>> "Schedule allocate volume: Could not find any available >> weighted backend". >> >>>>>>>>> >> >>>>>>>>> Indeed then I go to the cinder log and it indicates: >> >>>>>>>>> >> >>>>>>>>> "volume service is down - host: rbd:volumes at ceph-rbd”. >> >>>>>>>>> >> >>>>>>>>> I check with: >> >>>>>>>>> >> >>>>>>>>> "openstack volume service list” in which state are the >> services and I see that indeed this happens: >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> | cinder-volume | rbd:volumes at ceph-rbd | nova | enabled | down >> | 2021-04-29T09:48:42.000000 | >> >>>>>>>>> >> >>>>>>>>> And stopped since 2021-04-29 ! >> >>>>>>>>> >> >>>>>>>>> I have checked Ceph (monitors,managers, osds. etc) and there >> are no problems with the Ceph BackEnd, everything is apparently working. >> >>>>>>>>> >> >>>>>>>>> This happened after an uncontrolled outage.So my question is >> how do I restart only cinder-volumes (I also have cinder-backup, >> cinder-scheduler but they are ok). >> >>>>>>>>> >> >>>>>>>>> Thank you very much in advance. Regards. >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>> >> >> >> >> >> > >> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ignaziocassano at gmail.com Sat May 15 17:41:24 2021 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Sat, 15 May 2021 19:41:24 +0200 Subject: [stein][neutron] l3 agent error Message-ID: Hello Guys, I've just upgraded from openstack queens to rocky and then to stein on centos 7. In my configuration I have router high availability. After the upgrade and rebooting each controller one by one I get the following errors on all my 3 controllers under /var/log/neutron/l3-agent.log http://paste.openstack.org/show/805407/ If I run openstack router show for for one of uuid in the log: http://paste.openstack.org/show/805408/ Namespace for router in present on all 3 controllers. After the controllers reboot,some routers lost their routing tables, but restarting l3-agent they went ok. Is it possible router ha stopped working? Any idea,please ? Ignazio -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke.camilleri at zylacomputing.com Sun May 16 19:36:13 2021 From: luke.camilleri at zylacomputing.com (Luke Camilleri) Date: Sun, 16 May 2021 21:36:13 +0200 Subject: [Octavia][Victoria] No service listening on port 9443 in the amphora instance In-Reply-To: References: <038c74c9-1365-0c08-3b5b-93b4d175dcb3@zylacomputing.com> <326471ef-287b-d937-a174-0b1ccbbd6273@zylacomputing.com>

<8ce76cbd-ca2c-033a-5406-5a1557d84302@zylacomputing.com>

Message-ID: <7fe049df-098f-1378-d65e-d52817da729d@zylacomputing.com> HI Michael thanks as always for your input, below please find my replies: 1- I do not know why is this status being show as DOWN, the deployment for the entire cloud platform is a manual one (or bare metal installation as sometimes I have seen this being called). There must be a script somewhere that is checking the status of this port. Although I have no apparent issue from this I would like to see how I can get the status of this port in an ACTIVE state so any pointers would be helpful. 2- Agreed and that is in fact how it is setup, thanks. 3- Below please find commands run from amphora (192.168.1.11 is an ubuntu web server configured as a member server): # export HAPROXY_SERVER_ADDR=192.168.1.11 # ip netns exec amphora-haproxy /var/lib/octavia/ping-wrapper.sh # ip netns exec amphora-haproxy echo $? 0 # ip netns exec amphora-haproxy /usr/sbin/ping -q -n -w 1 -c 1 $HAPROXY_SERVER_ADDR PING 192.168.1.11 (192.168.1.11) 56(84) bytes of data. --- 192.168.1.11 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.183/0.183/0.183/0.000 ms And yet I still receive the below: (openstack) loadbalancer healthmonitor list | 34e1e2cf-826d-41b4-98b7-7866b680eea6 | hm3-ping | 35a0fa65de1741619709485c5f6d989b | PING | True (openstack) loadbalancer member list pool3 +--------------------------------------+----------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ | id | name | project_id | provisioning_status | address | protocol_port | operating_status | weight | +--------------------------------------+----------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ | 2ce664e2-c84c-4e71-a903-d5f650b0f0e7 | ubuntu-2 | 35a0fa65de1741619709485c5f6d989b | ACTIVE | 192.168.1.11 | 80 | ERROR | 1 | | 376070b6-d290-455f-a718-aa957864e456 | ubuntu-1 | 35a0fa65de1741619709485c5f6d989b | ACTIVE | 192.168.1.235 | 80 | ERROR | 1 | +--------------------------------------+----------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ Ping is no rocket science and the amphora interface does not even need any routing to reach the member since they are on the same layer-2 network. Below you can also see the MAC addresses of both members from the amphora: # ip netns exec amphora-haproxy arp -n Address HWtype HWaddress Flags Mask Iface 192.168.1.235 ether fa:16:3e:70:70:8e C eth1 192.168.1.11 ether fa:16:3e:cc:03:05 C eth1 Deleting the health-monitor of type ping and adding an HTTP healthcheck on "/" works immediately (which clearly shows reachability to the member nodes). I agree with you 100% that a ping health-check in this day and age is something that one should not even consider but I want to make sure that I am not excluding a bigger issue here.... Thanks in advance On 14/05/2021 20:12, Michael Johnson wrote: > Hi Luke, > > 1. The "octavia-health-manager listent-port"(s) should be "ACTIVE" in > neutron. Something may have gone wrong in the deployment tooling or in > neutron for those ports. > 2. As for the VIP ports on the amphora, the base port should be > "ACTIVE", but the VIP port we use to store the VIP IP address should > be "DOWN". The "ACTIVE" base ports will list the VIP IP as it's > "allowed-address-pairs" port/ip. > 3. On the issue with the health monitor of type PING, it's rarely used > outside of some of our tests as it's a poor gauge of the health of an > endpoint (https://docs.openstack.org/octavia/latest/user/guides/basic-cookbook.html#other-health-monitors). > That said, it should be working. It's an external test in the amphora > provider that runs "ping/ping6", so it's interesting that you can ping > from the netns successfully. > Inside the amphora network namespace, can you try running the ping > script directly? > export HAPROXY_SERVER_ADDR= > /var/lib/octavia/ping-wrapper.sh > echo $? > An answer of 0 means the ping was successful, 1 a failure. > > Michael > > On Fri, May 14, 2021 at 8:15 AM Luke Camilleri > wrote: >> Hi Michael, thanks as always for the below, I have watched the video and configured all the requirements as shown in those guides. Working great right now. >> >> I have noticed the following points and would like to know if you can give me some feedback please: >> >> In the Octavia project at the networks screen --> lb-mgmt-net --> Ports --> octavia-health-manager-listen-port (the IP bound to the health-manager service) has its status Down. It does not create any sort of issue but was wondering if this was normal behavior? >> Similarly to the above point, in the tenant networks screen, every port that is "Attached Device - Octavia" has its status reported as "Down" ( these are the VIP addresses assigned to the amphora). Just need to confirm that this is normal behaviour >> Creating a health monitor of type ping fails to get the operating status of the nodes and the nodes are in error (horizon) and the amphora reports that there are no backends and hence it is not working (I am using the same backend nodes with another loadbalancer but with an HTTP check and it is working fine. a security group is setup to allow ping from 0.0.0.0/0 and from the amphora-haproxy network namespace on the amphora instance I can ping both nodes without issues ). Below the amphora's haproxy.log >> >> May 14 15:00:50 amphora-9658d9ec-3bf1-407f-a134-86304899c015 haproxy[1984]: Server c0092bf4-d2a2-431f-8b7f-9dc3ace52933:e268db93-2d20-4395-bd6f-f6d835bce769/f04824b7-6fdf-46dc-bc83-b98b3b9f5be0 is DOWN, reason: Socket error, info: "Resource temporarily unavailable", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. >> >> May 14 15:00:50 amphora-9658d9ec-3bf1-407f-a134-86304899c015 haproxy[1984]: Server c0092bf4-d2a2-431f-8b7f-9dc3ace52933:e268db93-2d20-4395-bd6f-f6d835bce769/f04824b7-6fdf-46dc-bc83-b98b3b9f5be0 is DOWN, reason: Socket error, info: "Resource temporarily unavailable", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. >> >> May 14 15:00:50 amphora-9658d9ec-3bf1-407f-a134-86304899c015 haproxy[1984]: backend c0092bf4-d2a2-431f-8b7f-9dc3ace52933:e268db93-2d20-4395-bd6f-f6d835bce769 has no server available! >> >> May 14 15:00:50 amphora-9658d9ec-3bf1-407f-a134-86304899c015 haproxy[1984]: backend c0092bf4-d2a2-431f-8b7f-9dc3ace52933:e268db93-2d20-4395-bd6f-f6d835bce769 has no server available! >> >> Thanks in advance >> >> On 13/05/2021 18:33, Michael Johnson wrote: >> >> You are correct that two IPs are being allocated for the VIP, one is a >> secondary IP which neutron implements as an "allowed address pairs" >> port. We do this to allow failover of the amphora instance should nova >> fail the service VM. We hold the VIP IP in a special port so the IP is >> not lost while we rebuild the service VM. >> If you are using the active/standby topology (or an Octavia flavor >> with active/standby enabled), this failover is accelerated with nearly >> no visible impact to the flows through the load balancer. >> Active/Standby has been an Octavia feature since the Mitaka release. I >> gave a demo of it at the Tokoyo summit here: >> https://youtu.be/8n7FGhtOiXk?t=1420 >> >> You can enable active/standby as the default by setting the >> "loadbalancer_topology" setting in the configuration file >> (https://docs.openstack.org/octavia/latest/configuration/configref.html#controller_worker.loadbalancer_topology) >> or by creating an Octavia flavor that creates the load balancer with >> an active/standby topology >> (https://docs.openstack.org/octavia/latest/admin/flavors.html). >> >> Michael >> >> On Thu, May 13, 2021 at 4:23 AM Luke Camilleri >> wrote: >> >> HI Michael, thanks a lot for the below information it is very helpful. I >> ended up setting the o-hm0 interface statically in the >> octavia-interface.sh script which is called by the service and also >> added a delay to make sure that the bridges are up before trying to >> create a veth pair and connect the endpoints. >> >> Also I edited the unit section of the health-manager service and at the >> after option I added octavia-interface.service or else on startup the >> health manager will not bind to the lb-mgmt-net since it would not be up yet >> >> The floating IPs part was a bit tricky until I understood what was >> really going on with the VIP concept and how better and more flexible it >> is to set the VIP on the tenant network and then associate with public >> ip to the VIP. >> >> With this being said I noticed that 2 IPs are being assigned to the >> amphora instance and that the actual port assigned to the instance has >> an allowed pair with the VIP port. I checked online and it seems that >> there is an active/standby project going on with VRRP/keepalived and in >> fact the keepalived daemon is running in the amphora instance. >> >> Am I on the right track with the active/standby feature and if so do you >> have any installation/project links to share please so that I can test it? >> >> Regards >> >> On 12/05/2021 08:37, Michael Johnson wrote: >> >> Answers inline below. >> >> Michael >> >> On Mon, May 10, 2021 at 5:15 PM Luke Camilleri >> wrote: >> >> Hi Michael and thanks a lot for the detailed answer below. >> >> I believe I have got most of this sorted out apart from some small issues below: >> >> If the o-hm0 interface gets the IP information from the DHCP server setup by neutron for the lb-mgmt-net, then the management node will always have 2 default gateways and this will bring along issues, the same DHCP settings when deployed to the amphora do not have the same issue since the amphora only has 1 IP assigned on the lb-mgmt-net. Can you please confirm this? >> >> The amphorae do not have issues with DHCP and gateways as we control >> the DHCP client configuration inside the amphora. It does only have >> one IP on the lb-mgmt-net, it will honor gateways provided by neutron >> for the lb-mgmt-net traffic, but a gateway is not required on the >> lb-mgmt-network unless you are routing the lb-mgmt-net traffic across >> subnets. >> >> How does the amphora know where to locate the worker and housekeeping processes or does the traffic originate from the services instead? Maybe the addresses are "injected" from the config file? >> >> The worker and housekeeping processes only create connections to the >> amphora, they do not receive connections from them. The amphora send a >> heartbeat packet to the health manager endpoints every ten seconds by >> default. The list of valid health manager endpoints is included in the >> amphora agent configuration file that is injected into the service VM >> at boot time. It can be updated using the Octavia admin API for >> refreshing the amphora agent configuration. >> >> Can you please confirm if the same floating IP concept runs from public (external) IP to the private (tenant) and from private to lb-mgmt-net please? >> >> Octavia does not use floating IPs. Users can create and assign >> floating IPs via neutron if they would like, but they are not >> necessary. Octavia VIPs can be created directly on neutron "external" >> networks, avoiding the NAT overhead of floating IPs. >> There is no practical reason to assign a floating IP to a port on the >> lb-mgmt-net as tenant traffic is never on or accessible from that >> network. >> >> Thanks in advance for any feedback >> >> On 06/05/2021 22:46, Michael Johnson wrote: >> >> Hi Luke, >> >> 1. I agree that DHCP is technically unnecessary for the o-hm0 >> interface if you can manage your address allocation on the network you >> are using for the lb-mgmt-net. >> I don't have detailed information about the Ubuntu install >> instructions, but I suspect it was done to simplify the IPAM to be >> managed by whatever is providing DHCP on the lb-mgmt-net provided (be >> it neutron or some other resource on a provider network). >> The lb-mgmt-net is simply a neutron network that the amphora >> management address is on. It is routable and does not require external >> access. The only tricky part to it is the worker, health manager, and >> housekeeping processes need to be reachable from the amphora, and the >> controllers need to reach the amphora over the network(s). There are >> many ways to accomplish this. >> >> 2. See my above answer. Fundamentally the lb-mgmt-net is just a >> neutron network that nova can use to attach an interface to the >> amphora instances for command and control traffic. As long as the >> controllers can reach TCP 9433 on the amphora, and the amphora can >> send UDP 5555 back to the health manager endpoints, it will work fine. >> >> 3. Octavia, with the amphora driver, does not require any special >> configuration in Neutron (beyond the advanced services RBAC policy >> being available for the neutron service account used in your octavia >> configuration file). The neutron_lbaas.conf and services_lbaas.conf >> are legacy configuration files/settings that were used for >> neutron-lbaas which is now end of life. See the wiki page for >> information on the deprecation of neutron-lbaas: >> https://wiki.openstack.org/wiki/Neutron/LBaaS/Deprecation. >> >> Michael >> >> On Thu, May 6, 2021 at 12:30 PM Luke Camilleri >> wrote: >> >> Hi Michael and thanks a lot for your help on this, after following your >> steps the agent got deployed successfully in the amphora-image. >> >> I have some other queries that I would like to ask mainly related to the >> health-manager/load-balancer network setup and IP assignment. First of >> all let me point out that I am using a manual installation process, and >> it might help others to understand the underlying infrastructure >> required to make this component work as expected. >> >> 1- The installation procedure contains this step: >> >> $ sudo cp octavia/etc/dhcp/dhclient.conf /etc/dhcp/octavia >> >> which is later on called to assign the IP to the o-hm0 interface which >> is connected to the lb-management network as shown below: >> >> $ sudo dhclient -v o-hm0 -cf /etc/dhcp/octavia >> >> Apart from having a dhcp config for a single IP seems a bit of an >> overkill, using these steps is injecting an additional routing table >> into the default namespace as shown below in my case: >> >> # route -n >> Kernel IP routing table >> Destination Gateway Genmask Flags Metric Ref Use >> Iface >> 0.0.0.0 172.16.0.1 0.0.0.0 UG 0 0 0 o-hm0 >> 0.0.0.0 10.X.X.1 0.0.0.0 UG 100 0 0 ensX >> 10.X.X.0 0.0.0.0 255.255.255.0 U 100 0 0 ensX >> 169.254.169.254 172.16.0.100 255.255.255.255 UGH 0 0 0 o-hm0 >> 172.16.0.0 0.0.0.0 255.240.0.0 U 0 0 0 o-hm0 >> >> Since the load-balancer management network does not need any external >> connectivity (but only communication between health-manager service and >> amphora-agent), why is a gateway required and why isn't the IP address >> allocated as part of the interface creation script which is called when >> the service is started or stopped (example below)? >> >> --- >> >> #!/bin/bash >> >> set -ex >> >> MAC=$MGMT_PORT_MAC >> BRNAME=$BRNAME >> >> if [ "$1" == "start" ]; then >> ip link add o-hm0 type veth peer name o-bhm0 >> brctl addif $BRNAME o-bhm0 >> ip link set o-bhm0 up >> ip link set dev o-hm0 address $MAC >> *** ip addr add 172.16.0.2/12 dev o-hm0 >> ***ip link set o-hm0 mtu 1500 >> ip link set o-hm0 up >> iptables -I INPUT -i o-hm0 -p udp --dport 5555 -j ACCEPT >> elif [ "$1" == "stop" ]; then >> ip link del o-hm0 >> else >> brctl show $BRNAME >> ip a s dev o-hm0 >> fi >> >> --- >> >> 2- Is there a possibility to specify a fixed vlan outside of tenant >> range for the load balancer management network? >> >> 3- Are the configuration changes required only in neutron.conf or also >> in additional config files like neutron_lbaas.conf and >> services_lbaas.conf, similar to the vpnaas configuration? >> >> Thanks in advance for any assistance, but its like putting together a >> puzzle of information :-) >> >> On 05/05/2021 20:25, Michael Johnson wrote: >> >> Hi Luke. >> >> Yes, the amphora-agent will listen on 9443 in the amphorae instances. >> It uses TLS mutual authentication, so you can get a TLS response, but >> it will not let you into the API without a valid certificate. A simple >> "openssl s_client" is usually enough to prove that it is listening and >> requesting the client certificate. >> >> I can't talk to the "openstack-octavia-diskimage-create" package you >> found in centos, but I can discuss how to build an amphora image using >> the OpenStack tools. >> >> If you get Octavia from git or via a release tarball, we provide a >> script to build the amphora image. This is how we build our images for >> the testing gates, etc. and is the recommended way (at least from the >> OpenStack Octavia community) to create amphora images. >> >> https://opendev.org/openstack/octavia/src/branch/master/diskimage-create >> >> For CentOS 8, the command would be: >> >> diskimage-create.sh -g stable/victoria -i centos-minimal -d 8 -s 3 (3 >> is the minimum disk size for centos images, you may want more if you >> are not offloading logs) >> >> I just did a run on a fresh centos 8 instance: >> git clone https://opendev.org/openstack/octavia >> python3 -m venv dib >> source dib/bin/activate >> pip3 install diskimage-builder PyYAML six >> sudo dnf install yum-utils >> ./diskimage-create.sh -g stable/victoria -i centos-minimal -d 8 -s 3 >> >> This built an image. >> >> Off and on we have had issues building CentOS images due to issues in >> the tools we rely on. If you run into issues with this image, drop us >> a note back. >> >> Michael >> >> On Wed, May 5, 2021 at 9:37 AM Luke Camilleri >> wrote: >> >> Hi there, i am trying to get Octavia running on a Victoria deployment on >> CentOS 8. It was a bit rough getting to the point to launch an instance >> mainly due to the load-balancer management network and the lack of >> documentation >> (https://docs.openstack.org/octavia/victoria/install/install.html) to >> deploy this oN CentOS. I will try to fix this once I have my deployment >> up and running to help others on the way installing and configuring this :-) >> >> At this point a LB can be launched by the tenant and the instance is >> spawned in the Octavia project and I can ping and SSH into the amphora >> instance from the Octavia node where the octavia-health-manager service >> is running using the IP within the same subnet of the amphoras >> (172.16.0.0/12). >> >> Unfortunately I keep on getting these errors in the log file of the >> worker log (/var/log/octavia/worker.log): >> >> 2021-05-05 01:54:49.368 14521 WARNING >> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect >> to instance. Retrying.: requests.exceptions.ConnectionError: >> HTTPSConnectionPool(host='172.16.4.46', p >> ort=9443): Max retries exceeded with url: // (Caused by >> NewConnectionError('> at 0x7f83e0181550>: Failed to establish a new connection: [Errno 111] >> Connection ref >> used',)) >> >> 2021-05-05 01:54:54.374 14521 ERROR >> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Connection retries >> (currently set to 120) exhausted. The amphora is unavailable. Reason: >> HTTPSConnectionPool(host='172.16 >> .4.46', port=9443): Max retries exceeded with url: // (Caused by >> NewConnectionError('> at 0x7f83e0181550>: Failed to establish a new connection: [Errno 111] Conne >> ction refused',)) >> >> 2021-05-05 01:54:54.374 14521 ERROR >> octavia.controller.worker.v1.tasks.amphora_driver_tasks [-] Amphora >> compute instance failed to become reachable. This either means the >> compute driver failed to fully boot the >> instance inside the timeout interval or the instance is not reachable >> via the lb-mgmt-net.: >> octavia.amphorae.driver_exceptions.exceptions.TimeOutException: >> contacting the amphora timed out >> >> obviously the instance is deleted then and the task fails from the >> tenant's perspective. >> >> The main issue here is that there is no service running on port 9443 on >> the amphora instance. I am assuming that this is in fact the >> amphora-agent service that is running on the instance which should be >> listening on this port 9443 but the service does not seem to be up or >> not installed at all. >> >> To create the image I have installed the CentOS package >> "openstack-octavia-diskimage-create" which provides the utility >> disk-image-create but from what I can conclude the amphora-agent is not >> being installed (thought this was done automatically by default :-( ) >> >> Can anyone let me know if the amphora-agent is what gets queried on port >> 9443 ? >> >> If the agent is not installed/injected by default when building the >> amphora image? >> >> The command to inject the amphora-agent into the amphora image when >> using the disk-image-create command? >> >> Thanks in advance for any assistance >> >> From ignaziocassano at gmail.com Sat May 15 17:13:27 2021 From: ignaziocassano at gmail.com (Ignazio Cassano) Date: Sat, 15 May 2021 19:13:27 +0200 Subject: [neutron][stein] l3 agent errors Message-ID: Hello Guys, I've just upgraded from openstack queens to rocky and then to stein on centos 7. In my configuration I have router high availability. After the upgrade I get the following errors on all my 3 controllers under /var/log/neutron/l3-agent.log 2021-05-15 19:04:28.599 13219 ERROR neutron.agent.linux.external_process [-] ip_monitor for router with uuid 8c137f17-52e6-47f9-bcaa-0eac94a9acd7 not found. The process should not have died 2021-05-15 19:04:28.600 13219 WARNING neutron.agent.linux.external_process [-] Respawning ip_monitor for uuid 8c137f17-52e6-47f9-bcaa-0eac94a9acd7 2021-05-15 19:04:29.905 13219 ERROR neutron.agent.linux.external_process [-] ip_monitor for router with uuid 303c88e4-1c94-4ebc-b592-91063b371db9 not found. The process should not have died 2021-05-15 19:04:29.905 13219 WARNING neutron.agent.linux.external_process [-] Respawning ip_monitor for uuid 303c88e4-1c94-4ebc-b592-91063b371db9 2021-05-15 19:04:31.267 13219 ERROR neutron.agent.linux.external_process [-] ip_monitor for router with uuid 103f3588-a775-4bca-b7a5-093a05068f64 not f

Extra Specs¶

Nova Cells¶