[kolla] neutron-l3-agent namespace NAT table not working?
Hi there, I've got a weird problem with the neutron-l3-agent container on my deployment. It comes up, sets up the iptables rules in the qrouter namespace (and I can see these using "ip netns...") but traffic isn't having DNAT or SNAT applied. What's most strange is that manually adding a LOG jump target to the iptables nat PRE/POSTROUTING chains (after enabling nf logging sent to the host kernel, confirmed that works) doesn't result in any log entries. It's as if the nat table isn't being applied at all for any packets traversing the qrouter namespace. This is driving me crazy :) Anyone got some quick suggestions? (assume I tried the obvious stuff). Jon. -- Computer Architect
Hi, Is this qrouter namespace created with all those rules in container or in the host directly? Do You have qr-xxx and qg-xxx ports from br-int in this qrouter namespace?
On 4 Jan 2020, at 05:44, Jon Masters <jcm@jonmasters.org> wrote:
Hi there,
I've got a weird problem with the neutron-l3-agent container on my deployment. It comes up, sets up the iptables rules in the qrouter namespace (and I can see these using "ip netns...") but traffic isn't having DNAT or SNAT applied. What's most strange is that manually adding a LOG jump target to the iptables nat PRE/POSTROUTING chains (after enabling nf logging sent to the host kernel, confirmed that works) doesn't result in any log entries. It's as if the nat table isn't being applied at all for any packets traversing the qrouter namespace. This is driving me crazy :)
Anyone got some quick suggestions? (assume I tried the obvious stuff).
Jon.
-- Computer Architect
— Slawek Kaplonski Senior software engineer Red Hat
On Sat, 2020-01-04 at 10:46 +0100, Slawek Kaplonski wrote:
Hi,
Is this qrouter namespace created with all those rules in container or in the host directly? Do You have qr-xxx and qg-xxx ports from br-int in this qrouter namespace? in kolla the l3 agent should be running with net=host so the container should be useing the hosts root namespace and it will create network namespaces as needed for the different routers.
the ip table rules should be in the router sub namespaces.
On 4 Jan 2020, at 05:44, Jon Masters <jcm@jonmasters.org> wrote:
Hi there,
I've got a weird problem with the neutron-l3-agent container on my deployment. It comes up, sets up the iptables rules in the qrouter namespace (and I can see these using "ip netns...") but traffic isn't having DNAT or SNAT applied. What's most strange is that manually adding a LOG jump target to the iptables nat PRE/POSTROUTING chains (after enabling nf logging sent to the host kernel, confirmed that works) doesn't result in any log entries. It's as if the nat table isn't being applied at all for any packets traversing the qrouter namespace. This is driving me crazy :)
Anyone got some quick suggestions? (assume I tried the obvious stuff).
Jon.
-- Computer Architect
— Slawek Kaplonski Senior software engineer Red Hat
Excuse top posting on my phone. Also, yes, the namespaces are as described. It’s just that the (correct) nat rules for the qrouter netns are never running, in spite of the two interfaces existing in that ns and correctly attached to the vswitch. -- Computer Architect
On Jan 4, 2020, at 07:56, Sean Mooney <smooney@redhat.com> wrote:
On Sat, 2020-01-04 at 10:46 +0100, Slawek Kaplonski wrote:
Hi,
Is this qrouter namespace created with all those rules in container or in the host directly? Do You have qr-xxx and qg-xxx ports from br-int in this qrouter namespace? in kolla the l3 agent should be running with net=host so the container should be useing the hosts root namespace and it will create network namespaces as needed for the different routers.
the ip table rules should be in the router sub namespaces.
On 4 Jan 2020, at 05:44, Jon Masters <jcm@jonmasters.org> wrote:
Hi there,
I've got a weird problem with the neutron-l3-agent container on my deployment. It comes up, sets up the iptables rules in the qrouter namespace (and I can see these using "ip netns...") but traffic isn't having DNAT or SNAT applied. What's most strange is that manually adding a LOG jump target to the iptables nat PRE/POSTROUTING chains (after enabling nf logging sent to the host kernel, confirmed that works) doesn't result in any log entries. It's as if the nat table isn't being applied at all for any packets traversing the qrouter namespace. This is driving me crazy :)
Anyone got some quick suggestions? (assume I tried the obvious stuff).
Jon.
-- Computer Architect
— Slawek Kaplonski Senior software engineer Red Hat
This turns out to a not well documented bug in the CentOS7.7 kernel that causes exactly nat rules not to run as I was seeing. Oh dear god was this nasty as whatever to find and workaround. -- Computer Architect
On Jan 4, 2020, at 10:39, Jon Masters <jcm@jonmasters.org> wrote:
Excuse top posting on my phone. Also, yes, the namespaces are as described. It’s just that the (correct) nat rules for the qrouter netns are never running, in spite of the two interfaces existing in that ns and correctly attached to the vswitch.
-- Computer Architect
On Jan 4, 2020, at 07:56, Sean Mooney <smooney@redhat.com> wrote:
On Sat, 2020-01-04 at 10:46 +0100, Slawek Kaplonski wrote: Hi,
Is this qrouter namespace created with all those rules in container or in the host directly? Do You have qr-xxx and qg-xxx ports from br-int in this qrouter namespace? in kolla the l3 agent should be running with net=host so the container should be useing the hosts root namespace and it will create network namespaces as needed for the different routers.
the ip table rules should be in the router sub namespaces.
On 4 Jan 2020, at 05:44, Jon Masters <jcm@jonmasters.org> wrote:
Hi there,
I've got a weird problem with the neutron-l3-agent container on my deployment. It comes up, sets up the iptables rules in the qrouter namespace (and I can see these using "ip netns...") but traffic isn't having DNAT or SNAT applied. What's most strange is that manually adding a LOG jump target to the iptables nat PRE/POSTROUTING chains (after enabling nf logging sent to the host kernel, confirmed that works) doesn't result in any log entries. It's as if the nat table isn't being applied at all for any packets traversing the qrouter namespace. This is driving me crazy :)
Anyone got some quick suggestions? (assume I tried the obvious stuff).
Jon.
-- Computer Architect
— Slawek Kaplonski Senior software engineer Red Hat
Do you happen to have the bug ID for Centos? On Sun, Jan 5, 2020 at 2:11 PM Jon Masters <jcm@jonmasters.org> wrote:
This turns out to a not well documented bug in the CentOS7.7 kernel that causes exactly nat rules not to run as I was seeing. Oh dear god was this nasty as whatever to find and workaround.
-- Computer Architect
On Jan 4, 2020, at 10:39, Jon Masters <jcm@jonmasters.org> wrote:
Excuse top posting on my phone. Also, yes, the namespaces are as described. It’s just that the (correct) nat rules for the qrouter netns are never running, in spite of the two interfaces existing in that ns and correctly attached to the vswitch.
-- Computer Architect
On Jan 4, 2020, at 07:56, Sean Mooney <smooney@redhat.com> wrote:
On Sat, 2020-01-04 at 10:46 +0100, Slawek Kaplonski wrote: Hi,
Is this qrouter namespace created with all those rules in container or in the host directly? Do You have qr-xxx and qg-xxx ports from br-int in this qrouter namespace? in kolla the l3 agent should be running with net=host so the container should be useing the hosts root namespace and it will create network namespaces as needed for the different routers.
the ip table rules should be in the router sub namespaces.
On 4 Jan 2020, at 05:44, Jon Masters <jcm@jonmasters.org> wrote:
Hi there,
I've got a weird problem with the neutron-l3-agent container on my
rules in the qrouter namespace (and I can see these using "ip netns...") but traffic isn't having DNAT or SNAT applied. What's most strange is that manually adding a LOG jump target to the iptables nat PRE/POSTROUTING chains (after enabling nf logging sent to the host kernel, confirmed that works) doesn't result in any log entries. It's as if the nat table isn't being applied at all for any packets
deployment. It comes up, sets up the iptables traversing the qrouter namespace. This is driving me
crazy :)
Anyone got some quick suggestions? (assume I tried the obvious stuff).
Jon.
-- Computer Architect
— Slawek Kaplonski Senior software engineer Red Hat
Could you send us more details about your deployment - e.g. kolla version and image info? And please try to check neutron-openvswitch-agent log - errors regarding applying iptables rules should be there. I've encountered similar behavior when trying to run a nftables OS image (Debian 10) on iptables OS image (Ubuntu 16.04). You can try it by running sudo update-alternatives --query iptables If it's the case, option to force legacy iptables has been added - https://review.opendev.org/#/c/685967/. Best regards, Jan Dne po 6. 1. 2020 0:56 uživatel Laurent Dumont <laurentfdumont@gmail.com> napsal:
Do you happen to have the bug ID for Centos?
On Sun, Jan 5, 2020 at 2:11 PM Jon Masters <jcm@jonmasters.org> wrote:
This turns out to a not well documented bug in the CentOS7.7 kernel that causes exactly nat rules not to run as I was seeing. Oh dear god was this nasty as whatever to find and workaround.
-- Computer Architect
On Jan 4, 2020, at 10:39, Jon Masters <jcm@jonmasters.org> wrote:
Excuse top posting on my phone. Also, yes, the namespaces are as described. It’s just that the (correct) nat rules for the qrouter netns are never running, in spite of the two interfaces existing in that ns and correctly attached to the vswitch.
-- Computer Architect
On Jan 4, 2020, at 07:56, Sean Mooney <smooney@redhat.com> wrote:
On Sat, 2020-01-04 at 10:46 +0100, Slawek Kaplonski wrote: Hi,
Is this qrouter namespace created with all those rules in container or in the host directly? Do You have qr-xxx and qg-xxx ports from br-int in this qrouter namespace? in kolla the l3 agent should be running with net=host so the container should be useing the hosts root namespace and it will create network namespaces as needed for the different routers.
the ip table rules should be in the router sub namespaces.
> On 4 Jan 2020, at 05:44, Jon Masters <jcm@jonmasters.org> wrote:
Hi there,
I've got a weird problem with the neutron-l3-agent container on my
rules in the qrouter namespace (and I can see these using "ip netns...") but traffic isn't having DNAT or SNAT applied. What's most strange is that manually adding a LOG jump target to the iptables nat PRE/POSTROUTING chains (after enabling nf logging sent to the host kernel, confirmed that works) doesn't result in any log entries. It's as if the nat table isn't being applied at all for any packets
deployment. It comes up, sets up the iptables traversing the qrouter namespace. This is driving me
crazy :)
Anyone got some quick suggestions? (assume I tried the obvious stuff).
Jon.
-- Computer Architect
— Slawek Kaplonski Senior software engineer Red Hat
There’s no bug ID that I’m aware of. But I’ll go look for one or file one. -- Computer Architect
On Jan 5, 2020, at 18:51, Laurent Dumont <laurentfdumont@gmail.com> wrote:
Do you happen to have the bug ID for Centos?
On Sun, Jan 5, 2020 at 2:11 PM Jon Masters <jcm@jonmasters.org> wrote: This turns out to a not well documented bug in the CentOS7.7 kernel that causes exactly nat rules not to run as I was seeing. Oh dear god was this nasty as whatever to find and workaround.
-- Computer Architect
On Jan 4, 2020, at 10:39, Jon Masters <jcm@jonmasters.org> wrote:
Excuse top posting on my phone. Also, yes, the namespaces are as described. It’s just that the (correct) nat rules for the qrouter netns are never running, in spite of the two interfaces existing in that ns and correctly attached to the vswitch.
-- Computer Architect
On Jan 4, 2020, at 07:56, Sean Mooney <smooney@redhat.com> wrote:
On Sat, 2020-01-04 at 10:46 +0100, Slawek Kaplonski wrote: Hi,
Is this qrouter namespace created with all those rules in container or in the host directly? Do You have qr-xxx and qg-xxx ports from br-int in this qrouter namespace? in kolla the l3 agent should be running with net=host so the container should be useing the hosts root namespace and it will create network namespaces as needed for the different routers.
the ip table rules should be in the router sub namespaces.
> On 4 Jan 2020, at 05:44, Jon Masters <jcm@jonmasters.org> wrote:
Hi there,
I've got a weird problem with the neutron-l3-agent container on my deployment. It comes up, sets up the iptables rules in the qrouter namespace (and I can see these using "ip netns...") but traffic isn't having DNAT or SNAT applied. What's most strange is that manually adding a LOG jump target to the iptables nat PRE/POSTROUTING chains (after enabling nf logging sent to the host kernel, confirmed that works) doesn't result in any log entries. It's as if the nat table isn't being applied at all for any packets traversing the qrouter namespace. This is driving me crazy :)
Anyone got some quick suggestions? (assume I tried the obvious stuff).
Jon.
-- Computer Architect
— Slawek Kaplonski Senior software engineer Red Hat
If it's RHEL kernel's bug, then Red Hat would likely want to know about it (if not knowing already). I have my kolla deployment on c7.7 and I don't encounter this issue, though there is a pending kernel update so now I'm worried about applying it... -yoctozepto pon., 6 sty 2020 o 03:34 Jon Masters <jcm@jonmasters.org> napisał(a):
There’s no bug ID that I’m aware of. But I’ll go look for one or file one.
-- Computer Architect
On Jan 5, 2020, at 18:51, Laurent Dumont <laurentfdumont@gmail.com> wrote:
Do you happen to have the bug ID for Centos?
On Sun, Jan 5, 2020 at 2:11 PM Jon Masters <jcm@jonmasters.org> wrote:
This turns out to a not well documented bug in the CentOS7.7 kernel that causes exactly nat rules not to run as I was seeing. Oh dear god was this nasty as whatever to find and workaround.
-- Computer Architect
On Jan 4, 2020, at 10:39, Jon Masters <jcm@jonmasters.org> wrote:
Excuse top posting on my phone. Also, yes, the namespaces are as described. It’s just that the (correct) nat rules for the qrouter netns are never running, in spite of the two interfaces existing in that ns and correctly attached to the vswitch.
-- Computer Architect
On Jan 4, 2020, at 07:56, Sean Mooney <smooney@redhat.com> wrote:
On Sat, 2020-01-04 at 10:46 +0100, Slawek Kaplonski wrote: Hi,
Is this qrouter namespace created with all those rules in container or in the host directly? Do You have qr-xxx and qg-xxx ports from br-int in this qrouter namespace? in kolla the l3 agent should be running with net=host so the container should be useing the hosts root namespace and it will create network namespaces as needed for the different routers.
the ip table rules should be in the router sub namespaces.
> On 4 Jan 2020, at 05:44, Jon Masters <jcm@jonmasters.org> wrote:
Hi there,
I've got a weird problem with the neutron-l3-agent container on my deployment. It comes up, sets up the iptables rules in the qrouter namespace (and I can see these using "ip netns...") but traffic isn't having DNAT or SNAT applied. What's most strange is that manually adding a LOG jump target to the iptables nat PRE/POSTROUTING chains (after enabling nf logging sent to the host kernel, confirmed that works) doesn't result in any log entries. It's as if the nat table isn't being applied at all for any packets traversing the qrouter namespace. This is driving me crazy :)
Anyone got some quick suggestions? (assume I tried the obvious stuff).
Jon.
-- Computer Architect
— Slawek Kaplonski Senior software engineer Red Hat
On Mon, 2020-01-06 at 10:11 +0100, Radosław Piliszek wrote:
If it's RHEL kernel's bug, then Red Hat would likely want to know about it (if not knowing already). I have my kolla deployment on c7.7 and I don't encounter this issue, though there is a pending kernel update so now I'm worried about applying it... it sound more like a confilct between legacy iptables and the new nftables based replacement. if you mix the two then it will appear as if the rules are installed but only some of the rules will run. so the container images and the host need to be both configured to use the same versions.
that said fi you are using centos images on a centos host they should be providing your usnign centos 7 or centos 8 on both. if you try to use centos 7 image on a centos 8 host or centos 8 images on a centos 7 host it would likely have issues due to the fact centos 8 uses a differt iptables implemeantion
-yoctozepto
pon., 6 sty 2020 o 03:34 Jon Masters <jcm@jonmasters.org> napisał(a):
There’s no bug ID that I’m aware of. But I’ll go look for one or file one.
-- Computer Architect
On Jan 5, 2020, at 18:51, Laurent Dumont <laurentfdumont@gmail.com> wrote:
Do you happen to have the bug ID for Centos?
On Sun, Jan 5, 2020 at 2:11 PM Jon Masters <jcm@jonmasters.org> wrote:
This turns out to a not well documented bug in the CentOS7.7 kernel that causes exactly nat rules not to run as I was seeing. Oh dear god was this nasty as whatever to find and workaround.
-- Computer Architect
On Jan 4, 2020, at 10:39, Jon Masters <jcm@jonmasters.org> wrote:
Excuse top posting on my phone. Also, yes, the namespaces are as described. It’s just that the (correct) nat rules for the qrouter netns are never running, in spite of the two interfaces existing in that ns and correctly attached to the vswitch.
-- Computer Architect
On Jan 4, 2020, at 07:56, Sean Mooney <smooney@redhat.com> wrote:
On Sat, 2020-01-04 at 10:46 +0100, Slawek Kaplonski wrote: Hi,
Is this qrouter namespace created with all those rules in container or in the host directly? Do You have qr-xxx and qg-xxx ports from br-int in this qrouter namespace?
in kolla the l3 agent should be running with net=host so the container should be useing the hosts root namespace and it will create network namespaces as needed for the different routers.
the ip table rules should be in the router sub namespaces.
> > On 4 Jan 2020, at 05:44, Jon Masters <jcm@jonmasters.org> wrote: > > Hi there, > > I've got a weird problem with the neutron-l3-agent container on my deployment. It comes up, sets up the > iptables > rules in the qrouter namespace (and I can see these using "ip netns...") but traffic isn't having DNAT or > SNAT > applied. What's most strange is that manually adding a LOG jump target to the iptables nat PRE/POSTROUTING > chains > (after enabling nf logging sent to the host kernel, confirmed that works) doesn't result in any log > entries. It's as > if the nat table isn't being applied at all for any packets traversing the qrouter namespace. This is > driving me > crazy :) > > Anyone got some quick suggestions? (assume I tried the obvious stuff). > > Jon. > > -- > Computer Architect
— Slawek Kaplonski Senior software engineer Red Hat
po 6. 1. 2020 v 12:46 odesílatel Sean Mooney <smooney@redhat.com> napsal:
On Mon, 2020-01-06 at 10:11 +0100, Radosław Piliszek wrote:
If it's RHEL kernel's bug, then Red Hat would likely want to know about it (if not knowing already). I have my kolla deployment on c7.7 and I don't encounter this issue, though there is a pending kernel update so now I'm worried about applying it... it sound more like a confilct between legacy iptables and the new nftables based replacement. if you mix the two then it will appear as if the rules are installed but only some of the rules will run. so the container images and the host need to be both configured to use the same versions.
that said fi you are using centos images on a centos host they should be providing your usnign centos 7 or centos 8 on both. if you try to use centos 7 image on a centos 8 host or centos 8 images on a centos 7 host it would likely have issues due to the fact centos 8 uses a differt iptables implemeantion
As I wrote before this scenario has already been covered in following patches: https://review.opendev.org/#/c/685967/ https://review.opendev.org/#/c/683679/ To force iptables legacy in neutron containers put following line into globals.yml file: neutron_legacy_iptables: "yes" Beware currently there is an issue in applying changes in enviromental variables for already running containers so you may have to manually delete neutron containers and recreate them using reconfigure or - if possible - destroy and redeploy whole deployment. J.V.
I did specifically check for such a conflict tho before proceeding down the path I went :) -- Computer Architect
On Jan 6, 2020, at 03:40, Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2020-01-06 at 10:11 +0100, Radosław Piliszek wrote:
If it's RHEL kernel's bug, then Red Hat would likely want to know about it (if not knowing already). I have my kolla deployment on c7.7 and I don't encounter this issue, though there is a pending kernel update so now I'm worried about applying it... it sound more like a confilct between legacy iptables and the new nftables based replacement. if you mix the two then it will appear as if the rules are installed but only some of the rules will run. so the container images and the host need to be both configured to use the same versions.
that said fi you are using centos images on a centos host they should be providing your usnign centos 7 or centos 8 on both. if you try to use centos 7 image on a centos 8 host or centos 8 images on a centos 7 host it would likely have issues due to the fact centos 8 uses a differt iptables implemeantion
-yoctozepto
pon., 6 sty 2020 o 03:34 Jon Masters <jcm@jonmasters.org> napisał(a):
There’s no bug ID that I’m aware of. But I’ll go look for one or file one.
-- Computer Architect
On Jan 5, 2020, at 18:51, Laurent Dumont <laurentfdumont@gmail.com> wrote:
Do you happen to have the bug ID for Centos?
On Sun, Jan 5, 2020 at 2:11 PM Jon Masters <jcm@jonmasters.org> wrote:
This turns out to a not well documented bug in the CentOS7.7 kernel that causes exactly nat rules not to run as I was seeing. Oh dear god was this nasty as whatever to find and workaround.
-- Computer Architect
On Jan 4, 2020, at 10:39, Jon Masters <jcm@jonmasters.org> wrote:
Excuse top posting on my phone. Also, yes, the namespaces are as described. It’s just that the (correct) nat rules for the qrouter netns are never running, in spite of the two interfaces existing in that ns and correctly attached to the vswitch.
-- Computer Architect
> On Jan 4, 2020, at 07:56, Sean Mooney <smooney@redhat.com> wrote: > > On Sat, 2020-01-04 at 10:46 +0100, Slawek Kaplonski wrote: > Hi, > > Is this qrouter namespace created with all those rules in container or in the host directly? > Do You have qr-xxx and qg-xxx ports from br-int in this qrouter namespace?
in kolla the l3 agent should be running with net=host so the container should be useing the hosts root namespace and it will create network namespaces as needed for the different routers.
the ip table rules should be in the router sub namespaces.
> >>> On 4 Jan 2020, at 05:44, Jon Masters <jcm@jonmasters.org> wrote: >> >> Hi there, >> >> I've got a weird problem with the neutron-l3-agent container on my deployment. It comes up, sets up the >> iptables >> rules in the qrouter namespace (and I can see these using "ip netns...") but traffic isn't having DNAT or >> SNAT >> applied. What's most strange is that manually adding a LOG jump target to the iptables nat PRE/POSTROUTING >> chains >> (after enabling nf logging sent to the host kernel, confirmed that works) doesn't result in any log >> entries. It's as >> if the nat table isn't being applied at all for any packets traversing the qrouter namespace. This is >> driving me >> crazy :) >> >> Anyone got some quick suggestions? (assume I tried the obvious stuff). >> >> Jon. >> >> -- >> Computer Architect > > — > Slawek Kaplonski > Senior software engineer > Red Hat > >
Folks, this seems to be about C7, not C8, and "neutron_legacy_iptables" does not apply here. @Jon - what is the kernel bug you mentioned but never referenced? -yoctozepto pon., 6 sty 2020 o 13:13 Jon Masters <jcm@jonmasters.org> napisał(a):
I did specifically check for such a conflict tho before proceeding down the path I went :)
-- Computer Architect
On Jan 6, 2020, at 03:40, Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2020-01-06 at 10:11 +0100, Radosław Piliszek wrote:
If it's RHEL kernel's bug, then Red Hat would likely want to know about it (if not knowing already). I have my kolla deployment on c7.7 and I don't encounter this issue, though there is a pending kernel update so now I'm worried about applying it... it sound more like a confilct between legacy iptables and the new nftables based replacement. if you mix the two then it will appear as if the rules are installed but only some of the rules will run. so the container images and the host need to be both configured to use the same versions.
that said fi you are using centos images on a centos host they should be providing your usnign centos 7 or centos 8 on both. if you try to use centos 7 image on a centos 8 host or centos 8 images on a centos 7 host it would likely have issues due to the fact centos 8 uses a differt iptables implemeantion
-yoctozepto
pon., 6 sty 2020 o 03:34 Jon Masters <jcm@jonmasters.org> napisał(a):
There’s no bug ID that I’m aware of. But I’ll go look for one or file one.
-- Computer Architect
On Jan 5, 2020, at 18:51, Laurent Dumont <laurentfdumont@gmail.com> wrote:
Do you happen to have the bug ID for Centos?
On Sun, Jan 5, 2020 at 2:11 PM Jon Masters <jcm@jonmasters.org> wrote:
This turns out to a not well documented bug in the CentOS7.7 kernel that causes exactly nat rules not to run as I was seeing. Oh dear god was this nasty as whatever to find and workaround.
-- Computer Architect
On Jan 4, 2020, at 10:39, Jon Masters <jcm@jonmasters.org> wrote:
Excuse top posting on my phone. Also, yes, the namespaces are as described. It’s just that the (correct) nat rules for the qrouter netns are never running, in spite of the two interfaces existing in that ns and correctly attached to the vswitch.
-- Computer Architect
>> On Jan 4, 2020, at 07:56, Sean Mooney <smooney@redhat.com> wrote: >> >> On Sat, 2020-01-04 at 10:46 +0100, Slawek Kaplonski wrote: >> Hi, >> >> Is this qrouter namespace created with all those rules in container or in the host directly? >> Do You have qr-xxx and qg-xxx ports from br-int in this qrouter namespace? > > in kolla the l3 agent should be running with net=host so the container should be useing the hosts > root namespace and it will create network namespaces as needed for the different routers. > > the ip table rules should be in the router sub namespaces. > >> >>>> On 4 Jan 2020, at 05:44, Jon Masters <jcm@jonmasters.org> wrote: >>> >>> Hi there, >>> >>> I've got a weird problem with the neutron-l3-agent container on my deployment. It comes up, sets up the >>> iptables >>> rules in the qrouter namespace (and I can see these using "ip netns...") but traffic isn't having DNAT or >>> SNAT >>> applied. What's most strange is that manually adding a LOG jump target to the iptables nat PRE/POSTROUTING >>> chains >>> (after enabling nf logging sent to the host kernel, confirmed that works) doesn't result in any log >>> entries. It's as >>> if the nat table isn't being applied at all for any packets traversing the qrouter namespace. This is >>> driving me >>> crazy :) >>> >>> Anyone got some quick suggestions? (assume I tried the obvious stuff). >>> >>> Jon. >>> >>> -- >>> Computer Architect >> >> — >> Slawek Kaplonski >> Senior software engineer >> Red Hat >> >>
On 1/6/20 7:33 AM, Radosław Piliszek wrote:
Folks, this seems to be about C7, not C8, and "neutron_legacy_iptables" does not apply here. @Jon - what is the kernel bug you mentioned but never referenced?
There was a previous kernel bug in a Centos kernel that broke DNAT, https://bugs.launchpad.net/neutron/+bug/1776778 but don't know if this is the same issue. I would have hoped no one was using that kernel by now, and/or it was blacklisted. -Brian
pon., 6 sty 2020 o 13:13 Jon Masters <jcm@jonmasters.org> napisał(a):
I did specifically check for such a conflict tho before proceeding down the path I went :)
-- Computer Architect
On Jan 6, 2020, at 03:40, Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2020-01-06 at 10:11 +0100, Radosław Piliszek wrote:
If it's RHEL kernel's bug, then Red Hat would likely want to know about it (if not knowing already). I have my kolla deployment on c7.7 and I don't encounter this issue, though there is a pending kernel update so now I'm worried about applying it... it sound more like a confilct between legacy iptables and the new nftables based replacement. if you mix the two then it will appear as if the rules are installed but only some of the rules will run. so the container images and the host need to be both configured to use the same versions.
that said fi you are using centos images on a centos host they should be providing your usnign centos 7 or centos 8 on both. if you try to use centos 7 image on a centos 8 host or centos 8 images on a centos 7 host it would likely have issues due to the fact centos 8 uses a differt iptables implemeantion
-yoctozepto
pon., 6 sty 2020 o 03:34 Jon Masters <jcm@jonmasters.org> napisał(a):
There’s no bug ID that I’m aware of. But I’ll go look for one or file one.
-- Computer Architect
On Jan 5, 2020, at 18:51, Laurent Dumont <laurentfdumont@gmail.com> wrote:
Do you happen to have the bug ID for Centos?
On Sun, Jan 5, 2020 at 2:11 PM Jon Masters <jcm@jonmasters.org> wrote:
This turns out to a not well documented bug in the CentOS7.7 kernel that causes exactly nat rules not to run as I was seeing. Oh dear god was this nasty as whatever to find and workaround.
-- Computer Architect
> On Jan 4, 2020, at 10:39, Jon Masters <jcm@jonmasters.org> wrote: > > Excuse top posting on my phone. Also, yes, the namespaces are as described. It’s just that the (correct) nat > rules for the qrouter netns are never running, in spite of the two interfaces existing in that ns and correctly > attached to the vswitch. > > -- > Computer Architect > > >>> On Jan 4, 2020, at 07:56, Sean Mooney <smooney@redhat.com> wrote: >>> >>> On Sat, 2020-01-04 at 10:46 +0100, Slawek Kaplonski wrote: >>> Hi, >>> >>> Is this qrouter namespace created with all those rules in container or in the host directly? >>> Do You have qr-xxx and qg-xxx ports from br-int in this qrouter namespace? >> >> in kolla the l3 agent should be running with net=host so the container should be useing the hosts >> root namespace and it will create network namespaces as needed for the different routers. >> >> the ip table rules should be in the router sub namespaces. >> >>> >>>>> On 4 Jan 2020, at 05:44, Jon Masters <jcm@jonmasters.org> wrote: >>>> >>>> Hi there, >>>> >>>> I've got a weird problem with the neutron-l3-agent container on my deployment. It comes up, sets up the >>>> iptables >>>> rules in the qrouter namespace (and I can see these using "ip netns...") but traffic isn't having DNAT or >>>> SNAT >>>> applied. What's most strange is that manually adding a LOG jump target to the iptables nat PRE/POSTROUTING >>>> chains >>>> (after enabling nf logging sent to the host kernel, confirmed that works) doesn't result in any log >>>> entries. It's as >>>> if the nat table isn't being applied at all for any packets traversing the qrouter namespace. This is >>>> driving me >>>> crazy :) >>>> >>>> Anyone got some quick suggestions? (assume I tried the obvious stuff). >>>> >>>> Jon. >>>> >>>> -- >>>> Computer Architect >>> >>> — >>> Slawek Kaplonski >>> Senior software engineer >>> Red Hat >>> >>>
Hi,
On 6 Jan 2020, at 16:15, Brian Haley <haleyb.dev@gmail.com> wrote:
On 1/6/20 7:33 AM, Radosław Piliszek wrote:
Folks, this seems to be about C7, not C8, and "neutron_legacy_iptables" does not apply here. @Jon - what is the kernel bug you mentioned but never referenced?
There was a previous kernel bug in a Centos kernel that broke DNAT, https://bugs.launchpad.net/neutron/+bug/1776778 but don't know if this is the same issue. I would have hoped no one was using that kernel by now, and/or it was blacklisted.
This one also came to my mind when I read about kernel bug here. But this old bug was affecting only DNAT on dvr routers IIRC so IMO it doesn’t seems like same issue.
-Brian
pon., 6 sty 2020 o 13:13 Jon Masters <jcm@jonmasters.org> napisał(a):
I did specifically check for such a conflict tho before proceeding down the path I went :)
-- Computer Architect
On Jan 6, 2020, at 03:40, Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2020-01-06 at 10:11 +0100, Radosław Piliszek wrote:
If it's RHEL kernel's bug, then Red Hat would likely want to know about it (if not knowing already). I have my kolla deployment on c7.7 and I don't encounter this issue, though there is a pending kernel update so now I'm worried about applying it... it sound more like a confilct between legacy iptables and the new nftables based replacement. if you mix the two then it will appear as if the rules are installed but only some of the rules will run. so the container images and the host need to be both configured to use the same versions.
that said fi you are using centos images on a centos host they should be providing your usnign centos 7 or centos 8 on both. if you try to use centos 7 image on a centos 8 host or centos 8 images on a centos 7 host it would likely have issues due to the fact centos 8 uses a differt iptables implemeantion
-yoctozepto
pon., 6 sty 2020 o 03:34 Jon Masters <jcm@jonmasters.org> napisał(a):
There’s no bug ID that I’m aware of. But I’ll go look for one or file one.
-- Computer Architect
> On Jan 5, 2020, at 18:51, Laurent Dumont <laurentfdumont@gmail.com> wrote:
Do you happen to have the bug ID for Centos?
On Sun, Jan 5, 2020 at 2:11 PM Jon Masters <jcm@jonmasters.org> wrote: > > This turns out to a not well documented bug in the CentOS7.7 kernel that causes exactly nat rules not to run as I > was seeing. Oh dear god was this nasty as whatever to find and workaround. > > -- > Computer Architect > > >> On Jan 4, 2020, at 10:39, Jon Masters <jcm@jonmasters.org> wrote: >> >> Excuse top posting on my phone. Also, yes, the namespaces are as described. It’s just that the (correct) nat >> rules for the qrouter netns are never running, in spite of the two interfaces existing in that ns and correctly >> attached to the vswitch. >> >> -- >> Computer Architect >> >> >>>> On Jan 4, 2020, at 07:56, Sean Mooney <smooney@redhat.com> wrote: >>>> >>>> On Sat, 2020-01-04 at 10:46 +0100, Slawek Kaplonski wrote: >>>> Hi, >>>> >>>> Is this qrouter namespace created with all those rules in container or in the host directly? >>>> Do You have qr-xxx and qg-xxx ports from br-int in this qrouter namespace? >>> >>> in kolla the l3 agent should be running with net=host so the container should be useing the hosts >>> root namespace and it will create network namespaces as needed for the different routers. >>> >>> the ip table rules should be in the router sub namespaces. >>> >>>> >>>>>> On 4 Jan 2020, at 05:44, Jon Masters <jcm@jonmasters.org> wrote: >>>>> >>>>> Hi there, >>>>> >>>>> I've got a weird problem with the neutron-l3-agent container on my deployment. It comes up, sets up the >>>>> iptables >>>>> rules in the qrouter namespace (and I can see these using "ip netns...") but traffic isn't having DNAT or >>>>> SNAT >>>>> applied. What's most strange is that manually adding a LOG jump target to the iptables nat PRE/POSTROUTING >>>>> chains >>>>> (after enabling nf logging sent to the host kernel, confirmed that works) doesn't result in any log >>>>> entries. It's as >>>>> if the nat table isn't being applied at all for any packets traversing the qrouter namespace. This is >>>>> driving me >>>>> crazy :) >>>>> >>>>> Anyone got some quick suggestions? (assume I tried the obvious stuff). >>>>> >>>>> Jon. >>>>> >>>>> -- >>>>> Computer Architect >>>> >>>> — >>>> Slawek Kaplonski >>>> Senior software engineer >>>> Red Hat >>>> >>>>
— Slawek Kaplonski Senior software engineer Red Hat
https://bugs.launchpad.net/kolla/+bug/1858505 On Mon, Jan 6, 2020 at 11:56 AM Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
On 6 Jan 2020, at 16:15, Brian Haley <haleyb.dev@gmail.com> wrote:
On 1/6/20 7:33 AM, Radosław Piliszek wrote:
Folks, this seems to be about C7, not C8, and "neutron_legacy_iptables" does not apply here. @Jon - what is the kernel bug you mentioned but never referenced?
There was a previous kernel bug in a Centos kernel that broke DNAT, https://bugs.launchpad.net/neutron/+bug/1776778 but don't know if this is the same issue. I would have hoped no one was using that kernel by now, and/or it was blacklisted.
This one also came to my mind when I read about kernel bug here. But this old bug was affecting only DNAT on dvr routers IIRC so IMO it doesn’t seems like same issue.
-Brian
pon., 6 sty 2020 o 13:13 Jon Masters <jcm@jonmasters.org> napisał(a):
I did specifically check for such a conflict tho before proceeding
-- Computer Architect
On Jan 6, 2020, at 03:40, Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2020-01-06 at 10:11 +0100, Radosław Piliszek wrote:
If it's RHEL kernel's bug, then Red Hat would likely want to know about it (if not knowing already). I have my kolla deployment on c7.7 and I don't encounter this issue, though there is a pending kernel update so now I'm worried about applying it... it sound more like a confilct between legacy iptables and the new
nftables based replacement.
if you mix the two then it will appear as if the rules are installed but only some of the rules will run. so the container images and the host need to be both configured to use the same versions.
that said fi you are using centos images on a centos host they should be providing your usnign centos 7 or centos 8 on both. if you try to use centos 7 image on a centos 8 host or centos 8 images on a centos 7 host it would likely have issues due to the fact centos 8 uses a differt iptables implemeantion
-yoctozepto
pon., 6 sty 2020 o 03:34 Jon Masters <jcm@jonmasters.org>
napisał(a):
> > There’s no bug ID that I’m aware of. But I’ll go look for one or file one. > > -- > Computer Architect > > >> On Jan 5, 2020, at 18:51, Laurent Dumont <laurentfdumont@gmail.com> wrote: > > > Do you happen to have the bug ID for Centos? > > On Sun, Jan 5, 2020 at 2:11 PM Jon Masters <jcm@jonmasters.org> wrote: >> >> This turns out to a not well documented bug in the CentOS7.7 kernel that causes exactly nat rules not to run as I >> was seeing. Oh dear god was this nasty as whatever to find and workaround. >> >> -- >> Computer Architect >> >> >>> On Jan 4, 2020, at 10:39, Jon Masters <jcm@jonmasters.org> wrote: >>> >>> Excuse top posting on my phone. Also, yes, the namespaces are as described. It’s just that the (correct) nat >>> rules for the qrouter netns are never running, in spite of the two interfaces existing in that ns and correctly >>> attached to the vswitch. >>> >>> -- >>> Computer Architect >>> >>> >>>>> On Jan 4, 2020, at 07:56, Sean Mooney <smooney@redhat.com> wrote: >>>>> >>>>> On Sat, 2020-01-04 at 10:46 +0100, Slawek Kaplonski wrote: >>>>> Hi, >>>>> >>>>> Is this qrouter namespace created with all those rules in container or in the host directly? >>>>> Do You have qr-xxx and qg-xxx ports from br-int in this qrouter namespace? >>>> >>>> in kolla the l3 agent should be running with net=host so the container should be useing the hosts >>>> root namespace and it will create network namespaces as needed for the different routers. >>>> >>>> the ip table rules should be in the router sub namespaces. >>>> >>>>> >>>>>>> On 4 Jan 2020, at 05:44, Jon Masters <jcm@jonmasters.org> wrote: >>>>>> >>>>>> Hi there, >>>>>> >>>>>> I've got a weird problem with the neutron-l3-agent container on my deployment. It comes up, sets up the >>>>>> iptables >>>>>> rules in the qrouter namespace (and I can see these using "ip netns...") but traffic isn't having DNAT or >>>>>> SNAT >>>>>> applied. What's most strange is that manually adding a LOG jump target to the iptables nat PRE/POSTROUTING >>>>>> chains >>>>>> (after enabling nf logging sent to the host kernel, confirmed
>>>>>> entries. It's as >>>>>> if the nat table isn't being applied at all for any packets
down the path I went :) that works) doesn't result in any log traversing the qrouter namespace. This is
>>>>>> driving me >>>>>> crazy :) >>>>>> >>>>>> Anyone got some quick suggestions? (assume I tried the obvious stuff). >>>>>> >>>>>> Jon. >>>>>> >>>>>> -- >>>>>> Computer Architect >>>>> >>>>> — >>>>> Slawek Kaplonski >>>>> Senior software engineer >>>>> Red Hat >>>>> >>>>>
— Slawek Kaplonski Senior software engineer Red Hat
-- Computer Architect
Thanks, Jon, though it's still too general to deduce anything more. Please see my comments on bug. -yoctozepto pon., 6 sty 2020 o 23:27 Jon Masters <jcm@jonmasters.org> napisał(a):
https://bugs.launchpad.net/kolla/+bug/1858505
On Mon, Jan 6, 2020 at 11:56 AM Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
On 6 Jan 2020, at 16:15, Brian Haley <haleyb.dev@gmail.com> wrote:
On 1/6/20 7:33 AM, Radosław Piliszek wrote:
Folks, this seems to be about C7, not C8, and "neutron_legacy_iptables" does not apply here. @Jon - what is the kernel bug you mentioned but never referenced?
There was a previous kernel bug in a Centos kernel that broke DNAT, https://bugs.launchpad.net/neutron/+bug/1776778 but don't know if this is the same issue. I would have hoped no one was using that kernel by now, and/or it was blacklisted.
This one also came to my mind when I read about kernel bug here. But this old bug was affecting only DNAT on dvr routers IIRC so IMO it doesn’t seems like same issue.
-Brian
pon., 6 sty 2020 o 13:13 Jon Masters <jcm@jonmasters.org> napisał(a):
I did specifically check for such a conflict tho before proceeding down the path I went :)
-- Computer Architect
On Jan 6, 2020, at 03:40, Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2020-01-06 at 10:11 +0100, Radosław Piliszek wrote: > If it's RHEL kernel's bug, then Red Hat would likely want to know > about it (if not knowing already). > I have my kolla deployment on c7.7 and I don't encounter this issue, > though there is a pending kernel update so now I'm worried about > applying it... it sound more like a confilct between legacy iptables and the new nftables based replacement. if you mix the two then it will appear as if the rules are installed but only some of the rules will run. so the container images and the host need to be both configured to use the same versions.
that said fi you are using centos images on a centos host they should be providing your usnign centos 7 or centos 8 on both. if you try to use centos 7 image on a centos 8 host or centos 8 images on a centos 7 host it would likely have issues due to the fact centos 8 uses a differt iptables implemeantion
> > -yoctozepto > > pon., 6 sty 2020 o 03:34 Jon Masters <jcm@jonmasters.org> napisał(a): >> >> There’s no bug ID that I’m aware of. But I’ll go look for one or file one. >> >> -- >> Computer Architect >> >> >>> On Jan 5, 2020, at 18:51, Laurent Dumont <laurentfdumont@gmail.com> wrote: >> >> >> Do you happen to have the bug ID for Centos? >> >> On Sun, Jan 5, 2020 at 2:11 PM Jon Masters <jcm@jonmasters.org> wrote: >>> >>> This turns out to a not well documented bug in the CentOS7.7 kernel that causes exactly nat rules not to run as I >>> was seeing. Oh dear god was this nasty as whatever to find and workaround. >>> >>> -- >>> Computer Architect >>> >>> >>>> On Jan 4, 2020, at 10:39, Jon Masters <jcm@jonmasters.org> wrote: >>>> >>>> Excuse top posting on my phone. Also, yes, the namespaces are as described. It’s just that the (correct) nat >>>> rules for the qrouter netns are never running, in spite of the two interfaces existing in that ns and correctly >>>> attached to the vswitch. >>>> >>>> -- >>>> Computer Architect >>>> >>>> >>>>>> On Jan 4, 2020, at 07:56, Sean Mooney <smooney@redhat.com> wrote: >>>>>> >>>>>> On Sat, 2020-01-04 at 10:46 +0100, Slawek Kaplonski wrote: >>>>>> Hi, >>>>>> >>>>>> Is this qrouter namespace created with all those rules in container or in the host directly? >>>>>> Do You have qr-xxx and qg-xxx ports from br-int in this qrouter namespace? >>>>> >>>>> in kolla the l3 agent should be running with net=host so the container should be useing the hosts >>>>> root namespace and it will create network namespaces as needed for the different routers. >>>>> >>>>> the ip table rules should be in the router sub namespaces. >>>>> >>>>>> >>>>>>>> On 4 Jan 2020, at 05:44, Jon Masters <jcm@jonmasters.org> wrote: >>>>>>> >>>>>>> Hi there, >>>>>>> >>>>>>> I've got a weird problem with the neutron-l3-agent container on my deployment. It comes up, sets up the >>>>>>> iptables >>>>>>> rules in the qrouter namespace (and I can see these using "ip netns...") but traffic isn't having DNAT or >>>>>>> SNAT >>>>>>> applied. What's most strange is that manually adding a LOG jump target to the iptables nat PRE/POSTROUTING >>>>>>> chains >>>>>>> (after enabling nf logging sent to the host kernel, confirmed that works) doesn't result in any log >>>>>>> entries. It's as >>>>>>> if the nat table isn't being applied at all for any packets traversing the qrouter namespace. This is >>>>>>> driving me >>>>>>> crazy :) >>>>>>> >>>>>>> Anyone got some quick suggestions? (assume I tried the obvious stuff). >>>>>>> >>>>>>> Jon. >>>>>>> >>>>>>> -- >>>>>>> Computer Architect >>>>>> >>>>>> — >>>>>> Slawek Kaplonski >>>>>> Senior software engineer >>>>>> Red Hat >>>>>> >>>>>> > >
— Slawek Kaplonski Senior software engineer Red Hat
-- Computer Architect
Hello, Im following quick start guide on deploying Kolla ansible and getting failure on deploy stage: https://docs.openstack.org/kolla-ansible/latest/user/quickstart.html kolla-ansible -i ./multinode deploy TASK [mariadb : Creating haproxy mysql user] ******************************************************************************************************************************** fatal: [control01]: FAILED! => {"changed": false, "msg": "Can not parse the inner module output: localhost | SUCCESS => {\n \"changed\": false, \n \"user\": \"haproxy\"\n}\n"} I deploy to Centos7 with latest updates. [user@master ~]$ pip list DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support Package Version -------------------------------- ---------- ansible 2.9.1 Babel 2.8.0 backports.ssl-match-hostname 3.7.0.1 certifi 2019.11.28 cffi 1.13.2 chardet 3.0.4 configobj 4.7.2 cryptography 2.8 debtcollector 1.22.0 decorator 3.4.0 docker 4.1.0 enum34 1.1.6 funcsigs 1.0.2 httplib2 0.9.2 idna 2.8 iniparse 0.4 ipaddress 1.0.23 IPy 0.75 iso8601 0.1.12 Jinja2 2.10.3 jmespath 0.9.4 kitchen 1.1.1 kolla-ansible 9.0.0 MarkupSafe 1.1.1 monotonic 1.5 netaddr 0.7.19 netifaces 0.10.9 oslo.config 6.12.0 oslo.i18n 3.25.0 oslo.utils 3.42.1 paramiko 2.1.1 pbr 5.4.4 perf 0.1 pip 19.3.1 ply 3.4 policycoreutils-default-encoding 0.1 pyasn1 0.1.9 pycparser 2.19 pycurl 7.19.0 pygobject 3.22.0 pygpgme 0.3 pyliblzma 0.5.3 pyparsing 2.4.6 python-linux-procfs 0.4.9 pytz 2019.3 pyudev 0.15 pyxattr 0.5.1 PyYAML 5.2 requests 2.22.0 rfc3986 1.3.2 schedutils 0.4 seobject 0.1 sepolicy 1.1 setuptools 44.0.0 six 1.13.0 slip 0.4.0 slip.dbus 0.4.0 stevedore 1.31.0 urlgrabber 3.10 urllib3 1.25.7 websocket-client 0.57.0 wrapt 1.11.2 yum-metadata-parser 1.1.4 and it looks like Im not alone with such issue: https://q.cnblogs.com/q/125213/ Thanks for your attention, Andrei Perepiolkin
Hi Andrei, I see you use kolla-ansible for Train, yet it looks as if you are deploying Stein there. Could you confirm that? If you prefer to deploy Stein, please use the Stein branch of kolla-ansible or analogically the 8.* releases from PyPI. Otherwise try deploying Train. -yoctozepto pon., 6 sty 2020 o 05:58 Andrei Perapiolkin <andrei.perepiolkin@open-e.com> napisał(a):
Hello,
Im following quick start guide on deploying Kolla ansible and getting failure on deploy stage:
https://docs.openstack.org/kolla-ansible/latest/user/quickstart.html
kolla-ansible -i ./multinode deploy
TASK [mariadb : Creating haproxy mysql user] ********************************************************************************************************************************
fatal: [control01]: FAILED! => {"changed": false, "msg": "Can not parse the inner module output: localhost | SUCCESS => {\n \"changed\": false, \n \"user\": \"haproxy\"\n}\n"}
I deploy to Centos7 with latest updates.
[user@master ~]$ pip list DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support Package Version -------------------------------- ---------- ansible 2.9.1 Babel 2.8.0 backports.ssl-match-hostname 3.7.0.1 certifi 2019.11.28 cffi 1.13.2 chardet 3.0.4 configobj 4.7.2 cryptography 2.8 debtcollector 1.22.0 decorator 3.4.0 docker 4.1.0 enum34 1.1.6 funcsigs 1.0.2 httplib2 0.9.2 idna 2.8 iniparse 0.4 ipaddress 1.0.23 IPy 0.75 iso8601 0.1.12 Jinja2 2.10.3 jmespath 0.9.4 kitchen 1.1.1 kolla-ansible 9.0.0 MarkupSafe 1.1.1 monotonic 1.5 netaddr 0.7.19 netifaces 0.10.9 oslo.config 6.12.0 oslo.i18n 3.25.0 oslo.utils 3.42.1 paramiko 2.1.1 pbr 5.4.4 perf 0.1 pip 19.3.1 ply 3.4 policycoreutils-default-encoding 0.1 pyasn1 0.1.9 pycparser 2.19 pycurl 7.19.0 pygobject 3.22.0 pygpgme 0.3 pyliblzma 0.5.3 pyparsing 2.4.6 python-linux-procfs 0.4.9 pytz 2019.3 pyudev 0.15 pyxattr 0.5.1 PyYAML 5.2 requests 2.22.0 rfc3986 1.3.2 schedutils 0.4 seobject 0.1 sepolicy 1.1 setuptools 44.0.0 six 1.13.0 slip 0.4.0 slip.dbus 0.4.0 stevedore 1.31.0 urlgrabber 3.10 urllib3 1.25.7 websocket-client 0.57.0 wrapt 1.11.2 yum-metadata-parser 1.1.4
and it looks like Im not alone with such issue: https://q.cnblogs.com/q/125213/
Thanks for your attention,
Andrei Perepiolkin
Hi Radosław, Thanks for answering me. Yes I was deploying "Stein". And Yes, after setting openstack_release to Train error disappeared. Many thanks again Radosław. Andrei Perepiolkin On 1/6/20 11:08 AM, Radosław Piliszek wrote:
Hi Andrei,
I see you use kolla-ansible for Train, yet it looks as if you are deploying Stein there. Could you confirm that? If you prefer to deploy Stein, please use the Stein branch of kolla-ansible or analogically the 8.* releases from PyPI. Otherwise try deploying Train.
-yoctozepto
pon., 6 sty 2020 o 05:58 Andrei Perapiolkin <andrei.perepiolkin@open-e.com> napisał(a):
Hello,
Im following quick start guide on deploying Kolla ansible and getting failure on deploy stage:
https://docs.openstack.org/kolla-ansible/latest/user/quickstart.html
kolla-ansible -i ./multinode deploy
TASK [mariadb : Creating haproxy mysql user] ********************************************************************************************************************************
fatal: [control01]: FAILED! => {"changed": false, "msg": "Can not parse the inner module output: localhost | SUCCESS => {\n \"changed\": false, \n \"user\": \"haproxy\"\n}\n"}
I deploy to Centos7 with latest updates.
[user@master ~]$ pip list DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support Package Version -------------------------------- ---------- ansible 2.9.1 Babel 2.8.0 backports.ssl-match-hostname 3.7.0.1 certifi 2019.11.28 cffi 1.13.2 chardet 3.0.4 configobj 4.7.2 cryptography 2.8 debtcollector 1.22.0 decorator 3.4.0 docker 4.1.0 enum34 1.1.6 funcsigs 1.0.2 httplib2 0.9.2 idna 2.8 iniparse 0.4 ipaddress 1.0.23 IPy 0.75 iso8601 0.1.12 Jinja2 2.10.3 jmespath 0.9.4 kitchen 1.1.1 kolla-ansible 9.0.0 MarkupSafe 1.1.1 monotonic 1.5 netaddr 0.7.19 netifaces 0.10.9 oslo.config 6.12.0 oslo.i18n 3.25.0 oslo.utils 3.42.1 paramiko 2.1.1 pbr 5.4.4 perf 0.1 pip 19.3.1 ply 3.4 policycoreutils-default-encoding 0.1 pyasn1 0.1.9 pycparser 2.19 pycurl 7.19.0 pygobject 3.22.0 pygpgme 0.3 pyliblzma 0.5.3 pyparsing 2.4.6 python-linux-procfs 0.4.9 pytz 2019.3 pyudev 0.15 pyxattr 0.5.1 PyYAML 5.2 requests 2.22.0 rfc3986 1.3.2 schedutils 0.4 seobject 0.1 sepolicy 1.1 setuptools 44.0.0 six 1.13.0 slip 0.4.0 slip.dbus 0.4.0 stevedore 1.31.0 urlgrabber 3.10 urllib3 1.25.7 websocket-client 0.57.0 wrapt 1.11.2 yum-metadata-parser 1.1.4
and it looks like Im not alone with such issue: https://q.cnblogs.com/q/125213/
Thanks for your attention,
Andrei Perepiolkin
participants (8)
-
Andrei Perapiolkin
-
Brian Haley
-
Jan Vondra
-
Jon Masters
-
Laurent Dumont
-
Radosław Piliszek
-
Sean Mooney
-
Slawek Kaplonski