Open Stack

Friday, resolved the haproxy / vip issue. It was due to loosing
net.ipv4.ip_forward [1] and then we regressed back to not working at
all in HA or Simple. There was also an issue with the neutron keystone
call being cached oddly causing a trace. Removing the ability to cache
the object appears to have stopped the error, but the changes leading
into learning about this issue may have steered us into this not
working again.

Today, fought on the regression

HA deployment testing is completely blocked by [2], [3]. Usually this
occurs after puppet re-runs the controller after an error. This
frequently occurs due to swift [4].

Simple deployment testing is plagued by neutron notifiers auth failures [5].

There is some minor annoyance with [6] that doesn't have to be fixed,
but is annoying. There is another variant where other, non-recoverable
(usually malformed-syntax errors) are also caught in the retry when
they should be outright aborted on. This is likely due to the regex in
neutron.rb being overly grabby.

[1] https://bugs.launchpad.net/fuel/+bug/1340968
[2] https://gist.github.com/xarses/219be742ab04faeb7f53#file-pacemaker-corosync-error
[3] https://gist.github.com/xarses/219be742ab04faeb7f53#file-rabbit-service-failure
[4] https://gist.github.com/xarses/219be742ab04faeb7f53#file-swift-error
[5] https://gist.github.com/xarses/219be742ab04faeb7f53#file-neutron-vif-plug-error
[6] https://gist.github.com/xarses/219be742ab04faeb7f53#file-neutron_network-changes-shared-on-existing-network

On Fri, Jul 11, 2014 at 12:08 AM, Andrew Woodward <xarses at gmail.com> wrote:
> Retested today
> ubuntu single nova vlan - works
> centos single nova dhcp - works
> ubuntu single neutron gre - works
> centos single neutron vlan - works
> centos ha(1) neutron vlan - fail haproxy issue
> ubuntu ha(1) neutron gre - fail haproxy issue.
>
> haproxy / vip issue:
>
> due to whatever reason that I haven't been able to track down yet, the ip
> netns namespaces (public and management) ns_IPaddr2 vips can not ping or
> otherwise communicate with nodes remote to who ever owns the respective vip.
> Once this issue is resolved, I believe that CI should pass given that the
> build appears 100% functional except that computes cant connect to the vip
> properly.
>
>
> On Thu, Jul 10, 2014 at 1:05 AM, Mike Scherbakov <mscherbakov at mirantis.com>
> wrote:
>>
>> We had a call between Andrew (@xarses), Vladimir Kuklin (@aglarendil) and
>> myself (@mihgen) today to finally sort out Neutron ML2 integration in Fuel.
>> We didn't have @xenolog on the call, but hopefully he is more or less fine
>> with all below, and kindly request him to confirm :)
>> Discussed following topics, with an agreement from all participants:
>>
>> Risks of merging https://review.openstack.org/#/c/103280 (@xarses,
>> upstream puppet module, will further refer as "280") vs
>> https://review.openstack.org/#/c/103947 (@xenolog, extending existing puppet
>> module with ML2 support, will further refer as "947")
>>
>> We all agree, that 280 is strategically the way to go. It was so by
>> design, and 947 was done only as risk mitigation plan in case if 280 is not
>> ready in time
>> Both 280 and 947 were manually verified in combinations of
>> ubuntu/centos/vlan/gre/ha, 280 needs to be verified with nova-network
>> 947 was ready a week ago and considered to be more stable solution
>> 280 is has much higher risks to introduce regressions, as it is basically
>> rearchitecturing of Neutron puppet module in Fuel
>> 947 has backward compatibility support with legacy OVS, while 280 doesn't
>> have it at the moment
>>
>> Mellanox & VMWare NSX dependency on ML2 implementation rebase time
>>
>> Rebase itself should not be hard
>> It has to be tested and may take up to next WE to do all
>> rebases/testing/fixing
>> As both Mellanox & NSX Neutron pieces are isolated, it can be an exception
>> for merging by next TH
>>
>> Discussed sanitize_network_config [1]
>>
>> @aglarendil points out that we need to verify input params which puppet
>> receives in advance, before waiting hours of deploy
>> @mihgen's point of view is that we need to consider each Fuel component as
>> module, and verify output of it with certain input params. So there is no
>> point to verify input in puppet, if it's being verified in output of
>> Nailgun.
>> @xarses says that we need to verify configuration files created in system
>> after module execution, to check if it matches with module input params
>> Overall topic has been discussed much more extensively, and it needs
>> further follow up. @aglarendil confirms that it is Ok to remove sanitize for
>> now, but start writing much more tests to check that 280 deploys correctly
>> right away
>>
>> Action plan we've come up with:
>>
>> Merge 947, as less risky at the moment, also considering the fact that 280
>> doesn't pass CI
>>
>> this will immediately unblock Mellanox & NSX teams so they can rebase on
>> ML2 implementation
>>
>> In parallel, create a test plan for 280 together with QA team
>> @xenolog to join 280's effort, start fixing issues discovered there,
>> including [2]
>> Build an ISO, based on 280, and start testing it according to the plan
>> Merge 280 once test plan is executed and issues fixed. Expected on Monday.
>>
>> [1] https://review.openstack.org/#/c/99807/1/specs/5.1/ml2-neutron.rst
>> [2]
>> https://fuel-jenkins.mirantis.com/job/master_fuellib_review_systest_ubuntu/1608/
>>
>> Thanks,
>>
>>
>> On Wed, Jul 9, 2014 at 10:31 PM, Andrew Woodward <xarses at gmail.com> wrote:
>>>>
>>>> Teams from MElanox and VMware based their work on current implementation
>>>> of Neutron
>>>
>>> Mlnx appears to not set any neutron settings, so wont be enabled
>>> correctly with out some more TLC
>>> (
>>> https://wiki.openstack.org/wiki/Mellanox-Neutron-Icehouse-Redhat#Neutron_Server_Node
>>> )
>>>
>>> NSX appears to be using legacy OVS, but assumes just whatever the default
>>> core_plugin is so it will need some TLC too.
>>>
>>>> to merge https://review.openstack.org/#/c/103280 in fuel-5.1
>>>
>>> Link is wrong should be https://review.openstack.org/#/c/103947
>>>
>>>>
>>>> To base Neutron implementation for 6.0 get
>>>> https://review.openstack.org/#/c/103280/ and start working for adapt this
>>>> for Juno Neutron.
>>>
>>> I still think we should merge it
>>>>
>>>> Also we should discuss how to make HA-wrapper more pluggable.
>>>
>>> https://review.openstack.org/#/c/103279/ makes it very pluggable, and
>>> felt like the best start, what else do we need to add in the near term?
>>> Should we just extend it as we move more components to it?
>>>
>>>
>>> On Wed, Jul 9, 2014 at 9:21 AM, Sergey Vasilenko
>>> <svasilenko at mirantis.com> wrote:
>>>>
>>>> After review https://review.openstack.org/#/c/103280/ I have following
>>>> thoughts. This patch looks fine, but huge and has lot of changes in
>>>> deployment logic. Parallel teams work (vmware, melanox) depends on current
>>>> implementation of Neutron in FUEL. I have a some dreads, that they can’t
>>>> refactor his code in the short time remaining.
>>>>
>>>> Implementation https://review.openstack.org/#/c/103280 has considerably
>>>> less changes in current code of FUEL. Teams from MElanox and VMware based
>>>> their work on current implementation of Neutron. Also this implementation
>>>> was already tested in most deployment modes.
>>>>
>>>> On this basis, I suggest:
>>>>
>>>> to merge https://review.openstack.org/#/c/103280 in fuel-5.1
>>>> To base Neutron implementation for 6.0 get
>>>> https://review.openstack.org/#/c/103280/ and start working for adapt this
>>>> for Juno Neutron.
>>>> Also we should discuss how to make HA-wrapper more pluggable.
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>
>>>
>>> --
>>> Andrew
>>> Mirantis
>>> Ceph community
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>>
>> --
>> Mike Scherbakov
>> #mihgen
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
> Andrew
> Mirantis
> Ceph community



-- 
Andrew
Mirantis
Ceph community