Open Stack

Andrew,

[2] may be due to agents failing to start. Incorrect agents configuration
will lead to agents unability to start and pacemaker timeouts. [3] I do not
see failures in rabbit service as I see that it succesfully transitioned
from stopped to running state. [4] Swift error shows that you have
incorrect rings configuration.


On Tue, Jul 15, 2014 at 9:23 AM, Andrew Woodward <xarses at gmail.com> wrote:

> Friday, resolved the haproxy / vip issue. It was due to loosing
> net.ipv4.ip_forward [1] and then we regressed back to not working at
> all in HA or Simple. There was also an issue with the neutron keystone
> call being cached oddly causing a trace. Removing the ability to cache
> the object appears to have stopped the error, but the changes leading
> into learning about this issue may have steered us into this not
> working again.
>
> Today, fought on the regression
>
> HA deployment testing is completely blocked by [2], [3]. Usually this
> occurs after puppet re-runs the controller after an error. This
> frequently occurs due to swift [4].
>
> Simple deployment testing is plagued by neutron notifiers auth failures
> [5].
>
> There is some minor annoyance with [6] that doesn't have to be fixed,
> but is annoying. There is another variant where other, non-recoverable
> (usually malformed-syntax errors) are also caught in the retry when
> they should be outright aborted on. This is likely due to the regex in
> neutron.rb being overly grabby.
>
> [1] https://bugs.launchpad.net/fuel/+bug/1340968
> [2]
> https://gist.github.com/xarses/219be742ab04faeb7f53#file-pacemaker-corosync-error
> [3]
> https://gist.github.com/xarses/219be742ab04faeb7f53#file-rabbit-service-failure
> [4] https://gist.github.com/xarses/219be742ab04faeb7f53#file-swift-error
> [5]
> https://gist.github.com/xarses/219be742ab04faeb7f53#file-neutron-vif-plug-error
> [6]
> https://gist.github.com/xarses/219be742ab04faeb7f53#file-neutron_network-changes-shared-on-existing-network
>
> On Fri, Jul 11, 2014 at 12:08 AM, Andrew Woodward <xarses at gmail.com>
> wrote:
> > Retested today
> > ubuntu single nova vlan - works
> > centos single nova dhcp - works
> > ubuntu single neutron gre - works
> > centos single neutron vlan - works
> > centos ha(1) neutron vlan - fail haproxy issue
> > ubuntu ha(1) neutron gre - fail haproxy issue.
> >
> > haproxy / vip issue:
> >
> > due to whatever reason that I haven't been able to track down yet, the ip
> > netns namespaces (public and management) ns_IPaddr2 vips can not ping or
> > otherwise communicate with nodes remote to who ever owns the respective
> vip.
> > Once this issue is resolved, I believe that CI should pass given that the
> > build appears 100% functional except that computes cant connect to the
> vip
> > properly.
> >
> >
> > On Thu, Jul 10, 2014 at 1:05 AM, Mike Scherbakov <
> mscherbakov at mirantis.com>
> > wrote:
> >>
> >> We had a call between Andrew (@xarses), Vladimir Kuklin (@aglarendil)
> and
> >> myself (@mihgen) today to finally sort out Neutron ML2 integration in
> Fuel.
> >> We didn't have @xenolog on the call, but hopefully he is more or less
> fine
> >> with all below, and kindly request him to confirm :)
> >> Discussed following topics, with an agreement from all participants:
> >>
> >> Risks of merging https://review.openstack.org/#/c/103280 (@xarses,
> >> upstream puppet module, will further refer as "280") vs
> >> https://review.openstack.org/#/c/103947 (@xenolog, extending existing
> puppet
> >> module with ML2 support, will further refer as "947")
> >>
> >> We all agree, that 280 is strategically the way to go. It was so by
> >> design, and 947 was done only as risk mitigation plan in case if 280 is
> not
> >> ready in time
> >> Both 280 and 947 were manually verified in combinations of
> >> ubuntu/centos/vlan/gre/ha, 280 needs to be verified with nova-network
> >> 947 was ready a week ago and considered to be more stable solution
> >> 280 is has much higher risks to introduce regressions, as it is
> basically
> >> rearchitecturing of Neutron puppet module in Fuel
> >> 947 has backward compatibility support with legacy OVS, while 280
> doesn't
> >> have it at the moment
> >>
> >> Mellanox & VMWare NSX dependency on ML2 implementation rebase time
> >>
> >> Rebase itself should not be hard
> >> It has to be tested and may take up to next WE to do all
> >> rebases/testing/fixing
> >> As both Mellanox & NSX Neutron pieces are isolated, it can be an
> exception
> >> for merging by next TH
> >>
> >> Discussed sanitize_network_config [1]
> >>
> >> @aglarendil points out that we need to verify input params which puppet
> >> receives in advance, before waiting hours of deploy
> >> @mihgen's point of view is that we need to consider each Fuel component
> as
> >> module, and verify output of it with certain input params. So there is
> no
> >> point to verify input in puppet, if it's being verified in output of
> >> Nailgun.
> >> @xarses says that we need to verify configuration files created in
> system
> >> after module execution, to check if it matches with module input params
> >> Overall topic has been discussed much more extensively, and it needs
> >> further follow up. @aglarendil confirms that it is Ok to remove
> sanitize for
> >> now, but start writing much more tests to check that 280 deploys
> correctly
> >> right away
> >>
> >> Action plan we've come up with:
> >>
> >> Merge 947, as less risky at the moment, also considering the fact that
> 280
> >> doesn't pass CI
> >>
> >> this will immediately unblock Mellanox & NSX teams so they can rebase on
> >> ML2 implementation
> >>
> >> In parallel, create a test plan for 280 together with QA team
> >> @xenolog to join 280's effort, start fixing issues discovered there,
> >> including [2]
> >> Build an ISO, based on 280, and start testing it according to the plan
> >> Merge 280 once test plan is executed and issues fixed. Expected on
> Monday.
> >>
> >> [1] https://review.openstack.org/#/c/99807/1/specs/5.1/ml2-neutron.rst
> >> [2]
> >>
> https://fuel-jenkins.mirantis.com/job/master_fuellib_review_systest_ubuntu/1608/
> >>
> >> Thanks,
> >>
> >>
> >> On Wed, Jul 9, 2014 at 10:31 PM, Andrew Woodward <xarses at gmail.com>
> wrote:
> >>>>
> >>>> Teams from MElanox and VMware based their work on current
> implementation
> >>>> of Neutron
> >>>
> >>> Mlnx appears to not set any neutron settings, so wont be enabled
> >>> correctly with out some more TLC
> >>> (
> >>>
> https://wiki.openstack.org/wiki/Mellanox-Neutron-Icehouse-Redhat#Neutron_Server_Node
> >>> )
> >>>
> >>> NSX appears to be using legacy OVS, but assumes just whatever the
> default
> >>> core_plugin is so it will need some TLC too.
> >>>
> >>>> to merge https://review.openstack.org/#/c/103280 in fuel-5.1
> >>>
> >>> Link is wrong should be https://review.openstack.org/#/c/103947
> >>>
> >>>>
> >>>> To base Neutron implementation for 6.0 get
> >>>> https://review.openstack.org/#/c/103280/ and start working for adapt
> this
> >>>> for Juno Neutron.
> >>>
> >>> I still think we should merge it
> >>>>
> >>>> Also we should discuss how to make HA-wrapper more pluggable.
> >>>
> >>> https://review.openstack.org/#/c/103279/ makes it very pluggable, and
> >>> felt like the best start, what else do we need to add in the near term?
> >>> Should we just extend it as we move more components to it?
> >>>
> >>>
> >>> On Wed, Jul 9, 2014 at 9:21 AM, Sergey Vasilenko
> >>> <svasilenko at mirantis.com> wrote:
> >>>>
> >>>> After review https://review.openstack.org/#/c/103280/ I have
> following
> >>>> thoughts. This patch looks fine, but huge and has lot of changes in
> >>>> deployment logic. Parallel teams work (vmware, melanox) depends on
> current
> >>>> implementation of Neutron in FUEL. I have a some dreads, that they
> can’t
> >>>> refactor his code in the short time remaining.
> >>>>
> >>>> Implementation https://review.openstack.org/#/c/103280 has
> considerably
> >>>> less changes in current code of FUEL. Teams from MElanox and VMware
> based
> >>>> their work on current implementation of Neutron. Also this
> implementation
> >>>> was already tested in most deployment modes.
> >>>>
> >>>> On this basis, I suggest:
> >>>>
> >>>> to merge https://review.openstack.org/#/c/103280 in fuel-5.1
> >>>> To base Neutron implementation for 6.0 get
> >>>> https://review.openstack.org/#/c/103280/ and start working for adapt
> this
> >>>> for Juno Neutron.
> >>>> Also we should discuss how to make HA-wrapper more pluggable.
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> OpenStack-dev mailing list
> >>>> OpenStack-dev at lists.openstack.org
> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Andrew
> >>> Mirantis
> >>> Ceph community
> >>>
> >>> _______________________________________________
> >>> OpenStack-dev mailing list
> >>> OpenStack-dev at lists.openstack.org
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> >>
> >>
> >>
> >> --
> >> Mike Scherbakov
> >> #mihgen
> >>
> >>
> >> _______________________________________________
> >> OpenStack-dev mailing list
> >> OpenStack-dev at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> >
> >
> > --
> > Andrew
> > Mirantis
> > Ceph community
>
>
>
> --
> Andrew
> Mirantis
> Ceph community
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Yours Faithfully,
Vladimir Kuklin,
Fuel Library Tech Lead,
Mirantis, Inc.
+7 (495) 640-49-04
+7 (926) 702-39-68
Skype kuklinvv
45bk3, Vorontsovskaya Str.
Moscow, Russia,
www.mirantis.com <http://www.mirantis.ru/>
www.mirantis.ru
vkuklin at mirantis.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140715/d1e9b27d/attachment.html>