Open Stack

Hi, Alex!
Good to know someone is doing that.

One question about Your initiative:
Are You going to reimplement all MOS HA features?
Seems first implementation of 'upstream on Fuel' will be HA-less.


On Thu, Nov 12, 2015 at 1:51 AM, Alex Schultz <aschultz at mirantis.com> wrote:

> On Tue, Nov 10, 2015 at 11:10 AM, Vladimir Kuklin <vkuklin at mirantis.com>
> wrote:
> > Alex
> >
> > Thanks for your very detailed answer - it clarified things a bit. So, if
> you
> > would allow me to rephrase it - you are actually conducting a research on
> > what is the actual gap between our downstream/fork and upstream
> > UCA/puppet-openstack. This seems to be a very promising initiative. Are
> you
> > going to come up with some external-user readable report soon?
> >
> > Also, regarding multiple distros support.  I think, we need to come up
> with
> > an approach of making 'release manager' piece of Nailgun data driven and
> > just allow a user to run any distribution he or she wants. Just register
> a
> > set of repos with packages and run it. Actually, we already have it - we
> > need to make base-image generation for RPM more flexible and any RPM/DEB
> > based distro should work ok with it.
> >
> > The remaining piece is to actually support distro-specific things, e.g.
> > CentOS/RHEL networking configuration, e.g. l23 network stored config
> puppet
> > providers. But this is a distro-supporter/community burden.
> >
> >
>
> Yes I hope to have something together by the end of the week.  I've
> managed to get a controller and compute/cinder nodes up (and passing
> basic OSTF tests) with what appears to be some minor adjustments to
> the fuel-library code.  The one thing that gets dropped because of
> lack of upstream support is nova floating network ranges. There's a
> pending review that'll get that back in but I also don't know how
> important it would be to support for this type of configuration.
> Another issue is the upstream murano module is still a work in
> progress so that won't work right now either. Hopefully that'll get
> sorted out in time for the official release of the liberty puppet
> modules.
>
> As I've been working through this, I've noticed that it would be
> possible to use fuel-plugins to only apply UCA packages to specific
> nodes via a plugin role. An interesting follow on to this effort would
> be to use MOS packages for controllers and UCA for Compute or vice
> versa.  But that should probably be more an academic exercise rather
> than production one.
>
> -Alex
>
> >
> > On Tue, Nov 10, 2015 at 6:25 PM, Alex Schultz <aschultz at mirantis.com>
> wrote:
> >>
> >> Hey Vladimir,
> >>
> >> On Tue, Nov 10, 2015 at 5:56 AM, Vladimir Kuklin <vkuklin at mirantis.com>
> >> wrote:
> >> > Alex
> >> >
> >> > That's great to hear that. But be aware - making all of the components
> >> > work
> >> > exactly the way they work within MOS is actually identical to
> >> > upstreaming
> >> > MOS. We are using some components of different versions to satisfy
> many
> >> > requirements for our Reference Architecutre implementation. It will
> not
> >> > be
> >> > so easy to base them upon existing 3rd party distributions. For
> example,
> >> > `read timeout` for SQL requests is crucial for our HA as it handles
> >> > cases
> >> > when an SQL node dies while serving the request. And you will find
> >> > myriads
> >> > of them. And as we do not control things in upstream, we will always
> >> > have
> >> > some downstream divergence.
> >> >
> >>
> >> Yes, I'm aware that it'll be an effort to make it work identically to
> >> MOS.  Currently that's not my goal. My goal is to get it working at
> >> all and be able to document the deficiencies when using upstream
> >> packages/modules vs MOS provided ones.  Once we have documented these
> >> differences we will be able to make decisions as to what efforts
> >> should be made if we choose to address the differences.  The read
> >> timeout thing seems to be an issue with what mysql python driver is
> >> used so that could easily be configurable based on a packages or a
> >> configuration option.
> >>
> >> > I guess, the optimal solution is to figure out the actual divergence
> >> > between
> >> > upstream and downstream and try to push things upstream as hard as we
> >> > can,
> >> > while retaining overrides for some packages and manifests on top of
> >> > upstream
> >> > versions. Do not get me wrong, but it seems there is exactly 0 (zero)
> >> > ways
> >> > you can get Fuel working with upstream packages unless they support
> >> > exactly
> >> > the same feature set and fix the same bugs in various components that
> >> > Fuel
> >> > expect them to support or fix. By 'working' I mean passing the same
> set
> >> > of
> >> > at least smoke and destructive tests, let alone passing scale tests.
> >> >
> >>
> >> So I think this is where we are currently backwards in the way we're
> >> doings. As we hope to push Fuel as a community project, we need to be
> >> more open to supporting the distributions versions of these packages
> >> as being supported.  If we want to continue with specific versions of
> >> things we need to be able to modularize them and apply them to a
> >> downstream version of Fuel that we can promote with the MOS package
> >> set.  I agree that right now it's highly unlikely that one would be
> >> able to use Fuel without any MOS packages. Because I don't think it's
> >> possible right now, what I'm attempting to do is be able to deploy an
> >> alternate OpenStack package set but not all of the other pieces as
> >> they relate to our reference architecture.  That seems to be a more
> >> achievable goal right now and allows us to document the gaps where
> >> Fuel/MOS are a head (or behind depending on your views) with upstream.
> >>
> >> > Simon
> >> >
> >> > AFAIK, this patch is really simple and can be easily applied to any
> >> > version
> >> > of HAProxy. So it is not too hard handle it while it provides
> sufficient
> >> > benefits to our deployment engineers. If you have any better solution,
> >> > please feel free to share the code with us.
> >> >
> >>
> >> Unfortunately the haproxy fork that we're currently running with also
> >> includes our forked version of the puppet haproxy module. I'm not sure
> >> the includes configs patch is really worth carrying the additional
> >> work/forks but I think that is a conversation for another day.
> >> Personally I think it would better if fuel-library supported the
> >> currently available upstream versions of haproxy & haproxy puppet
> >> module and if we want to continue using our copies of haproxy & puppet
> >> haproxy module that we turn that into a downstream that is applied for
> >> the MOS specific version of Fuel.  That's the advantage of the
> >> Puppetfile we've been working on. We can change out the modules with
> >> our own version downstream.
> >>
> >> Just to add some additional updates issues that I've run into.
> >>
> >> - nova-vncproxy package names (nova-consoleproxy vs nova-novncproxy)
> >> but that's easily solved with the proposed os_package_type fact.
> >> - heat-docker package does not exist in UCA. Haven't looked at what
> >> exactly is in this package yet.
> >> - horizon UCA package runs into
> >> https://bugs.launchpad.net/cloud-archive/+bug/1479340 (we solved this
> >> for MOS but it's still broken in UCA)
> >> - neutron UCA package seems to suffer from
> >> https://bugs.launchpad.net/mos/+bug/1510844
> >> - Need to solve for the url issues with the openstack command that are
> >> showing up in the CI as I just hit it with my deployment.
> >>
> >> Thanks,
> >> -Alex
> >>
> >> > On Tue, Nov 10, 2015 at 12:54 PM, Stanislaw Bogatkin
> >> > <sbogatkin at mirantis.com> wrote:
> >> >>
> >> >> Hi Simon,
> >> >>
> >> >> using 'include' in HAProxy is damn conveniently, I don't think it
> >> >> should
> >> >> die. There is just one way I know to use haproxy with several
> different
> >> >> conf's - to construct looooooooooong command line with all of them -
> >> >> and
> >> >> it's really inconvenient.
> >> >>
> >> >> On Tue, Nov 10, 2015 at 12:20 PM, Simon Pasquier
> >> >> <spasquier at mirantis.com>
> >> >> wrote:
> >> >>>
> >> >>> Hello Alex!
> >> >>>
> >> >>> On Mon, Nov 9, 2015 at 9:29 PM, Alex Schultz <aschultz at mirantis.com
> >
> >> >>> wrote:
> >> >>>>
> >> >>>> Hey folks,
> >> >>>>
> >> >>>> I'm testing[0] out flipping our current method of consuming
> upstream
> >> >>>> puppet modules from using pinned versions hosted on fuel-infra to
> be
> >> >>>> able to use the ones directly from upstream (master).  This work is
> >> >>>> primarily to be closer aligned with the other OpenStack projects as
> >> >>>> well as switching the current way we manage Fuel into a downstream
> of
> >> >>>> the upstream community version. As part of this work we have also
> >> >>>> been
> >> >>>> working towards improving the upstream modules support different
> >> >>>> package sets.  Specifically running Debian packages on
> Ubuntu[1][2].
> >> >>>> This work is the start of being able to allow a user of Fuel to be
> >> >>>> able to specify a specific package set and having it be able to
> work.
> >> >>>> If we can properly split out the puppet modules and package
> >> >>>> dependencies this will make Fuel a more flexible deployment engine
> as
> >> >>>> I believe we would be better positioned to support multiple
> versions
> >> >>>> of OpenStack for a given Fuel release.
> >> >>>>
> >> >>>> I'm currently working to get a PoC of Fuel consuming upstream
> puppet
> >> >>>> modules and the UCA packages together and documenting all of the
> >> >>>> issues so that we can address them.  So far I have been able to
> >> >>>> deploy
> >> >>>> the upstream modules via a custom ISO using the MOS package set and
> >> >>>> it
> >> >>>> works locally. Unfortunately the CI seems to be hitting some issues
> >> >>>> that I think might be related to recently merged keystone changes
> but
> >> >>>> I did not run into the same problem when running a manual
> deployment.
> >> >>>> As I work through this PoC, I'm also attempting to develop a small
> >> >>>> plugin that could be used to capture the work arounds to the
> >> >>>> deployment process. I've run into a few issues so far as I work to
> >> >>>> switch out the package sets.
> >> >>>>
> >> >>>> For the sake of providing additional visibility into this work,
> here
> >> >>>> are the issues that I've hit so far.
> >> >>>>
> >> >>>> The first issue I ran across is that currently the MOS repositories
> >> >>>> contain packages for both OpenStack and other system dependencies
> for
> >> >>>> creating our HA implementation.  This is problematic when we want
> to
> >> >>>> switch out the OpenStack packages but still want the MOS packages
> for
> >> >>>> our HA items.  I'm working around this by adding the UCA repository
> >> >>>> as
> >> >>>> a higher priorities for deployments.  As such I've run into an
> issue
> >> >>>> with the haproxy package that MOS provides vs the upstream Ubuntu
> >> >>>> package.  To get around this, I've pinned the MOS version for now
> >> >>>> until I can circle back around and figure out if the difference is
> a
> >> >>>> config or functionality issue.
> >> >>>
> >> >>>
> >> >>> I know at least for one difference between the MOS and Ubuntu
> >> >>> versions:
> >> >>> HAProxy from MOS has a patch to support the "include" configuration
> >> >>> parameter.
> >> >>> This is unrelated to your mail but I think this fork should die
> since
> >> >>> it
> >> >>> will never be accepted upstream and there are other ways to address
> >> >>> the use
> >> >>> case [0].
> >> >>>
> >> >>> HTH,
> >> >>> Simon
> >> >>>
> >> >>> [0] http://marc.info/?l=haproxy&m=130817444025140&w=2
> >> >>>
> >> >>>>
> >> >>>>
> >> >>>> The second issue that I have run across is that we are appending
> >> >>>> read_timeout=60 to our mysql connection strings for our
> >> >>>> configurations. This seems to be unsupported by the libraries and
> >> >>>> OpenStack components provided by the UCA package set.  I'm not sure
> >> >>>> how the priority of python-pymsql vs python-mysqldb is resolved as
> I
> >> >>>> had both packages installed but it continued to fail on
> read_timeout
> >> >>>> being in the connection string.  For now I've updated the
> >> >>>> fuel-library
> >> >>>> code to remove this item from the connection strings and will be
> >> >>>> circling back around to figure out the correct 'fix' for this
> issue.
> >> >>>>
> >> >>>> I'm hoping to be able to have a working ISO, plugin and a set of
> >> >>>> instructions that can be used to deploy a basic cloud using Fuel by
> >> >>>> the end of this week.
> >> >>>>
> >> >>>> Thanks,
> >> >>>> -Alex
> >> >>>>
> >> >>>> [0] https://review.openstack.org/#/c/240325/
> >> >>>> [1] https://review.openstack.org/#/c/241615/
> >> >>>> [2] https://review.openstack.org/#/c/241741/
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> __________________________________________________________________________
> >> >>>> OpenStack Development Mailing List (not for usage questions)
> >> >>>> Unsubscribe:
> >> >>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> __________________________________________________________________________
> >> >>> OpenStack Development Mailing List (not for usage questions)
> >> >>> Unsubscribe:
> >> >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >>
> __________________________________________________________________________
> >> >> OpenStack Development Mailing List (not for usage questions)
> >> >> Unsubscribe:
> >> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Yours Faithfully,
> >> > Vladimir Kuklin,
> >> > Fuel Library Tech Lead,
> >> > Mirantis, Inc.
> >> > +7 (495) 640-49-04
> >> > +7 (926) 702-39-68
> >> > Skype kuklinvv
> >> > 35bk3, Vorontsovskaya Str.
> >> > Moscow, Russia,
> >> > www.mirantis.com
> >> > www.mirantis.ru
> >> > vkuklin at mirantis.com
> >> >
> >> >
> >> >
> __________________________________________________________________________
> >> > OpenStack Development Mailing List (not for usage questions)
> >> > Unsubscribe:
> >> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >
> >>
> >>
> __________________________________________________________________________
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> >
> > --
> > Yours Faithfully,
> > Vladimir Kuklin,
> > Fuel Library Tech Lead,
> > Mirantis, Inc.
> > +7 (495) 640-49-04
> > +7 (926) 702-39-68
> > Skype kuklinvv
> > 35bk3, Vorontsovskaya Str.
> > Moscow, Russia,
> > www.mirantis.com
> > www.mirantis.ru
> > vkuklin at mirantis.com
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 

Kind Regards,

Alexandr Kostrikov,

Mirantis, Inc.

35b/3, Vorontsovskaya St., 109147, Moscow, Russia


Tel.: +7 (495) 640-49-04
Tel.: +7 (925) 716-64-52 <%2B7%20%28906%29%20740-64-79>

Skype: akostrikov_mirantis

E-mail: akostrikov at mirantis.com <elogutova at mirantis.com>

*www.mirantis.com <http://www.mirantis.ru/>*
*www.mirantis.ru <http://www.mirantis.ru/>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151113/0330e083/attachment.html>