Open Stack

On Fri, Nov 13, 2015 at 10:17 AM, Alexander Kostrikov
<akostrikov at mirantis.com> wrote:
> Hi, Alex!
> Good to know someone is doing that.
>
> One question about Your initiative:
> Are You going to reimplement all MOS HA features?
> Seems first implementation of 'upstream on Fuel' will be HA-less.
>

My first pass included the MOS HA features as we will still be
leveraging the MOS packages for these items (mysql/haproxy).  My PoC
allows for a different set of OpenStack packages to be set as the
preferred package to use over the current MOS packages which are the
Fuel default. I think eventually we'll need to reevaluate how we can
split out the MOS packages for these features and allow a user to
select which package set to use and still have it 'work'.


Thanks,
-Alex

>
> On Thu, Nov 12, 2015 at 1:51 AM, Alex Schultz <aschultz at mirantis.com> wrote:
>>
>> On Tue, Nov 10, 2015 at 11:10 AM, Vladimir Kuklin <vkuklin at mirantis.com>
>> wrote:
>> > Alex
>> >
>> > Thanks for your very detailed answer - it clarified things a bit. So, if
>> > you
>> > would allow me to rephrase it - you are actually conducting a research
>> > on
>> > what is the actual gap between our downstream/fork and upstream
>> > UCA/puppet-openstack. This seems to be a very promising initiative. Are
>> > you
>> > going to come up with some external-user readable report soon?
>> >
>> > Also, regarding multiple distros support.  I think, we need to come up
>> > with
>> > an approach of making 'release manager' piece of Nailgun data driven and
>> > just allow a user to run any distribution he or she wants. Just register
>> > a
>> > set of repos with packages and run it. Actually, we already have it - we
>> > need to make base-image generation for RPM more flexible and any RPM/DEB
>> > based distro should work ok with it.
>> >
>> > The remaining piece is to actually support distro-specific things, e.g.
>> > CentOS/RHEL networking configuration, e.g. l23 network stored config
>> > puppet
>> > providers. But this is a distro-supporter/community burden.
>> >
>> >
>>
>> Yes I hope to have something together by the end of the week.  I've
>> managed to get a controller and compute/cinder nodes up (and passing
>> basic OSTF tests) with what appears to be some minor adjustments to
>> the fuel-library code.  The one thing that gets dropped because of
>> lack of upstream support is nova floating network ranges. There's a
>> pending review that'll get that back in but I also don't know how
>> important it would be to support for this type of configuration.
>> Another issue is the upstream murano module is still a work in
>> progress so that won't work right now either. Hopefully that'll get
>> sorted out in time for the official release of the liberty puppet
>> modules.
>>
>> As I've been working through this, I've noticed that it would be
>> possible to use fuel-plugins to only apply UCA packages to specific
>> nodes via a plugin role. An interesting follow on to this effort would
>> be to use MOS packages for controllers and UCA for Compute or vice
>> versa.  But that should probably be more an academic exercise rather
>> than production one.
>>
>> -Alex
>>
>> >
>> > On Tue, Nov 10, 2015 at 6:25 PM, Alex Schultz <aschultz at mirantis.com>
>> > wrote:
>> >>
>> >> Hey Vladimir,
>> >>
>> >> On Tue, Nov 10, 2015 at 5:56 AM, Vladimir Kuklin <vkuklin at mirantis.com>
>> >> wrote:
>> >> > Alex
>> >> >
>> >> > That's great to hear that. But be aware - making all of the
>> >> > components
>> >> > work
>> >> > exactly the way they work within MOS is actually identical to
>> >> > upstreaming
>> >> > MOS. We are using some components of different versions to satisfy
>> >> > many
>> >> > requirements for our Reference Architecutre implementation. It will
>> >> > not
>> >> > be
>> >> > so easy to base them upon existing 3rd party distributions. For
>> >> > example,
>> >> > `read timeout` for SQL requests is crucial for our HA as it handles
>> >> > cases
>> >> > when an SQL node dies while serving the request. And you will find
>> >> > myriads
>> >> > of them. And as we do not control things in upstream, we will always
>> >> > have
>> >> > some downstream divergence.
>> >> >
>> >>
>> >> Yes, I'm aware that it'll be an effort to make it work identically to
>> >> MOS.  Currently that's not my goal. My goal is to get it working at
>> >> all and be able to document the deficiencies when using upstream
>> >> packages/modules vs MOS provided ones.  Once we have documented these
>> >> differences we will be able to make decisions as to what efforts
>> >> should be made if we choose to address the differences.  The read
>> >> timeout thing seems to be an issue with what mysql python driver is
>> >> used so that could easily be configurable based on a packages or a
>> >> configuration option.
>> >>
>> >> > I guess, the optimal solution is to figure out the actual divergence
>> >> > between
>> >> > upstream and downstream and try to push things upstream as hard as we
>> >> > can,
>> >> > while retaining overrides for some packages and manifests on top of
>> >> > upstream
>> >> > versions. Do not get me wrong, but it seems there is exactly 0 (zero)
>> >> > ways
>> >> > you can get Fuel working with upstream packages unless they support
>> >> > exactly
>> >> > the same feature set and fix the same bugs in various components that
>> >> > Fuel
>> >> > expect them to support or fix. By 'working' I mean passing the same
>> >> > set
>> >> > of
>> >> > at least smoke and destructive tests, let alone passing scale tests.
>> >> >
>> >>
>> >> So I think this is where we are currently backwards in the way we're
>> >> doings. As we hope to push Fuel as a community project, we need to be
>> >> more open to supporting the distributions versions of these packages
>> >> as being supported.  If we want to continue with specific versions of
>> >> things we need to be able to modularize them and apply them to a
>> >> downstream version of Fuel that we can promote with the MOS package
>> >> set.  I agree that right now it's highly unlikely that one would be
>> >> able to use Fuel without any MOS packages. Because I don't think it's
>> >> possible right now, what I'm attempting to do is be able to deploy an
>> >> alternate OpenStack package set but not all of the other pieces as
>> >> they relate to our reference architecture.  That seems to be a more
>> >> achievable goal right now and allows us to document the gaps where
>> >> Fuel/MOS are a head (or behind depending on your views) with upstream.
>> >>
>> >> > Simon
>> >> >
>> >> > AFAIK, this patch is really simple and can be easily applied to any
>> >> > version
>> >> > of HAProxy. So it is not too hard handle it while it provides
>> >> > sufficient
>> >> > benefits to our deployment engineers. If you have any better
>> >> > solution,
>> >> > please feel free to share the code with us.
>> >> >
>> >>
>> >> Unfortunately the haproxy fork that we're currently running with also
>> >> includes our forked version of the puppet haproxy module. I'm not sure
>> >> the includes configs patch is really worth carrying the additional
>> >> work/forks but I think that is a conversation for another day.
>> >> Personally I think it would better if fuel-library supported the
>> >> currently available upstream versions of haproxy & haproxy puppet
>> >> module and if we want to continue using our copies of haproxy & puppet
>> >> haproxy module that we turn that into a downstream that is applied for
>> >> the MOS specific version of Fuel.  That's the advantage of the
>> >> Puppetfile we've been working on. We can change out the modules with
>> >> our own version downstream.
>> >>
>> >> Just to add some additional updates issues that I've run into.
>> >>
>> >> - nova-vncproxy package names (nova-consoleproxy vs nova-novncproxy)
>> >> but that's easily solved with the proposed os_package_type fact.
>> >> - heat-docker package does not exist in UCA. Haven't looked at what
>> >> exactly is in this package yet.
>> >> - horizon UCA package runs into
>> >> https://bugs.launchpad.net/cloud-archive/+bug/1479340 (we solved this
>> >> for MOS but it's still broken in UCA)
>> >> - neutron UCA package seems to suffer from
>> >> https://bugs.launchpad.net/mos/+bug/1510844
>> >> - Need to solve for the url issues with the openstack command that are
>> >> showing up in the CI as I just hit it with my deployment.
>> >>
>> >> Thanks,
>> >> -Alex
>> >>
>> >> > On Tue, Nov 10, 2015 at 12:54 PM, Stanislaw Bogatkin
>> >> > <sbogatkin at mirantis.com> wrote:
>> >> >>
>> >> >> Hi Simon,
>> >> >>
>> >> >> using 'include' in HAProxy is damn conveniently, I don't think it
>> >> >> should
>> >> >> die. There is just one way I know to use haproxy with several
>> >> >> different
>> >> >> conf's - to construct looooooooooong command line with all of them -
>> >> >> and
>> >> >> it's really inconvenient.
>> >> >>
>> >> >> On Tue, Nov 10, 2015 at 12:20 PM, Simon Pasquier
>> >> >> <spasquier at mirantis.com>
>> >> >> wrote:
>> >> >>>
>> >> >>> Hello Alex!
>> >> >>>
>> >> >>> On Mon, Nov 9, 2015 at 9:29 PM, Alex Schultz
>> >> >>> <aschultz at mirantis.com>
>> >> >>> wrote:
>> >> >>>>
>> >> >>>> Hey folks,
>> >> >>>>
>> >> >>>> I'm testing[0] out flipping our current method of consuming
>> >> >>>> upstream
>> >> >>>> puppet modules from using pinned versions hosted on fuel-infra to
>> >> >>>> be
>> >> >>>> able to use the ones directly from upstream (master).  This work
>> >> >>>> is
>> >> >>>> primarily to be closer aligned with the other OpenStack projects
>> >> >>>> as
>> >> >>>> well as switching the current way we manage Fuel into a downstream
>> >> >>>> of
>> >> >>>> the upstream community version. As part of this work we have also
>> >> >>>> been
>> >> >>>> working towards improving the upstream modules support different
>> >> >>>> package sets.  Specifically running Debian packages on
>> >> >>>> Ubuntu[1][2].
>> >> >>>> This work is the start of being able to allow a user of Fuel to be
>> >> >>>> able to specify a specific package set and having it be able to
>> >> >>>> work.
>> >> >>>> If we can properly split out the puppet modules and package
>> >> >>>> dependencies this will make Fuel a more flexible deployment engine
>> >> >>>> as
>> >> >>>> I believe we would be better positioned to support multiple
>> >> >>>> versions
>> >> >>>> of OpenStack for a given Fuel release.
>> >> >>>>
>> >> >>>> I'm currently working to get a PoC of Fuel consuming upstream
>> >> >>>> puppet
>> >> >>>> modules and the UCA packages together and documenting all of the
>> >> >>>> issues so that we can address them.  So far I have been able to
>> >> >>>> deploy
>> >> >>>> the upstream modules via a custom ISO using the MOS package set
>> >> >>>> and
>> >> >>>> it
>> >> >>>> works locally. Unfortunately the CI seems to be hitting some
>> >> >>>> issues
>> >> >>>> that I think might be related to recently merged keystone changes
>> >> >>>> but
>> >> >>>> I did not run into the same problem when running a manual
>> >> >>>> deployment.
>> >> >>>> As I work through this PoC, I'm also attempting to develop a small
>> >> >>>> plugin that could be used to capture the work arounds to the
>> >> >>>> deployment process. I've run into a few issues so far as I work to
>> >> >>>> switch out the package sets.
>> >> >>>>
>> >> >>>> For the sake of providing additional visibility into this work,
>> >> >>>> here
>> >> >>>> are the issues that I've hit so far.
>> >> >>>>
>> >> >>>> The first issue I ran across is that currently the MOS
>> >> >>>> repositories
>> >> >>>> contain packages for both OpenStack and other system dependencies
>> >> >>>> for
>> >> >>>> creating our HA implementation.  This is problematic when we want
>> >> >>>> to
>> >> >>>> switch out the OpenStack packages but still want the MOS packages
>> >> >>>> for
>> >> >>>> our HA items.  I'm working around this by adding the UCA
>> >> >>>> repository
>> >> >>>> as
>> >> >>>> a higher priorities for deployments.  As such I've run into an
>> >> >>>> issue
>> >> >>>> with the haproxy package that MOS provides vs the upstream Ubuntu
>> >> >>>> package.  To get around this, I've pinned the MOS version for now
>> >> >>>> until I can circle back around and figure out if the difference is
>> >> >>>> a
>> >> >>>> config or functionality issue.
>> >> >>>
>> >> >>>
>> >> >>> I know at least for one difference between the MOS and Ubuntu
>> >> >>> versions:
>> >> >>> HAProxy from MOS has a patch to support the "include" configuration
>> >> >>> parameter.
>> >> >>> This is unrelated to your mail but I think this fork should die
>> >> >>> since
>> >> >>> it
>> >> >>> will never be accepted upstream and there are other ways to address
>> >> >>> the use
>> >> >>> case [0].
>> >> >>>
>> >> >>> HTH,
>> >> >>> Simon
>> >> >>>
>> >> >>> [0] http://marc.info/?l=haproxy&m=130817444025140&w=2
>> >> >>>
>> >> >>>>
>> >> >>>>
>> >> >>>> The second issue that I have run across is that we are appending
>> >> >>>> read_timeout=60 to our mysql connection strings for our
>> >> >>>> configurations. This seems to be unsupported by the libraries and
>> >> >>>> OpenStack components provided by the UCA package set.  I'm not
>> >> >>>> sure
>> >> >>>> how the priority of python-pymsql vs python-mysqldb is resolved as
>> >> >>>> I
>> >> >>>> had both packages installed but it continued to fail on
>> >> >>>> read_timeout
>> >> >>>> being in the connection string.  For now I've updated the
>> >> >>>> fuel-library
>> >> >>>> code to remove this item from the connection strings and will be
>> >> >>>> circling back around to figure out the correct 'fix' for this
>> >> >>>> issue.
>> >> >>>>
>> >> >>>> I'm hoping to be able to have a working ISO, plugin and a set of
>> >> >>>> instructions that can be used to deploy a basic cloud using Fuel
>> >> >>>> by
>> >> >>>> the end of this week.
>> >> >>>>
>> >> >>>> Thanks,
>> >> >>>> -Alex
>> >> >>>>
>> >> >>>> [0] https://review.openstack.org/#/c/240325/
>> >> >>>> [1] https://review.openstack.org/#/c/241615/
>> >> >>>> [2] https://review.openstack.org/#/c/241741/
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> __________________________________________________________________________
>> >> >>>> OpenStack Development Mailing List (not for usage questions)
>> >> >>>> Unsubscribe:
>> >> >>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> __________________________________________________________________________
>> >> >>> OpenStack Development Mailing List (not for usage questions)
>> >> >>> Unsubscribe:
>> >> >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >> >>>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> __________________________________________________________________________
>> >> >> OpenStack Development Mailing List (not for usage questions)
>> >> >> Unsubscribe:
>> >> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Yours Faithfully,
>> >> > Vladimir Kuklin,
>> >> > Fuel Library Tech Lead,
>> >> > Mirantis, Inc.
>> >> > +7 (495) 640-49-04
>> >> > +7 (926) 702-39-68
>> >> > Skype kuklinvv
>> >> > 35bk3, Vorontsovskaya Str.
>> >> > Moscow, Russia,
>> >> > www.mirantis.com
>> >> > www.mirantis.ru
>> >> > vkuklin at mirantis.com
>> >> >
>> >> >
>> >> >
>> >> > __________________________________________________________________________
>> >> > OpenStack Development Mailing List (not for usage questions)
>> >> > Unsubscribe:
>> >> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >> >
>> >>
>> >>
>> >> __________________________________________________________________________
>> >> OpenStack Development Mailing List (not for usage questions)
>> >> Unsubscribe:
>> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> >
>> >
>> >
>> > --
>> > Yours Faithfully,
>> > Vladimir Kuklin,
>> > Fuel Library Tech Lead,
>> > Mirantis, Inc.
>> > +7 (495) 640-49-04
>> > +7 (926) 702-39-68
>> > Skype kuklinvv
>> > 35bk3, Vorontsovskaya Str.
>> > Moscow, Russia,
>> > www.mirantis.com
>> > www.mirantis.ru
>> > vkuklin at mirantis.com
>> >
>> >
>> > __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> --
>
> Kind Regards,
>
> Alexandr Kostrikov,
>
>
> Mirantis, Inc.
>
> 35b/3, Vorontsovskaya St., 109147, Moscow, Russia
>
>
> Tel.: +7 (495) 640-49-04
> Tel.: +7 (925) 716-64-52
>
> Skype: akostrikov_mirantis
>
> E-mail: akostrikov at mirantis.com
>
> www.mirantis.com
> www.mirantis.ru
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>