<div dir="ltr">Hi, Alex!<div>Good to know someone is doing that.</div><div><br></div><div>One question about Your initiative:</div><div>Are You going to reimplement all MOS HA features?</div><div>Seems first implementation of 'upstream on Fuel' will be HA-less.</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 12, 2015 at 1:51 AM, Alex Schultz <span dir="ltr"><<a href="mailto:aschultz@mirantis.com" target="_blank">aschultz@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Tue, Nov 10, 2015 at 11:10 AM, Vladimir Kuklin <<a href="mailto:vkuklin@mirantis.com">vkuklin@mirantis.com</a>> wrote:<br>
> Alex<br>
><br>
</span><span class="">> Thanks for your very detailed answer - it clarified things a bit. So, if you<br>
> would allow me to rephrase it - you are actually conducting a research on<br>
> what is the actual gap between our downstream/fork and upstream<br>
> UCA/puppet-openstack. This seems to be a very promising initiative. Are you<br>
> going to come up with some external-user readable report soon?<br>
><br>
> Also, regarding multiple distros support. I think, we need to come up with<br>
> an approach of making 'release manager' piece of Nailgun data driven and<br>
> just allow a user to run any distribution he or she wants. Just register a<br>
> set of repos with packages and run it. Actually, we already have it - we<br>
> need to make base-image generation for RPM more flexible and any RPM/DEB<br>
> based distro should work ok with it.<br>
><br>
> The remaining piece is to actually support distro-specific things, e.g.<br>
> CentOS/RHEL networking configuration, e.g. l23 network stored config puppet<br>
> providers. But this is a distro-supporter/community burden.<br>
><br>
><br>
<br>
</span>Yes I hope to have something together by the end of the week. I've<br>
managed to get a controller and compute/cinder nodes up (and passing<br>
basic OSTF tests) with what appears to be some minor adjustments to<br>
the fuel-library code. The one thing that gets dropped because of<br>
lack of upstream support is nova floating network ranges. There's a<br>
pending review that'll get that back in but I also don't know how<br>
important it would be to support for this type of configuration.<br>
Another issue is the upstream murano module is still a work in<br>
progress so that won't work right now either. Hopefully that'll get<br>
sorted out in time for the official release of the liberty puppet<br>
modules.<br>
<br>
As I've been working through this, I've noticed that it would be<br>
possible to use fuel-plugins to only apply UCA packages to specific<br>
nodes via a plugin role. An interesting follow on to this effort would<br>
be to use MOS packages for controllers and UCA for Compute or vice<br>
versa. But that should probably be more an academic exercise rather<br>
than production one.<br>
<div class="HOEnZb"><div class="h5"><br>
-Alex<br>
<br>
><br>
> On Tue, Nov 10, 2015 at 6:25 PM, Alex Schultz <<a href="mailto:aschultz@mirantis.com">aschultz@mirantis.com</a>> wrote:<br>
>><br>
>> Hey Vladimir,<br>
>><br>
>> On Tue, Nov 10, 2015 at 5:56 AM, Vladimir Kuklin <<a href="mailto:vkuklin@mirantis.com">vkuklin@mirantis.com</a>><br>
>> wrote:<br>
>> > Alex<br>
>> ><br>
>> > That's great to hear that. But be aware - making all of the components<br>
>> > work<br>
>> > exactly the way they work within MOS is actually identical to<br>
>> > upstreaming<br>
>> > MOS. We are using some components of different versions to satisfy many<br>
>> > requirements for our Reference Architecutre implementation. It will not<br>
>> > be<br>
>> > so easy to base them upon existing 3rd party distributions. For example,<br>
>> > `read timeout` for SQL requests is crucial for our HA as it handles<br>
>> > cases<br>
>> > when an SQL node dies while serving the request. And you will find<br>
>> > myriads<br>
>> > of them. And as we do not control things in upstream, we will always<br>
>> > have<br>
>> > some downstream divergence.<br>
>> ><br>
>><br>
>> Yes, I'm aware that it'll be an effort to make it work identically to<br>
>> MOS. Currently that's not my goal. My goal is to get it working at<br>
>> all and be able to document the deficiencies when using upstream<br>
>> packages/modules vs MOS provided ones. Once we have documented these<br>
>> differences we will be able to make decisions as to what efforts<br>
>> should be made if we choose to address the differences. The read<br>
>> timeout thing seems to be an issue with what mysql python driver is<br>
>> used so that could easily be configurable based on a packages or a<br>
>> configuration option.<br>
>><br>
>> > I guess, the optimal solution is to figure out the actual divergence<br>
>> > between<br>
>> > upstream and downstream and try to push things upstream as hard as we<br>
>> > can,<br>
>> > while retaining overrides for some packages and manifests on top of<br>
>> > upstream<br>
>> > versions. Do not get me wrong, but it seems there is exactly 0 (zero)<br>
>> > ways<br>
>> > you can get Fuel working with upstream packages unless they support<br>
>> > exactly<br>
>> > the same feature set and fix the same bugs in various components that<br>
>> > Fuel<br>
>> > expect them to support or fix. By 'working' I mean passing the same set<br>
>> > of<br>
>> > at least smoke and destructive tests, let alone passing scale tests.<br>
>> ><br>
>><br>
>> So I think this is where we are currently backwards in the way we're<br>
>> doings. As we hope to push Fuel as a community project, we need to be<br>
>> more open to supporting the distributions versions of these packages<br>
>> as being supported. If we want to continue with specific versions of<br>
>> things we need to be able to modularize them and apply them to a<br>
>> downstream version of Fuel that we can promote with the MOS package<br>
>> set. I agree that right now it's highly unlikely that one would be<br>
>> able to use Fuel without any MOS packages. Because I don't think it's<br>
>> possible right now, what I'm attempting to do is be able to deploy an<br>
>> alternate OpenStack package set but not all of the other pieces as<br>
>> they relate to our reference architecture. That seems to be a more<br>
>> achievable goal right now and allows us to document the gaps where<br>
>> Fuel/MOS are a head (or behind depending on your views) with upstream.<br>
>><br>
>> > Simon<br>
>> ><br>
>> > AFAIK, this patch is really simple and can be easily applied to any<br>
>> > version<br>
>> > of HAProxy. So it is not too hard handle it while it provides sufficient<br>
>> > benefits to our deployment engineers. If you have any better solution,<br>
>> > please feel free to share the code with us.<br>
>> ><br>
>><br>
>> Unfortunately the haproxy fork that we're currently running with also<br>
>> includes our forked version of the puppet haproxy module. I'm not sure<br>
>> the includes configs patch is really worth carrying the additional<br>
>> work/forks but I think that is a conversation for another day.<br>
>> Personally I think it would better if fuel-library supported the<br>
>> currently available upstream versions of haproxy & haproxy puppet<br>
>> module and if we want to continue using our copies of haproxy & puppet<br>
>> haproxy module that we turn that into a downstream that is applied for<br>
>> the MOS specific version of Fuel. That's the advantage of the<br>
>> Puppetfile we've been working on. We can change out the modules with<br>
>> our own version downstream.<br>
>><br>
>> Just to add some additional updates issues that I've run into.<br>
>><br>
>> - nova-vncproxy package names (nova-consoleproxy vs nova-novncproxy)<br>
>> but that's easily solved with the proposed os_package_type fact.<br>
>> - heat-docker package does not exist in UCA. Haven't looked at what<br>
>> exactly is in this package yet.<br>
>> - horizon UCA package runs into<br>
>> <a href="https://bugs.launchpad.net/cloud-archive/+bug/1479340" rel="noreferrer" target="_blank">https://bugs.launchpad.net/cloud-archive/+bug/1479340</a> (we solved this<br>
>> for MOS but it's still broken in UCA)<br>
>> - neutron UCA package seems to suffer from<br>
>> <a href="https://bugs.launchpad.net/mos/+bug/1510844" rel="noreferrer" target="_blank">https://bugs.launchpad.net/mos/+bug/1510844</a><br>
>> - Need to solve for the url issues with the openstack command that are<br>
>> showing up in the CI as I just hit it with my deployment.<br>
>><br>
>> Thanks,<br>
>> -Alex<br>
>><br>
>> > On Tue, Nov 10, 2015 at 12:54 PM, Stanislaw Bogatkin<br>
>> > <<a href="mailto:sbogatkin@mirantis.com">sbogatkin@mirantis.com</a>> wrote:<br>
>> >><br>
>> >> Hi Simon,<br>
>> >><br>
>> >> using 'include' in HAProxy is damn conveniently, I don't think it<br>
>> >> should<br>
>> >> die. There is just one way I know to use haproxy with several different<br>
>> >> conf's - to construct looooooooooong command line with all of them -<br>
>> >> and<br>
>> >> it's really inconvenient.<br>
>> >><br>
>> >> On Tue, Nov 10, 2015 at 12:20 PM, Simon Pasquier<br>
>> >> <<a href="mailto:spasquier@mirantis.com">spasquier@mirantis.com</a>><br>
>> >> wrote:<br>
>> >>><br>
>> >>> Hello Alex!<br>
>> >>><br>
>> >>> On Mon, Nov 9, 2015 at 9:29 PM, Alex Schultz <<a href="mailto:aschultz@mirantis.com">aschultz@mirantis.com</a>><br>
>> >>> wrote:<br>
>> >>>><br>
>> >>>> Hey folks,<br>
>> >>>><br>
>> >>>> I'm testing[0] out flipping our current method of consuming upstream<br>
>> >>>> puppet modules from using pinned versions hosted on fuel-infra to be<br>
>> >>>> able to use the ones directly from upstream (master). This work is<br>
>> >>>> primarily to be closer aligned with the other OpenStack projects as<br>
>> >>>> well as switching the current way we manage Fuel into a downstream of<br>
>> >>>> the upstream community version. As part of this work we have also<br>
>> >>>> been<br>
>> >>>> working towards improving the upstream modules support different<br>
>> >>>> package sets. Specifically running Debian packages on Ubuntu[1][2].<br>
>> >>>> This work is the start of being able to allow a user of Fuel to be<br>
>> >>>> able to specify a specific package set and having it be able to work.<br>
>> >>>> If we can properly split out the puppet modules and package<br>
>> >>>> dependencies this will make Fuel a more flexible deployment engine as<br>
>> >>>> I believe we would be better positioned to support multiple versions<br>
>> >>>> of OpenStack for a given Fuel release.<br>
>> >>>><br>
>> >>>> I'm currently working to get a PoC of Fuel consuming upstream puppet<br>
>> >>>> modules and the UCA packages together and documenting all of the<br>
>> >>>> issues so that we can address them. So far I have been able to<br>
>> >>>> deploy<br>
>> >>>> the upstream modules via a custom ISO using the MOS package set and<br>
>> >>>> it<br>
>> >>>> works locally. Unfortunately the CI seems to be hitting some issues<br>
>> >>>> that I think might be related to recently merged keystone changes but<br>
>> >>>> I did not run into the same problem when running a manual deployment.<br>
>> >>>> As I work through this PoC, I'm also attempting to develop a small<br>
>> >>>> plugin that could be used to capture the work arounds to the<br>
>> >>>> deployment process. I've run into a few issues so far as I work to<br>
>> >>>> switch out the package sets.<br>
>> >>>><br>
>> >>>> For the sake of providing additional visibility into this work, here<br>
>> >>>> are the issues that I've hit so far.<br>
>> >>>><br>
>> >>>> The first issue I ran across is that currently the MOS repositories<br>
>> >>>> contain packages for both OpenStack and other system dependencies for<br>
>> >>>> creating our HA implementation. This is problematic when we want to<br>
>> >>>> switch out the OpenStack packages but still want the MOS packages for<br>
>> >>>> our HA items. I'm working around this by adding the UCA repository<br>
>> >>>> as<br>
>> >>>> a higher priorities for deployments. As such I've run into an issue<br>
>> >>>> with the haproxy package that MOS provides vs the upstream Ubuntu<br>
>> >>>> package. To get around this, I've pinned the MOS version for now<br>
>> >>>> until I can circle back around and figure out if the difference is a<br>
>> >>>> config or functionality issue.<br>
>> >>><br>
>> >>><br>
>> >>> I know at least for one difference between the MOS and Ubuntu<br>
>> >>> versions:<br>
>> >>> HAProxy from MOS has a patch to support the "include" configuration<br>
>> >>> parameter.<br>
>> >>> This is unrelated to your mail but I think this fork should die since<br>
>> >>> it<br>
>> >>> will never be accepted upstream and there are other ways to address<br>
>> >>> the use<br>
>> >>> case [0].<br>
>> >>><br>
>> >>> HTH,<br>
>> >>> Simon<br>
>> >>><br>
>> >>> [0] <a href="http://marc.info/?l=haproxy&m=130817444025140&w=2" rel="noreferrer" target="_blank">http://marc.info/?l=haproxy&m=130817444025140&w=2</a><br>
>> >>><br>
>> >>>><br>
>> >>>><br>
>> >>>> The second issue that I have run across is that we are appending<br>
>> >>>> read_timeout=60 to our mysql connection strings for our<br>
>> >>>> configurations. This seems to be unsupported by the libraries and<br>
>> >>>> OpenStack components provided by the UCA package set. I'm not sure<br>
>> >>>> how the priority of python-pymsql vs python-mysqldb is resolved as I<br>
>> >>>> had both packages installed but it continued to fail on read_timeout<br>
>> >>>> being in the connection string. For now I've updated the<br>
>> >>>> fuel-library<br>
>> >>>> code to remove this item from the connection strings and will be<br>
>> >>>> circling back around to figure out the correct 'fix' for this issue.<br>
>> >>>><br>
>> >>>> I'm hoping to be able to have a working ISO, plugin and a set of<br>
>> >>>> instructions that can be used to deploy a basic cloud using Fuel by<br>
>> >>>> the end of this week.<br>
>> >>>><br>
>> >>>> Thanks,<br>
>> >>>> -Alex<br>
>> >>>><br>
>> >>>> [0] <a href="https://review.openstack.org/#/c/240325/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/240325/</a><br>
>> >>>> [1] <a href="https://review.openstack.org/#/c/241615/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/241615/</a><br>
>> >>>> [2] <a href="https://review.openstack.org/#/c/241741/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/241741/</a><br>
>> >>>><br>
>> >>>><br>
>> >>>><br>
>> >>>> __________________________________________________________________________<br>
>> >>>> OpenStack Development Mailing List (not for usage questions)<br>
>> >>>> Unsubscribe:<br>
>> >>>> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>> >>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>> >>><br>
>> >>><br>
>> >>><br>
>> >>><br>
>> >>><br>
>> >>> __________________________________________________________________________<br>
>> >>> OpenStack Development Mailing List (not for usage questions)<br>
>> >>> Unsubscribe:<br>
>> >>> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>> >>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>> >>><br>
>> >><br>
>> >><br>
>> >><br>
>> >> __________________________________________________________________________<br>
>> >> OpenStack Development Mailing List (not for usage questions)<br>
>> >> Unsubscribe:<br>
>> >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>> >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>> >><br>
>> ><br>
>> ><br>
>> ><br>
>> > --<br>
>> > Yours Faithfully,<br>
>> > Vladimir Kuklin,<br>
>> > Fuel Library Tech Lead,<br>
>> > Mirantis, Inc.<br>
>> > +7 (495) 640-49-04<br>
>> > +7 (926) 702-39-68<br>
>> > Skype kuklinvv<br>
>> > 35bk3, Vorontsovskaya Str.<br>
>> > Moscow, Russia,<br>
>> > <a href="http://www.mirantis.com" rel="noreferrer" target="_blank">www.mirantis.com</a><br>
>> > <a href="http://www.mirantis.ru" rel="noreferrer" target="_blank">www.mirantis.ru</a><br>
>> > <a href="mailto:vkuklin@mirantis.com">vkuklin@mirantis.com</a><br>
>> ><br>
>> ><br>
>> > __________________________________________________________________________<br>
>> > OpenStack Development Mailing List (not for usage questions)<br>
>> > Unsubscribe:<br>
>> > <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>> ><br>
>><br>
>> __________________________________________________________________________<br>
>> OpenStack Development Mailing List (not for usage questions)<br>
>> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
><br>
> --<br>
> Yours Faithfully,<br>
> Vladimir Kuklin,<br>
> Fuel Library Tech Lead,<br>
> Mirantis, Inc.<br>
> +7 (495) 640-49-04<br>
> +7 (926) 702-39-68<br>
> Skype kuklinvv<br>
> 35bk3, Vorontsovskaya Str.<br>
> Moscow, Russia,<br>
> <a href="http://www.mirantis.com" rel="noreferrer" target="_blank">www.mirantis.com</a><br>
> <a href="http://www.mirantis.ru" rel="noreferrer" target="_blank">www.mirantis.ru</a><br>
> <a href="mailto:vkuklin@mirantis.com">vkuklin@mirantis.com</a><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><p style="margin-bottom:0.0001pt"><span style="font-size:10pt;font-family:Arial,sans-serif" lang="EN-US">Kind Regards,<br></span></p><p style="margin-bottom:0.0001pt"><span style="font-size:10pt;font-family:Arial,sans-serif" lang="EN-US">Alexandr Kostrikov,<br></span></p><span style="font-size:10pt;font-family:Arial,sans-serif" lang="EN-US"><br>Mirantis, Inc.</span><span style="font-family:Arial,sans-serif;color:rgb(31,73,125)" lang="EN-US"></span>
<p style="margin:0cm 0cm 0.0001pt"><span style="font-size:10pt;font-family:Arial,sans-serif" lang="EN-US">35b/3, Vorontsovskaya
St., 109147, Moscow, Russia</span><span style="font-family:Arial,sans-serif;color:rgb(31,73,125)" lang="EN-US"></span></p>
<p style="margin:0cm 0cm 0.0001pt"><span style="font-size:10pt;font-family:Arial,sans-serif" lang="EN-US"><br>
Tel.: <a href="tel:%2B7%20%28495%29%20640-49-04" value="+74956404904" target="_blank">+7 (495) 640-49-04</a><br>
Tel.: <a href="tel:%2B7%20%28906%29%20740-64-79" value="+79067406479" target="_blank">+7 (925) 716-64-52</a></span><span style="font-size:10pt;font-family:Arial,sans-serif" lang="EN-US"></span></p>
<p style="margin:0cm 0cm 0.0001pt"><span style="font-size:10pt;font-family:Arial,sans-serif" lang="EN-US">Skype: akostrikov_mirantis</span></p>
<p style="margin:0cm 0cm 0.0001pt"><span style="font-size:10pt;font-family:Arial,sans-serif" lang="EN-US">E-mail:<span> </span></span><span style="font-size:10pt;font-family:Arial,sans-serif"><a href="mailto:elogutova@mirantis.com" target="_blank"><span style="color:rgb(17,85,204)" lang="EN-US"><span><span><span>akostrikov@mirantis.com</span></span></span></span></a></span><span style="font-family:Arial,sans-serif" lang="EN-US"></span></p>
<p style="margin-bottom:0.0001pt"><u><span style="font-size:10pt;font-family:Arial,sans-serif"><a href="http://www.mirantis.ru/" target="_blank"><span style="color:rgb(17,85,204)" lang="EN-US">www.mirantis.com</span></a></span></u><u><span style="font-size:10pt;font-family:Arial,sans-serif" lang="EN-US"><br>
</span></u><u><span style="font-size:10pt;font-family:Arial,sans-serif"><a href="http://www.mirantis.ru/" target="_blank"><span style="color:rgb(17,85,204)" lang="EN-US">www.mirantis.ru</span></a></span></u><span style="font-family:Arial,sans-serif" lang="EN-US"></span></p></div></div>
</div>