<div dir="ltr">Alex<div><br></div><div>Thanks for your very detailed answer - it clarified things a bit. So, if you would allow me to rephrase it - you are actually conducting a research on what is the actual gap between our downstream/fork and upstream UCA/puppet-openstack. This seems to be a very promising initiative. Are you going to come up with some external-user readable report soon?</div><div><br></div><div>Also, regarding multiple distros support. I think, we need to come up with an approach of making 'release manager' piece of Nailgun data driven and just allow a user to run any distribution he or she wants. Just register a set of repos with packages and run it. Actually, we already have it - we need to make base-image generation for RPM more flexible and any RPM/DEB based distro should work ok with it.</div><div><br></div><div>The remaining piece is to actually support distro-specific things, e.g. CentOS/RHEL networking configuration, e.g. l23 network stored config puppet providers. But this is a distro-supporter/community burden.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 10, 2015 at 6:25 PM, Alex Schultz <span dir="ltr"><<a href="mailto:aschultz@mirantis.com" target="_blank">aschultz@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey Vladimir,<br>
<span class=""><br>
On Tue, Nov 10, 2015 at 5:56 AM, Vladimir Kuklin <<a href="mailto:vkuklin@mirantis.com">vkuklin@mirantis.com</a>> wrote:<br>
> Alex<br>
><br>
> That's great to hear that. But be aware - making all of the components work<br>
> exactly the way they work within MOS is actually identical to upstreaming<br>
> MOS. We are using some components of different versions to satisfy many<br>
> requirements for our Reference Architecutre implementation. It will not be<br>
> so easy to base them upon existing 3rd party distributions. For example,<br>
> `read timeout` for SQL requests is crucial for our HA as it handles cases<br>
> when an SQL node dies while serving the request. And you will find myriads<br>
> of them. And as we do not control things in upstream, we will always have<br>
> some downstream divergence.<br>
><br>
<br>
</span>Yes, I'm aware that it'll be an effort to make it work identically to<br>
MOS. Currently that's not my goal. My goal is to get it working at<br>
all and be able to document the deficiencies when using upstream<br>
packages/modules vs MOS provided ones. Once we have documented these<br>
differences we will be able to make decisions as to what efforts<br>
should be made if we choose to address the differences. The read<br>
timeout thing seems to be an issue with what mysql python driver is<br>
used so that could easily be configurable based on a packages or a<br>
configuration option.<br>
<span class=""><br>
> I guess, the optimal solution is to figure out the actual divergence between<br>
> upstream and downstream and try to push things upstream as hard as we can,<br>
> while retaining overrides for some packages and manifests on top of upstream<br>
> versions. Do not get me wrong, but it seems there is exactly 0 (zero) ways<br>
> you can get Fuel working with upstream packages unless they support exactly<br>
> the same feature set and fix the same bugs in various components that Fuel<br>
> expect them to support or fix. By 'working' I mean passing the same set of<br>
> at least smoke and destructive tests, let alone passing scale tests.<br>
><br>
<br>
</span>So I think this is where we are currently backwards in the way we're<br>
doings. As we hope to push Fuel as a community project, we need to be<br>
more open to supporting the distributions versions of these packages<br>
as being supported. If we want to continue with specific versions of<br>
things we need to be able to modularize them and apply them to a<br>
downstream version of Fuel that we can promote with the MOS package<br>
set. I agree that right now it's highly unlikely that one would be<br>
able to use Fuel without any MOS packages. Because I don't think it's<br>
possible right now, what I'm attempting to do is be able to deploy an<br>
alternate OpenStack package set but not all of the other pieces as<br>
they relate to our reference architecture. That seems to be a more<br>
achievable goal right now and allows us to document the gaps where<br>
Fuel/MOS are a head (or behind depending on your views) with upstream.<br>
<span class=""><br>
> Simon<br>
><br>
> AFAIK, this patch is really simple and can be easily applied to any version<br>
> of HAProxy. So it is not too hard handle it while it provides sufficient<br>
> benefits to our deployment engineers. If you have any better solution,<br>
> please feel free to share the code with us.<br>
><br>
<br>
</span>Unfortunately the haproxy fork that we're currently running with also<br>
includes our forked version of the puppet haproxy module. I'm not sure<br>
the includes configs patch is really worth carrying the additional<br>
work/forks but I think that is a conversation for another day.<br>
Personally I think it would better if fuel-library supported the<br>
currently available upstream versions of haproxy & haproxy puppet<br>
module and if we want to continue using our copies of haproxy & puppet<br>
haproxy module that we turn that into a downstream that is applied for<br>
the MOS specific version of Fuel. That's the advantage of the<br>
Puppetfile we've been working on. We can change out the modules with<br>
our own version downstream.<br>
<br>
Just to add some additional updates issues that I've run into.<br>
<br>
- nova-vncproxy package names (nova-consoleproxy vs nova-novncproxy)<br>
but that's easily solved with the proposed os_package_type fact.<br>
- heat-docker package does not exist in UCA. Haven't looked at what<br>
exactly is in this package yet.<br>
- horizon UCA package runs into<br>
<a href="https://bugs.launchpad.net/cloud-archive/+bug/1479340" rel="noreferrer" target="_blank">https://bugs.launchpad.net/cloud-archive/+bug/1479340</a> (we solved this<br>
for MOS but it's still broken in UCA)<br>
- neutron UCA package seems to suffer from<br>
<a href="https://bugs.launchpad.net/mos/+bug/1510844" rel="noreferrer" target="_blank">https://bugs.launchpad.net/mos/+bug/1510844</a><br>
- Need to solve for the url issues with the openstack command that are<br>
showing up in the CI as I just hit it with my deployment.<br>
<br>
Thanks,<br>
-Alex<br>
<div class="HOEnZb"><div class="h5"><br>
> On Tue, Nov 10, 2015 at 12:54 PM, Stanislaw Bogatkin<br>
> <<a href="mailto:sbogatkin@mirantis.com">sbogatkin@mirantis.com</a>> wrote:<br>
>><br>
>> Hi Simon,<br>
>><br>
>> using 'include' in HAProxy is damn conveniently, I don't think it should<br>
>> die. There is just one way I know to use haproxy with several different<br>
>> conf's - to construct looooooooooong command line with all of them - and<br>
>> it's really inconvenient.<br>
>><br>
>> On Tue, Nov 10, 2015 at 12:20 PM, Simon Pasquier <<a href="mailto:spasquier@mirantis.com">spasquier@mirantis.com</a>><br>
>> wrote:<br>
>>><br>
>>> Hello Alex!<br>
>>><br>
>>> On Mon, Nov 9, 2015 at 9:29 PM, Alex Schultz <<a href="mailto:aschultz@mirantis.com">aschultz@mirantis.com</a>><br>
>>> wrote:<br>
>>>><br>
>>>> Hey folks,<br>
>>>><br>
>>>> I'm testing[0] out flipping our current method of consuming upstream<br>
>>>> puppet modules from using pinned versions hosted on fuel-infra to be<br>
>>>> able to use the ones directly from upstream (master). This work is<br>
>>>> primarily to be closer aligned with the other OpenStack projects as<br>
>>>> well as switching the current way we manage Fuel into a downstream of<br>
>>>> the upstream community version. As part of this work we have also been<br>
>>>> working towards improving the upstream modules support different<br>
>>>> package sets. Specifically running Debian packages on Ubuntu[1][2].<br>
>>>> This work is the start of being able to allow a user of Fuel to be<br>
>>>> able to specify a specific package set and having it be able to work.<br>
>>>> If we can properly split out the puppet modules and package<br>
>>>> dependencies this will make Fuel a more flexible deployment engine as<br>
>>>> I believe we would be better positioned to support multiple versions<br>
>>>> of OpenStack for a given Fuel release.<br>
>>>><br>
>>>> I'm currently working to get a PoC of Fuel consuming upstream puppet<br>
>>>> modules and the UCA packages together and documenting all of the<br>
>>>> issues so that we can address them. So far I have been able to deploy<br>
>>>> the upstream modules via a custom ISO using the MOS package set and it<br>
>>>> works locally. Unfortunately the CI seems to be hitting some issues<br>
>>>> that I think might be related to recently merged keystone changes but<br>
>>>> I did not run into the same problem when running a manual deployment.<br>
>>>> As I work through this PoC, I'm also attempting to develop a small<br>
>>>> plugin that could be used to capture the work arounds to the<br>
>>>> deployment process. I've run into a few issues so far as I work to<br>
>>>> switch out the package sets.<br>
>>>><br>
>>>> For the sake of providing additional visibility into this work, here<br>
>>>> are the issues that I've hit so far.<br>
>>>><br>
>>>> The first issue I ran across is that currently the MOS repositories<br>
>>>> contain packages for both OpenStack and other system dependencies for<br>
>>>> creating our HA implementation. This is problematic when we want to<br>
>>>> switch out the OpenStack packages but still want the MOS packages for<br>
>>>> our HA items. I'm working around this by adding the UCA repository as<br>
>>>> a higher priorities for deployments. As such I've run into an issue<br>
>>>> with the haproxy package that MOS provides vs the upstream Ubuntu<br>
>>>> package. To get around this, I've pinned the MOS version for now<br>
>>>> until I can circle back around and figure out if the difference is a<br>
>>>> config or functionality issue.<br>
>>><br>
>>><br>
>>> I know at least for one difference between the MOS and Ubuntu versions:<br>
>>> HAProxy from MOS has a patch to support the "include" configuration<br>
>>> parameter.<br>
>>> This is unrelated to your mail but I think this fork should die since it<br>
>>> will never be accepted upstream and there are other ways to address the use<br>
>>> case [0].<br>
>>><br>
>>> HTH,<br>
>>> Simon<br>
>>><br>
>>> [0] <a href="http://marc.info/?l=haproxy&m=130817444025140&w=2" rel="noreferrer" target="_blank">http://marc.info/?l=haproxy&m=130817444025140&w=2</a><br>
>>><br>
>>>><br>
>>>><br>
>>>> The second issue that I have run across is that we are appending<br>
>>>> read_timeout=60 to our mysql connection strings for our<br>
>>>> configurations. This seems to be unsupported by the libraries and<br>
>>>> OpenStack components provided by the UCA package set. I'm not sure<br>
>>>> how the priority of python-pymsql vs python-mysqldb is resolved as I<br>
>>>> had both packages installed but it continued to fail on read_timeout<br>
>>>> being in the connection string. For now I've updated the fuel-library<br>
>>>> code to remove this item from the connection strings and will be<br>
>>>> circling back around to figure out the correct 'fix' for this issue.<br>
>>>><br>
>>>> I'm hoping to be able to have a working ISO, plugin and a set of<br>
>>>> instructions that can be used to deploy a basic cloud using Fuel by<br>
>>>> the end of this week.<br>
>>>><br>
>>>> Thanks,<br>
>>>> -Alex<br>
>>>><br>
>>>> [0] <a href="https://review.openstack.org/#/c/240325/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/240325/</a><br>
>>>> [1] <a href="https://review.openstack.org/#/c/241615/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/241615/</a><br>
>>>> [2] <a href="https://review.openstack.org/#/c/241741/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/241741/</a><br>
>>>><br>
>>>><br>
>>>> __________________________________________________________________________<br>
>>>> OpenStack Development Mailing List (not for usage questions)<br>
>>>> Unsubscribe:<br>
>>>> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>><br>
>>><br>
>>><br>
>>><br>
>>> __________________________________________________________________________<br>
>>> OpenStack Development Mailing List (not for usage questions)<br>
>>> Unsubscribe:<br>
>>> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>><br>
>><br>
>><br>
>> __________________________________________________________________________<br>
>> OpenStack Development Mailing List (not for usage questions)<br>
>> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>><br>
><br>
><br>
><br>
> --<br>
> Yours Faithfully,<br>
> Vladimir Kuklin,<br>
> Fuel Library Tech Lead,<br>
> Mirantis, Inc.<br>
> +7 (495) 640-49-04<br>
> +7 (926) 702-39-68<br>
> Skype kuklinvv<br>
> 35bk3, Vorontsovskaya Str.<br>
> Moscow, Russia,<br>
> <a href="http://www.mirantis.com" rel="noreferrer" target="_blank">www.mirantis.com</a><br>
> <a href="http://www.mirantis.ru" rel="noreferrer" target="_blank">www.mirantis.ru</a><br>
> <a href="mailto:vkuklin@mirantis.com">vkuklin@mirantis.com</a><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr">Yours Faithfully,<br>Vladimir Kuklin,<br>Fuel Library Tech Lead,<br>Mirantis, Inc.<br>+7 (495) 640-49-04<br>+7 (926) 702-39-68<br>Skype kuklinvv<br>35bk3, Vorontsovskaya Str.<br>Moscow, Russia,<br><a href="http://www.mirantis.ru/" target="_blank">www.mirantis.com</a><br><a href="http://www.mirantis.ru/" target="_blank">www.mirantis.ru</a><br><a href="mailto:vkuklin@mirantis.com" target="_blank">vkuklin@mirantis.com</a></div></div></div></div>
</div>