Open Stack

Hi,

Zane Bitter <zbitter at redhat.com> writes:

> On 17/02/18 16:40, Dan Prince wrote:
>> Thanks for the update Emilien. A couple of things to add:
>> 
>> 1) This was really difficult to pin-point via the Heat stack error
>> message ('list index out of range'). I actually had to go and add
>> LOG.debug statements to Heat to get to the bottom of it. I aim to sync
>> with a few of the Heat folks next week on this to see if we can do
>> better here.
>
> The message itself is pretty much all we get from yaql, even in its own 
> interpreter:
>
> (py27) cat yaql_data.json
> {"data": [{"foo": "bar"}]}
> (py27) yaql -d yaql_data.json
> Yet Another Query Language - command-line query tool
> Version 1.1.3
> Copyright (c) 2013-2017 Mirantis, Inc
>
> yaql> dict($.data.where($ != 
> null).flatten().selectMany($.items()).groupBy($[0], $[1], $.flatten()))
> {
>      "foo": [
>          "bar"
>      ]
> }
> yaql> dict($.data.where($ != 
> null).flatten().selectMany($.items()).groupBy($[0], $[1], [$[0], 
> $[1].flatten()]))
> Execution exception: list index out of range
> yaql>
>
> (Note that different lengths of data will give you different errors though.)
>
> The big issue here though is that for failures in validation we report 
> the path in the template to the function that failed, but we don't do 
> the same for failures in actually resolving the function at runtime. A 
> comprehensive fix is challenging without breaking what is supposed to be 
> a stable third-party plugin API, but it might be possible. Was that the 
> information you needed to debug this?
>
> We do report which resource failed, but for something with a huge 
> definition like allNodesConfig I can see why that might not help as much 
> as you'd hope.
>
>> 2) I had initially thought it would have been much better to revert
>> the (breaking) change to python-yaql. That said it was from 2016! So I
>> think our window of opportunity for the revert is probably way too
>> large to consider that. Sounds like we need to publish the yaql
>> package more often in RDO, etc. So your patch to update our queries is
>> probably our only option.
>
> I _think_ this should be OK for upgrades, as long as you never do a 
> stack update using the existing (Pike) templates after upgrading the 
> undercloud to Queens, but... sadface.

So as can be seen there[1] during a P->M upgrade this is not safe for
upgrade :)  Beside the fact that it breaks all hope of having any kind
of mixed upgrade CI testing (where the undercloud is N and expected to
deploy overcloud N-1) it breaks mixed version operation as well. 

[1] http://logs.openstack.org/62/545762/3/experimental/tripleo-ci-centos-7-scenario001-multinode-oc-upgrade/afc98a5/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-02-19_09_48_54


> I think we need to either merge Thomas's patch that gets rid of this 
> function altogether (https://review.openstack.org/#/c/545856/)

I'm currently testing the pike's backport of this[2] in the experimental
pipeline.  I'll report inside the review.

[2] https://review.openstack.org/#/c/546094/

>>> On Mon, Feb 19, 2018 at 03:24:24PM +0100, Bogdan Dobrelya wrote:
>>>
>>> With a backport of the YAQL fixes for tht made for Pike, would it be the
>>> full fix to make a backport of yaql 1.1.3 for Pike repos as well? Or am I
>>> missing something?
>>
>> At some level that should be fine.  In the broader OpenSteck perspective
>> we've been gating with yaql 1.1.3 from pypi for releases > newton.  So
>> while updating the yaql package on pike to 1.1.3 isn't guaranteed to be
>> safe it should be fine.

So we fast forward upgrade, we need to make sure that master undercloud
is able to deploy the newton templates for CI testing and support of
some mixed version operations.

So if Thomas' patch enables us to support both yaql version that would
be ideal from an upgrade perspective.  It will need backport all the way
to newton.

> backport it to older versions of t-h-t, or make yaql itself 
> backward-compatible by doing something like 
> https://review.openstack.org/#/c/545996/
>
> cheers,
> Zane.
>
>> On Fri, Feb 16, 2018 at 8:36 PM, Emilien Macchi <emilien at redhat.com> wrote:
>>> Upgrading YAQL from 1.1.0 to 1.1.3 breaks advanced queries with groupBy
>>> aggregation.
>>>
>>> The commit that broke it is
>>> https://github.com/openstack/yaql/commit/3fb91784018de335440b01b3b069fe45dc53e025
>>>
>>> It broke TripleO: https://bugs.launchpad.net/tripleo/+bug/1750032
>>> But Alex and I figured (after a strong headache) that we needed to update
>>> the query like this: https://review.openstack.org/545498
>>>
>>> It would be great to avoid this kind of change within minor versions, please
>>> please.
>>>
>>> Happy weekend,
>>>
>>> PS: I'm adding YAQL to my linkedin profile right now.
>> 
>> Be careful here. Do you really want to write YAQL queries all day!
>> 
>> Dan
>> 
>>> --
>>> Emilien Macchi
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>> 
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-- 
Sofer