[requirements][mistral] Add yamlloader to global requirements
Hi! Can you please add yamlloader library to global requirements? https://pypi.org/project/yamlloader/
It provides ability to preserve key order in dicts, it supports either python 2.7 and python 3.x, it provides better performance than built-in functions. Thank you.
Yes, it would help us implement a useful feature.
Thanks
Renat Akhmerov @Nokia On 14 Feb 2019, 21:02 +0700, Oleg Ovcharuk vgvoleg@gmail.com, wrote:
Hi! Can you please add yamlloader library to global requirements? https://pypi.org/project/yamlloader/
It provides ability to preserve key order in dicts, it supports either python 2.7 and python 3.x, it provides better performance than built-in functions. Thank you.
Anyone can propose to add a new library to global-requirements. Open a review in the requirements repo[1] and make sure the commit message answers the questions in the docs[2].
1: http://git.openstack.org/cgit/openstack/requirements/ 2: https://docs.openstack.org/project-team-guide/dependency-management.html#rev...
On 2/14/19 9:10 AM, Renat Akhmerov wrote:
Yes, it would help us implement a useful feature.
Thanks
Renat Akhmerov @Nokia On 14 Feb 2019, 21:02 +0700, Oleg Ovcharuk vgvoleg@gmail.com, wrote:
Hi! Can you please add yamlloader library to global requirements? https://pypi.org/project/yamlloader/
It provides ability to preserve key order in dicts, it supports either python 2.7 and python 3.x, it provides better performance than built-in functions. Thank you.
On 19-02-14 16:58:49, Oleg Ovcharuk wrote:
Hi! Can you please add yamlloader library to global requirements? https://pypi.org/project/yamlloader/
It provides ability to preserve key order in dicts, it supports either python 2.7 and python 3.x, it provides better performance than built-in functions. Thank you.
I'd like to know a little more about why we need this, yaml as a spec itself doesn't guarantee order so order should be stored somewhere else.
If all you need is ordereddict support something like this may be better then adding yet another lib.
https://gist.github.com/enaeseth/844388
Matthew, we use not only load, but also dump. We can't use custom constructor and default representer - the output will be terrible. This custom constructor contains about 50 lines of code, representer would have a similar count. Also, we should think about compatibility with Python 2.7, 3.x and about it's performance. Summary, we would have about 150 lines of code, which is just copy-paste from `yamlloader` library. IMHO, it is better to use existing solutions.
чт, 14 февр. 2019 г. в 19:14, Matthew Thode mthode@mthode.org:
On 19-02-14 16:58:49, Oleg Ovcharuk wrote:
Hi! Can you please add yamlloader library to global requirements? https://pypi.org/project/yamlloader/
It provides ability to preserve key order in dicts, it supports either python 2.7 and python 3.x, it provides better performance than built-in functions. Thank you.
I'd like to know a little more about why we need this, yaml as a spec itself doesn't guarantee order so order should be stored somewhere else.
If all you need is ordereddict support something like this may be better then adding yet another lib.
https://gist.github.com/enaeseth/844388
-- Matthew Thode
On 2/14/19 10:46 AM, Oleg Ovcharuk wrote:
Matthew, we use not only load, but also dump. We can't use custom constructor and default representer - the output will be terrible. This custom constructor contains about 50 lines of code, representer would have a similar count. Also, we should think about compatibility with Python 2.7, 3.x and about it's performance. Summary, we would have about 150 lines of code, which is just copy-paste from `yamlloader` library. IMHO, it is better to use existing solutions.
You don't need a complex representer to dump OrderedDicts. It can be done in about three lines: https://github.com/cybertron/tripleo-scripts/blob/105381d4f080394e68a40327c3...
That's the code I used when I wanted to dump dicts in a particular order. Once you add the representer OrderedDicts are handled as you would expect.
чт, 14 февр. 2019 г. в 19:14, Matthew Thode <mthode@mthode.org mailto:mthode@mthode.org>:
On 19-02-14 16:58:49, Oleg Ovcharuk wrote: > Hi! Can you please add yamlloader library to global requirements? > https://pypi.org/project/yamlloader/ > > It provides ability to preserve key order in dicts, it supports either > python 2.7 and python 3.x, it provides better performance than built-in > functions. > Thank you. I'd like to know a little more about why we need this, yaml as a spec itself doesn't guarantee order so order should be stored somewhere else. If all you need is ordereddict support something like this may be better then adding yet another lib. https://gist.github.com/enaeseth/844388 -- Matthew Thode
On 19-02-14 11:15:59, Ben Nemec wrote:
On 2/14/19 10:46 AM, Oleg Ovcharuk wrote:
Matthew, we use not only load, but also dump. We can't use custom constructor and default representer - the output will be terrible. This custom constructor contains about 50 lines of code, representer would have a similar count. Also, we should think about compatibility with Python 2.7, 3.x and about it's performance. Summary, we would have about 150 lines of code, which is just copy-paste from `yamlloader` library. IMHO, it is better to use existing solutions.
You don't need a complex representer to dump OrderedDicts. It can be done in about three lines: https://github.com/cybertron/tripleo-scripts/blob/105381d4f080394e68a40327c3...
That's the code I used when I wanted to dump dicts in a particular order. Once you add the representer OrderedDicts are handled as you would expect.
чт, 14 февр. 2019 г. в 19:14, Matthew Thode <mthode@mthode.org mailto:mthode@mthode.org>:
On 19-02-14 16:58:49, Oleg Ovcharuk wrote: > Hi! Can you please add yamlloader library to global requirements? > https://pypi.org/project/yamlloader/ > > It provides ability to preserve key order in dicts, it supports either > python 2.7 and python 3.x, it provides better performance than built-in > functions. > Thank you. I'd like to know a little more about why we need this, yaml as a spec itself doesn't guarantee order so order should be stored somewhere else. If all you need is ordereddict support something like this may be better then adding yet another lib. https://gist.github.com/enaeseth/844388 -- Matthew Thode
Thanks for this, hope this can be used instead of adding yet another lib to track
On 2019-02-14 11:15:59 -0600 (-0600), Ben Nemec wrote: [...]
You don't need a complex representer to dump OrderedDicts. It can be done in about three lines
[...]
What's more, once we're exclusively on Python >=3.6 (not that much longer, really!) ordering is guaranteed by the normal dict type with no special handling required.
This looks like a solution, yes. Thanks.
@Oleg, please take a look. Maybe we really decided to add a new yaml lib too early.
Renat On 15 Feb 2019, 00:16 +0700, Ben Nemec openstack@nemebean.com, wrote:
On 2/14/19 10:46 AM, Oleg Ovcharuk wrote:
Matthew, we use not only load, but also dump. We can't use custom constructor and default representer - the output will be terrible. This custom constructor contains about 50 lines of code, representer would have a similar count. Also, we should think about compatibility with Python 2.7, 3.x and about it's performance. Summary, we would have about 150 lines of code, which is just copy-paste from `yamlloader` library. IMHO, it is better to use existing solutions.
You don't need a complex representer to dump OrderedDicts. It can be done in about three lines: https://github.com/cybertron/tripleo-scripts/blob/105381d4f080394e68a40327c3...
That's the code I used when I wanted to dump dicts in a particular order. Once you add the representer OrderedDicts are handled as you would expect.
чт, 14 февр. 2019 г. в 19:14, Matthew Thode <mthode@mthode.org mailto:mthode@mthode.org>:
On 19-02-14 16:58:49, Oleg Ovcharuk wrote:
Hi! Can you please add yamlloader library to global requirements? https://pypi.org/project/yamlloader/
It provides ability to preserve key order in dicts, it supports
either
python 2.7 and python 3.x, it provides better performance than
built-in
functions. Thank you.
I'd like to know a little more about why we need this, yaml as a spec itself doesn't guarantee order so order should be stored somewhere else.
If all you need is ordereddict support something like this may be better then adding yet another lib.
https://gist.github.com/enaeseth/844388
-- Matthew Thode
I’ve checked this solution. It doesn’t work as well in Python 2.7 - dumped yaml includes this ugly parts "!!python/object/apply:collections.OrderedDict». Yes, it could be fixed by including another hundred lines of code, but it still smells like we just copy and paste existing solution.
15 февр. 2019 г., в 8:28, Renat Akhmerov renat.akhmerov@gmail.com написал(а):
This looks like a solution, yes. Thanks.
@Oleg, please take a look. Maybe we really decided to add a new yaml lib too early.
Renat On 15 Feb 2019, 00:16 +0700, Ben Nemec openstack@nemebean.com, wrote:
On 2/14/19 10:46 AM, Oleg Ovcharuk wrote:
Matthew, we use not only load, but also dump. We can't use custom constructor and default representer - the output will be terrible. This custom constructor contains about 50 lines of code, representer would have a similar count. Also, we should think about compatibility with Python 2.7, 3.x and about it's performance. Summary, we would have about 150 lines of code, which is just copy-paste from `yamlloader` library. IMHO, it is better to use existing solutions.
You don't need a complex representer to dump OrderedDicts. It can be done in about three lines: https://github.com/cybertron/tripleo-scripts/blob/105381d4f080394e68a40327c3...
That's the code I used when I wanted to dump dicts in a particular order. Once you add the representer OrderedDicts are handled as you would expect.
чт, 14 февр. 2019 г. в 19:14, Matthew Thode <mthode@mthode.org mailto:mthode@mthode.org>:
On 19-02-14 16:58:49, Oleg Ovcharuk wrote:
Hi! Can you please add yamlloader library to global requirements? https://pypi.org/project/yamlloader/
It provides ability to preserve key order in dicts, it supports
either
python 2.7 and python 3.x, it provides better performance than
built-in
functions. Thank you.
I'd like to know a little more about why we need this, yaml as a spec itself doesn't guarantee order so order should be stored somewhere else.
If all you need is ordereddict support something like this may be better then adding yet another lib.
https://gist.github.com/enaeseth/844388
-- Matthew Thode
On 2019-03-28 00:56:51 +0300 (+0300), Oleg Ovcharuk wrote:
I’ve checked this solution. It doesn’t work as well in Python 2.7
- dumped yaml includes this ugly parts
"!!python/object/apply:collections.OrderedDict». Yes, it could be fixed by including another hundred lines of code, but it still smells like we just copy and paste existing solution.
[...]
I don't really have an opinion either way, but if support for implicitly ordering associative arrays with normal dict objects in Python 2.7 is the main concern, keep in mind that whatever compatibility code you include for it only needs to be maintained one more cycle. After that we should be able to drop support for Python<3.6 and so get implicit ordering for free (first by chance in 3.6, but guaranteed by specification in 3.7 onward).
On Thu, Mar 28, 2019 at 12:56:51AM +0300, Oleg Ovcharuk wrote:
I’ve checked this solution. It doesn’t work as well in Python 2.7 - dumped yaml includes this ugly parts "!!python/object/apply:collections.OrderedDict». Yes, it could be fixed by including another hundred lines of code, but it still smells like we just copy and paste existing solution.
Can you supply a link to your change. I use OrderedDict in both python2 and 3 and while I recall that issue I also know I fixed it and I don't recall writing 'another hundred lines of code'
Yours Tony.
On 27/03/19 5:56 PM, Oleg Ovcharuk wrote:
I’ve checked this solution. It doesn’t work as well in Python 2.7 - dumped yaml includes this ugly parts "!!python/object/apply:collections.OrderedDict».
It looks like you accidentally used yaml.dump() instead of yaml.safe_dump(), after adding the representer to only the SafeDumper class.
Yes, it could be fixed by including another hundred lines of code, but it still smells like we just copy and paste existing solution.
I've implemented this myself a couple of times and it is exactly 5 lines of code:
https://github.com/openstack/tripleo-heat-templates/blob/master/network/endp... https://github.com/openstack/heat/blob/master/heat/common/template_format.py...
Loading is more complicated though because you have to reimplement construct_mapping for maximum efficiency. The simplest version I can come up with is 9 lines:
class OrderedLoader(yaml.SafeLoader): def construct_yaml_map_ordered(self, node): data = collections.OrderedDict() yield data value = self.construct_mapping(node) keys = (self.construct_object(kn) for kn, vn in node.value) data.update((k, value[k]) for k in keys)
OrderedLoader.add_constructor(u'tag:yaml.org,2002:map', OrderedLoader.construct_yaml_map_ordered)
but it obviously has the disadvantage of having to call construct_object() twice on each key.
cheers, Zane.
15 февр. 2019 г., в 8:28, Renat Akhmerov <renat.akhmerov@gmail.com mailto:renat.akhmerov@gmail.com> написал(а):
This looks like a solution, yes. Thanks.
@Oleg, please take a look. Maybe we really decided to add a new yaml lib too early.
Renat On 15 Feb 2019, 00:16 +0700, Ben Nemec <openstack@nemebean.com mailto:openstack@nemebean.com>, wrote:
On 2/14/19 10:46 AM, Oleg Ovcharuk wrote:
Matthew, we use not only load, but also dump. We can't use custom constructor and default representer - the output will be terrible. This custom constructor contains about 50 lines of code, representer would have a similar count. Also, we should think about compatibility with Python 2.7, 3.x and about it's performance. Summary, we would have about 150 lines of code, which is just copy-paste from `yamlloader` library. IMHO, it is better to use existing solutions.
You don't need a complex representer to dump OrderedDicts. It can be done in about three lines: https://github.com/cybertron/tripleo-scripts/blob/105381d4f080394e68a40327c3...
That's the code I used when I wanted to dump dicts in a particular order. Once you add the representer OrderedDicts are handled as you would expect.
чт, 14 февр. 2019 г. в 19:14, Matthew Thode <mthode@mthode.org mailto:mthode@mthode.org mailto:mthode@mthode.org>:
On 19-02-14 16:58:49, Oleg Ovcharuk wrote:
Hi! Can you please add yamlloader library to global requirements? https://pypi.org/project/yamlloader/
It provides ability to preserve key order in dicts, it supports
either
python 2.7 and python 3.x, it provides better performance than
built-in
functions. Thank you.
I'd like to know a little more about why we need this, yaml as a spec itself doesn't guarantee order so order should be stored somewhere else.
If all you need is ordereddict support something like this may be better then adding yet another lib.
https://gist.github.com/enaeseth/844388
-- Matthew Thode
чт, 14 февр. 2019 г. в 18:15, Matthew Thode mthode@mthode.org:
On 19-02-14 16:58:49, Oleg Ovcharuk wrote:
Hi! Can you please add yamlloader library to global requirements? https://pypi.org/project/yamlloader/
It provides ability to preserve key order in dicts, it supports either python 2.7 and python 3.x, it provides better performance than built-in functions. Thank you.
I'd like to know a little more about why we need this, yaml as a spec itself doesn't guarantee order so order should be stored somewhere else.
If all you need is ordereddict support something like this may be better then adding yet another lib.
<trolling>
<joke> > this may be better then adding yet another lib If someone raised this question before, probably, we would have less oslo libs. </joke> </trolling>
https://gist.github.com/enaeseth/844388
-- Matthew Thode
participants (8)
-
Andrey Kurilin
-
Ben Nemec
-
Jeremy Stanley
-
Matthew Thode
-
Oleg Ovcharuk
-
Renat Akhmerov
-
Tony Breeds
-
Zane Bitter