[charms] Zaza bundle tests
I was writing a patch recently and in order to test it, I needed to make changes to the test bundles. I ended up making the same change across several files, and missing one (thanks to the reviewer for noticing that!).
Some of the other projects I'm involved with use symlinks to a base bundle with overlays:
./tests$ ls -l bundles/ total 20 -rw-rw---- 2 xav xav 5046 Sep 29 14:56 base.yaml lrwxrwxrwx 1 xav xav 9 Sep 29 10:01 bionic.yaml -> base.yaml lrwxrwxrwx 1 xav xav 9 Sep 29 10:01 focal.yaml -> base.yaml drwxrwx--x 2 xav xav 4096 Sep 29 10:01 overlays lrwxrwxrwx 1 xav xav 9 Sep 29 10:01 xenial.yaml -> base.yaml ./tests$ ls bundles/overlays/ bionic.yaml.j2 focal.yaml.j2 local-charm-overlay.yaml.j2 xenial.yaml.j2
This means that I can edit base.yaml just once, and if a change is specific to any of the particular bundles there's a place for that in the individual overlays. When we have bundles for each release going back to Mitaka, this could be quite an effort saver.
I gather there's been some discussion around this already, and would be keen to see if there's a reason folks might avoid that pattern?
On 04-10-2020 08:54, Xav Paice wrote:
I was writing a patch recently and in order to test it, I needed to make changes to the test bundles. I ended up making the same change across several files, and missing one (thanks to the reviewer for noticing that!).
Some of the other projects I'm involved with use symlinks to a base bundle with overlays:
-- snip --
This means that I can edit base.yaml just once, and if a change is specific to any of the particular bundles there's a place for that in the individual overlays. When we have bundles for each release going back to Mitaka, this could be quite an effort saver.
I'd be quite interested in seeing where this could go, as there is a lot of duplication in the charms' test code that could probably be dramatically reduced by taking this approach! Could you propose a change to one of the repos as an example that we could functionally validate, as well as confirming the assumption that the only differences between the bundles is the series, and openstack-origin/source configs?
Chris MacNaughton
absolutely - could do designate, which I'm working on now anyway, and has a large number of bundles. You're right, an illustration and something to check against would be helpful.
On Mon, 5 Oct 2020 at 19:04, Chris MacNaughton < chris.macnaughton@canonical.com> wrote:
On 04-10-2020 08:54, Xav Paice wrote:
I was writing a patch recently and in order to test it, I needed to make changes to the test bundles. I ended up making the same change across several files, and missing one (thanks to the reviewer for noticing that!).
Some of the other projects I'm involved with use symlinks to a base bundle with overlays:
-- snip --
This means that I can edit base.yaml just once, and if a change is specific to any of the particular bundles there's a place for that in the individual overlays. When we have bundles for each release going back to Mitaka, this could be quite an effort saver.
I'd be quite interested in seeing where this could go, as there is a lot of duplication in the charms' test code that could probably be dramatically reduced by taking this approach! Could you propose a change to one of the repos as an example that we could functionally validate, as well as confirming the assumption that the only differences between the bundles is the series, and openstack-origin/source configs?
Chris MacNaughton
On Mon, 5 Oct 2020 at 19:04, Chris MacNaughton < chris.macnaughton@canonical.com> wrote:
On 04-10-2020 08:54, Xav Paice wrote:
I was writing a patch recently and in order to test it, I needed to make changes to the test bundles. I ended up making the same change across several files, and missing one (thanks to the reviewer for noticing that!).
Some of the other projects I'm involved with use symlinks to a base bundle with overlays:
-- snip --
This means that I can edit base.yaml just once, and if a change is specific to any of the particular bundles there's a place for that in the individual overlays. When we have bundles for each release going back to Mitaka, this could be quite an effort saver.
I'd be quite interested in seeing where this could go, as there is a lot of duplication in the charms' test code that could probably be dramatically reduced by taking this approach! Could you propose a change to one of the repos as an example that we could functionally validate, as well as confirming the assumption that the only differences between the bundles is the series, and openstack-origin/source configs?
Here's a simple example, where there's two base bundles and a pile of overlays - https://review.opendev.org/#/c/756399/
Chris MacNaughton
Hi
I think I remember that a conscious decision was made to avoid using symlinks for the bundles due to the hell that openstack-mojo-specs descended into? Liam may want to wade in on this?
Cheers Alex.
On Sun, Oct 4, 2020 at 7:57 AM Xav Paice xavpaice@gmail.com wrote:
I was writing a patch recently and in order to test it, I needed to make changes to the test bundles. I ended up making the same change across several files, and missing one (thanks to the reviewer for noticing that!).
Some of the other projects I'm involved with use symlinks to a base bundle with overlays:
./tests$ ls -l bundles/ total 20 -rw-rw---- 2 xav xav 5046 Sep 29 14:56 base.yaml lrwxrwxrwx 1 xav xav 9 Sep 29 10:01 bionic.yaml -> base.yaml lrwxrwxrwx 1 xav xav 9 Sep 29 10:01 focal.yaml -> base.yaml drwxrwx--x 2 xav xav 4096 Sep 29 10:01 overlays lrwxrwxrwx 1 xav xav 9 Sep 29 10:01 xenial.yaml -> base.yaml ./tests$ ls bundles/overlays/ bionic.yaml.j2 focal.yaml.j2 local-charm-overlay.yaml.j2 xenial.yaml.j2
This means that I can edit base.yaml just once, and if a change is specific to any of the particular bundles there's a place for that in the individual overlays. When we have bundles for each release going back to Mitaka, this could be quite an effort saver.
I gather there's been some discussion around this already, and would be keen to see if there's a reason folks might avoid that pattern?
I think I remember that a conscious decision was made to avoid using symlinks for the bundles due to the hell that openstack-mojo-specs descended into? Liam may want to wade in on this?
Broadly, this is one of the reasons I proposed submitting a review of a single charm to be a practical example of what this would look like, and how complex it would be. It should also help us, as a project, identify if the repetition is enough that symlinks, or refactoring the library, would be worthwhile.
Chris
FWIW, I have used sym links in a couple of charms [0] [1]. This seems like a perfectly rational thing to do. I suspect it could be leveraged even further as @Xav Paice suggests.
@Alex Kavanagh, I think this is a separate issue from mojo, as this primarily pertains to Zaza tests. Also, the build process removes the sym-links and creates separate files for us in a built charm.
[0] https://github.com/openstack/charm-mysql-innodb-cluster/tree/master/src/test... [1] https://github.com/openstack-charmers/charm-ceph-benchmarking/tree/master/te...
-- David Ames
On Mon, Oct 5, 2020 at 4:07 AM Chris MacNaughton chris.macnaughton@canonical.com wrote:
I think I remember that a conscious decision was made to avoid using symlinks for the bundles due to the hell that openstack-mojo-specs descended into? Liam may want to wade in on this?
Broadly, this is one of the reasons I proposed submitting a review of a single charm to be a practical example of what this would look like, and how complex it would be. It should also help us, as a project, identify if the repetition is enough that symlinks, or refactoring the library, would be worthwhile.
Chris
Hi
So I'm not massively against the idea, but I would like to present some potential disadvantages for consideration:
I have to admit to not being keen to using symlinks for the functional test yaml files. My main objection is maintenance as new openstack and ubuntu releases occur and bundles are added and removed from the charm.
At present, without symlinks (apart from in the overlays), the bundle for an ubuntu-openstack version is a plain file. To remove a version, it is just deleted. If there are symlinks then the 'base.yaml' version represents the one that the charm starts with (say bionic-queens). And then bionic-rocky is a symlink (perhaps with an overlay) and bionic-stein is another symlink, etc. However, at some point in the future bionic-queens will eventually be removed. base.yaml is the 'bionic-queens'. So what is done with base.yaml? Do we make it 'focal-ussuri' and change all the overlays? Leave it as is? Have a new base for each Ubuntu LTS and work from that?
Whilst the current system isn't DRY, it does make it simple to see what's in a particular test bundle for a variation.
Having said all of the above, it is a bit of a pain to manage all the separate files as well, especially when there are changes across multiple versions of the tests.
Thanks Alex.
On Mon, Oct 5, 2020 at 5:02 PM David Ames david.ames@canonical.com wrote:
FWIW, I have used sym links in a couple of charms [0] [1]. This seems like a perfectly rational thing to do. I suspect it could be leveraged even further as @Xav Paice suggests.
@Alex Kavanagh, I think this is a separate issue from mojo, as this primarily pertains to Zaza tests. Also, the build process removes the sym-links and creates separate files for us in a built charm.
[0] https://github.com/openstack/charm-mysql-innodb-cluster/tree/master/src/test... [1] https://github.com/openstack-charmers/charm-ceph-benchmarking/tree/master/te...
-- David Ames
On Mon, Oct 5, 2020 at 4:07 AM Chris MacNaughton chris.macnaughton@canonical.com wrote:
I think I remember that a conscious decision was made to avoid using symlinks for the bundles due to the hell that openstack-mojo-specs descended into? Liam may want to wade in on this?
Broadly, this is one of the reasons I proposed submitting a review of a single charm to be a practical example of what this would look like, and how complex it would be. It should also help us, as a project, identify if the repetition is enough that symlinks, or refactoring the library, would be worthwhile.
Chris
On Wed, 7 Oct 2020 at 02:37, Alex Kavanagh alex.kavanagh@canonical.com wrote:
Hi
So I'm not massively against the idea, but I would like to present some potential disadvantages for consideration:
I have to admit to not being keen to using symlinks for the functional test yaml files. My main objection is maintenance as new openstack and ubuntu releases occur and bundles are added and removed from the charm.
At present, without symlinks (apart from in the overlays), the bundle for an ubuntu-openstack version is a plain file. To remove a version, it is just deleted. If there are symlinks then the 'base.yaml' version represents the one that the charm starts with (say bionic-queens). And then bionic-rocky is a symlink (perhaps with an overlay) and bionic-stein is another symlink, etc. However, at some point in the future bionic-queens will eventually be removed. base.yaml is the 'bionic-queens'. So what is done with base.yaml? Do we make it 'focal-ussuri' and change all the overlays? Leave it as is? Have a new base for each Ubuntu LTS and work from that?
Take a quick look at https://review.opendev.org/#/c/756399/ for an example of what that might look like - to remove a bundle, e.g. the Trusty bundle, we could just rm trusty-mitaka.yaml (which we would do anyway) and rm overlays/trusty-mitaka.yaml.j2. Job done. If there's a bundle with no more symlinks pointing at it, that is something which could easily be missed though, and that's really up to folks that work with these charms on a daily basis (like yourself) to decide if that's an issue or not.
Whilst the current system isn't DRY, it does make it simple to see what's
in a particular test bundle for a variation.
Having said all of the above, it is a bit of a pain to manage all the separate files as well, especially when there are changes across multiple versions of the tests.
Thanks Alex.
participants (4)
-
Alex Kavanagh
-
Chris MacNaughton
-
David Ames
-
Xav Paice