Open Stack

On Wed, Feb 17, 2016 at 10:23 AM, Bogdan Dobrelya <bdobrelia at mirantis.com>
wrote:

> > So we'll have tons of conditionals in composition layer, right? Even if
> > some puppet-openstack class have just one new parameter in new release,
> > then we'll have to write a conditional and duplicate class declaration.
> Or
> > write complex parameters hash definitions/merges and use
> > create_resources(). The more releases we want to support the more
> > complicated composition layer will become. That won't make contribution
> to
> > fuel-library easier and even can greatly reduce development speed. Also
> are
> > we going to add new features to stable releases using this workflow with
> > single composition layer?
>
> As I can see from an example composition [0], such code would be an
> unmaintainable burden for development and QA process. Next imagine a
> case for incompatible *providers* like network transformations - shall
> we put multiple if/case to the ruby providers as well?..
>
> That is not a way to go for a composition, sorry. While the idea may be
> doable, I agree, but perhaps another way.
>

So I agree that the provided example isn't exactly the cleanest
implementation but I think the point is that we need to start considering
supporting multiple versions of OpenStack for a given set of fuel-library
manifests. I think the provided example begins to show some issues with the
way we currently structure our modular tasks.  We basically don't have any
organized structure which leads to having to sprinkle random if/elses
throughout the code to try and provide some semblance of backwards
compatibility or deprecation for tasks.  As I see it today in fuel-library
we are too closely tied to the current development version of OpenStack and
with every release we are not providing proper deprecation for
configuration tasks between each release of fuel. I'm sure fuel plugin
developers could probably attest to the amount of pain each release brings
trying to re-reverse engineer the deployment process to see what has
changed.

I think it's unrealistic to expect that all users are going to want the
latest version of OpenStack as soon as it's released. Fuel is
an OpenStack deployment tool, so why can't we deploy different versions
with a single version of fuel?  Technically we should have a way to do with
with releases in Fuel but it seems that we've handicapped any ability to
try and leverage this information to support a different version of
OpenStack.  Much like the UCA work where we started to extract out the
package/repo information, we need to extract out the OpenStack release
information (version/repos/etc). We should allow a user to specify that
they want a build a cloud using X fuel release to deploy Y os with Z
OpenStack release.  If we can't provide this type of flexibility I'm not
seeing why someone would want to use Fuel over their own OS
provisioning+puppet/ansible/chef deployment method.



> (tl;dr)
> By the way, this reminded me "The wrong abstraction" [1] article and
> discussion. I agree with the author and believe one should not group
> code (here it is versioned puppet modules & compositions) in a way which
> introduces abstractions (here a super-composition) with multiple
> if/else/case and hardcoded things to switch the execution flow based on
> version of things. Just keep code as is - partially duplicated by
> different releases in separate directories with separate modules and
> composition layers and think of better solutions please.
>

Completely duplicating the fuel-library for each OpenStack release is
probably even more unmanageable then beginning to structure our composition
layer into something that supports multiple OpenStack resources but I guess
that could be an option...


>
> There is also a nice comment: "...try to optimize my code around
> reducing state, coupling, complexity and code, in that order". I
> understood that like a set of "golden rules":
> - Make it coupled more tight to decrease (shared) state
> - Make it more complex to decrease coupling
> - Make it duplicated to decrease complexity (e.g. abstractions)
>
> (tl;dr, I mean it)
> So, bringing those here.
> - The shared state is perhaps the Nailgun's world view of all data and
> versioned serializers for supported releases, which know how to convert
> the only latest existing data to any of its supported previous versions.
> - Decoupling we do by putting modules with its compositions to different
> versioned /etc/puppet subdirectories. I'm not sure how do we decouple
> Nailgun serializers though.
> - Complexity is how we compose those modules / write logic of serializers.
> - Duplication is puppet classes (and providers) with slightly different
> call parameters from a version to version. Sometimes even not backwards
> compatible. Probably same to the serializers?
>
> So, we're going to *increase complexity* by introducing
> super-compositions for multi OpenStack releases. Not sure about what to
> happen to the serializers, any volunteers to clarify an impact?. And the
> Rules "allow" us to do so only in order to decrease either coupling or
> shared state, which is not the case, AFAICT. Modules with compositions
> are separated well by OpenStack versions, nothing to decrease. Might
> that change to decrease a shared state? I'm not sure if it even applies
> here. Puppet versioning shares nothing. Only Nailgun folks may know the
> answer.
>
>
I don't think we have to increase complexity if we properly structure
fuel-library. I think we could also structure it in a way that can be
tested in an automated fashion. I think we can come up with a set of rules
for tasks and how to implement a new task that can reduce complexity and
actually allow for code reuse.  If you take a look at our tasks today there
is actually an excessive amount of duplication between tasks[0][1] around
variable and configuration gathering before we even call the upstream
OpenStack modules.  The tasks we have today are basically just undocumented
freeform puppet that can only be fully tested via multiple deployment tests
because of the lack of decent test coverage.  Additionally, we have no real
way of testing deprecation between releases today because there's no formal
api contract for modular tasks themselves.

I would be interested in investigating a restructure of the tasks that
might go something like....
1) Restructure fuel-library tasks into a configuration
generation/formatting method (extracting data from hiera/globals/etc) and
the actual configuration application logic
2) For the configuration gathering items that we traditionally load up at
the top of our tasks, we could investigate leveraging hiera for providing
data to classes or structure into an osnailyfacter::config::<thing> class.
2) Move the actual configuration application logic into an
osnailyfacter::task::<thing> method with a proper documented api contract
with deprecation policy. These classes could use something like
create_resources(...) to dynamically load the classes with the provided
parameters to support the class api changes between upstream module
versions.
3) Add release specific overrides into
osnailyfacter::task::<thing>::<release> where ::<release> could be
automatically included based on something like hiera('openstack_version')

There are a few added benefits of restructuring the tasks into traditional
puppet classes.
1) If we need to support any sort of traditional puppet master LCM for
nodes, by moving the tasks into specific class we can actually just
leverage our already written task code to ensure configurations.
2) Testability around idempotency as we can write beaker tests to be able
to test deployment tasks for idempotency and deployment without having to
stand up all the other pieces of fuel.  Similar to the Puppet OpenStack's
puppet-openstack-integration[2] module.
3) Better defined separation and containment of release specific items so
when we are moving forward.  It's much easier to remove n-X release as we
could just 'find deployment/puppet/osnailyfacter/manifests/ -name 'kilo.pp'
-delete'

That being said, this would be a lot of work but it's mostly taking what we
have today and reorganizing it rather than having to write things from
scratch and providing some policy rules around structure of tasks.  I think
in the long run we could benefit by being able to test fuel-library tasks
with existing tools rather than continually having to rely on noop or
actual deployment tests with nailgun/astute/isos/etc.  This is just a
starting point for a conversation and I think we should have a serious
discussion about this topic and not just dismiss it because it's hard or
might add complexity.


Thanks,
-Alex

[0]
https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular/glance/glance.pp#L3-L99
[1]
https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular/keystone/keystone.pp#L3-L115
[2] https://github.com/openstack/puppet-openstack-integration/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160217/e3626098/attachment.html>