Open Stack

I've finally compiled a spec for this topic

https://review.openstack.org/#/c/281557/

On Wed, Feb 17, 2016 at 2:13 PM Alex Schultz <aschultz at mirantis.com> wrote:

> On Wed, Feb 17, 2016 at 10:23 AM, Bogdan Dobrelya <bdobrelia at mirantis.com>
> wrote:
>
>> > So we'll have tons of conditionals in composition layer, right? Even if
>> > some puppet-openstack class have just one new parameter in new release,
>> > then we'll have to write a conditional and duplicate class declaration.
>> Or
>> > write complex parameters hash definitions/merges and use
>> > create_resources(). The more releases we want to support the more
>> > complicated composition layer will become. That won't make contribution
>> to
>> > fuel-library easier and even can greatly reduce development speed. Also
>> are
>> > we going to add new features to stable releases using this workflow with
>> > single composition layer?
>>
>> As I can see from an example composition [0], such code would be an
>> unmaintainable burden for development and QA process. Next imagine a
>> case for incompatible *providers* like network transformations - shall
>> we put multiple if/case to the ruby providers as well?..
>>
>> That is not a way to go for a composition, sorry. While the idea may be
>> doable, I agree, but perhaps another way.
>>
>
> So I agree that the provided example isn't exactly the cleanest
> implementation but I think the point is that we need to start considering
> supporting multiple versions of OpenStack for a given set of fuel-library
> manifests. I think the provided example begins to show some issues with the
> way we currently structure our modular tasks.  We basically don't have any
> organized structure which leads to having to sprinkle random if/elses
> throughout the code to try and provide some semblance of backwards
> compatibility or deprecation for tasks.  As I see it today in fuel-library
> we are too closely tied to the current development version of OpenStack and
> with every release we are not providing proper deprecation for
> configuration tasks between each release of fuel. I'm sure fuel plugin
> developers could probably attest to the amount of pain each release brings
> trying to re-reverse engineer the deployment process to see what has
> changed.
>
> I think it's unrealistic to expect that all users are going to want the
> latest version of OpenStack as soon as it's released. Fuel is
> an OpenStack deployment tool, so why can't we deploy different versions
> with a single version of fuel?  Technically we should have a way to do with
> with releases in Fuel but it seems that we've handicapped any ability to
> try and leverage this information to support a different version of
> OpenStack.  Much like the UCA work where we started to extract out the
> package/repo information, we need to extract out the OpenStack release
> information (version/repos/etc). We should allow a user to specify that
> they want a build a cloud using X fuel release to deploy Y os with Z
> OpenStack release.  If we can't provide this type of flexibility I'm not
> seeing why someone would want to use Fuel over their own OS
> provisioning+puppet/ansible/chef deployment method.
>
>
>
>> (tl;dr)
>> By the way, this reminded me "The wrong abstraction" [1] article and
>> discussion. I agree with the author and believe one should not group
>> code (here it is versioned puppet modules & compositions) in a way which
>> introduces abstractions (here a super-composition) with multiple
>> if/else/case and hardcoded things to switch the execution flow based on
>> version of things. Just keep code as is - partially duplicated by
>> different releases in separate directories with separate modules and
>> composition layers and think of better solutions please.
>>
>
> Completely duplicating the fuel-library for each OpenStack release is
> probably even more unmanageable then beginning to structure our composition
> layer into something that supports multiple OpenStack resources but I guess
> that could be an option...
>
>
>>
>> There is also a nice comment: "...try to optimize my code around
>> reducing state, coupling, complexity and code, in that order". I
>> understood that like a set of "golden rules":
>> - Make it coupled more tight to decrease (shared) state
>> - Make it more complex to decrease coupling
>> - Make it duplicated to decrease complexity (e.g. abstractions)
>>
>> (tl;dr, I mean it)
>> So, bringing those here.
>> - The shared state is perhaps the Nailgun's world view of all data and
>> versioned serializers for supported releases, which know how to convert
>> the only latest existing data to any of its supported previous versions.
>> - Decoupling we do by putting modules with its compositions to different
>> versioned /etc/puppet subdirectories. I'm not sure how do we decouple
>> Nailgun serializers though.
>> - Complexity is how we compose those modules / write logic of serializers.
>> - Duplication is puppet classes (and providers) with slightly different
>> call parameters from a version to version. Sometimes even not backwards
>> compatible. Probably same to the serializers?
>>
>> So, we're going to *increase complexity* by introducing
>> super-compositions for multi OpenStack releases. Not sure about what to
>> happen to the serializers, any volunteers to clarify an impact?. And the
>> Rules "allow" us to do so only in order to decrease either coupling or
>> shared state, which is not the case, AFAICT. Modules with compositions
>> are separated well by OpenStack versions, nothing to decrease. Might
>> that change to decrease a shared state? I'm not sure if it even applies
>> here. Puppet versioning shares nothing. Only Nailgun folks may know the
>> answer.
>>
>>
> I don't think we have to increase complexity if we properly structure
> fuel-library. I think we could also structure it in a way that can be
> tested in an automated fashion. I think we can come up with a set of rules
> for tasks and how to implement a new task that can reduce complexity and
> actually allow for code reuse.  If you take a look at our tasks today there
> is actually an excessive amount of duplication between tasks[0][1] around
> variable and configuration gathering before we even call the upstream
> OpenStack modules.  The tasks we have today are basically just undocumented
> freeform puppet that can only be fully tested via multiple deployment tests
> because of the lack of decent test coverage.  Additionally, we have no real
> way of testing deprecation between releases today because there's no formal
> api contract for modular tasks themselves.
>
> I would be interested in investigating a restructure of the tasks that
> might go something like....
> 1) Restructure fuel-library tasks into a configuration
> generation/formatting method (extracting data from hiera/globals/etc) and
> the actual configuration application logic
> 2) For the configuration gathering items that we traditionally load up at
> the top of our tasks, we could investigate leveraging hiera for providing
> data to classes or structure into an osnailyfacter::config::<thing> class.
> 2) Move the actual configuration application logic into an
> osnailyfacter::task::<thing> method with a proper documented api contract
> with deprecation policy. These classes could use something like
> create_resources(...) to dynamically load the classes with the provided
> parameters to support the class api changes between upstream module
> versions.
> 3) Add release specific overrides into
> osnailyfacter::task::<thing>::<release> where ::<release> could be
> automatically included based on something like hiera('openstack_version')
>
> There are a few added benefits of restructuring the tasks into traditional
> puppet classes.
> 1) If we need to support any sort of traditional puppet master LCM for
> nodes, by moving the tasks into specific class we can actually just
> leverage our already written task code to ensure configurations.
> 2) Testability around idempotency as we can write beaker tests to be able
> to test deployment tasks for idempotency and deployment without having to
> stand up all the other pieces of fuel.  Similar to the Puppet OpenStack's
> puppet-openstack-integration[2] module.
> 3) Better defined separation and containment of release specific items so
> when we are moving forward.  It's much easier to remove n-X release as we
> could just 'find deployment/puppet/osnailyfacter/manifests/ -name 'kilo.pp'
> -delete'
>
> That being said, this would be a lot of work but it's mostly taking what
> we have today and reorganizing it rather than having to write things from
> scratch and providing some policy rules around structure of tasks.  I think
> in the long run we could benefit by being able to test fuel-library tasks
> with existing tools rather than continually having to rely on noop or
> actual deployment tests with nailgun/astute/isos/etc.  This is just a
> starting point for a conversation and I think we should have a serious
> discussion about this topic and not just dismiss it because it's hard or
> might add complexity.
>
>
> Thanks,
> -Alex
>
> [0]
> https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular/glance/glance.pp#L3-L99
> [1]
> https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular/keystone/keystone.pp#L3-L115
> [2] https://github.com/openstack/puppet-openstack-integration/
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 

--

Andrew Woodward

Mirantis

Fuel Community Ambassador

Ceph Community
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160217/7ba6e364/attachment.html>