[openstack-dev] [Murano] Repositoris re-organization

Sergey Lukjanov slukjanov at mirantis.com
Tue Feb 18 18:06:37 UTC 2014


Ruslan,

I'm absolutely agree with you, only one correction, I think
murano-guestagent will better fit the repo content.

Thanks.


On Tue, Feb 18, 2014 at 7:23 PM, Alexander Tivelkov
<ativelkov at mirantis.com>wrote:

> Hi Ruslan,
>
> Thanks for your feedback. I completely agree with these arguments:
> actually, these were the reasons why I've initiated this discussion.
>
> Team, let's discuss this on the IRC meeting today.
>
> --
> Regards,
> Alexander Tivelkov
>
>
> On Tue, Feb 18, 2014 at 5:55 PM, Ruslan Kamaldinov <
> rkamaldinov at mirantis.com> wrote:
>
>> I'd suggest to reduce number of Murano repositories for several reasons:
>>
>> * All other OpenStack projects have a single repo per project. While this
>> point might look like something not worth mentioning, it's really
>> important:
>> - unified project structure simplifies life for new developers. once they
>> get familiar with one project, they can expect something similar from
>> another project
>> - unified project structure simplifies life for deployers. similar project
>> structure simplifies packaging and deployment automation
>>
>> * Another important reason is to simplify gated testing. Just take a look
>> at
>> Solum layout [1], they have everything needed (contrib, functionaltests)
>> to
>> run dvsm job in a single repo. One simple job definition [2] allows to
>> install Solum in DevStack and run Tempest tests for Solum.
>>
>> * As a side-effect, this approach will improve integrity of project
>> components. Having murano-common in the same repo with other components
>> will
>> help to catch integration issues earlier.
>>
>>
>> In an ideal world there will be only the following repos:
>> - murano (api, common, conductor, docs, repository, tests)
>> - python-muranoclient
>> - murano-dashboard
>> - murano-agent
>> - puppet-murano (optional, but nice to have)
>>
>>
>> [1] https://github.com/stackforge/solum
>> [2]
>> https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/jenkins_job_builder/config/solum.yaml
>>
>>
>> Thanks,
>> Ruslan
>>
>>
>> On Thu, Feb 6, 2014 at 3:14 PM, Serg Melikyan <smelikyan at mirantis.com>wrote:
>>
>>> Hi, Alexander,
>>>
>>> In general I am completely agree with Clint and Robert, and as one of
>>> contributors of Murano I don't see any practical reasons for repositories
>>> reorganization. And regarding of your proposal I have a few thoughts that I
>>> would like to share below:
>>>
>>> >This enourmous amount of repositories adds too much infrustructural
>>> complexity
>>> Creating a new repository is a quick, easy and completely automated
>>> procedure that requires only simple commit to Zuul configuration. All
>>> infrastructure related to repositories is handled by Openstack CI and
>>> supported by Openstack Infra Team, and actually don't require anything from
>>> project development team. About what infrastructure complexity you are
>>> talking about?
>>>
>>> >I actually think keeping them separate is a great way to make sure you
>>> have ongoing API stability. (c) Clint
>>> I would like to share a little statistic gathered by Stan Lagun
>>> a little time ago regarding repositories count in different PaaS solution.
>>> If you are concerned about large number of repositories used by Murano, you
>>> will be quite amused:
>>>
>>>    - https://github.com/heroku - 275
>>>    - https://github.com/cloudfoundry - 132
>>>     - https://github.com/openshift - 49
>>>    - https://github.com/CloudifySource - 46
>>>
>>> >First of all, I would suggest to have a single reposository for all
>>> the three main components of Murano: main murano API (the contents of the
>>> present), workflow execution engine (currently murano-conductor; also it
>>> was suggested to rename the component itself to murano-engine for more
>>> consistent naming) and metadata repository (currently murano-repository).
>>>
>>> *murano-api* and *murano-repository* have many things in common, they
>>> are both present HTTP API to the user, and I hope would be rewritten to
>>> common framework (Pecan?). But *murano-conductor* have only one thing
>>> in common with other two components: code shared under *murano-common*.
>>> That repository may be eventually eliminated by moving to Oslo (as it
>>> should be done).
>>>
>>> >Also, it has been suggested to move our agents (both windows and
>>> unified python) into the main repository as well - just to put them into a
>>> separate subfolder. I don't see any reasons why they should be separated
>>> from core Murano: I don't believe we are going to have any third-party
>>> implementations of our "Unified agent" proposals, while this possibility
>>> was the main reason for separatinng them.
>>>
>>> Main reason for murano-agent to have separate repository was not a
>>> possibility to have another implementation, but that all sources that
>>> should be able to be built as package, have tests and can be uploaded to
>>> PyPI (or any other gate job) should be placed in different repository.
>>> OpenStack CI have several rules regarding how repositories should be
>>> organized to support running different gate jobs. For example, to run tests
>>> *tox.ini* is need to be present in root directory, to build package
>>> *setup.py* should be present in root directory. So we could not simply
>>> move them to separate directories in main repository and have same
>>> capabilities as in separate repository.
>>>
>>> >Next, deployment scripts and auto-generated docs: are there reasons
>>> why they should be in their own repositories, instead of "docs" and
>>> "tools/deployment" folders of the primary repo? I would prefer the latter:
>>> docs and deployment scripts have no meaning without the sources which they
>>> document/deploy - so it is better to have them consistent.
>>> We have *developers documentation* alongside with all sources:
>>> murano-conductor<https://github.com/stackforge/murano-conductor/tree/master/doc/source>,
>>> murano-api<https://github.com/stackforge/murano-api/tree/master/doc/source> and
>>> so on. It is true that we have not so much documentation there, and not
>>> much code is documented to add auto-generated documentation. Documentation
>>> that is found in *murano-docs* repository actually is a docbook
>>> documentation, that is presented in book manner, and follows documentation
>>> patterns found in core projects itself: openstack-manuals<https://github.com/openstack/openstack-manuals/tree/master/doc>
>>> .
>>>
>>> *murano-deployment* contains scripts and other artefacts related to
>>> deployment, but not necessary to source code. This repository don't use
>>> much of CI capabilities, but raise it is logical place where we can place
>>> different thing related to deployment: various scripts, specs, patches and
>>> so on. Also with separate repository we can not to spam our deployment
>>> engineers with software engineers related commits.
>>>
>>>
>>>
>>> On Tue, Jan 21, 2014 at 11:55 PM, Alexander Tivelkov <
>>> ativelkov at mirantis.com> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> As we are moving towards incubation application, I took a closer look
>>>> at what is going on with our repositories.
>>>> An here is what I found. We currently have 11 repositories at
>>>> stackforge:
>>>>
>>>>    - murano-api
>>>>    - murano-conductor
>>>>    - murano-repository
>>>>    - murano-dashboard
>>>>    - murano-common
>>>>    - python-muranoclient
>>>>    - murano-metadataclient
>>>>    - murano-agent
>>>>    - murano-docs
>>>>    - murano-tests
>>>>    - murano-deployment
>>>>
>>>> This enourmous amount of repositories adds too much infrustructural
>>>> complexity, and maintaining the changes in in consistent and reliable
>>>> manner becomes a really tricky tasks. We often have changes which require
>>>> modifing two or more repositories - and thus we have to make several
>>>> changesets in gerrit, targeting different repositories. Quite often the
>>>> dependencies between these changesets are not obvious, the patches get
>>>> reviewed and approved on wrong order (yes, this also questions the quality
>>>> of the code review, but that is a different topic), which causes in
>>>> inconsostent state of the repositories.
>>>>
>>>> Well, anyway, this has to be changed in some way or another.
>>>> I suggest to initiate the discussions on how to do all this.
>>>>
>>>> Below you may find my position. This is not final in any meaning, just
>>>> a proposal. Please, feel free to discuss :)
>>>>
>>>> First of all, I would suggest to have a single reposository for all the
>>>> three main components of Murano: main murano API (the contents of the
>>>> present), workflow execution engine (currently murano-conductor; also it
>>>> was suggested to rename the component itself to murano-engine for more
>>>> consistent naming) and metadata repository (currently murano-repository).
>>>> These should remain as independent modules, being able to run as
>>>> different daemons, but stored within a single repository (similar to how
>>>> heat has heat-api, heat-cfn and heat-engine under the same hood). The name
>>>> of this repository is tentative: I think none of the existing match, so I
>>>> would suggest to create a new repo (simple "murano" seems to fit the best),
>>>> and then relocate all the content from other 3 repos and remove them
>>>> aftwerwards.
>>>>
>>>> When the api, the repository and the engine are merged into a single
>>>> repo, there will be no sense in having murano-common repo for storing their
>>>> common classes: instead, there should be a "common" package inside the main
>>>> murano repository.
>>>>
>>>> Also, it has been suggested to move our agents (both windows and
>>>> unified python) into the main repository as well - just to put them into a
>>>> separate subfolder. I don't see any reasons why they should be separated
>>>> from core Murano: I don't believe we are going to have any third-party
>>>> implementations of our "Unified agent" proposals, while this possibility
>>>> was the main reason for separatinng them.
>>>>
>>>> Next, deployment scripts and auto-generated docs: are there reasons why
>>>> they should be in their own repositories, instead of "docs" and
>>>> "tools/deployment" folders of the primary repo? I would prefer the latter:
>>>> docs and deployment scripts have no meaning without the sources which they
>>>> document/deploy - so it is better to have them consistent.
>>>>
>>>>
>>>> Then, the python bindings: "There can be only one" (c). Yes, the
>>>> metadata API and the main murano API are different indeed, however there is
>>>> no reason in having two repositories for their clients: let's have a single
>>>> repo, containing two packages inside. Are there any technical reasons
>>>> preventing us from doing that?
>>>> CLI should be common as well - I think there should be a single
>>>> command-line utility ("murano" should be the name), allowing to query both
>>>> APIs. This CLI will eventually evolve into the developer's utility: it will
>>>> get commands to package, sign and submit application packages.
>>>>
>>>> Openstack Dashboard plugin - aka Murano-dashboard - should remain in a
>>>> separated repo, I have no objections here :)
>>>>
>>>> murano-tests may reamin independent as well - however, this repository
>>>> is not likely to be transferred when we go to incubation: incubated
>>>> projects should have tempest test in their repositories, shoudn't they? Our
>>>> our test may remain on stackforge - this is irrelevant.
>>>>
>>>> And finally, we need some place to store sources of our metadata
>>>> objects: the definition of  core murano library, as well as example
>>>> services, with all their stuff -  metadata and ui definitions, heat
>>>> templates, scripts etc. Here I propose to create a new repo, specially
>>>> dedicated for this purpose. If we succeed in building the ecosystem for
>>>> application developers and publishers, this will be the repo in which they
>>>> should contribute, while the core murano repo's will remain relativele
>>>> stable.
>>>>
>>>>
>>>> So, this brings us to the following list of repos:
>>>>
>>>>    - *murano* - main services, common, agents docs, deployments scripts
>>>>    - *python-muranoclient* - python bindings and CLI
>>>>    - *murano-dashboard* - OS Dashboard plugin
>>>>    - *murano-apps* - new repo for metadata, including core library and
>>>>    example apps.
>>>>    - *murano-tests* - existing test-repo, not going to be transferred
>>>>    when incubated.
>>>>
>>>>
>>>> This leaves us with just 4 repositories (plus one additional which will
>>>> remain on stackforge), with clear separation of concerns.
>>>>
>>>> There may be technical issues in doing this mergement (we do not want
>>>> to loose revision history, do we?), but they should be solvable (I'll write
>>>> to infra asking on what is possible and what is not), but in general this
>>>> is the direction in which we should be moving, as it seems to me.
>>>>
>>>> --
>>>> Regards,
>>>> Alexander Tivelkov
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>>>
>>> --
>>> Serg Melikyan, Senior Software Engineer at Mirantis, Inc.
>>> http://mirantis.com | smelikyan at mirantis.com
>>>
>>> +7 (495) 640-4904, 0261
>>> +7 (903) 156-0836
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Sincerely yours,
Sergey Lukjanov
Savanna Technical Lead
Mirantis Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140218/47e0eaa9/attachment.html>


More information about the OpenStack-dev mailing list