Open Stack

I'd suggest to reduce number of Murano repositories for several reasons:

* All other OpenStack projects have a single repo per project. While this
point might look like something not worth mentioning, it's really important:
- unified project structure simplifies life for new developers. once they
get familiar with one project, they can expect something similar from
another project
- unified project structure simplifies life for deployers. similar project
structure simplifies packaging and deployment automation

* Another important reason is to simplify gated testing. Just take a look at
Solum layout [1], they have everything needed (contrib, functionaltests) to
run dvsm job in a single repo. One simple job definition [2] allows to
install Solum in DevStack and run Tempest tests for Solum.

* As a side-effect, this approach will improve integrity of project
components. Having murano-common in the same repo with other components will
help to catch integration issues earlier.


In an ideal world there will be only the following repos:
- murano (api, common, conductor, docs, repository, tests)
- python-muranoclient
- murano-dashboard
- murano-agent
- puppet-murano (optional, but nice to have)


[1] https://github.com/stackforge/solum
[2]
https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/jenkins_job_builder/config/solum.yaml


Thanks,
Ruslan


On Thu, Feb 6, 2014 at 3:14 PM, Serg Melikyan <smelikyan at mirantis.com>wrote:

> Hi, Alexander,
>
> In general I am completely agree with Clint and Robert, and as one of
> contributors of Murano I don't see any practical reasons for repositories
> reorganization. And regarding of your proposal I have a few thoughts that I
> would like to share below:
>
> >This enourmous amount of repositories adds too much infrustructural
> complexity
> Creating a new repository is a quick, easy and completely automated
> procedure that requires only simple commit to Zuul configuration. All
> infrastructure related to repositories is handled by Openstack CI and
> supported by Openstack Infra Team, and actually don't require anything from
> project development team. About what infrastructure complexity you are
> talking about?
>
> >I actually think keeping them separate is a great way to make sure you
> have ongoing API stability. (c) Clint
> I would like to share a little statistic gathered by Stan Lagun
> a little time ago regarding repositories count in different PaaS solution.
> If you are concerned about large number of repositories used by Murano, you
> will be quite amused:
>
>    - https://github.com/heroku - 275
>    - https://github.com/cloudfoundry - 132
>     - https://github.com/openshift - 49
>    - https://github.com/CloudifySource - 46
>
> >First of all, I would suggest to have a single reposository for all the
> three main components of Murano: main murano API (the contents of the
> present), workflow execution engine (currently murano-conductor; also it
> was suggested to rename the component itself to murano-engine for more
> consistent naming) and metadata repository (currently murano-repository).
>
> *murano-api* and *murano-repository* have many things in common, they are
> both present HTTP API to the user, and I hope would be rewritten to common
> framework (Pecan?). But *murano-conductor* have only one thing in common
> with other two components: code shared under *murano-common*. That
> repository may be eventually eliminated by moving to Oslo (as it should be
> done).
>
> >Also, it has been suggested to move our agents (both windows and unified
> python) into the main repository as well - just to put them into a separate
> subfolder. I don't see any reasons why they should be separated from core
> Murano: I don't believe we are going to have any third-party
> implementations of our "Unified agent" proposals, while this possibility
> was the main reason for separatinng them.
>
> Main reason for murano-agent to have separate repository was not a
> possibility to have another implementation, but that all sources that
> should be able to be built as package, have tests and can be uploaded to
> PyPI (or any other gate job) should be placed in different repository.
> OpenStack CI have several rules regarding how repositories should be
> organized to support running different gate jobs. For example, to run tests
> *tox.ini* is need to be present in root directory, to build package
> *setup.py* should be present in root directory. So we could not simply
> move them to separate directories in main repository and have same
> capabilities as in separate repository.
>
> >Next, deployment scripts and auto-generated docs: are there reasons why
> they should be in their own repositories, instead of "docs" and
> "tools/deployment" folders of the primary repo? I would prefer the latter:
> docs and deployment scripts have no meaning without the sources which they
> document/deploy - so it is better to have them consistent.
> We have *developers documentation* alongside with all sources:
> murano-conductor<https://github.com/stackforge/murano-conductor/tree/master/doc/source>,
> murano-api<https://github.com/stackforge/murano-api/tree/master/doc/source> and
> so on. It is true that we have not so much documentation there, and not
> much code is documented to add auto-generated documentation. Documentation
> that is found in *murano-docs* repository actually is a docbook
> documentation, that is presented in book manner, and follows documentation
> patterns found in core projects itself: openstack-manuals<https://github.com/openstack/openstack-manuals/tree/master/doc>
> .
>
> *murano-deployment* contains scripts and other artefacts related to
> deployment, but not necessary to source code. This repository don't use
> much of CI capabilities, but raise it is logical place where we can place
> different thing related to deployment: various scripts, specs, patches and
> so on. Also with separate repository we can not to spam our deployment
> engineers with software engineers related commits.
>
>
>
> On Tue, Jan 21, 2014 at 11:55 PM, Alexander Tivelkov <
> ativelkov at mirantis.com> wrote:
>
>> Hi folks,
>>
>> As we are moving towards incubation application, I took a closer look at
>> what is going on with our repositories.
>> An here is what I found. We currently have 11 repositories at stackforge:
>>
>>    - murano-api
>>    - murano-conductor
>>    - murano-repository
>>    - murano-dashboard
>>    - murano-common
>>    - python-muranoclient
>>    - murano-metadataclient
>>    - murano-agent
>>    - murano-docs
>>    - murano-tests
>>    - murano-deployment
>>
>> This enourmous amount of repositories adds too much infrustructural
>> complexity, and maintaining the changes in in consistent and reliable
>> manner becomes a really tricky tasks. We often have changes which require
>> modifing two or more repositories - and thus we have to make several
>> changesets in gerrit, targeting different repositories. Quite often the
>> dependencies between these changesets are not obvious, the patches get
>> reviewed and approved on wrong order (yes, this also questions the quality
>> of the code review, but that is a different topic), which causes in
>> inconsostent state of the repositories.
>>
>> Well, anyway, this has to be changed in some way or another.
>> I suggest to initiate the discussions on how to do all this.
>>
>> Below you may find my position. This is not final in any meaning, just a
>> proposal. Please, feel free to discuss :)
>>
>> First of all, I would suggest to have a single reposository for all the
>> three main components of Murano: main murano API (the contents of the
>> present), workflow execution engine (currently murano-conductor; also it
>> was suggested to rename the component itself to murano-engine for more
>> consistent naming) and metadata repository (currently murano-repository).
>> These should remain as independent modules, being able to run as
>> different daemons, but stored within a single repository (similar to how
>> heat has heat-api, heat-cfn and heat-engine under the same hood). The name
>> of this repository is tentative: I think none of the existing match, so I
>> would suggest to create a new repo (simple "murano" seems to fit the best),
>> and then relocate all the content from other 3 repos and remove them
>> aftwerwards.
>>
>> When the api, the repository and the engine are merged into a single
>> repo, there will be no sense in having murano-common repo for storing their
>> common classes: instead, there should be a "common" package inside the main
>> murano repository.
>>
>> Also, it has been suggested to move our agents (both windows and unified
>> python) into the main repository as well - just to put them into a separate
>> subfolder. I don't see any reasons why they should be separated from core
>> Murano: I don't believe we are going to have any third-party
>> implementations of our "Unified agent" proposals, while this possibility
>> was the main reason for separatinng them.
>>
>> Next, deployment scripts and auto-generated docs: are there reasons why
>> they should be in their own repositories, instead of "docs" and
>> "tools/deployment" folders of the primary repo? I would prefer the latter:
>> docs and deployment scripts have no meaning without the sources which they
>> document/deploy - so it is better to have them consistent.
>>
>>
>> Then, the python bindings: "There can be only one" (c). Yes, the metadata
>> API and the main murano API are different indeed, however there is no
>> reason in having two repositories for their clients: let's have a single
>> repo, containing two packages inside. Are there any technical reasons
>> preventing us from doing that?
>> CLI should be common as well - I think there should be a single
>> command-line utility ("murano" should be the name), allowing to query both
>> APIs. This CLI will eventually evolve into the developer's utility: it will
>> get commands to package, sign and submit application packages.
>>
>> Openstack Dashboard plugin - aka Murano-dashboard - should remain in a
>> separated repo, I have no objections here :)
>>
>> murano-tests may reamin independent as well - however, this repository is
>> not likely to be transferred when we go to incubation: incubated projects
>> should have tempest test in their repositories, shoudn't they? Our our test
>> may remain on stackforge - this is irrelevant.
>>
>> And finally, we need some place to store sources of our metadata objects:
>> the definition of  core murano library, as well as example services, with
>> all their stuff -  metadata and ui definitions, heat templates, scripts
>> etc. Here I propose to create a new repo, specially dedicated for this
>> purpose. If we succeed in building the ecosystem for application developers
>> and publishers, this will be the repo in which they should contribute,
>> while the core murano repo's will remain relativele stable.
>>
>>
>> So, this brings us to the following list of repos:
>>
>>    - *murano* - main services, common, agents docs, deployments scripts
>>    - *python-muranoclient* - python bindings and CLI
>>    - *murano-dashboard* - OS Dashboard plugin
>>    - *murano-apps* - new repo for metadata, including core library and
>>    example apps.
>>    - *murano-tests* - existing test-repo, not going to be transferred
>>    when incubated.
>>
>>
>> This leaves us with just 4 repositories (plus one additional which will
>> remain on stackforge), with clear separation of concerns.
>>
>> There may be technical issues in doing this mergement (we do not want to
>> loose revision history, do we?), but they should be solvable (I'll write to
>> infra asking on what is possible and what is not), but in general this is
>> the direction in which we should be moving, as it seems to me.
>>
>> --
>> Regards,
>> Alexander Tivelkov
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Serg Melikyan, Senior Software Engineer at Mirantis, Inc.
> http://mirantis.com | smelikyan at mirantis.com
>
> +7 (495) 640-4904, 0261
> +7 (903) 156-0836
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140218/1bd34e8e/attachment.html>