Open Stack

Hi, Alexander,

In general I am completely agree with Clint and Robert, and as one of
contributors of Murano I don't see any practical reasons for repositories
reorganization. And regarding of your proposal I have a few thoughts that I
would like to share below:

>This enourmous amount of repositories adds too much infrustructural
complexity
Creating a new repository is a quick, easy and completely automated
procedure that requires only simple commit to Zuul configuration. All
infrastructure related to repositories is handled by Openstack CI and
supported by Openstack Infra Team, and actually don't require anything from
project development team. About what infrastructure complexity you are
talking about?

>I actually think keeping them separate is a great way to make sure you
have ongoing API stability. (c) Clint
I would like to share a little statistic gathered by Stan Lagun
a little time ago regarding repositories count in different PaaS solution.
If you are concerned about large number of repositories used by Murano, you
will be quite amused:

   - https://github.com/heroku - 275
   - https://github.com/cloudfoundry - 132
   - https://github.com/openshift - 49
   - https://github.com/CloudifySource - 46

>First of all, I would suggest to have a single reposository for all the
three main components of Murano: main murano API (the contents of the
present), workflow execution engine (currently murano-conductor; also it
was suggested to rename the component itself to murano-engine for more
consistent naming) and metadata repository (currently murano-repository).

*murano-api* and *murano-repository* have many things in common, they are
both present HTTP API to the user, and I hope would be rewritten to common
framework (Pecan?). But *murano-conductor* have only one thing in common
with other two components: code shared under *murano-common*. That
repository may be eventually eliminated by moving to Oslo (as it should be
done).

>Also, it has been suggested to move our agents (both windows and unified
python) into the main repository as well - just to put them into a separate
subfolder. I don't see any reasons why they should be separated from core
Murano: I don't believe we are going to have any third-party
implementations of our "Unified agent" proposals, while this possibility
was the main reason for separatinng them.

Main reason for murano-agent to have separate repository was not a
possibility to have another implementation, but that all sources that
should be able to be built as package, have tests and can be uploaded to
PyPI (or any other gate job) should be placed in different repository.
OpenStack CI have several rules regarding how repositories should be
organized to support running different gate jobs. For example, to run tests
*tox.ini* is need to be present in root directory, to build package
*setup.py* should be present in root directory. So we could not simply move
them to separate directories in main repository and have same capabilities
as in separate repository.

>Next, deployment scripts and auto-generated docs: are there reasons why
they should be in their own repositories, instead of "docs" and
"tools/deployment" folders of the primary repo? I would prefer the latter:
docs and deployment scripts have no meaning without the sources which they
document/deploy - so it is better to have them consistent.
We have *developers documentation* alongside with all sources:
murano-conductor<https://github.com/stackforge/murano-conductor/tree/master/doc/source>,
murano-api <https://github.com/stackforge/murano-api/tree/master/doc/source>
and
so on. It is true that we have not so much documentation there, and not
much code is documented to add auto-generated documentation. Documentation
that is found in *murano-docs* repository actually is a docbook
documentation, that is presented in book manner, and follows documentation
patterns found in core projects itself:
openstack-manuals<https://github.com/openstack/openstack-manuals/tree/master/doc>
.

*murano-deployment* contains scripts and other artefacts related to
deployment, but not necessary to source code. This repository don't use
much of CI capabilities, but raise it is logical place where we can place
different thing related to deployment: various scripts, specs, patches and
so on. Also with separate repository we can not to spam our deployment
engineers with software engineers related commits.



On Tue, Jan 21, 2014 at 11:55 PM, Alexander Tivelkov <ativelkov at mirantis.com
> wrote:

> Hi folks,
>
> As we are moving towards incubation application, I took a closer look at
> what is going on with our repositories.
> An here is what I found. We currently have 11 repositories at stackforge:
>
>    - murano-api
>    - murano-conductor
>    - murano-repository
>    - murano-dashboard
>    - murano-common
>    - python-muranoclient
>    - murano-metadataclient
>    - murano-agent
>    - murano-docs
>    - murano-tests
>    - murano-deployment
>
> This enourmous amount of repositories adds too much infrustructural
> complexity, and maintaining the changes in in consistent and reliable
> manner becomes a really tricky tasks. We often have changes which require
> modifing two or more repositories - and thus we have to make several
> changesets in gerrit, targeting different repositories. Quite often the
> dependencies between these changesets are not obvious, the patches get
> reviewed and approved on wrong order (yes, this also questions the quality
> of the code review, but that is a different topic), which causes in
> inconsostent state of the repositories.
>
> Well, anyway, this has to be changed in some way or another.
> I suggest to initiate the discussions on how to do all this.
>
> Below you may find my position. This is not final in any meaning, just a
> proposal. Please, feel free to discuss :)
>
> First of all, I would suggest to have a single reposository for all the
> three main components of Murano: main murano API (the contents of the
> present), workflow execution engine (currently murano-conductor; also it
> was suggested to rename the component itself to murano-engine for more
> consistent naming) and metadata repository (currently murano-repository).
> These should remain as independent modules, being able to run as different
> daemons, but stored within a single repository (similar to how heat has
> heat-api, heat-cfn and heat-engine under the same hood). The name of this
> repository is tentative: I think none of the existing match, so I would
> suggest to create a new repo (simple "murano" seems to fit the best), and
> then relocate all the content from other 3 repos and remove them
> aftwerwards.
>
> When the api, the repository and the engine are merged into a single repo,
> there will be no sense in having murano-common repo for storing their
> common classes: instead, there should be a "common" package inside the main
> murano repository.
>
> Also, it has been suggested to move our agents (both windows and unified
> python) into the main repository as well - just to put them into a separate
> subfolder. I don't see any reasons why they should be separated from core
> Murano: I don't believe we are going to have any third-party
> implementations of our "Unified agent" proposals, while this possibility
> was the main reason for separatinng them.
>
> Next, deployment scripts and auto-generated docs: are there reasons why
> they should be in their own repositories, instead of "docs" and
> "tools/deployment" folders of the primary repo? I would prefer the latter:
> docs and deployment scripts have no meaning without the sources which they
> document/deploy - so it is better to have them consistent.
>
>
> Then, the python bindings: "There can be only one" (c). Yes, the metadata
> API and the main murano API are different indeed, however there is no
> reason in having two repositories for their clients: let's have a single
> repo, containing two packages inside. Are there any technical reasons
> preventing us from doing that?
> CLI should be common as well - I think there should be a single
> command-line utility ("murano" should be the name), allowing to query both
> APIs. This CLI will eventually evolve into the developer's utility: it will
> get commands to package, sign and submit application packages.
>
> Openstack Dashboard plugin - aka Murano-dashboard - should remain in a
> separated repo, I have no objections here :)
>
> murano-tests may reamin independent as well - however, this repository is
> not likely to be transferred when we go to incubation: incubated projects
> should have tempest test in their repositories, shoudn't they? Our our test
> may remain on stackforge - this is irrelevant.
>
> And finally, we need some place to store sources of our metadata objects:
> the definition of  core murano library, as well as example services, with
> all their stuff -  metadata and ui definitions, heat templates, scripts
> etc. Here I propose to create a new repo, specially dedicated for this
> purpose. If we succeed in building the ecosystem for application developers
> and publishers, this will be the repo in which they should contribute,
> while the core murano repo's will remain relativele stable.
>
>
> So, this brings us to the following list of repos:
>
>    - *murano* - main services, common, agents docs, deployments scripts
>    - *python-muranoclient* - python bindings and CLI
>    - *murano-dashboard* - OS Dashboard plugin
>    - *murano-apps* - new repo for metadata, including core library and
>    example apps.
>    - *murano-tests* - existing test-repo, not going to be transferred
>    when incubated.
>
>
> This leaves us with just 4 repositories (plus one additional which will
> remain on stackforge), with clear separation of concerns.
>
> There may be technical issues in doing this mergement (we do not want to
> loose revision history, do we?), but they should be solvable (I'll write to
> infra asking on what is possible and what is not), but in general this is
> the direction in which we should be moving, as it seems to me.
>
> --
> Regards,
> Alexander Tivelkov
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Serg Melikyan, Senior Software Engineer at Mirantis, Inc.
http://mirantis.com | smelikyan at mirantis.com

+7 (495) 640-4904, 0261
+7 (903) 156-0836
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140206/369593ce/attachment.html>