Open Stack

Hi Adrian 

Sounds like there's a need for a true workflow engine here. I suspect that for any use case beyond trivial, each user will have her own flavor of CD (TripleO is a good example for this) - some will prefer to use a canary server and test it manually before deploying to other servers, some will deploy to specific servers based on the subset of users they're serving, some will do a red/black type of deployment (Which maps well into environments below), etc. But I think that there should be an user-configurable mechanism to define and tweak these workflows. There will also be additional related workflows - e.g. rolling back a deployment, or updating the infrastructure components of a running application (e.g. database version upgrade). 
I might be raising a topic that's already been discussed (sorry if I am), but I was wondering how this would fit into the architecture. 

Thanks! 
Uri 

On Oct 31, 2013, at 6:42 AM, Adrian Otto <adrian.otto at rackspace.com> wrote:

> Robert,
> 
> We would be interested in exploring your use case for CD, and help you judge what would be the best fit. I know that Monty Taylor has identified some parts of openstack-infra that we could potentially leverage in Solum to address our CI/CD goals. The design of that feature is still pending, so we are looking forward to sourcing its of input on that.
> 
> I don't anticipate any fundament gaps that would prevent you from using a future version of Solum to help narrow your scope. The whole point of our focus is to try to make these goals easier to achieve, and try to get maximum leverage of what the OpenStack ecosystem already offers.
> 
> The high level concept behind the GIt Integration feature currently conceived for Solum is to offer application developers a way to commit code to a Git repo and upon a git push command, have a remote system trigger a modular CI/CD pipeline that ends with that application running on an OpenStack cloud in one or more "environments" (dev/test/stage/prod/etc.). In a simple case that may be (in the case of a dynamic language like Python) the testing of the code, arrangement of the commit into a deployable bundle, triggering of the Solum API to trigger a deployment, automatic generation of a HOT artifact, and stimulation of the Heat API and any other API calls needed.
> 
> In a more complex scenario, you may want additional features and capabilities, such as a system like Gerrit to trigger the commit/push, integrate an artifact repository for storing the builds, etc. Using Zuul for that would be appropriate for sure.
> 
> There are also dev shops who already have a lot invested into their Jenkins setups, and will want to bypass portions of what I described above, so there will be a use case supported where you can just integrate with the solum API, and pass in a pre-built bundle artifact and bypass part or all of Solum's CI features, and just use it for deployment and re-deployment and promotion of a deployment between various environments (dev/test/pros/etc.). You would expect features like canary deploy in that case that might make it more attractive than just generating your own HOT artifact in your Jenkins workflow and passing that in directly for orchestration.
> 
> You should expect a higher level of control for management of the deployed system with abstracts that let you easily extend the system for monitoring and management reasons. 
> 
> Note that not all this functionality belongs in Solum. Some of it may actually come from Heat, and other downstream systems. The idea is that we address the key gaps.
> 
> Adrian
> 
> On Oct 30, 2013, at 9:16 PM, Robert Collins <robertc at robertcollins.net>
> wrote:
> 
>> Hi Adrian,
>>   I'm very interested in Solum, particularly as it relates to
>> TripleO : as you know the TripleO story is to treat OpenStack as just
>> another application to install via Heat/Nova etc - and we've got a
>> major story around pulling from git -> CI + CD pipeline -> automated
>> deployment to production.
>> 
>> So - where we can say 'hey, this should be part of Solum', we can hive
>> stuff out of the TripleO program into Solum - but only if we've got
>> compatible stories.
>> 
>> Right now, our design for CD is basically Gerrit + Zuul - exactly the
>> same as gating checks for OpenStack itself. When I look at
>> https://wiki.openstack.org/wiki/Solum/FeatureBlueprints/GitIntegration#Flow_Boundaries
>> I'm having a little trouble mapping that into your design.
>> 
>> I'm wondering if you can expand on how that hangs together?
>> 
>> Thanks!
>> Rob
>> 
>> On 31 October 2013 14:01, Adrian Otto <adrian.otto at rackspace.com> wrote:
>>> Clayton,
>>> 
>>> Thanks for adding the diagram illustrating the flow. I expect that "respond to client" may not be a synchronous flow, rather that if creation of a repo takes a while that the API may return a 202 Accepted, and the client can poll a status attribute to determine completion and learn the actual location of the repo created. So in that case "respond to client" also might mean "update async status". This will be particularly important in cases where an external SCM system is used (perhaps Github).
>>> 
>>> I agree that this will be a helpful reference to guide our upcoming interactive design sessions. I have created the first call for participation, which will also be separately announced:
>>> 
>>> https://wiki.openstack.org/wiki/Solum/BreakoutMeetings
>>> 
>>> Adrian
>>> 
>>> On Oct 30, 2013, at 2:45 PM, Clayton Coleman <ccoleman at redhat.com> wrote:
>>> 
>>>> In the IRC meeting yesterday [1] we discussed splitting the individual topics for the git deploy blueprint [2] into rough sub areas.  To help frame the abstractions we've been discussing I roughed out a flow based on two user inputs, REST API create and a git push and then tried to draw boxes around the related bits.  The abstractions roughly correspond to the potential areas that can be the focus of break out discussions (and if they don't a good discussion of why is always helpful).
>>>> 
>>>> Squiggly boxes are potential plugin points and/or strong responsibility boundaries.  Feedback desirable.
>>>> 
>>>> [1] http://irclogs.solum.io/2013/solum.2013-10-29-16.01.txt
>>>> [2] https://wiki.openstack.org/wiki/Solum/FeatureBlueprints/GitIntegration#Flow_Boundaries
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> 
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> 
>> -- 
>> Robert Collins <rbtcollins at hp.com>
>> Distinguished Technologist
>> HP Converged Cloud
>> 
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev