<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 01/06/2016 11:48 AM, Dougal Matthews
wrote:<br>
</div>
<blockquote
cite="mid:CAPMB-2THenVJzBTHP3f140ipDRj49C1F3Yoj0NaG=5Pv=eLdmA@mail.gmail.com"
type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 5 January 2016 at 17:09, Jiri
Tomasek <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:jtomasek@redhat.com" target="_blank">jtomasek@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div class="">
<div class="h5">On 12/23/2015 07:40 PM, Steven Hardy
wrote:<br>
</div>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<div class="h5">
On Wed, Dec 23, 2015 at 11:05:05AM -0600, Ben Nemec
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
On 12/23/2015 10:26 AM, Steven Hardy wrote:<br>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
On Wed, Dec 23, 2015 at 09:28:59AM -0600, Ben
Nemec wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
On 12/23/2015 03:19 AM, Dougal Matthews wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<br>
On 22 December 2015 at 17:59, Ben Nemec <<a
moz-do-not-send="true"
href="mailto:openstack@nemebean.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:openstack@nemebean.com">openstack@nemebean.com</a></a><br>
<mailto:<a moz-do-not-send="true"
href="mailto:openstack@nemebean.com"
target="_blank">openstack@nemebean.com</a>>>
wrote:<br>
<br>
Can we just do git like I've been
suggesting all along? ;-)<br>
<br>
More serious discussion inline. :-)<br>
<br>
On 12/22/2015 09:36 AM, Dougal Matthews
wrote:<br>
> Hi all,<br>
><br>
> This topic came up in the
2015-12-15 meeting[1], and again briefly<br>
today.<br>
> After working with the code that
came out of the deployment library<br>
> spec[2] I<br>
> had some concerns with how we are
storing the templates.<br>
><br>
> Simply put, when we are dealing
with 100+ files from<br>
tripleo-heat-templates<br>
> how can we ensure consistency in
Swift without any atomicity or<br>
> transactions.<br>
> I think this is best explained
with a couple of examples.<br>
><br>
> - When we create a new deployment
plan (upload all the templates<br>
to swift)<br>
> how do we handle the case where
there is an error? For example,<br>
if we are<br>
> uploading 10 files - what do we
do if the 5th one fails for<br>
some reason?<br>
> There is a patch to do a manual
rollback[3], but I have<br>
concerns about<br>
> doing this in Python. If Swift
is completely inaccessible for a<br>
short<br>
> period the rollback wont work
either.<br>
><br>
> - When deploying to Heat, we need
to download all the YAML files from<br>
> Swift.<br>
> This can take a couple of
seconds. What happens if somebody<br>
starts to<br>
> upload a new version of the
plan in the middle? We could end up<br>
trying to<br>
> deploy half old and half new
files. We wouldn't have a<br>
consistent view of<br>
> the database.<br>
><br>
> We had a few suggestions in the
meeting:<br>
><br>
> - Add a locking mechanism. I
would be concerned about deadlocks or<br>
> having to<br>
> lock for the full duration of a
deploy.<br>
<br>
There should be no need to lock the
plan for the entire deploy. It's<br>
not like we're re-reading the templates
at the end of the deploy today.<br>
It's a one-shot read and then the plan
could be unlocked, at least as<br>
far as I know.<br>
<br>
<br>
Good point. That would be holding the lock
for longer than we need.<br>
<br>
The only option where we wouldn't need
locking at all is the<br>
read-copy-update model Clint mentions,
which might be a valid option as<br>
well. Whatever we do, there are going
to be concurrency issues though.<br>
For example, what happens if two users
try to make updates to the plan<br>
at the same time? If you don't either
merge the changes or disallow one<br>
of them completely then one user's
changes might be lost.<br>
<br>
TBH, this is further convincing me that
we should just make this git<br>
backed and let git handle the merging
and conflict resolution (never<br>
mind the fact that it gets us a
well-understood version control system<br>
for "free"). For updates that don't
conflict with other changes, git<br>
can merge them automatically, but for
merge conflicts you just return a<br>
rebase error to the user and make them
resolve it. I have a feeling<br>
this is the behavior we'll converge on
eventually anyway, and rather<br>
than reimplement git, let's just use
the real thing.<br>
<br>
<br>
I'd be curious to hear more how you would go
about doing this with git. I've<br>
never automated git to this level, so I am
concerned about what issues we<br>
might hit.<br>
</blockquote>
TBH I haven't thought it through to that
extent yet. I'm mostly<br>
suggesting it because it seems like a fit for
the template storage<br>
requirements - we know we want version
control, we want to be able to<br>
merge changes from multiple sources, and we
want some way to handle<br>
merge conflicts. Git does all of this
already.<br>
<br>
That said, I'm not sure about everything
here. For example, how would<br>
you expose merge conflicts to the user? I
don't know that I would want<br>
to force a user to learn git in order to use
TripleO (although that<br>
would be the devops-y thing to do), but maybe
just passing them back the<br>
files with the merge conflict markers and
having them resolve those<br>
locally and retry the update would work. I'm
not sure how that would<br>
map to the current version of the API though.
Do we provide any way to<br>
pass templates back to the user? I feel like
that was kind of a one-way<br>
street.<br>
</blockquote>
What part of the deployment API workflow could
result in merge conflicts?<br>
<br>
My understanding was that it's something like:<br>
<br>
1. Take copy of reference templates tree<br>
2. Introspect tempalates, expose required
parameters so user can be<br>
prompted for them<br>
3. Create environment files(s) derived from the
user input<br>
4. Validate the combination of (1) and (3)<br>
5. Deploy the templates+environments<br>
<br>
On update, (1) would be "overwrite existing
version of templates"<br>
</blockquote>
This update policy means you may have just blown
away someone else's<br>
work, unless you rebase on the plan's templates
immediately before<br>
updating (and even then there's a race if two
people submit updates at<br>
the same time).<br>
</blockquote>
What has been proposed to date is somewhat more
limited in scope than what<br>
you're hinting at (which I think is more of a
colloborate-on-templates<br>
requirement?)<br>
<br>
<a moz-do-not-send="true"
href="https://github.com/openstack/tripleo-specs/blob/master/specs/mitaka/tripleo-overcloud-deployment-library.rst"
rel="noreferrer" target="_blank">https://github.com/openstack/tripleo-specs/blob/master/specs/mitaka/tripleo-overcloud-deployment-library.rst</a><br>
<br>
Here, you would expect any template collaboration to
happen outside of the<br>
scope of the actual deployment workflow, so e.g step
1 above consumes<br>
either a packaged version of tripleo-heat-templates
(which we don't expect<br>
to be routinely modified), or another location on
the local filesystem<br>
(such as a repository managed by e.g git, outside of
the deployment<br>
workflow).<br>
<br>
The "plan" then takes a copy of the golden tree,
prompts for additional<br>
inputs, validates and deploys it.<br>
<br>
You are right though, if we allow concurrent update
of the plan, it's<br>
possible that environments added to two versions of
the plan would have to<br>
be merged, which could mean either conflicts or
validation errors (if two<br>
operators select mutually exclusive configurations
for example).<br>
<br>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
Possible example: Two operators are working on
enabling separate<br>
features in their cloud, and need to make
configuration changes to the<br>
plan to do so. Let's say one decides they need to
enable the Storage<br>
network, while the other decides to enable the
Tenant network. The<br>
first operator makes their changes, sends the
update and thinks their<br>
work is done. The second operator, working from
the same base set of<br>
templates as the first, makes their changes and
sends the update. Using<br>
the "overwrite" method of conflict resolution the
first operator's<br>
changes have just been silently destroyed with no
indication to either<br>
user that anything bad happened.<br>
</blockquote>
Ok, so separating the two requirements alluded to
here may help improve<br>
clarity:<br>
<br>
1. Multiple users collaborating on the t-h-t tree as
a whole.<br>
<br>
2. Enabling multiple features via updates and
avoiding mid-air-collisions<br>
<br>
I think (2) may simpler problem to consider,
particularly if a lock<br>
of some sort is considered acceptable, e.g we
explcitly do not allow multiple<br>
operators actively modifying the cloud concurrently.<br>
<br>
That would also be consistent with the current heat
behavior, e.g even if<br>
you did allow multiple operators to concurrently
change a plan, they cannot<br>
concurrently update the overcloud via heat anyway
(this will change<br>
eventually with convergence).<br>
<br>
(1) is a much harder problem, and I can't help
thinking it'd be better<br>
solved with existing tools (e.g document how to use
git, gerrit, jenkins &<br>
CI test your own t-h-t tree, potentially allowing
for semi-automated<br>
promotion of things between environments, a staging
workflow).<br>
<br>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
I guess you could tell users "don't do that", but
unless you have<br>
exactly one person making updates to the templates
there's going to be<br>
the possibility of conflicts, and in the Swift
case all it takes is two<br>
people editing the same file, even in completely
different areas, for<br>
someone's changes to be lost.<br>
</blockquote>
Ok, good point, I think I'd been assuming more of a
serialized workflow as<br>
a given, so it's definitely something to consider,
thanks for clarifying.<br>
<br>
Steve<br>
<br>
</div>
</div>
<span class="">
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage
questions)<br>
Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
</span><a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
<br>
To add the information here and maybe (hopefully) clear
things a bit, the current workflow does not manipulate
the templates and environments content.<br>
We only set the metadata about certain
templates/environments and create single temporary
environment file:<br>
<br>
1. Upload files (using git, it means provide git url) and
identify capabilities-map file (capabilities_map.yaml) and
set it's 'type' metadata to 'capabilities-map'<br>
</blockquote>
<div><br>
</div>
<div>I think we have multiple ideas related to git floating
around - using git as an external input source, or using
git as a data store that we update and manage and store on
the undercloud. Both seem valid.<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
2. based on the capabilities-map information, identify
'root-template' (overcloud.yaml), 'root-environment'
(overcloud-resource-registry-puppet.yaml), 'environment'
(environments/*.yaml) and store this information in those
files 'type' metadata.<br>
</blockquote>
<div><br>
</div>
<div>I don't think we need to set this metadata. We can use
the capabilities-map as an index and look up that file
each time we need this information.<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
Good point, that get's us rid of having to store those.<br>
<br>
<blockquote
cite="mid:CAPMB-2THenVJzBTHP3f140ipDRj49C1F3Yoj0NaG=5Pv=eLdmA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
3. Let user select from optional environments ('type' is
'environment') based on the constraints defined in
capabilities-map. Store the information about selected
environments in 'enabled' meta.<br>
</blockquote>
<div><br>
</div>
<div>The metadata for enabled is environments is important,
but I'll come back to this below.<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
4. Generate a list of parameters by sending templates,
root-environment and _enabled_ optional environments to
heat-validate (nested). Let user set values for those
parameters and store the parameter values in newly created
temporary environment's parameter_defaults block. Upload
this template to Swift and set it's 'type' meta to
'temp-environment'.<br>
5. Deploy - take everything from Swift, process templates
(to resolve the urls in get_file etc.) and merge
environments in order: root environment < enabled
optional environments < temporary environment. And send
this to Heat API's Stack Create.<br>
<br>
So you can see, that we don't really manipulate the
template files, we just add a metadata and create single
temporary environment that holds the parameter values, </blockquote>
<div><br>
</div>
<div>Don't we allow users to upload new template files or
update them? If users need to delete a plan and create a
new one for each version that sounds painful.<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<br>
<br>
<blockquote
cite="mid:CAPMB-2THenVJzBTHP3f140ipDRj49C1F3Yoj0NaG=5Pv=eLdmA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">although this is not
really necessary and can be replaced by storing the
parameter values in DB and then send this as 'parameters'
param to Heat. I think that storing files in Git is good
idea as it is what we already have (t-h-t) but we probably
need to use DB to store the metadata because the metadata
are plan-specific, whereas the Git repository is not (or
is it meant to be? That would mean creating separate git
repo for every deploymeny attempt.)<br>
</blockquote>
<div><br>
</div>
<div>I think we need to be careful how we store any
metadata. They key advantage (AFAICT) with storing the
files in git is that operators can easily access and
deploy them manually. However, if they need to understand
our bespoke metadata or extract it from a database to
understand the deploy then that advantage is lost. Maybe
rather than metadata we can update a file (or users can
add this file) that defines the deployment, this would be
similar to one that has been proposed to
python-tripleoclient[1]. If we can then support this file
in python-heatclient it would mean a deploy could easily
be understood from the API, python-tripleoclient and
python-heatclient. Even without heatclient supporting this
file, it is easy to look at and see how you would call
heatclient.<br>
</div>
<div><br>
[1]: <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/249222/">https://review.openstack.org/#/c/249222/</a><br>
<br>
</div>
<div>When we make a deploy, we will want to store the sha
that we have deployed, I am not sure where we want to
store this information. <br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
Ok, so this approach involves branching the git repo with a Plan
creation and the Plan metadata would get stored in the answers file
that gets committed to that branch. Sounds good. <br>
<br>
In regards to uploading/updating new templates, this sounds somewhat
counterproductive to me. Is there a use case for adding/changing
template as part of Plan design? IMO if we want to add template it
is usually done globally in t-h-t and not in Plan specific branch. I
don't see when we could need to do this. Adding environment is more
valid probably, but that would involve also updating the
capabilities map. We have the feature to add additional files to
plan currently because we use Swift and we have this step of
uploading files as part of plan creation. Using GIT, Plan creation
is just a matter of pointing to git repo.<br>
<br>
This is why I tend to not touch the files and just store the
metadata. Tying the metadata to the git repo (using answers file and
branching repo on Plan creation) is totally valid point.<br>
<br>
<blockquote
cite="mid:CAPMB-2THenVJzBTHP3f140ipDRj49C1F3Yoj0NaG=5Pv=eLdmA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
To make sure, that Plan is in sync with Git repo (t-h-t)
we can create the Plan is tied to not just specific
repository, but also to a specific tag or commit. This way
if the user updates the templates repository with changes
he wants to use, he needs to create a new Plan and start
over the deployment process.<br>
<br>
Correct me if I am wrong, but I think this approach
resolves the problems with merge conflicts. The Files and
Plan (Deployment) are separate thing - Files are stored in
Git and Plan is stored in DB, holds the files metadata and
is tied to a Git repo commit/tag.<br>
<br>
Any changes that involve the changes in templates themself
should be done in Git repo and I am not convinced that we
want to introduce anything like that in GUI/CLI deployment
workflow, as as it was agreed before, Git is best tool for
doing/tracking such changes.<br>
<br>
Jirka<span class=""><br>
<br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage
questions)<br>
Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
</span><a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
Jirka<br>
</body>
</html>