<html>
  <head>
    <meta content="text/html; charset=windows-1252"
      http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <div class="moz-cite-prefix">On 01/06/2016 11:48 AM, Dougal Matthews
      wrote:<br>
    </div>
    <blockquote
cite="mid:CAPMB-2THenVJzBTHP3f140ipDRj49C1F3Yoj0NaG=5Pv=eLdmA@mail.gmail.com"
      type="cite">
      <div dir="ltr"><br>
        <div class="gmail_extra"><br>
          <div class="gmail_quote">On 5 January 2016 at 17:09, Jiri
            Tomasek <span dir="ltr"><<a moz-do-not-send="true"
                href="mailto:jtomasek@redhat.com" target="_blank">jtomasek@redhat.com</a>></span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
              0.8ex;border-left:1px solid
              rgb(204,204,204);padding-left:1ex">
              <div class="">
                <div class="h5">On 12/23/2015 07:40 PM, Steven Hardy
                  wrote:<br>
                </div>
              </div>
              <blockquote class="gmail_quote" style="margin:0px 0px 0px
                0.8ex;border-left:1px solid
                rgb(204,204,204);padding-left:1ex">
                <div>
                  <div class="h5">
                    On Wed, Dec 23, 2015 at 11:05:05AM -0600, Ben Nemec
                    wrote:<br>
                    <blockquote class="gmail_quote" style="margin:0px
                      0px 0px 0.8ex;border-left:1px solid
                      rgb(204,204,204);padding-left:1ex">
                      On 12/23/2015 10:26 AM, Steven Hardy wrote:<br>
                      <blockquote class="gmail_quote" style="margin:0px
                        0px 0px 0.8ex;border-left:1px solid
                        rgb(204,204,204);padding-left:1ex">
                        On Wed, Dec 23, 2015 at 09:28:59AM -0600, Ben
                        Nemec wrote:<br>
                        <blockquote class="gmail_quote"
                          style="margin:0px 0px 0px
                          0.8ex;border-left:1px solid
                          rgb(204,204,204);padding-left:1ex">
                          On 12/23/2015 03:19 AM, Dougal Matthews wrote:<br>
                          <blockquote class="gmail_quote"
                            style="margin:0px 0px 0px
                            0.8ex;border-left:1px solid
                            rgb(204,204,204);padding-left:1ex">
                            <br>
                            On 22 December 2015 at 17:59, Ben Nemec <<a
                              moz-do-not-send="true"
                              href="mailto:openstack@nemebean.com"
                              target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:openstack@nemebean.com">openstack@nemebean.com</a></a><br>
                            <mailto:<a moz-do-not-send="true"
                              href="mailto:openstack@nemebean.com"
                              target="_blank">openstack@nemebean.com</a>>>
                            wrote:<br>
                            <br>
                                 Can we just do git like I've been
                            suggesting all along? ;-)<br>
                            <br>
                                 More serious discussion inline. :-)<br>
                            <br>
                                 On 12/22/2015 09:36 AM, Dougal Matthews
                            wrote:<br>
                                 > Hi all,<br>
                                 ><br>
                                 > This topic came up in the
                            2015-12-15 meeting[1], and again briefly<br>
                                 today.<br>
                                 > After working with the code that
                            came out of the deployment library<br>
                                 > spec[2] I<br>
                                 > had some concerns with how we are
                            storing the templates.<br>
                                 ><br>
                                 > Simply put, when we are dealing
                            with 100+ files from<br>
                                 tripleo-heat-templates<br>
                                 > how can we ensure consistency in
                            Swift without any atomicity or<br>
                                 > transactions.<br>
                                 > I think this is best explained
                            with a couple of examples.<br>
                                 ><br>
                                 >  - When we create a new deployment
                            plan (upload all the templates<br>
                                 to swift)<br>
                                 >    how do we handle the case where
                            there is an error? For example,<br>
                                 if we are<br>
                                 >    uploading 10 files - what do we
                            do if the 5th one fails for<br>
                                 some reason?<br>
                                 >    There is a patch to do a manual
                            rollback[3], but I have<br>
                                 concerns about<br>
                                 >    doing this in Python. If Swift
                            is completely inaccessible for a<br>
                                 short<br>
                                 >    period the rollback wont work
                            either.<br>
                                 ><br>
                                 >  - When deploying to Heat, we need
                            to download all the YAML files from<br>
                                 > Swift.<br>
                                 >    This can take a couple of
                            seconds. What happens if somebody<br>
                                 starts to<br>
                                 >    upload a new version of the
                            plan in the middle? We could end up<br>
                                 trying to<br>
                                 >    deploy half old and half new
                            files. We wouldn't have a<br>
                                 consistent view of<br>
                                 >    the database.<br>
                                 ><br>
                                 > We had a few suggestions in the
                            meeting:<br>
                                 ><br>
                                 >  - Add a locking mechanism. I
                            would be concerned about deadlocks or<br>
                                 > having to<br>
                                 >    lock for the full duration of a
                            deploy.<br>
                            <br>
                                 There should be no need to lock the
                            plan for the entire deploy.  It's<br>
                                 not like we're re-reading the templates
                            at the end of the deploy today.<br>
                                  It's a one-shot read and then the plan
                            could be unlocked, at least as<br>
                                 far as I know.<br>
                            <br>
                            <br>
                            Good point. That would be holding the lock
                            for longer than we need.<br>
                              <br>
                                 The only option where we wouldn't need
                            locking at all is the<br>
                                 read-copy-update model Clint mentions,
                            which might be a valid option as<br>
                                 well.  Whatever we do, there are going
                            to be concurrency issues though.<br>
                                  For example, what happens if two users
                            try to make updates to the plan<br>
                                 at the same time?  If you don't either
                            merge the changes or disallow one<br>
                                 of them completely then one user's
                            changes might be lost.<br>
                            <br>
                                 TBH, this is further convincing me that
                            we should just make this git<br>
                                 backed and let git handle the merging
                            and conflict resolution (never<br>
                                 mind the fact that it gets us a
                            well-understood version control system<br>
                                 for "free").  For updates that don't
                            conflict with other changes, git<br>
                                 can merge them automatically, but for
                            merge conflicts you just return a<br>
                                 rebase error to the user and make them
                            resolve it.  I have a feeling<br>
                                 this is the behavior we'll converge on
                            eventually anyway, and rather<br>
                                 than reimplement git, let's just use
                            the real thing.<br>
                            <br>
                            <br>
                            I'd be curious to hear more how you would go
                            about doing this with git. I've<br>
                            never automated git to this level, so I am
                            concerned about what issues we<br>
                            might hit.<br>
                          </blockquote>
                          TBH I haven't thought it through to that
                          extent yet.  I'm mostly<br>
                          suggesting it because it seems like a fit for
                          the template storage<br>
                          requirements - we know we want version
                          control, we want to be able to<br>
                          merge changes from multiple sources, and we
                          want some way to handle<br>
                          merge conflicts.  Git does all of this
                          already.<br>
                          <br>
                          That said, I'm not sure about everything
                          here.  For example, how would<br>
                          you expose merge conflicts to the user?  I
                          don't know that I would want<br>
                          to force a user to learn git in order to use
                          TripleO (although that<br>
                          would be the devops-y thing to do), but maybe
                          just passing them back the<br>
                          files with the merge conflict markers and
                          having them resolve those<br>
                          locally and retry the update would work.  I'm
                          not sure how that would<br>
                          map to the current version of the API though. 
                          Do we provide any way to<br>
                          pass templates back to the user?  I feel like
                          that was kind of a one-way<br>
                          street.<br>
                        </blockquote>
                        What part of the deployment API workflow could
                        result in merge conflicts?<br>
                        <br>
                        My understanding was that it's something like:<br>
                        <br>
                        1. Take copy of reference templates tree<br>
                        2. Introspect tempalates, expose required
                        parameters so user can be<br>
                        prompted for them<br>
                        3. Create environment files(s) derived from the
                        user input<br>
                        4. Validate the combination of (1) and (3)<br>
                        5. Deploy the templates+environments<br>
                        <br>
                        On update, (1) would be "overwrite existing
                        version of templates"<br>
                      </blockquote>
                      This update policy means you may have just blown
                      away someone else's<br>
                      work, unless you rebase on the plan's templates
                      immediately before<br>
                      updating (and even then there's a race if two
                      people submit updates at<br>
                      the same time).<br>
                    </blockquote>
                    What has been proposed to date is somewhat more
                    limited in scope than what<br>
                    you're hinting at (which I think is more of a
                    colloborate-on-templates<br>
                    requirement?)<br>
                    <br>
                    <a moz-do-not-send="true"
href="https://github.com/openstack/tripleo-specs/blob/master/specs/mitaka/tripleo-overcloud-deployment-library.rst"
                      rel="noreferrer" target="_blank">https://github.com/openstack/tripleo-specs/blob/master/specs/mitaka/tripleo-overcloud-deployment-library.rst</a><br>
                    <br>
                    Here, you would expect any template collaboration to
                    happen outside of the<br>
                    scope of the actual deployment workflow, so e.g step
                    1 above consumes<br>
                    either a packaged version of tripleo-heat-templates
                    (which we don't expect<br>
                    to be routinely modified), or another location on
                    the local filesystem<br>
                    (such as a repository managed by e.g git, outside of
                    the deployment<br>
                    workflow).<br>
                    <br>
                    The "plan" then takes a copy of the golden tree,
                    prompts for additional<br>
                    inputs, validates and deploys it.<br>
                    <br>
                    You are right though, if we allow concurrent update
                    of the plan, it's<br>
                    possible that environments added to two versions of
                    the plan would have to<br>
                    be merged, which could mean either conflicts or
                    validation errors (if two<br>
                    operators select mutually exclusive configurations
                    for example).<br>
                    <br>
                    <blockquote class="gmail_quote" style="margin:0px
                      0px 0px 0.8ex;border-left:1px solid
                      rgb(204,204,204);padding-left:1ex">
                      Possible example: Two operators are working on
                      enabling separate<br>
                      features in their cloud, and need to make
                      configuration changes to the<br>
                      plan to do so.  Let's say one decides they need to
                      enable the Storage<br>
                      network, while the other decides to enable the
                      Tenant network.  The<br>
                      first operator makes their changes, sends the
                      update and thinks their<br>
                      work is done.  The second operator, working from
                      the same base set of<br>
                      templates as the first, makes their changes and
                      sends the update.  Using<br>
                      the "overwrite" method of conflict resolution the
                      first operator's<br>
                      changes have just been silently destroyed with no
                      indication to either<br>
                      user that anything bad happened.<br>
                    </blockquote>
                    Ok, so separating the two requirements alluded to
                    here may help improve<br>
                    clarity:<br>
                    <br>
                    1. Multiple users collaborating on the t-h-t tree as
                    a whole.<br>
                    <br>
                    2. Enabling multiple features via updates and
                    avoiding mid-air-collisions<br>
                    <br>
                    I think (2) may simpler problem to consider,
                    particularly if a lock<br>
                    of some sort is considered acceptable, e.g we
                    explcitly do not allow multiple<br>
                    operators actively modifying the cloud concurrently.<br>
                    <br>
                    That would also be consistent with the current heat
                    behavior, e.g even if<br>
                    you did allow multiple operators to concurrently
                    change a plan, they cannot<br>
                    concurrently update the overcloud via heat anyway
                    (this will change<br>
                    eventually with convergence).<br>
                    <br>
                    (1) is a much harder problem, and I can't help
                    thinking it'd be better<br>
                    solved with existing tools (e.g document how to use
                    git, gerrit, jenkins &<br>
                    CI test your own t-h-t tree, potentially allowing
                    for semi-automated<br>
                    promotion of things between environments, a staging
                    workflow).<br>
                    <br>
                    <blockquote class="gmail_quote" style="margin:0px
                      0px 0px 0.8ex;border-left:1px solid
                      rgb(204,204,204);padding-left:1ex">
                      I guess you could tell users "don't do that", but
                      unless you have<br>
                      exactly one person making updates to the templates
                      there's going to be<br>
                      the possibility of conflicts, and in the Swift
                      case all it takes is two<br>
                      people editing the same file, even in completely
                      different areas, for<br>
                      someone's changes to be lost.<br>
                    </blockquote>
                    Ok, good point, I think I'd been assuming more of a
                    serialized workflow as<br>
                    a given, so it's definitely something to consider,
                    thanks for clarifying.<br>
                    <br>
                    Steve<br>
                    <br>
                  </div>
                </div>
                <span class="">
__________________________________________________________________________<br>
                  OpenStack Development Mailing List (not for usage
                  questions)<br>
                  Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
                    rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
                </span><a moz-do-not-send="true"
                  href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
                  rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
              </blockquote>
              <br>
              To add the information here and maybe (hopefully) clear
              things a bit,  the current workflow does not manipulate
              the templates and environments content.<br>
              We only set the metadata about certain
              templates/environments and create single temporary
              environment file:<br>
              <br>
              1. Upload files (using git, it means provide git url) and
              identify capabilities-map file (capabilities_map.yaml) and
              set it's 'type' metadata to 'capabilities-map'<br>
            </blockquote>
            <div><br>
            </div>
            <div>I think we have multiple ideas related to git floating
              around - using git as an external input source, or using
              git as a data store that we update and manage and store on
              the undercloud. Both seem valid.<br>
            </div>
            <div> </div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
              0.8ex;border-left:1px solid
              rgb(204,204,204);padding-left:1ex">
              2. based on the capabilities-map information, identify
              'root-template' (overcloud.yaml), 'root-environment'
              (overcloud-resource-registry-puppet.yaml), 'environment'
              (environments/*.yaml) and store this information in those
              files 'type' metadata.<br>
            </blockquote>
            <div><br>
            </div>
            <div>I don't think we need to set this metadata. We can use
              the capabilities-map as an index and look up that file
              each time we need this information.<br>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    Good point, that get's us rid of having to store those.<br>
    <br>
    <blockquote
cite="mid:CAPMB-2THenVJzBTHP3f140ipDRj49C1F3Yoj0NaG=5Pv=eLdmA@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">
            <div> </div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
              0.8ex;border-left:1px solid
              rgb(204,204,204);padding-left:1ex">
              3. Let user select from optional environments ('type' is
              'environment') based on the constraints defined in
              capabilities-map. Store the information about selected
              environments in 'enabled' meta.<br>
            </blockquote>
            <div><br>
            </div>
            <div>The metadata for enabled is environments is important,
              but I'll come back to this below.<br>
            </div>
            <div> </div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
              0.8ex;border-left:1px solid
              rgb(204,204,204);padding-left:1ex">
              4. Generate a list of parameters by sending templates,
              root-environment and _enabled_ optional environments to
              heat-validate (nested). Let user set values for those
              parameters and store the parameter values in newly created
              temporary environment's parameter_defaults block. Upload
              this template to Swift and set it's 'type' meta to
              'temp-environment'.<br>
              5. Deploy - take everything from Swift, process templates
              (to resolve the urls in get_file etc.) and merge
              environments in order: root environment < enabled
              optional environments < temporary environment. And send
              this to Heat API's Stack Create.<br>
              <br>
              So you can see, that we don't really manipulate the
              template files, we just add a metadata and create single
              temporary environment that holds the parameter values, </blockquote>
            <div><br>
            </div>
            <div>Don't we allow users to upload new template files or
              update them? If users need to delete a plan and create a
              new one for each version that sounds painful.<br>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    <br>
    <br>
    <blockquote
cite="mid:CAPMB-2THenVJzBTHP3f140ipDRj49C1F3Yoj0NaG=5Pv=eLdmA@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">
            <div> </div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
              0.8ex;border-left:1px solid
              rgb(204,204,204);padding-left:1ex">although this is not
              really necessary and can be replaced by storing the
              parameter values in DB and then send this as 'parameters'
              param to Heat.  I think that storing files in Git is good
              idea as it is what we already have (t-h-t) but we probably
              need to use DB to store the metadata because the metadata
              are plan-specific, whereas the Git repository is not (or
              is it meant to be? That would mean creating separate git
              repo for every deploymeny attempt.)<br>
            </blockquote>
            <div><br>
            </div>
            <div>I think we need to be careful how we store any
              metadata. They key advantage (AFAICT) with storing the
              files in git is that operators can easily access and
              deploy them manually. However, if they need to understand
              our bespoke metadata or extract it from a database to
              understand the deploy then that advantage is lost. Maybe
              rather than metadata we can update a file (or users can
              add this file) that defines the deployment, this would be
              similar to one that has been proposed to
              python-tripleoclient[1]. If we can then support this file
              in python-heatclient it would mean a deploy could easily
              be understood from the API, python-tripleoclient and
              python-heatclient. Even without heatclient supporting this
              file, it is easy to look at and see how you would call
              heatclient.<br>
            </div>
            <div><br>
              [1]: <a moz-do-not-send="true"
                href="https://review.openstack.org/#/c/249222/">https://review.openstack.org/#/c/249222/</a><br>
              <br>
            </div>
            <div>When we make a deploy, we will want to store the sha
              that we have deployed, I am not sure where we want to
              store this information.  <br>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    Ok, so this approach involves branching the git repo with a Plan
    creation and the Plan metadata would get stored in the answers file
    that gets committed to that branch. Sounds good. <br>
    <br>
    In regards to uploading/updating new templates, this sounds somewhat
    counterproductive to me. Is there a use case for adding/changing
    template as part of Plan design? IMO if we want to add template it
    is usually done globally in t-h-t and not in Plan specific branch. I
    don't see when we could need to do this. Adding environment is more
    valid probably, but that would involve also updating the
    capabilities map. We have the feature to add additional files to
    plan currently because we use Swift and we have this step of
    uploading files as part of plan creation. Using GIT, Plan creation
    is just a matter of pointing to git repo.<br>
    <br>
    This is why I tend to not touch the files and just store the
    metadata. Tying the metadata to the git repo (using answers file and
    branching repo on Plan creation) is totally valid point.<br>
    <br>
    <blockquote
cite="mid:CAPMB-2THenVJzBTHP3f140ipDRj49C1F3Yoj0NaG=5Pv=eLdmA@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">
            <div><br>
              <br>
            </div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
              0.8ex;border-left:1px solid
              rgb(204,204,204);padding-left:1ex">
              To make sure, that Plan is in sync with Git repo (t-h-t)
              we can create the Plan is tied to not just specific
              repository, but also to a specific tag or commit. This way
              if the user updates the templates repository with changes
              he wants to use, he needs to create a new Plan and start
              over the deployment process.<br>
              <br>
              Correct me if I am wrong, but I think this approach
              resolves the problems with merge conflicts. The Files and
              Plan (Deployment) are separate thing - Files are stored in
              Git and Plan is stored in DB, holds the files metadata and
              is tied to a Git repo commit/tag.<br>
              <br>
              Any changes that involve the changes in templates themself
              should be done in Git repo and I am not convinced that we
              want to introduce anything like that in GUI/CLI deployment
              workflow, as as it was agreed before, Git is best tool for
              doing/tracking such changes.<br>
              <br>
              Jirka<span class=""><br>
                <br>
                <br>
                <br>
__________________________________________________________________________<br>
                OpenStack Development Mailing List (not for usage
                questions)<br>
                Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
                  rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
              </span><a moz-do-not-send="true"
                href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
                rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
            </blockquote>
          </div>
          <br>
        </div>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
    </blockquote>
    <br>
    Jirka<br>
  </body>
</html>