<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 15/01/14 06:21, Clint Byrum wrote:<br>
</div>
<blockquote cite="mid:1389716983-sup-8015@fewbar.com" type="cite">
<pre wrap="">Hey Steve, it has taken me about a month to get enough time to go
through this. Thanks for doing it, comments in-line.
Excerpts from Steve Baker's message of 2013-12-13 15:46:48 -0800:
</pre>
<blockquote type="cite">
<pre wrap="">I've been working on a POC in heat for resources which perform software
configuration, with the aim of implementing this spec
<a class="moz-txt-link-freetext" href="https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-spec">https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-spec</a>
The code to date is here:
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/q/topic:bp/hot-software-config,n,z">https://review.openstack.org/#/q/topic:bp/hot-software-config,n,z</a>
What would be helpful now is reviews which give the architectural
approach enough of a blessing to justify fleshing this POC out into a
ready to merge changeset.
Currently it is possible to:
- create templates containing OS::Heat::SoftwareConfig and
OS::Heat::SoftwareDeployment resources
- deploy configs to OS::Nova::Server, where the deployment resource
remains in an IN_PROGRESS state until it is signalled with the output values
- write configs which execute shell scripts and report back with output
values that other resources can have access to.
What follows is an overview of the architecture and implementation to
help with your reviews.
REST API
========
Like many heat resources, OS::Heat::SoftwareConfig and
OS::Heat::SoftwareDeployment are backed by "real" resources that are
invoked via a REST API. However in this case, the API that is called is
heat itself.
The REST API for these resources really just act as structured storage
for config and deployments, and the entities are managed via the REST
paths /{tenant_id}/software_configs and /{tenant_id}/software_deployments:
<a class="moz-txt-link-rfc2396E" href="https://review.openstack.org/#/c/58878/7/heat/api/openstack/v1/__init__.py"><https://review.openstack.org/#/c/58878/7/heat/api/openstack/v1/__init__.py></a><a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/58878/">https://review.openstack.org/#/c/58878/</a>
RPC layer of REST API:
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/58877/">https://review.openstack.org/#/c/58877/</a>
DB layer of REST API:
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/58876">https://review.openstack.org/#/c/58876</a>
heatclient lib access to REST API:
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/58885">https://review.openstack.org/#/c/58885</a>
This data could be stored in a less structured datastore like swift, but
this API has a couple of important implementation details which I think
justify it existing:
- SoftwareConfig resources are immutable once created. There is no
update API to modify an existing config. This gives confidence that a
config can have a long lifecycle without changing, and a certainty of
what exactly is deployed on a server with a given config.
- Fetching all the deployments and configs for a given server is an
operation done repeatedly throughout the lifecycle of the stack, so is
optimized to be able to do in a single operation. This is called by
using the deployments index API call,
/{tenant_id}/software_deployments?server_id=<server_id>. The resulting
list of deployments include the their associated config data[1].
</pre>
</blockquote>
<pre wrap="">
I'm curious if we can use the existing Metadata for this. That would
be attractive to me, as that would keep the software deployments data
available to CFN-API based tools. For instance ohai can read the CFN
API already, but not the Heat native in-instance API (since such a thing
basically does not exist).
I'm not suggesting they should both work at the same time. But I can't
see a time where they need to, so it seems logical enough to me to just
have the deployments bit populate the resource metadata, albeit with a
bit more structure. I understand that means updating resource metadata
after booting the server which may be the very reason not having them be
the same thing is simpler.
</pre>
</blockquote>
I *think* you've just describe exactly how it works. There is no
Metadata property in the HOT spec, but it is still in the model and
polling metadata via cfn-api is how the instance triggers deployment
work. One important point is that only the metadata for the server
resource needs to be polled, since this returns the deployment data
for all the deployments on that server.<br>
<blockquote cite="mid:1389716983-sup-8015@fewbar.com" type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">OS::Heat::SoftwareConfig resource
=================================
OS::Heat::SoftwareConfig can be used directly in a template, but it may
end be more frequently used in a resource provider template which
provides a resource aimed at a particular configuration management tool.
<a class="moz-txt-link-freetext" href="http://docs-draft.openstack.org/79/58879/7/check/gate-heat-docs/911a250/doc/build/html/template_guide/openstack.html#OS::Heat::SoftwareConfig">http://docs-draft.openstack.org/79/58879/7/check/gate-heat-docs/911a250/doc/build/html/template_guide/openstack.html#OS::Heat::SoftwareConfig</a>
The contents of the config property will depend on the CM tool being
used, but at least one value in the config map will be the actual script
that the CM tool invokes. An inputs and outputs schema is also defined
here. The group property is used when the deployments data is actually
delivered to the server (more on that later).
</pre>
</blockquote>
<pre wrap="">
Can you elaborate on "the actual script that the CM tool invokes" ?
</pre>
</blockquote>
In this context it means the CM tool's own configuration syntax,
such as a puppet manifest or a chef recipe.<br>
<blockquote cite="mid:1389716983-sup-8015@fewbar.com" type="cite">
<pre wrap="">
I have an extremely strong desire to never embed script code in an
orchestration template as it quickly becomes a quagmire of formatting
problems and poorly defined interfaces, IMO. </pre>
</blockquote>
Totally agreed, that is why I've been sidetracked recently on
implementing the get-file blueprint[1]. It is an inclusion mechanism
for getting non-heat syntax content from their own files into heat
requests using a new intrinsic function get_file.<br>
<blockquote cite="mid:1389716983-sup-8015@fewbar.com" type="cite">
<pre wrap="">Also my whole reason for
doing golden image based deployment is to have code all live in the
image and stay immutable through the deployed servers' life time.
So I want to make sure I don't _have_ to do the script for some reason.
</pre>
</blockquote>
No, you don't have to. For tools which just need a configuration
data structure like OAC then that data structure would be the
"script" in that case.<br>
<blockquote cite="mid:1389716983-sup-8015@fewbar.com" type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">Since a config is immutable, any changes to a OS::Heat::SoftwareConfig
on stack update result in replacement.
OS::Heat::SoftwareDeployment resource
=====================================
OS::Heat::SoftwareDeployment joins a OS::Heat::SoftwareConfig resource
with a OS::Nova::Server resource. It allows server-specific input values
to be specified that map to the OS::Heat::SoftwareConfig inputs schema.
Output values that are signaled to the deployment resource are exposed
as resource attributes, using the names specified in the outputs schema.
The OS::Heat::SoftwareDeployment resource remains in an IN_PROGRESS
state until it receives a signal (containing any outputs) from the server.
<a class="moz-txt-link-freetext" href="http://docs-draft.openstack.org/79/58879/7/check/gate-heat-docs/911a250/doc/build/html/template_guide/openstack.html#OS::Heat::SoftwareDeployment">http://docs-draft.openstack.org/79/58879/7/check/gate-heat-docs/911a250/doc/build/html/template_guide/openstack.html#OS::Heat::SoftwareDeployment</a>
A deployment has its own actions and statuses that are specific to what
a deployment does, and OS::Heat::SoftwareDeployment maps this to heat
resource statuses and actions:
actions:
DEPLOY -> CREATE
REDEPLOY -> UPDATE
UNDEPLOY -> DELETE
status (these could use some bikeshedding):
WAITING -> IN_PROGRESS
RECEIVED -> COMPLETE
FAILED -> FAILED
In the config outputs schema there is a special flag for error_output.
If the signal response contains any value for any of these error_output
outputs then the deployment resource is put into the FAILED state.
The SoftwareDeployment class subclasses SignalResponder which means that
a SoftwareDeployment creates an associated user and ec2 keypair. Since
the SoftwareDeployment needs to use the resource_id for the deployment
resource uuid, the user_id needs to be stored in resource-date instead.
This non-wip change enables that:
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/61902/">https://review.openstack.org/#/c/61902/</a>
During create, the deployment REST API is polled until status goes from
WAITING to RECEIVED. When handle_signal is called, the deployment is
updated via the REST API to set the status to RECEIVED (or FAILED),
along with any output values that were received.
One alarming consequence of having a deployments API is that any tenant
user can create a deployment for any heat-created nova server and that
software will be deployed to that server, which is, um, powerful.
</pre>
</blockquote>
<pre wrap="">
Seems like there is a whole policy can of worms there that all OpenStack
services probably need to address. AFAIK, any same-tenant user can just
delete any other user in the same tenant's servers, right? I'm sure
somebody with Keystone knowledge (I'm looking at you Mr. Hardy) would be
able to tell us if such things exist now and whether or not we could
make a stronger bond between user and stack.
</pre>
</blockquote>
Maybe for implementation zero we punt on the policy and add a check
which says a deployment resource must exist in the same stack as the
server resource (although it would be nice to support deployment and
server being in the same nested stack tree).<br>
<blockquote cite="mid:1389716983-sup-8015@fewbar.com" type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">There will need to be a deployment policy (probably an OS::Nova::Server
property) which limits to scope of what deployments are allowed on that
server. This could default to deployments in the same stack, but could
still allow deployments from anywhere.
OS::Nova::Server support
========================
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/58880">https://review.openstack.org/#/c/58880</a>
A new user_data_format=SOFTWARE_CONFIG is currently used to denote that
this server is configured via software config deployments. Like
user_data_format=HEAT_CFNTOOLS, nova_utils.build_userdata is used to
build the cloud-init parts required to support software config. However
like user_data_format=RAW anything specified in user_data will be parsed
as cloud-init data. If user_data is multi-part data then the parts will
be appended to the parts created in nova_utils.build_userdata.
The agent used currently is os-collect-config. This is typically
configured to poll for metadata from a particular heat resource via the
CFN API using the configured ec2 keypair. In the current implementation
the resource which is polled is the OS::Nova::Server itself, since this
is the only resource known to exist at server boot time (deployment
resources depend on server resources, so have not been created yet). The
ec2 keypair comes from a user created implicitly with the server
(similar to SignalResponder resources). This means the template author
doesn't need to include User/AccessKey/AccessPolicy resources in their
templates just to enable os-collect-config metadata polling.
</pre>
</blockquote>
<pre wrap="">
\o/
</pre>
<blockquote type="cite">
<pre wrap="">Until now, polling the metadata for a resource just returns the metadata
which has been stored in the stack resource database. This
implementation changes metadata polling to actually query the
deployments API to return the latest deployments data. This means
deployment state can be stored in one place, and there is no need to
keep various metadata stores updated with any changed state.
</pre>
</blockquote>
<pre wrap="">
I do like that very much. :)
</pre>
<blockquote type="cite">
<pre wrap="">An actual template
==================
<a class="moz-txt-link-freetext" href="http://paste.openstack.org/show/54988/">http://paste.openstack.org/show/54988/</a>
This template contains:
- a config resource
- 2 deployments which deploy that config with 2 different sets of inputs
- stack outputs which output the results of the deployments
- a server resource
- an os-refresh-config script delivered via cloud-config[2] which
executes config scripts with deployment inputs and signals outputs to
the provided webhook.
</pre>
</blockquote>
<pre wrap="">
You have demonstrated exactly what I need here. One config definition, two
deployments of it on the same server with different parameters. Huzzah!
I'm a little confused how the config.script gets access to $bar and $foo
though. ?
</pre>
</blockquote>
In
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a href="http://paste.openstack.org/show/54988/#120">http://paste.openstack.org/show/54988/</a>
the orc script on line 120 takes the<br>
deployment inputs and builds the shell environment from that. Hooks
for<br>
different CM tools would do something else, such as building up a
command-line<br>
invocation which transforms the inputs to invocation parameters.<br>
<a href="http://paste.openstack.org/show/54988/#120"></a>
<blockquote cite="mid:1389716983-sup-8015@fewbar.com" type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">
/opt/stack/os-config-refresh/configure.d/55-heat-config-bash is a hook
specific for performing configuration via shell scripts, and only acts
on software config which has group=Heat::Shell. Each configuration
management tool will have its own hook, and will act on its own group
namespace. Each configuration management tool will also have its own way
of passing inputs and outputs. The hooks job is to invoke the CM tool
with the given inputs and script, then extract the outputs and signal heat.
The server needs to have the CM tool and the hook already installed,
either by building a golden image or by using cloud-config during boot.
Next steps
==========
There is a lot left to do and I'd like to spread the development load.
What happens next entirely depends on feedback to this POC, but here is
my ideal scenario:
- any feedback which causes churn on many of the current changes I will
address
- a volunteer is found to take the REST API/RPC/DB/heatclient changes
and make them ready to merge
</pre>
</blockquote>
<pre wrap="">
Using this is a mid-term goal for me, but the Tuskar devs would probably
love to drop my old terrible merge.py and start working on translating
the TripleO templates to using this. So you will find a mid-term
volunteer in me, but you might find a sooner volunteer in them. :)
</pre>
</blockquote>
I'd be happy to help out if anyone decides to tackle this.<br>
<br>
<blockquote cite="mid:1389716983-sup-8015@fewbar.com" type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">- we continue to discuss and refine the resources, the changes to
OS::Nova::Server, and the example shell hook
- volunteers write hooks for different CM tools, Chef and Puppet hooks
will need to be attempted soon to validate this approach.
Vaguely related changes include:
- Some solution for specifying cloud-init config, either the intrinsic
functions or cloud-init heat resources
- Some heatclient file inclusion mechanism - writing that python hook in
a heat yaml template was a bit painful ;)
</pre>
</blockquote>
<pre wrap="">
This has been an open question for a long time. Absent golden images, I
think you just need a simple way to express where to fetch the hooks from.
That could be hidden behind the SoftwareConfig resource so that the url
has a default where the cloud operator has put tools. Users could then
just override that url when they want to write their own hooks.
</pre>
</blockquote>
So one thing which has moved on since I wrote this email is that the
cloud-config intrinsic functions have been ditched, and replaced
with cloud-config SoftwareConfig resources [2]. These are for
boot-only config, so instead of a deployment resource. I'll
demonstrate how this works at a later date, but this could fulfill
the above case either by referencing long-lived software config
resources which contain the hooks, or by defining the config
resource in your template and pulling in the hook with a get_file
call to a URL.<br>
<blockquote cite="mid:1389716983-sup-8015@fewbar.com" type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">
Trying for yourself
===================
- Using diskimage-builder, create an ubuntu image with
tripleo-image-elements os-apply-config, os-refresh-config and
os-collect-config
- Create a local heat branch containing
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/q/topic:bp/cloud-init-resource,n,z">https://review.openstack.org/#/q/topic:bp/cloud-init-resource,n,z</a> and
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/q/topic:bp/hot-software-config,n,z">https://review.openstack.org/#/q/topic:bp/hot-software-config,n,z</a>
- launch the above template with your created image
</pre>
</blockquote>
<pre wrap="">
At the time you sent this I think I got this working.
</pre>
</blockquote>
When it is easy to consume I'll send out another email describing
how to try this locally. The hooks need to be hosted somewhere - I
think the best place for now might be the heat-templates repo. I'll
attempt to come up with a directory structure to contain all the
appropriate templates, hooks, dib elements, READMEs and examples.<br>
<blockquote cite="mid:1389716983-sup-8015@fewbar.com" type="cite">
<pre wrap="">
Very excited to help move this forward as much as I can.
</pre>
</blockquote>
Jun Jie Nan has taken over the REST commits [3]. It would be good to
get as much of this as possible into i-2 so reviews on all of the
below would be very helpful.<br>
<br>
cheers<br>
<br>
[1]
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a href="https://review.openstack.org/#/q/topic:bp/get-file,n,z">https://review.openstack.org/#/q/topic:bp/get-file,n,z</a><br>
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a href="https://blueprints.launchpad.net/heat/+spec/get-file">https://blueprints.launchpad.net/heat/+spec/get-file</a><br>
[2]
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a href="https://review.openstack.org/#/c/63214/">https://review.openstack.org/#/c/63214/</a><br>
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a href="https://review.openstack.org/#/c/63215/">https://review.openstack.org/#/c/63215/</a><br>
[3]
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a
href="https://review.openstack.org/#/q/topic:bp/hot-software-config-rest,n,z">https://review.openstack.org/#/q/topic:bp/hot-software-config-rest,n,z</a><br>
<br>
<br>
</body>
</html>