<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 01/08/2014 06:57 PM, Prasad Vellanki
wrote:<br>
</div>
<blockquote
cite="mid:CADstcNaGpTcq57hZNvHGh6n99xuKJufqNo9p4JgqnonT6ie-VQ@mail.gmail.com"
type="cite">
<div dir="ltr">Clint & Steve
<div>One scenario we are trying to see is whether and how Heat
software-config enables deployment of images available from
third party as virtual appliances, providing network,
security or acceleration capabilities. The vendor in some
cases might not allow rebuilding and/or may not have the
cloud init capability.Sometimes changes to the image could run
into issues with licensing. Bootstrapping in such situations
is generally done via rest api or ssh once the appliance boots
up where one can bootstrap it further. </div>
<div><br>
</div>
<div>We are looking at how to automate deployment of such
service functions using new configuration and deployment
model in Heat which we really like. </div>
<div><br>
</div>
<div>One option is that software-config can provide an option in
Heat to trigger bootstrapping that can be done from outside
rather than inside, as done by cloud-init, and does
bootstrapping of appliances using ssh and/or rest. </div>
<div><br>
</div>
<div>Another option is there could be an agent outside that
recognizes this kind of service coming up and then inform Heat
to go to next state to configure the deployed resource. This
is more like a proxy model.</div>
<div><br>
</div>
<div>thanks</div>
<div>prasadv</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
</div>
</blockquote>
Prasad,<br>
<br>
Just to clarify, you want Heat to facilitate bootstrapping a black
box (third-party virtual appliance) that has no consistent
bootstrapping interface (such as cloud-init)? The solution you
propose I think goes along the lines of having Heat notify an
out-of-vm bootstrapping system (such as SSH) to connect to the black
box and execute the bootstrapping.<br>
<br>
If so, I see a problem with this approach:<br>
Heat requires the running of commands inside the virtual machine to
know when the virtual machine is done bootstrapping, for whatever
definition of bootstrapping you use (OS booted, vs OS loaded and
ready to provide service).<br>
<br>
This could be handled by modifying the init scripts to signal the
end of booting, but one constraint you mentioned was that images may
not be modified.<br>
<br>
Another approach that could be used today is to constantly connect
to the SSH port of the VM until you receive a connection. The
problem with this approach is who loads the ssh keys into the
image? SSH key injection is currently handled by the bootstrapping
process. This is a chicken-egg problem and a fundamental reason why
bootstrapping should be done internally to the virtual machine
driven by Heat. Assuming this model were used, a notification that
the booting process has completed is only an optimization to
indicate when SSH harassment should begin :)<br>
<br>
One possible workaround, mentioned in your use case is that the
virtual appliance contacts a REST server (to obtain bootstrapping
information, including possibly SSH keys). Since I assume these
virtual appliances come from different vendors, this would result in
REST bootstrapping server proliferation which is bad for operators
as each server has to be secure-ified, scale-ified, HA-ifed, and
documented.<br>
<br>
The path of least resistance in this case seems to be to influence
the appliance vendors to adopt cloud-init rather then do unnatural
acts inside infrastructure to support appliance vendors who are
unwilling to conform to Open Source choices made by a broad
community of technology experts (in this case, I mean not just the
OpenStack community, but rather nearly every cloud vendor has made
cloudinit central to their solutions). Since the appliance vendors
will add cloud-init to their image sooner or later due to operator /
customer pressure, it is also the right choice today.<br>
<br>
It is as simple as adding one package to the built image. In
exchange, from a bootstrapping perspective, their customers get a
simple secure reliable scalable highly available experience on
OpenStack and likely other IAAS platforms.<br>
<br>
Regards<br>
-steve<br>
<br>
<br>
<blockquote
cite="mid:CADstcNaGpTcq57hZNvHGh6n99xuKJufqNo9p4JgqnonT6ie-VQ@mail.gmail.com"
type="cite">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Jan 7, 2014 at 11:40 AM, Clint
Byrum <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:clint@fewbar.com" target="_blank">clint@fewbar.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">I'd say it
isn't so much cloud-init that you need, but "some kind<br>
of bootstrapper". The point of hot-software-config is to
help with<br>
in-instance orchestration. That's not going to happen
without some way<br>
to push the desired configuration into the instance.<br>
<br>
Excerpts from Susaant Kondapaneni's message of 2014-01-07
11:16:16 -0800:<br>
<div>
<div class="h5">> We work with images provided by
vendors over which we do not always have<br>
> control. So we are considering the cases where
vendor image does not come<br>
> installed with cloud-init. Is there a way to
support heat software config<br>
> in such scenarios?<br>
><br>
> Thanks<br>
> Susaant<br>
><br>
> On Mon, Jan 6, 2014 at 4:47 PM, Steve Baker <<a
moz-do-not-send="true" href="mailto:sbaker@redhat.com">sbaker@redhat.com</a>>
wrote:<br>
><br>
> > On 07/01/14 06:25, Susaant Kondapaneni wrote:<br>
> ><br>
> > Hi Steve,<br>
> ><br>
> > I am trying to understand the software config
implementation. Can you<br>
> > clarify the following:<br>
> ><br>
> > i. To use Software config and deploy in a
template, instance resource<br>
> > MUST always be accompanied by user_data.
User_data should specify how to<br>
> > bootstrap CM tool and signal it. Is that
correct?<br>
> ><br>
> > Yes, currently the user_data contains
cfn-init formatted metadata which<br>
> > tells os-collect-config how to poll for config
changes. What happens when<br>
> > new config is fetched depends on the
os-apply-config templates and<br>
> > os-refresh-config scripts which are already on
that image (or set up with<br>
> > cloud-init).<br>
> ><br>
> > ii. Supposing we were to use images which do
not have cloud-init<br>
> > packaged in them, (and a custom CM tool that
won't require bootstrapping on<br>
> > the instance itself), can we still use
software config and deploy resources<br>
> > to deploy software on such instances?<br>
> ><br>
> > Currently os-collect-config is more of a
requirement than cloud-init,<br>
> > but as Clint said cloud-init does a good job
of boot config so you'll need<br>
> > to elaborate on why you don't want to use it.<br>
> ><br>
> > iii. If ii. were possible who would signal
the deployment resource to<br>
> > indicate that the instance is ready for the
deployment?<br>
> ><br>
> > os-collect-config polls for the deployment
data, and triggers the<br>
> > resulting deployment/config changes. One day
this may be performed by a<br>
> > different agent like the unified agent that
has been discussed. Currently<br>
> > os-collect-collect polls via a heat-api-cfn
metadata call. This too may be<br>
> > done in any number of ways in the future such
as messaging or long-polling.<br>
> ><br>
> > So you *could* consume the supplied user_data
to know what to poll for<br>
> > subsequent config changes without cloud-init
or os-collect-config, but you<br>
> > would have to describe what you're doing in
detail for us to know if that<br>
> > sounds like a good idea.<br>
> ><br>
> ><br>
> ><br>
> > Thanks<br>
> > Susaant<br>
> ><br>
> ><br>
> > On Fri, Dec 13, 2013 at 3:46 PM, Steve Baker
<<a moz-do-not-send="true"
href="mailto:sbaker@redhat.com">sbaker@redhat.com</a>>
wrote:<br>
> ><br>
> >> I've been working on a POC in heat for
resources which perform software<br>
> >> configuration, with the aim of
implementing this spec<br>
> >> <a moz-do-not-send="true"
href="https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-spec"
target="_blank">https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-spec</a><br>
> >><br>
> >> The code to date is here:<br>
> >> <a moz-do-not-send="true"
href="https://review.openstack.org/#/q/topic:bp/hot-software-config,n,z"
target="_blank">https://review.openstack.org/#/q/topic:bp/hot-software-config,n,z</a><br>
> >><br>
> >> What would be helpful now is reviews which
give the architectural<br>
> >> approach enough of a blessing to justify
fleshing this POC out into a ready<br>
> >> to merge changeset.<br>
> >><br>
> >> Currently it is possible to:<br>
> >> - create templates containing
OS::Heat::SoftwareConfig and<br>
> >> OS::Heat::SoftwareDeployment resources<br>
> >> - deploy configs to OS::Nova::Server,
where the deployment resource<br>
> >> remains in an IN_PROGRESS state until it
is signalled with the output values<br>
> >> - write configs which execute shell
scripts and report back with output<br>
> >> values that other resources can have
access to.<br>
> >><br>
> >> What follows is an overview of the
architecture and implementation to<br>
> >> help with your reviews.<br>
> >><br>
> >> REST API<br>
> >> ========<br>
> >> Like many heat resources,
OS::Heat::SoftwareConfig and<br>
> >> OS::Heat::SoftwareDeployment are backed by
"real" resources that are<br>
> >> invoked via a REST API. However in this
case, the API that is called is<br>
> >> heat itself.<br>
> >><br>
> >> The REST API for these resources really
just act as structured storage<br>
> >> for config and deployments, and the
entities are managed via the REST paths<br>
> >> /{tenant_id}/software_configs and
/{tenant_id}/software_deployments:<br>
> >><br>
</div>
</div>
> >> <<a moz-do-not-send="true"
href="https://review.openstack.org/#/c/58878/7/heat/api/openstack/v1/__init__.py"
target="_blank">https://review.openstack.org/#/c/58878/7/heat/api/openstack/v1/__init__.py</a>><br>
<div>
<div class="h5">> >> <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/58878/"
target="_blank">https://review.openstack.org/#/c/58878/</a><br>
> >> RPC layer of REST API:<br>
> >> <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/58877/"
target="_blank">https://review.openstack.org/#/c/58877/</a><br>
> >> DB layer of REST API:<br>
> >> <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/58876"
target="_blank">https://review.openstack.org/#/c/58876</a><br>
> >> heatclient lib access to REST API:<br>
> >> <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/58885"
target="_blank">https://review.openstack.org/#/c/58885</a><br>
> >><br>
> >> This data could be stored in a less
structured datastore like swift, but<br>
> >> this API has a couple of important
implementation details which I think<br>
> >> justify it existing:<br>
> >> - SoftwareConfig resources are immutable
once created. There is no update<br>
> >> API to modify an existing config. This
gives confidence that a config can<br>
> >> have a long lifecycle without changing,
and a certainty of what exactly is<br>
> >> deployed on a server with a given config.<br>
> >> - Fetching all the deployments and configs
for a given server is an<br>
> >> operation done repeatedly throughout the
lifecycle of the stack, so is<br>
> >> optimized to be able to do in a single
operation. This is called by using<br>
> >> the deployments index API call,<br>
> >>
/{tenant_id}/software_deployments?server_id=<server_id>.
The resulting list<br>
> >> of deployments include the their
associated config data[1].<br>
> >><br>
> >> OS::Heat::SoftwareConfig resource<br>
> >> =================================<br>
> >> OS::Heat::SoftwareConfig can be used
directly in a template, but it may<br>
> >> end be more frequently used in a resource
provider template which provides<br>
> >> a resource aimed at a particular
configuration management tool.<br>
> >><br>
> >> <a moz-do-not-send="true"
href="http://docs-draft.openstack.org/79/58879/7/check/gate-heat-docs/911a250/doc/build/html/template_guide/openstack.html#OS::Heat::SoftwareConfig"
target="_blank">http://docs-draft.openstack.org/79/58879/7/check/gate-heat-docs/911a250/doc/build/html/template_guide/openstack.html#OS::Heat::SoftwareConfig</a><br>
> >> The contents of the config property will
depend on the CM tool being<br>
> >> used, but at least one value in the config
map will be the actual script<br>
> >> that the CM tool invokes. An inputs and
outputs schema is also defined<br>
> >> here. The group property is used when the
deployments data is actually<br>
> >> delivered to the server (more on that
later).<br>
> >><br>
> >> Since a config is immutable, any changes
to a OS::Heat::SoftwareConfig on<br>
> >> stack update result in replacement.<br>
> >><br>
> >> OS::Heat::SoftwareDeployment resource<br>
> >> =====================================<br>
> >> OS::Heat::SoftwareDeployment joins a
OS::Heat::SoftwareConfig resource<br>
> >> with a OS::Nova::Server resource. It
allows server-specific input values to<br>
> >> be specified that map to the
OS::Heat::SoftwareConfig inputs schema. Output<br>
> >> values that are signaled to the deployment
resource are exposed as resource<br>
> >> attributes, using the names specified in
the outputs schema. The<br>
> >> OS::Heat::SoftwareDeployment resource
remains in an IN_PROGRESS state until<br>
> >> it receives a signal (containing any
outputs) from the server.<br>
> >><br>
> >> <a moz-do-not-send="true"
href="http://docs-draft.openstack.org/79/58879/7/check/gate-heat-docs/911a250/doc/build/html/template_guide/openstack.html#OS::Heat::SoftwareDeployment"
target="_blank">http://docs-draft.openstack.org/79/58879/7/check/gate-heat-docs/911a250/doc/build/html/template_guide/openstack.html#OS::Heat::SoftwareDeployment</a><br>
> >><br>
> >> A deployment has its own actions and
statuses that are specific to what a<br>
> >> deployment does, and
OS::Heat::SoftwareDeployment maps this to heat<br>
> >> resource statuses and actions:<br>
> >> actions:<br>
> >> DEPLOY -> CREATE<br>
> >> REDEPLOY -> UPDATE<br>
> >> UNDEPLOY -> DELETE<br>
> >><br>
> >> status (these could use some
bikeshedding):<br>
> >> WAITING -> IN_PROGRESS<br>
> >> RECEIVED -> COMPLETE<br>
> >> FAILED -> FAILED<br>
> >><br>
> >> In the config outputs schema there is a
special flag for error_output. If<br>
> >> the signal response contains any value for
any of these error_output<br>
> >> outputs then the deployment resource is
put into the FAILED state.<br>
> >><br>
> >> The SoftwareDeployment class subclasses
SignalResponder which means that<br>
> >> a SoftwareDeployment creates an associated
user and ec2 keypair. Since the<br>
> >> SoftwareDeployment needs to use the
resource_id for the deployment resource<br>
> >> uuid, the user_id needs to be stored in
resource-date instead. This non-wip<br>
> >> change enables that:<br>
> >> <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/61902/"
target="_blank">https://review.openstack.org/#/c/61902/</a><br>
> >><br>
> >> During create, the deployment REST API is
polled until status goes from<br>
> >> WAITING to RECEIVED. When handle_signal is
called, the deployment is<br>
> >> updated via the REST API to set the status
to RECEIVED (or FAILED), along<br>
> >> with any output values that were received.<br>
> >><br>
> >> One alarming consequence of having a
deployments API is that any tenant<br>
> >> user can create a deployment for any
heat-created nova server and that<br>
> >> software will be deployed to that server,
which is, um, powerful.<br>
> >><br>
> >> There will need to be a deployment policy
(probably an OS::Nova::Server<br>
> >> property) which limits to scope of what
deployments are allowed on that<br>
> >> server. This could default to deployments
in the same stack, but could<br>
> >> still allow deployments from anywhere.<br>
> >><br>
> >> OS::Nova::Server support<br>
> >> ========================<br>
> >> <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/58880"
target="_blank">https://review.openstack.org/#/c/58880</a><br>
> >> A new user_data_format=SOFTWARE_CONFIG is
currently used to denote that<br>
> >> this server is configured via software
config deployments. Like<br>
> >> user_data_format=HEAT_CFNTOOLS,
nova_utils.build_userdata is used to build<br>
> >> the cloud-init parts required to support
software config. However like<br>
> >> user_data_format=RAW anything specified in
user_data will be parsed as<br>
> >> cloud-init data. If user_data is
multi-part data then the parts will be<br>
> >> appended to the parts created in
nova_utils.build_userdata.<br>
> >><br>
> >> The agent used currently is
os-collect-config. This is typically<br>
> >> configured to poll for metadata from a
particular heat resource via the CFN<br>
> >> API using the configured ec2 keypair. In
the current implementation the<br>
> >> resource which is polled is the
OS::Nova::Server itself, since this is the<br>
> >> only resource known to exist at server
boot time (deployment resources<br>
> >> depend on server resources, so have not
been created yet). The ec2 keypair<br>
> >> comes from a user created implicitly with
the server (similar to<br>
> >> SignalResponder resources). This means the
template author doesn't need to<br>
> >> include User/AccessKey/AccessPolicy
resources in their templates just to<br>
> >> enable os-collect-config metadata polling.<br>
> >><br>
> >> Until now, polling the metadata for a
resource just returns the metadata<br>
> >> which has been stored in the stack
resource database. This implementation<br>
> >> changes metadata polling to actually query
the deployments API to return<br>
> >> the latest deployments data. This means
deployment state can be stored in<br>
> >> one place, and there is no need to keep
various metadata stores updated<br>
> >> with any changed state.<br>
> >><br>
> >> An actual template<br>
> >> ==================<br>
> >> <a moz-do-not-send="true"
href="http://paste.openstack.org/show/54988/"
target="_blank">http://paste.openstack.org/show/54988/</a><br>
> >> This template contains:<br>
> >> - a config resource<br>
> >> - 2 deployments which deploy that config
with 2 different sets of inputs<br>
> >> - stack outputs which output the results
of the deployments<br>
> >> - a server resource<br>
> >> - an os-refresh-config script delivered
via cloud-config[2] which<br>
> >> executes config scripts with deployment
inputs and signals outputs to the<br>
> >> provided webhook.<br>
> >><br>
> >>
/opt/stack/os-config-refresh/configure.d/55-heat-config-bash
is a hook<br>
> >> specific for performing configuration via
shell scripts, and only acts on<br>
> >> software config which has
group=Heat::Shell. Each configuration management<br>
> >> tool will have its own hook, and will act
on its own group namespace. Each<br>
> >> configuration management tool will also
have its own way of passing inputs<br>
> >> and outputs. The hooks job is to invoke
the CM tool with the given inputs<br>
> >> and script, then extract the outputs and
signal heat.<br>
> >><br>
> >> The server needs to have the CM tool and
the hook already installed,<br>
> >> either by building a golden image or by
using cloud-config during boot.<br>
> >><br>
> >> Next steps<br>
> >> ==========<br>
> >> There is a lot left to do and I'd like to
spread the development load.<br>
> >> What happens next entirely depends on
feedback to this POC, but here is my<br>
> >> ideal scenario:<br>
> >> - any feedback which causes churn on many
of the current changes I will<br>
> >> address<br>
> >> - a volunteer is found to take the REST
API/RPC/DB/heatclient changes and<br>
> >> make them ready to merge<br>
> >> - we continue to discuss and refine the
resources, the changes to<br>
> >> OS::Nova::Server, and the example shell
hook<br>
> >> - volunteers write hooks for different CM
tools, Chef and Puppet hooks<br>
> >> will need to be attempted soon to validate
this approach.<br>
> >><br>
> >> Vaguely related changes include:<br>
> >> - Some solution for specifying cloud-init
config, either the intrinsic<br>
> >> functions or cloud-init heat resources<br>
> >> - Some heatclient file inclusion mechanism
- writing that python hook in<br>
> >> a heat yaml template was a bit painful ;)<br>
> >><br>
> >> Trying for yourself<br>
> >> ===================<br>
> >> - Using diskimage-builder, create an
ubuntu image with<br>
> >> tripleo-image-elements os-apply-config,
os-refresh-config and<br>
> >> os-collect-config<br>
> >> - Create a local heat branch containing<br>
> >> <a moz-do-not-send="true"
href="https://review.openstack.org/#/q/topic:bp/cloud-init-resource,n,z"
target="_blank">https://review.openstack.org/#/q/topic:bp/cloud-init-resource,n,z</a>
and<br>
> >> <a moz-do-not-send="true"
href="https://review.openstack.org/#/q/topic:bp/hot-software-config,n,z"
target="_blank">https://review.openstack.org/#/q/topic:bp/hot-software-config,n,z</a><br>
> >> - launch the above template with your
created image<br>
> >><br>
> >> cheers<br>
> >><br>
> >> [1] <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/58877/7/heat/engine/api.py"
target="_blank">https://review.openstack.org/#/c/58877/7/heat/engine/api.py</a><br>
> >> [2] This relies on these not-merged
intrinsic functions<br>
> >> <a moz-do-not-send="true"
href="https://review.openstack.org/#/q/topic:bp/cloud-init-resource,n,z"
target="_blank">https://review.openstack.org/#/q/topic:bp/cloud-init-resource,n,z</a><br>
> >><br>
> >>
_______________________________________________<br>
> >> OpenStack-dev mailing list<br>
> >> <a moz-do-not-send="true"
href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> >> <a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> >><br>
> >><br>
> ><br>
> ><br>
> >
_______________________________________________<br>
</div>
</div>
> > OpenStack-dev mailing
<a class="moz-txt-link-abbreviated" href="mailto:listOpenStack-dev@lists.openstack.orghttp://">listOpenStack-dev@lists.openstack.orghttp://</a><a
moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<div class="HOEnZb">
<div class="h5">> ><br>
> ><br>
> ><br>
> >
_______________________________________________<br>
> > OpenStack-dev mailing list<br>
> > <a moz-do-not-send="true"
href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> > <a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
> ><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>