<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">What about ability of service expert to plug-in remediation module?<o:p></o:p></p>
<p class="MsoNormal">If remediation action succeed – proceed, if not then stop.<o:p></o:p></p>
<p class="MsoNormal">Remediation module can be extended independently from main flow.<o:p></o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">Arkady<o:p></o:p></p>
<p>-----Original Message-----<br>
From: Steven Hardy [mailto:shardy@redhat.com] <br>
Sent: Wednesday, July 27, 2016 3:26 AM<br>
To: OpenStack Development Mailing List (not for usage questions) <br>
Subject: Re: [openstack-dev] [tripleo] service validation during deployment steps<br>
<br>
Hi Emilien,<br>
<br>
On Tue, Jul 26, 2016 at 03:59:33PM -0400, Emilien Macchi wrote:<br>
> I would love to hear some feedback about $topic, thanks.<br>
<br>
Sorry for the slow response, we did dicuss this on IRC, but providing that feedback and some other comments below:<br>
<br>
> On Fri, Jul 15, 2016 at 11:31 AM, Emilien Macchi wrote:<br>
> > Hi,<br>
> ><br>
> > Some people on the field brought interesting feedback:<br>
> ><br>
> > "As a TripleO User, I would like the deployment to stop immediately <br>
> > after an resource creation failure during a step of the deployment <br>
> > and be able to easily understand what service or resource failed to <br>
> > be installed".<br>
> ><br>
> > Example:<br>
> > If during step4 Puppet tries to deploy Neutron and OVS, but OVS <br>
> > fails to start for some reasons, deployment should stop at the end <br>
> > of the step.<br>
<br>
I don't think anyone will argue against this use-case, we absolutely want to enable a better "fail fast" for deployment problems, as well as better surfacing of why it failed.<br>
<br>
> > So there are 2 things in this user story:<br>
> ><br>
> > 1) Be able to run some service validation within a step deployment.<br>
> > Note about the implementation: make the validation composable per <br>
> > service (OVS, nova, etc) and not per role (compute, controller, etc).<br>
<br>
+1, now we have composable services we need any validations to be<br>
associated with the services, not the roles.<br>
<br>
That said, it's fairly easy to imagine an interface like step_config/config_settings could be used to wire in composable service validations on a per-role basis, e.g similar to what we do here, but<br>
per-step:<br>
<br>
https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud.yaml#L1144<br>
<br>
Similar to what was proposed (but never merged) here:<br>
<br>
https://review.openstack.org/#/c/174150/15/puppet/controller-post-puppet.yaml<br>
<br>
> > 2) Make this information readable and easy to access and understand <br>
> > for our users.<br>
> ><br>
> > I have a proof-of-concept for 1) and partially 2), with the example <br>
> > of<br>
> > OVS: https://review.openstack.org/#/c/342202/<br>
> > This patch will make sure OVS is actually usable at step 4 by <br>
> > running 'ovs-vsctl show' during the Puppet catalog and if it's <br>
> > working, it will create a Puppet anchor. This anchor is currently <br>
> > not useful but could be in future if we want to rely on it for orchestration.<br>
> > I wrote the service validation in Puppet 2 years ago when doing <br>
> > Spinal Stack with eNovance:<br>
> > https://github.com/openstack/puppet-openstacklib/blob/master/manifes<br>
> > ts/service_validation.pp I think we could re-use it very easily, it <br>
> > has been proven to work.<br>
> > Also, the code is within our Puppet profiles, so it's by design <br>
> > composable and we don't need to make any connection with our current <br>
> > services with some magic. Validation will reside within Puppet <br>
> > manifests.<br>
> > If you look my PoC, this code could even live in puppet-vswitch <br>
> > itself (we already have this code for puppet-nova, and some others).<br>
<br>
I think having the validations inside the puppet implementation is OK, but ideally I think we do want it to be part of the puppet modules themselves (not part of the puppet-tripleo abstraction layer).<br>
<br>
The issue I'd have with putting it in puppet-tripleo is that if we're going to do this in a tripleo specific way, it should probably be done via a method that's more config tool agnostic. Otherwise we'll have to recreate the same validations for future implementations
(I'm thinking specifically about containers here, and possibly ansible[1].<br>
<br>
So, in summary, I'm +1 on getting this integrated if it can be done with little overhead and it's something we can leverage via the puppet modules vs puppet-tripleo.<br>
<br>
> ><br>
> > Ok now, what if validation fails?<br>
> > I'm testing it here: https://review.openstack.org/#/c/342205/<br>
> > If you look at /var/log/messages, you'll see:<br>
> ><br>
> > Error: <br>
> > /Stage[main]/Tripleo::Profile::Base::Neutron::Ovs/Openstacklib::Serv<br>
> > ice_validation[openvswitch]/Exec[execute<br>
> > openvswitch validation]/returns: change from notrun to 0 failed<br>
> ><br>
> > So it's pretty clear by looking at logs that openvswitch service <br>
> > validation failed and something is wrong. You'll also notice in the <br>
> > logs that deployed stopped at step 4 since OVS is not considered to <br>
> > run.<br>
> > It's partially addressing 2) because we need to make it more <br>
> > explicit and readable. Dan Prince had the idea to use <br>
> > https://github.com/ripienaar/puppet-reportprint to print a nice <br>
> > report of Puppet catalog result (we haven't tried it yet). We could <br>
> > also use Operational Tools later to monitor Puppet logs and find <br>
> > Service validation failures.<br>
<br>
This all sounds good, but we do need to think beyond the puppet implementation, e.g how will we enable similar validations in a container based deployment?<br>
<br>
I remember SpinalStack also used serverspec, can you describe the differences between using that tool (was it only used for post-deploy validation of the whole server, not per-step validation?)<br>
<br>
I'm just wondering if the overhead of integrating per-service validations via a more generic tool (not necessarily serverspec but something like it, e.g I've been looking at testinfra which is a more python based tool aiming to do similar things[2]) would be
worth it?<br>
<br>
Maybe this is complementary to any per-step validation done inside the puppet modules, but we could do something like:<br>
<br>
outputs:<br>
role_data:<br>
description: Role data for the Heat Engine role.<br>
value:<br>
service_name: heat-api<br>
config_settings:<br>
heat::: foo<br>
step_config: |<br>
include ::tripleo::profile::base::heat::api<br>
validation:<br>
group: serverspec<br>
config:<br>
step_4: |<br>
Package "openstack-heat-api"<br>
should be installed<br>
Service "openstack-heat-api"<br>
should be enabled<br>
should be running<br>
Port "8004"<br>
should be listening<br>
<br>
<br>
Looking at the WIP container composable services patch[3], this can probably be directly reused:<br>
<br>
outputs:<br>
role_data:<br>
description: Role data for the Keystone API role.<br>
value:<br>
config_settings: <br>
step_config: <br>
puppet_tags: keystone_config<br>
docker_config:<br>
keystone:<br>
container_step_config: 1<br>
image:<br>
list_join:<br>
- '/'<br>
- [ {get_param: DockerNamespace}, {get_param:<br>
DockerKeystoneImage} ]<br>
net: host<br>
privileged: false<br>
restart: always<br>
volumes:<br>
- /run:/run<br>
- /var/lib/etc-data/json-config/keystone.json:/var/lib/kolla/config_files/keystone.json<br>
environment:<br>
- KOLLA_CONFIG_STRATEGY=COPY_ALWAYS<br>
validation:<br>
group: serverspec<br>
config:<br>
step_1: |<br>
Service "httpd"<br>
should be enabled<br>
should be running<br>
Port "5000"<br>
should be listening<br>
Port "35357"<br>
should be listening<br>
<br>
Anyway, just some ideas there - I'm not opposed to what you suggest re the puppet validations, but I'm very aware that we'll be left with a feature gap if we *only* do that, then (pretty soon) enable fully containerized deployments.<br>
<br>
Thanks,<br>
<br>
Steve<br>
<br>
[1] http://lists.openstack.org/pipermail/openstack-dev/2016-July/099564.html<br>
[2] https://testinfra.readthedocs.io/en/latest/<br>
[3] https://review.openstack.org/#/c/330659/12/docker/services/keystone.yaml<br>
<br>
<br>
> ><br>
> ><br>
> > So this email is a bootstrap of discussion, it's open for feedback.<br>
> > Don't take my PoC as something we'll implement. It's an idea and I <br>
> > think it's worth to look at it.<br>
> > I like it for 2 reasons:<br>
> > - the validation code reside within our profiles, so it's composable by design.<br>
> > - it's flexible and allow us to test everything. It can be a bash <br>
> > script, a shell command, a Puppet resource (provider, service, etc).<br>
> ><br>
> > Thanks for reading so far,<br>
> > --<br>
> > Emilien Macchi<br>
> <br>
> <br>
> <br>
> --<br>
> Emilien Macchi<br>
> <br>
> ______________________________________________________________________<br>
> ____ OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <br>
> OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<br>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br>
<br>
--<br>
Steve Hardy<br>
Red Hat Engineering, Cloud<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<br>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<o:p></o:p></p>
</div>
</body>
</html>