Open Stack

On Thu, Jun 8, 2017 at 1:47 PM, Sofer Athlan-Guyot <sathlang at redhat.com> wrote:
> Hi,
>
> Alex Schultz <aschultz at redhat.com> writes:
>
>> On Wed, Jun 7, 2017 at 5:20 AM, Sofer Athlan-Guyot <sathlang at redhat.com> wrote:
>>> Hi,
>>>
>>> Emilien Macchi <emilien at redhat.com> writes:
>>>
>>>> On Wed, Jun 7, 2017 at 12:45 PM, Sofer Athlan-Guyot <sathlang at redhat.com> wrote:
>>>>> Hi,
>>>>>
>>>>> I don't think we have such a job in place.  Basically that would check
>>>>> that re-running the "openstack deploy ..." command won't do anything.
>
> I've had a look at openstack-infra/tripleo-ci.  Should I test it in with
> ovb/quickstart or tripleo.sh.  Both way are fine by me, but I may be
> lacking context about which one is more relevant.
>
>>>>> We had such an error by the past[1], but I'm not sure this has been
>>>>> captured by an associated job.
>>>>>
>>>>> WDYT ?
>>>>
>>>> It would be interesting to measure how much time does it take to run
>>>> it again.
>>>
>>> Could you point out how such an experiment could be done ?
>>>
>>>> If it's short, we could add it to all our scenarios + ovb
>>>> jobs.  If it's long, maybe we need an additional job, but it would
>>>> take more resources, so maybe we could run it in periodic pipeline
>>>> (note that periodic jobs are not optimal since we could break
>>>> something quite easily).
>>>
>>> Just adding as context that the issue was already raised[1].  Beside
>>> time constraint, it was pointed out that we would also need to parse the
>>> log to find out if anything was restarted.  But it could be a second
>>> step.  For parsing, this code was pointed out[2].
>>>
>>
>> There's a few things that would need to be enabled in order to reuse
>> some of this work.  We'll need to add the ability to generate a report
>> on the puppet run[0]. And then we'll need to be able to capture it[1]
>> somewhere that we could then use that parsing code on.  From there,
>> just rerunning the installation would be a simple start to the
>> idempotency check.  In fuel, we had hacked in a special flag[2] that
>> we used in testing to actually rerun the task immediately to find when
>> a specific task was not idempotent in addition to also rerunning the
>> entire deployment. For tripleo a similar concept would be to rerun the
>> steps twice but that's usually not where the issues crop us for us. So
>> rerunning the entire installation deployment would be better as we
>> tend to have issues with configuration items between steps
>> conflicting.
>
> Maybe we could go with something equivalent to:
>
>   ts="$(date '+%F %T')"
>   ... re-run deploy command ...
>
>   sudo journalctl --since="${ts}" | egrep 'Stopping|Starting' | grep -v 'user.*slice' > restarted.log
>   wc -l restarted.log
>
> This should be 0 on every overcloud nodes.
>
> This is simpler to implement and should catch any unwanted service
> restart.
>
> WDYT ?

It's smart, for services. It doesn't cover configuration files changes
and other resources managed by Puppet, like Keystone resources, etc.
But it's an excellent start to me.

>>
>> Thanks,
>> -Alex
>>
>> [0] https://review.openstack.org/#/c/273740/4/mcagents/puppetd.rb@204
>> [1] https://review.openstack.org/#/c/273740/4/mcagents/puppetd.rb@102
>> [2] https://review.openstack.org/#/c/273737/
>>
>>> [1] http://lists.openstack.org/pipermail/openstack-dev/2017-March/114836.html
>>> [2] https://review.openstack.org/#/c/279271/9/fuelweb_test/helpers/astute_log_parser.py@212
>>>
>>>>
>>>>> [1] https://bugs.launchpad.net/tripleo/+bug/1664650
>>>>> --
>>>>> Sofer Athlan-Guyot
>>>>>
>>>>> __________________________________________________________________________
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>>
>>>> --
>>>> Emilien Macchi
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> --
>>> Sofer Athlan-Guyot
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> Thanks,
> --
> Sofer Athlan-Guyot
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Emilien Macchi