Open Stack

Screen is going away in Queens.

Making the dev / test runtimes as similar as possible is really
important. And there is so much weird debt around trying to make screen
launch things reliably (like random sleeps) because screen has funny
races in it.

It does mean some tricks people figured out in screen are going away.

Journalctl provides some pretty serious improvements in querying logs
https://www.freedesktop.org/software/systemd/man/journalctl.html - you
can search in time ranges, filter by units (one or more of them), and if
we get to the bottom of the eventlet interaction, we'll be able to
search by things like REQUEST_ID as well.

Plus every modern Linux system uses systemd now, so skills learned
around systemd and journalctl are transferable both from OpenStack to
other systems, as well as for new people coming in that understand how
this works outside of OpenStack. So it helps remove a difference from
the way we do things from the rest of the world.

	-Sean

On 05/03/2017 04:02 PM, Hongbin Lu wrote:
> Hi Sean,
> 
> I tried the new systemd devstack and frankly I don't like it. There are several handy operations in screen that seems to be impossible after switching to systemd. For example, freeze a process by "Ctrl + a + [". In addition, navigating though the logs seems difficult (perhaps I am not familiar with journalctl).
> 
> From my understanding, the plan is dropping screen entirely in devstack? I would argue that it is better to keep both screen and systemd, and let users choose one of them based on their preference.
> 
> Best regards,
> Hongbin
> 
>> -----Original Message-----
>> From: Sean Dague [mailto:sean at dague.net]
>> Sent: May-03-17 6:10 AM
>> To: openstack-dev at lists.openstack.org
>> Subject: Re: [openstack-dev] [devstack] [all] systemd in devstack by
>> default
>>
>> On 05/02/2017 08:30 AM, Sean Dague wrote:
>>> We started running systemd for devstack in the gate yesterday, so far
>>> so good.
>>>
>>> The following patch (which will hopefully land soon), will convert
>> the
>>> default local use of devstack to systemd as well -
>>> https://review.openstack.org/#/c/461716/. It also includes
>>> substantially updated documentation.
>>>
>>> Once you take this patch, a "./clean.sh" is recommended. Flipping
>>> modes can cause some cruft to build up, and ./clean.sh should be
>>> pretty good at eliminating them.
>>>
>>> https://review.openstack.org/#/c/461716/2/doc/source/development.rst
>>> is probably specifically interesting / useful for people to read, as
>>> it shows how the standard development workflows will change (for the
>>> better) with systemd.
>>>
>>> 	-Sean
>>
>> As a follow up, there are definitely a few edge conditions we've hit
>> with some jobs, so the following is provided as information in case you
>> have a job that seems to fail in one of these ways.
>>
>> Doing process stop / start
>> ==========================
>>
>> The nova live migration job is special, it was restarting services
>> manually, however it was doing so with some copy / pasted devstack code,
>> which means it didn't evolve with the rest of devstack. So the stop
>> code stopped working (and wasn't robust enough to make it clear that
>> was the issue).
>>
>> https://review.openstack.org/#/c/461803/ is the fix (merged)
>>
>> run_process limitations
>> =======================
>>
>> When doing the systemd conversion I looked for a path forward which was
>> going to make 90% of everything just work. The key trick here was that
>> services start as the "stack" user, and aren't daemonizing away from
>> the console. We can take the run_process command and make that the
>> ExecStart in a unit file.
>>
>> *Except* that only works if the command is specified by an *absolute
>> path*.
>>
>> So things like this in kuryr-libnetwork become an issue
>> https://github.com/openstack/kuryr-
>> libnetwork/blob/3e2891d6fc5d55b3712258c932a5a8b9b323f6c2/devstack/plugi
>> n.sh#L148
>>
>> There is also a second issue there, which is calling sudo in the
>> run_process line. If you need to run as a user/group different than the
>> default, you need to specify that directly.
>>
>> The run_process command now supports that -
>> https://github.com/openstack-
>> dev/devstack/blob/803acffcf9254e328426ad67380a99f4f5b164ec/functions-
>> common#L1531-L1535
>>
>> And lastly, run_process really always did expect that the thing you
>> started remained attached to the console. These are run as "simple"
>> services in systemd. If you are running a thing which already
>> daemonizes systemd is going to assume (correctly in this simple mode)
>> the fact that the process detatched from it means it died, and kill and
>> clean it up.
>>
>> This is the issue the OpenDaylight plugin ran into.
>> https://review.openstack.org/#/c/461889/ is the proposed fix.
>>
>>
>>
>> If you run into any other issues please pop into #openstack-qa (or
>> respond to this email) and we'll try to work through them.
>>
>> 	-Sean
>>
>> --
>> Sean Dague
>> http://dague.net
>>
>> _______________________________________________________________________
>> ___
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-
>> request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
Sean Dague
http://dague.net