[OSC][PTG] Summary: many things to do!
OpenStackClient held a session at the Denver PTG and despite not having much planned had plenty to talk about. Some of the highlights from the etherpad[0] are:
* Aretm is working on changing the Image commands to use OpenStackSDK. This is the work described in the cycle goal proposal[1] that he is planning to do anyway. I support going ahead with this even without an SDK 1.0 release as it lets us remove glanceclient and some of its unique dependencies.
* There was some discussion about image encryption and where the client-side bits of that may land. One option was to put it into os-brick; if that is where it winds up OSC will make that an optional dependency de to the number of other dependencies that will introduce. (ie, OSC currently uses very little of oslo, some of which brings in a number of things not otherwise needed client-side).
* Doug brought up the problems with load times due to scanning the entire import path on every invocation. I found in my notes almost exactly 2 years ago where we discussed this same topic. AS we did then, the idea of skipping entry points entirely for commands in the OSC repo is the best solution we have found. This would help some common cases but still leave all plugins with slow load times.
* Nate Johnston asked about supporting bulk create APIs, such as Neutron's bulk port create. After kicking around a couple of options the rough consensus is around using a YAML file (or JSON or both?) to define the resources to be created and giving it to a new top-level 'create' command (yes, verb only, the resource names will be in the YAML file). APIs that allow bulk creates will get a single call with the entire list, for other APIs we can loop and feed them one at a time. This would be very similar to using interactive mode and feeding in a list of commands, stopping at the first failure. Note that delete commands already take multiple resource identifiers, adding the ability to source that list from YAML would be an easy addition.
* OSC4 has been waiting in a feature branch for over 2 years (where has that PTL been???). I recently tried to merge master in to see how far off it was, it was enough that I think we should just cherry-pick the commites in that branch to master and move forward. So the current plan is to: * do one more release in the 3.x series to clean up outstanding things * switch to OSC4 development, cherry pick in amotoki's existing commits[2] (mostly changes to output formatting) * refresh and merge other reviews in the osc4 topic * remove all of the backward-compatibility code in the OSC authentication process so OSC will now work like all other pure keystoneauth- and sdk-using code.
Also relevant to OSC but covered in a Nova Forum session[3,4], highlights:
* boot-from-volume: Support type=image for the --block-device-mapping, and Add a --boot-from-volume option which will translate to a root --block-device-mapping using the provided --image value
* server migrate --live: deprecate the --live option and add a new --live-migration option and a --host option
* compute migration: begin exposing this resource in the CLI
dt
[0] https://etherpad.openstack.org/p/train-ptg-osc [1] https://review.opendev.org/#/c/639376/ [2] this series starts at https://review.opendev.org/#/c/657907/ [3] https://etherpad.openstack.org/p/DEN-osc-compute-api-gaps [4] http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005783.html
On Fri, 10 May 2019, Dean Troyer wrote:
- Nate Johnston asked about supporting bulk create APIs, such as
Neutron's bulk port create. After kicking around a couple of options the rough consensus is around using a YAML file (or JSON or both?) to define the resources to be created and giving it to a new top-level 'create' command (yes, verb only, the resource names will be in the YAML file).
+∞
There are presumably some error handling issues with that, but overall something like this would be very useful.
On Fri, May 10, 2019 at 11:55 AM Chris Dent cdent+os@anticdent.org wrote:
- Nate Johnston asked about supporting bulk create APIs, such as
Neutron's bulk port create. After kicking around a couple of options the rough consensus is around using a YAML file (or JSON or both?) to define the resources to be created and giving it to a new top-level 'create' command (yes, verb only, the resource names will be in the YAML file).
+∞
There are presumably some error handling issues with that, but overall something like this would be very useful.
I should admit that during this conversation I was well down the path of re-creating Ansible before I realized it... that is part of the reason we want to start with a simplistic equivalent to shell's 'set -e' and let the user sort it out.
Note that the delete command pattern is different in that it attempts them all once and returns a list of failures if any. I am certainly open to discussion on the preferred way to address errors without inventing complicated retry mechanisms; my feeling is if you need those use Ansible and the SDK directly.
dt
On 5/10/19 4:48 PM, Dean Troyer wrote:
OpenStackClient held a session at the Denver PTG and despite not having much planned had plenty to talk about. Some of the highlights from the etherpad[0] are:
Well, poo. Sorry I missed it.
- Aretm is working on changing the Image commands to use OpenStackSDK.
This is the work described in the cycle goal proposal[1] that he is planning to do anyway. I support going ahead with this even without an SDK 1.0 release as it lets us remove glanceclient and some of its unique dependencies.
- There was some discussion about image encryption and where the
client-side bits of that may land. One option was to put it into os-brick; if that is where it winds up OSC will make that an optional dependency de to the number of other dependencies that will introduce. (ie, OSC currently uses very little of oslo, some of which brings in a number of things not otherwise needed client-side).
We landed support for image signing in SDK:
https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/image...
so I don't think it would be an issue to also land support for encryption. I'd like to have support for signing so that people using SDK could do signing. However, like OSC, SDK doesn't use oslo deps, largely for the reason you mention - they're written for server side and vastly complicate the client-side story. SDK will not grow a dependency on os-brick, even an optional one.
How about if we put the encryption routines in SDK, then expose them to the server-side components via SDK?
- Doug brought up the problems with load times due to scanning the
entire import path on every invocation. I found in my notes almost exactly 2 years ago where we discussed this same topic. AS we did then, the idea of skipping entry points entirely for commands in the OSC repo is the best solution we have found. This would help some common cases but still leave all plugins with slow load times.
- Nate Johnston asked about supporting bulk create APIs, such as
Neutron's bulk port create. After kicking around a couple of options the rough consensus is around using a YAML file (or JSON or both?) to define the resources to be created and giving it to a new top-level 'create' command (yes, verb only, the resource names will be in the YAML file). APIs that allow bulk creates will get a single call with the entire list, for other APIs we can loop and feed them one at a time. This would be very similar to using interactive mode and feeding in a list of commands, stopping at the first failure. Note that delete commands already take multiple resource identifiers, adding the ability to source that list from YAML would be an easy addition.
- OSC4 has been waiting in a feature branch for over 2 years (where
has that PTL been???). I recently tried to merge master in to see how far off it was, it was enough that I think we should just cherry-pick the commites in that branch to master and move forward. So the current plan is to:
- do one more release in the 3.x series to clean up outstanding things
- switch to OSC4 development, cherry pick in amotoki's existing
commits[2] (mostly changes to output formatting)
- refresh and merge other reviews in the osc4 topic
- remove all of the backward-compatibility code in the OSC
authentication process so OSC will now work like all other pure keystoneauth- and sdk-using code.
Also relevant to OSC but covered in a Nova Forum session[3,4], highlights:
- boot-from-volume: Support type=image for the --block-device-mapping,
and Add a --boot-from-volume option which will translate to a root --block-device-mapping using the provided --image value
++
FWIW - https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/cloud... is where we define the similar parameters for server create in SDK. As we're making the new OSC params, if we can align naming and usage, that would likely lead to future joy - unless we can't, in which case it's fine.
- server migrate --live: deprecate the --live option and add a new
--live-migration option and a --host option
- compute migration: begin exposing this resource in the CLI
dt
[0] https://etherpad.openstack.org/p/train-ptg-osc [1] https://review.opendev.org/#/c/639376/ [2] this series starts at https://review.opendev.org/#/c/657907/ [3] https://etherpad.openstack.org/p/DEN-osc-compute-api-gaps [4] http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005783.html
-----Original Message----- From: Monty Taylor mordred@inaugust.com Sent: Friday, May 10, 2019 1:43 PM To: openstack-discuss@lists.openstack.org Subject: Re: [OSC][PTG] Summary: many things to do!
On 5/10/19 4:48 PM, Dean Troyer wrote:
OpenStackClient held a session at the Denver PTG and despite not having much planned had plenty to talk about. Some of the highlights from the etherpad[0] are:
Well, poo. Sorry I missed it.
- Aretm is working on changing the Image commands to use OpenStackSDK.
This is the work described in the cycle goal proposal[1] that he is planning to do anyway. I support going ahead with this even without an SDK 1.0 release as it lets us remove glanceclient and some of its unique dependencies.
- There was some discussion about image encryption and where the
client-side bits of that may land. One option was to put it into os-brick; if that is where it winds up OSC will make that an optional dependency de to the number of other dependencies that will introduce. (ie, OSC currently uses very little of oslo, some of which brings in a number of things not otherwise needed client-side).
We landed support for image signing in SDK:
https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/i mage/image_signer.py
The old python-glanceclient has an option to download the image to a local file. I missed that when using the clouds.yaml-based approach that directly accesses the Glance API. Hope we can add that option to the openstacksdk-based client.
Just my 2 cents.
Regards, Sundar
dt
[0] https://etherpad.openstack.org/p/train-ptg-osc [1] https://review.opendev.org/#/c/639376/ [2] this series starts at https://review.opendev.org/#/c/657907/ [3] https://etherpad.openstack.org/p/DEN-osc-compute-api-gaps [4] http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005783 .html
On 5/10/19 11:55 PM, Nadathur, Sundar wrote:
-----Original Message----- From: Monty Taylor mordred@inaugust.com Sent: Friday, May 10, 2019 1:43 PM To: openstack-discuss@lists.openstack.org Subject: Re: [OSC][PTG] Summary: many things to do!
On 5/10/19 4:48 PM, Dean Troyer wrote:
OpenStackClient held a session at the Denver PTG and despite not having much planned had plenty to talk about. Some of the highlights from the etherpad[0] are:
Well, poo. Sorry I missed it.
- Aretm is working on changing the Image commands to use OpenStackSDK.
This is the work described in the cycle goal proposal[1] that he is planning to do anyway. I support going ahead with this even without an SDK 1.0 release as it lets us remove glanceclient and some of its unique dependencies.
- There was some discussion about image encryption and where the
client-side bits of that may land. One option was to put it into os-brick; if that is where it winds up OSC will make that an optional dependency de to the number of other dependencies that will introduce. (ie, OSC currently uses very little of oslo, some of which brings in a number of things not otherwise needed client-side).
We landed support for image signing in SDK:
https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/i mage/image_signer.py
The old python-glanceclient has an option to download the image to a local file. I missed that when using the clouds.yaml-based approach that directly accesses the Glance API. Hope we can add that option to the openstacksdk-based client.
Yup - we have this.
The shade side of sdk has had this for a while. We just landed a refactor which pushes all of the shade image logic into the sdk layer (thanks to Artem) - so when 0.28.0 is cut later today, conn.image.download() should work properly in the sdk layer too. But in the meantime, you can just use conn.download_image and it'll work fine
Monty
On Fri, May 10, 2019 at 11:48:21AM -0500, Dean Troyer wrote:
OpenStackClient held a session at the Denver PTG and despite not having much planned had plenty to talk about. Some of the highlights from the etherpad[0] are:
- OSC4 has been waiting in a feature branch for over 2 years (where
has that PTL been???). I recently tried to merge master in to see how far off it was, it was enough that I think we should just cherry-pick the commites in that branch to master and move forward. So the current plan is to:
- do one more release in the 3.x series to clean up outstanding things
- switch to OSC4 development, cherry pick in amotoki's existing
commits[2] (mostly changes to output formatting)
I also have a few holding out there for the major release:
https://review.opendev.org/#/q/status:open+project:openstack/python-openstac...
I'll try to get to that one in merge conflict and I'll try to watch for other conflicts as we merge these different major changes in.
(sorry, didn't realize there was a feature branch for this work. Though that looks like it was OK given the current state. ;) )
On Fri, 2019-05-10 at 11:48 -0500, Dean Troyer wrote:
OpenStackClient held a session at the Denver PTG and despite not having much planned had plenty to talk about. Some of the highlights from the etherpad[0] are:
- Aretm is working on changing the Image commands to use OpenStackSDK.
This is the work described in the cycle goal proposal[1] that he is planning to do anyway. I support going ahead with this even without an SDK 1.0 release as it lets us remove glanceclient and some of its unique dependencies.
- There was some discussion about image encryption and where the
client-side bits of that may land. One option was to put it into os-brick; if that is where it winds up OSC will make that an optional dependency de to the number of other dependencies that will introduce. (ie, OSC currently uses very little of oslo, some of which brings in a number of things not otherwise needed client-side).
- Doug brought up the problems with load times due to scanning the
entire import path on every invocation. I found in my notes almost exactly 2 years ago where we discussed this same topic. AS we did then, the idea of skipping entry points entirely for commands in the OSC repo is the best solution we have found. This would help some common cases but still leave all plugins with slow load times.
- Nate Johnston asked about supporting bulk create APIs, such as
Neutron's bulk port create. After kicking around a couple of options the rough consensus is around using a YAML file (or JSON or both?) to define the resources to be created and giving it to a new top-level 'create' command (yes, verb only, the resource names will be in the YAML file). APIs that allow bulk creates will get a single call with the entire list, for other APIs we can loop and feed them one at a time. This would be very similar to using interactive mode and feeding in a list of commands, stopping at the first failure. Note that delete commands already take multiple resource identifiers, adding the ability to source that list from YAML would be an easy addition.
- OSC4 has been waiting in a feature branch for over 2 years (where
has that PTL been???). I recently tried to merge master in to see how far off it was, it was enough that I think we should just cherry-pick the commites in that branch to master and move forward. So the current plan is to:
- do one more release in the 3.x series to clean up outstanding things
- switch to OSC4 development, cherry pick in amotoki's existing
commits[2] (mostly changes to output formatting)
- refresh and merge other reviews in the osc4 topic
- remove all of the backward-compatibility code in the OSC
authentication process so OSC will now work like all other pure keystoneauth- and sdk-using code.
Also relevant to OSC but covered in a Nova Forum session[3,4], highlights:
- boot-from-volume: Support type=image for the --block-device-mapping,
and Add a --boot-from-volume option which will translate to a root --block-device-mapping using the provided --image value
- server migrate --live: deprecate the --live option and add a new
--live-migration option and a --host option
Could I suggest we don't bother deprecating the '--live' option in 3.x and simply rework it for 4.0? This is of course assuming the 4.0 release date is months away and not years, of course.
- compute migration: begin exposing this resource in the CLI
dt
[0] https://etherpad.openstack.org/p/train-ptg-osc [1] https://review.opendev.org/#/c/639376/ [2] this series starts at https://review.opendev.org/#/c/657907/ [3] https://etherpad.openstack.org/p/DEN-osc-compute-api-gaps [4] http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005783.html
On Mon, May 13, 2019 at 8:02 AM Stephen Finucane sfinucan@redhat.com wrote:
- server migrate --live: deprecate the --live option and add a new
--live-migration option and a --host option
Could I suggest we don't bother deprecating the '--live' option in 3.x and simply rework it for 4.0? This is of course assuming the 4.0 release date is months away and not years, of course.
Sure, my primary concern is that we do what makes the most sense from a user's point of view.
Re OSC4, I am cherry-picking the work that was done a while back on the feature/osc4 branch back to master, that should be done in a day or two. That said, I do not want to rush a 4.0 release, it has taken 2 years already, I want to make sure we include all of the things that we have been holding both explicitly and mentally. I am open to input on how long that should be...
dt
On 5/13/2019 8:02 AM, Stephen Finucane wrote:
- server migrate --live: deprecate the --live option and add a new
--live-migration option and a --host option
Could I suggest we don't bother deprecating the '--live' option in 3.x and simply rework it for 4.0? This is of course assuming the 4.0 release date is months away and not years, of course.
But, why? If we're going to get some stuff into 3.x yet, why not deprecate the bad thing so it's there in the version most people will be using for a long time before re-writing their stuff to work with 4.0? The deprecation is the easy bit.
Dean Troyer dtroyer@gmail.com writes:
- Doug brought up the problems with load times due to scanning the
entire import path on every invocation. I found in my notes almost exactly 2 years ago where we discussed this same topic. AS we did then, the idea of skipping entry points entirely for commands in the OSC repo is the best solution we have found. This would help some common cases but still leave all plugins with slow load times.
Because of the layering to handle versioned plugins this is going to be a little more complex than I expected and I'm unlikely to have time to work on it this cycle. I would be happy to talk about it if someone else wants to pick it up -- it's definitely possible, just not in the time I have available.
Regarding owners / progress on these...
On 5/10/2019 11:48 AM, Dean Troyer wrote:
Also relevant to OSC but covered in a Nova Forum session[3,4], highlights:
- boot-from-volume: Support type=image for the --block-device-mapping,
and Add a --boot-from-volume option which will translate to a root --block-device-mapping using the provided --image value
Artom signed up for this but I don't see patches yet.
- server migrate --live: deprecate the --live option and add a new
--live-migration option and a --host option
- compute migration: begin exposing this resource in the CLI
As far as I know we don't have owners for these yet. I'm a bit swamped right now otherwise I'd start working on the live migration plan we agreed on at the forum [1]. I can take a crack at starting a patch for the live migration thing but will likely need some peer developer synergy action super happy fun time for things like docs/tests/reno.
[1] https://etherpad.openstack.org/p/DEN-osc-compute-api-gaps
On 5/15/2019 3:45 PM, Matt Riedemann wrote:
- server migrate --live: deprecate the --live option and add a new
--live-migration option and a --host option
As far as I know we don't have owners for these yet. I'm a bit swamped right now otherwise I'd start working on the live migration plan we agreed on at the forum [1]. I can take a crack at starting a patch for the live migration thing but will likely need some peer developer synergy action super happy fun time for things like docs/tests/reno.
[1] https://etherpad.openstack.org/p/DEN-osc-compute-api-gaps
Despite my dramatic cry for help I've managed to crank out a start for the live migration fix:
https://review.opendev.org/#/c/659382/
participants (8)
-
Chris Dent
-
Dean Troyer
-
Doug Hellmann
-
Matt Riedemann
-
Monty Taylor
-
Nadathur, Sundar
-
Sean McGinnis
-
Stephen Finucane