Open Stack

Le 03/06/2015 15:15, Nikola Đipanov a écrit :
> On 06/02/2015 03:14 PM, Sylvain Bauza wrote:
>> Hi,
>>
>> Currently working on implementing the RequestSpec object BP [1], I had
>> some cool comments on my change here :
>> https://review.openstack.org/#/c/145528/12/nova/objects/request_spec.py,cm
>>
>> Since we didn't discussed on how to persist that RequestSpec object, I
>> think the comment is valuable.
>>
>> For the moment, the only agreed spec for persisting the object that we
>> have is [2] but there is also a corollar here that means that we would
>> have to persist more than the current fields
>> https://review.openstack.org/#/c/169901/3/specs/liberty/approved/add-buildrequest-obj.rst,cm
>>
>>
>> So, there are 2 possibilities :
>>   #1, we only persist the RequestSpec for the sole Scheduler and in that
>> case, we can leave as it is - only a few fields from Instance are stored
>>   #2, we consider that RequestSpec can be used for more than just the
>> Scheduler, and then we need to make sure that we will have all the
>> instance fields then.
>>
> So these are 2 possibilities if we agree that we need to make progress
> on the spec as is defined and merged now. What I was complaining
> yesterday is that we don't seem to have done enough of high level
> investigation into this stuff before embarking on writing a set of specs
> that then due to their format obscure the problems we are actually
> trying to solve.

Since Nova is big, it's pretty hard to take the big picture of all the 
problems that we have and provide a spec which can be fine grained 
enough for making sure it will take in account the overall problem.

At least, I'm seriously considering the RequestSpec object and how it is 
persisted as a first attempt to version a Scheduler API and provide an 
upgrade path for changing it.


> Work around the scheduler touches on a lot of issues that have only
> recently been noticed. While I am all for the incremental approach, it
> seems silly to completely disregard the issues we already know about. We
> should have a high level overview of the problems we know we want to
> solve, and then come up with an incremental way of solving them, but not
> without keeping an eye on the big picture at all times.

True story. That's why the biggest question I have is "do I actually add 
more technical debt to the project by writing RequestSpec, or is it 
something that helps reducing the debt ?"

So, that's why I'm very open to any comments explaining which kind of 
problem the current proposal could raise, and how we could prevent that.
IIUC, your concern is about duplicating some information within the 
RequestSpec object which would be persisted. My point is to consider 
that if the usage is well defined (ie. only a contract between conductor 
and scheduler, and not reused anywhere else) then that's a reasonable 
trade-off.

> An ad-hoc list of individual issues that we know about and should be
> trying to solve (in no particular order) that all seem related to the
> data model design problem we are trying to take a stab at here:
> 1/ RequestSpec is an unversioned dict even though it's the central piece
> of a placement request for the scheduler
Targeted by 
http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/request-spec-object.html
> 2/ There are scheduler_hints that are needed throughout the lifecycle of
> an instance but are never persisted so are lost after boot
Targeted by 
http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/persist-request-spec.html
> 3/ We have the Migration objects that are used both for resource
> tracking for instances being migrated, and as an indication of an
> instance being moved, but are not used in all the places we need this
> kind of book keeping (live migration, rebuild)
http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/robustify_evacuate.html 
is aiming to provide that evacuations, should we also propose to track 
rebuilds and live migrations as well ?
> 4/ Evacuate (an orchestrated rebuild) is especially problematic because
> it usually involves failure modes, which are difficult to identify and
> handle properly without a consistently used data model.
Could you please describe more the problem ? Is it due to the fact that 
we're not persisting the request spec so it's basically a wet finger for 
knowing if that's good or not ?
> 5/ Some of the recently added constraints that influence resource
> tracking (NUMA, CPU pinning) cannot simply be calculated from the flavor
> on the fly when tracking resources, but need to be persisted after a
> successful claim as they are dependent on the state of the host at that
> very moment (see [1])

I just wonder if a scheduler API called 'check_destination(RequestSpec, 
destination) could help ? I mean, when live-migrating, we know that we 
try to check what we can, but since neither the spec is persisted, nor 
the migration is verifying that the destination host can fulfill the 
request, there are many bugs with that.

FYI, I tried to propose that https://review.openstack.org/#/c/179224/ 
which still needs to be approved. Comments welcome. As I said in the 
spec proposal, it's half of the problem because the spec is not 
persisted yet. Once we will have *both* a destination check and a 
persisted spec, we should be able to verify those assertions, nope ?
> 6/ Related to the previous one - there is data related to the instance
> in addition to the flavor that need to follow the '_old' and '_new'
> pattern (needs the values related to both source and destination host
> persisted during a migration/resize/live migration/)

Could you please describe more the problem again ? Do we have a problem 
with the resource usage ? Do we have a bug which describes it ?
> 7/ The issues cells v2 folks are hitting (mentioned above) where they
> don't want to have any Instances in the top level cell but still need to
> persist stuff.

Agreed, that's why Andrew Laski proposed to amend 
http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/persist-request-spec.html 
to take in account the cells problem.
> 8/ Issues with having no access to individual instance UUIDs in the
> scheduler, but a lot of data access for more complex filtering revolves
> around it being present.

That's exactly why the objectification is necessary, because it will 
stabilize what we provide to the Scheduler IMHO.

> Most of the above have individual bugs that I can try to find and link
> here too.
>
> [1] https://bugs.launchpad.net/nova/+bug/1417667
>
> The overall theme of all the above is (to paraphrase alaski from IRC)
> how to organize the big blob of data that is an instance in all of it's
> possible states, in such a way that it makes sense, nothing is missing,
> there is as little duplication as possible, and access patterns of
> different services that require different bits can work without massive
> overhead.
+1 I think your concern is totally valid, I'm just trying to provide 
some efforts that, once linked, will make Nova more stronger than it is now.
>> I'm not strongly opiniated on that, I maybe consider that #2 is probably
>> the best option but there is a tie in my mind. Help me figuring out
>> what's the best option.
>>
> If we want to keep things moving forward on this particular BP - I'd go
> with adding the RequestSpec object and make sure the code that uses it
> is migrated. I believe that spike alone will leave us with much better
> idea about the problem.
>
> In addition - writing a high level spec/wiki that we can refer back to
> in individual BPs and see how they solve it would be massively helpful too.

Cool, do you want me to write this one ?

Thanks for all your insights, greatly appreciated.

-Sylvain

> N.
>
>> -Sylvain
>>
>> [1] :
>> http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/request-spec-object.html
>>
>> [2] :
>> http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/persist-request-spec.html
>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev