<div dir="ltr"><div><font face="courier new, monospace">Hi, ALL</font></div><div><font face="courier new, monospace"><br></font></div><font face="courier new, monospace">Recently I've been working on two blueprints[1][2], both involved with recording scheduling information. And would like to hear some comments for several design choices.<br>
<br>Problem Statement<br>--<br>* NoValidHost exception might masked out real failure reason to spin up an instance.<br><br>Consider following event sequence, "run_instance" on host1 failed to spin up an instance due to port allocation failure in neutron. The request casted back to scheduler to pick next available host. It failed again on host2 for the same reason of port allocation error. After Maximum 3 times to retry, instance is set in "ERROR" state with a NoValidHost exception. And there's no easy way to find out what is really going wrong.<br>
<br>* Current scheduling information are recorded in several different log items, which is difficult to lookup when debugging.<br><br>Design Proposal<br>--<br>1. Blueprint internal-scheduler[1] will try to address problem #1. After conductor retrieved selected destination hosts from scheduler, it will create a "scheduler_records_allocations" item in database, for each allocated instance/host allocation.</font><div>
<font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">Design choices:</font></div><div><font face="courier new, monospace">a) Correlate this </font><span style="font-family:'courier new',monospace">scheduler_records_allocations with the 'create' instance action, and generate a combined view with instance-action events.</span></div>
<div><font face="courier new, monospace">b) Add separate new API to retrieve this information.</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">I prefer the choice #a, because instance action events perfectly fits such usage case. And allocation records will supplement necessary information when viewing 'create' action events of an instance.</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">Thoughts?</font></div><div><br></div><div><div><font face="courier new, monospace">NOTE: Please find the following chart in link[3], in case of any format/display issue.<br>
<br> scheduler_records_allocations<br> +-----------------------------+<br> |allocation_id: 9001 |<br> |instance_uuid: inst1_uuid |<br>
scheduler_records |scheduler_record_id: 1210 |<br> +------------------------------+ |host: host1 |<br> |scheduler_record_id: 1210 | |weight: 197.0 | +---------------+<br>
|user_id: 'u_fakeid' | |result: Failed | |instance1 |<br> |project_id: 'p_fakeid' | |reason: 'No more IP addresses| +---------------+<br> |request_id: 'req-xxx' | +-----------------------------+<br>
|instance_uuids: [ | +-----------------------------+ +---------------+<br> | 'inst1_uuid', | |allocation_id: 9002 | |instance2 |<br> | 'inst2_uuid'] | |instance_uuid: inst2_uuid | +---------------+<br>
|request_spec: {...} | |scheduler_record_id: 1210 |<br> |filter_properties: {...} | |host: host2 |<br> |scheduler_records_allocations:| |weight: 128.0 |<br> | [9001, 9002] | |result: Success |<br>
|start_time: ... | |reason: |<br> |finish_time: ... | +-----------------------------+<br> +------------------------------+ +-----------------------------+<br> |allocation_id: 9003 |<br>
|instance_uuid: inst1_uuid |<br> |scheduler_record_id: 1210 |<br> |host: host2 |<br> |weight: 64.0 |<br>
|result: Failed |<br> |reason: 'No more IP addresses|<br> +-----------------------------+<br><br></font><div>
<font face="courier new, monospace">2. Blueprint record-scheduler-information[2] will try to solve the problem #2, to generate a structured information for each scheduler run.</font></div><div><span style="font-family:'courier new',monospace"><br>
</span></div><div><span style="font-family:'courier new',monospace">Design choices:</span></div><div><span style="font-family:'courier new',monospace">a) Record 'scheduler_records' info in database, which is easy to query, but introduce a great burden in terms of performance, extra database space usage, clean up/archiving policy, security relate issue[4], etc.</span></div>
<div><span style="font-family:'courier new',monospace">b) Record '</span><span style="font-family:'courier new',monospace">scheduler_records' into a separate log file, in JSON format, and each line for a single record of each scheduler run. And then add a new API extension to retrieve last n (as a query parameter) scheduler records. </span><span style="font-family:'courier new',monospace">The benefit of this approach avoided database issue, and plays well with external tooling, as well as provide a central place to view the log. But as a compromise, we won't be able to query logs for specific request_id.</span></div>
<div><span style="font-family:'courier new',monospace"><br></span></div><div><span style="font-family:'courier new',monospace">So the problem here is, is database storage solution still desirable? Or... implement backend driver which deployer could choose? However, in such case, API would be the minimum set to support both.</span></div>
<div><span style="font-family:'courier new',monospace"><br></span></div><div><span style="font-family:'courier new',monospace">Any comments or thoughts are highly appreciated. </span></div><font face="courier new, monospace"><br>
[1] <a href="https://blueprints.launchpad.net/nova/+spec/internal-scheduler">https://blueprints.launchpad.net/nova/+spec/internal-scheduler</a><br>[2] <a href="https://blueprints.launchpad.net/nova/+spec/record-scheduler-information">https://blueprints.launchpad.net/nova/+spec/record-scheduler-information</a></font></div>
<div><font face="courier new, monospace">[3] <a href="https://docs.google.com/document/d/1EsSNeq_tD-3NiX4IphCrQj4ii0_dO-8-Jn7NWHRJPNg/edit?usp=sharing">https://docs.google.com/document/d/1EsSNeq_tD-3NiX4IphCrQj4ii0_dO-8-Jn7NWHRJPNg/edit?usp=sharing</a></font></div>
<div><font face="courier new, monospace">[4] <a href="https://bugs.launchpad.net/nova/+bug/1175193">https://bugs.launchpad.net/nova/+bug/1175193</a><br><br>Thanks,<br>--<br>Qiu Yu</font></div></div></div>