[nova][kolla] questions on cells
Hi,
I'm currently working (with dougsz) on adding support for nova cells v2 in Kolla Ansible. I have a few questions, and things that I'd like confirming.
1. Is there any benefit to not having a superconductor? Presumably it's a little more efficient in the single cell case? Also IIUC it only requires a single message queue so is a little simpler?
2. Do console proxies need to live in the cells? This is what devstack does in superconductor mode. I did some digging through nova code, and it looks that way. Testing with novncproxy agrees. This suggests we need to expose a unique proxy endpoint for each cell, and configure all computes to use the right one via e.g. novncproxy_base_url, correct?
3. Should I upgrade the superconductor or conductor service first?
4. Does the cell conductor need access to the API DB?
5. What DB configuration should be used in nova.conf when running online data migrations? I can see some migrations that seem to need the API DB, and others that need a cell DB. If I just give it the API DB, will it use the cell mappings to get to each cell DB, or do I need to run it once for each cell?
6. After an upgrade, when can we restart services to unpin the compute RPC version? Looking at the compute RPC API, it looks like the super conductor will remain pinned until all computes have been upgraded. For a cell conductor, it looks like I could restart it to unpin after upgrading all computes in that cell, correct?
7. Which services require policy.{yml,json}? I can see policy referenced in API, conductor and compute.
I'm trying to design it such that it's possible to deploy a new cell in an existing environment, and upgrade one cell at a time. Getting the ordering right for this is fiddly, and I expect I'll be back to get some eyes on my plan.
Cheers, Mark
- Is there any benefit to not having a superconductor? Presumably
it's a little more efficient in the single cell case? Also IIUC it only requires a single message queue so is a little simpler?
In a multi-cell case you need it, but you're asking about the case where there's only one (real) cell yeah?
If the deployment is really small, then the overhead of having one is probably measurable and undesirable. I dunno what to tell you about where that cut-off is, unfortunately. However, once you're over a certain number of nodes, that probably shakes out a bit. The superconductor does things that the cell-specific ones won't have to do, so there's about the same amount of total load, just a potentially larger memory footprint for running extra services, which would be measurable at small scales. For a tiny deployment there's also overhead just in the complexity, but one of the goals of v2 has always been to get everyone on the same architecture, so having a "small mode" and a "large mode" brings with it its own complexity.
- Do console proxies need to live in the cells? This is what devstack
does in superconductor mode. I did some digging through nova code, and it looks that way. Testing with novncproxy agrees. This suggests we need to expose a unique proxy endpoint for each cell, and configure all computes to use the right one via e.g. novncproxy_base_url, correct?
I'll punt this to Melanie, as she's the console expert at this point, but I imagine you're right.
- Should I upgrade the superconductor or conductor service first?
Superconductor first, although they all kinda have to go around the same time. Superconductor, like the regular conductors, needs to look at the cell database directly, so if you were to upgrade superconductor before the cell database you'd likely have issues. I think probably the ideal would be to upgrade the db schema everywhere (which you can do without rolling code), then upgrade the top-level services (conductor, scheduler, api) and then you could probably get away with doing conductor in the cell along with computes, or whatever. If possible rolling the cell conductors with the top-level services would be ideal.
- Does the cell conductor need access to the API DB?
Technically it should not be allowed to talk to the API DB for "separation of concerns" reasons. However, there are a couple of features that still rely on the cell conductor being able to upcall to the API database, such as the late affinity check. If you can only choose one, then I'd say configure the cell conductors to talk to the API DB, but if there's a knob for "isolate them" it'd be better.
- What DB configuration should be used in nova.conf when running
online data migrations? I can see some migrations that seem to need the API DB, and others that need a cell DB. If I just give it the API DB, will it use the cell mappings to get to each cell DB, or do I need to run it once for each cell?
The API DB has its own set of migrations, so you obviously need API DB connection info to make that happen. There is no fanout to all the rest of the cells (currently), so you need to run it with a conf file pointing to the cell, for each cell you have. The latest attempt at making this fan out was abanoned in July with no explanation, so it dropped off my radar at least.
- After an upgrade, when can we restart services to unpin the compute
RPC version? Looking at the compute RPC API, it looks like the super conductor will remain pinned until all computes have been upgraded. For a cell conductor, it looks like I could restart it to unpin after upgrading all computes in that cell, correct?
Yeah.
- Which services require policy.{yml,json}? I can see policy
referenced in API, conductor and compute.
That's a good question. I would have thought it was just API, so maybe someone else can chime in here, although it's not specific to cells.
--Dan
On 9/30/2019 12:27 PM, Dan Smith wrote:
- Which services require policy.{yml,json}? I can see policy
referenced in API, conductor and compute.
That's a good question. I would have thought it was just API, so maybe someone else can chime in here, although it's not specific to cells.
I don't see any explicit policy checks in conductor. Maybe you're referring to the conductor code handling UnsupportedPolicyException? That's a bad exception class name but its really something that the scheduler service can raise when a server group is created with a given policy, e.g. anti-affinity, and the scheduler filter to enforce that policy is not enabled.
As for the nova-compute service, the only explicit policy check that I know of is here [1]. That's a late check since the API does not necessarily know which network we're going to boot on, we gather that information later in the build process on the compute service. Yes it's unfortunate and it came up in the dev ML a few years ago [2]. The only other "policy" stuff in the compute service is the late affinity check [3] that Dan mentioned which is about server group policy not RBAC.
[1] https://github.com/openstack/nova/blob/627c461a62ce722a4c95a44b181f40b8db198... [2] http://lists.openstack.org/pipermail/openstack-dev/2016-June/096919.html [3] https://github.com/openstack/nova/blob/627c461a62ce722a4c95a44b181f40b8db198...
On 9/30/2019 12:27 PM, Dan Smith wrote:
- Does the cell conductor need access to the API DB?
Technically it should not be allowed to talk to the API DB for "separation of concerns" reasons. However, there are a couple of features that still rely on the cell conductor being able to upcall to the API database, such as the late affinity check.
In case you haven't seen this yet, we have a list of operations requiring "up-calls" from compute/cell-conductor to the API DB in the docs here:
https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#operations-r...
Some have been fixed for awhile and some are still open because they are not default configuration we normally deal with (cross_az_attach=False) or hit in CI* runs (reschedules).
I think the biggest/hardest problem there to solve is the late affinity check which long-term should be solved with placement but no one is working on that. The reschedule stuff related to getting AZ/aggregate info is simpler but involves some RPC changes so it's not trivial and again no one is working on fixing that.
I think for those reasons CERN is running without a superconductor mode and can hit the API DB from the cells. Devstack superconductor mode is the ideal though for the separation of concerns Dan pointed out.
*Note we do hit the reschedule issue sometimes in multi-cell jobs:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A...
I think for those reasons CERN is running without a superconductor mode and can hit the API DB from the cells. Devstack superconductor mode is the ideal though for the separation of concerns Dan pointed out.
I'm not sure that CERN does run without a superconductor, but maybe. However, cells having API DB creds is a different issue -- you can give them those creds and still have a superconductor.
Without a superconductor, you'd have to choose one of your cells to receive the burden of cross-cell work (like instance boots, resizes, and anything we currently don't target to a cell. If you have one big "main" cell, and a bunch of smaller ones, that could make sense as you have extra capacity in the big one. From an organization point of view, and from the deployment tool's perspective, that makes it a little more lopsided than ideal. However, again, if the overhead of running those services is too great, then it makes sense.
--Dan
Hi, just to clarify,
CERN runs the superconductor. Yes, affinity check is an issue. We plan work on it in the next cycle. The metadata API runs per cell. The main reason is that we still run nova-network in few cells.
cheers, Belmiro
On Mon, Sep 30, 2019 at 8:56 PM Matt Riedemann mriedemos@gmail.com wrote:
On 9/30/2019 12:27 PM, Dan Smith wrote:
- Does the cell conductor need access to the API DB?
Technically it should not be allowed to talk to the API DB for "separation of concerns" reasons. However, there are a couple of features that still rely on the cell conductor being able to upcall to the API database, such as the late affinity check.
In case you haven't seen this yet, we have a list of operations requiring "up-calls" from compute/cell-conductor to the API DB in the docs here:
https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#operations-r...
Some have been fixed for awhile and some are still open because they are not default configuration we normally deal with (cross_az_attach=False) or hit in CI* runs (reschedules).
I think the biggest/hardest problem there to solve is the late affinity check which long-term should be solved with placement but no one is working on that. The reschedule stuff related to getting AZ/aggregate info is simpler but involves some RPC changes so it's not trivial and again no one is working on fixing that.
I think for those reasons CERN is running without a superconductor mode and can hit the API DB from the cells. Devstack superconductor mode is the ideal though for the separation of concerns Dan pointed out.
*Note we do hit the reschedule issue sometimes in multi-cell jobs:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A...
--
Thanks,
Matt
On 9/30/2019 12:27 PM, Dan Smith wrote:
- Do console proxies need to live in the cells? This is what devstack
does in superconductor mode. I did some digging through nova code, and it looks that way. Testing with novncproxy agrees. This suggests we need to expose a unique proxy endpoint for each cell, and configure all computes to use the right one via e.g. novncproxy_base_url, correct?
I'll punt this to Melanie, as she's the console expert at this point, but I imagine you're right.
Based on the Rocky spec [1] which says:
"instead we will resolve the cell database issue by running console proxies per cell instead of global to a deployment, such that the cell database is local to the console proxy"
Yes it's per-cell. There was stuff in the Rock release notes about this [2] and a lot of confusion around the deprecation of the nova-consoleauth service for which Mel knows the details, but it looks like we really should have something documented about this too, here [3] and/or here [4].
[1] https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/con... [2] https://docs.openstack.org/releasenotes/nova/rocky.html [3] https://docs.openstack.org/nova/latest/user/cellsv2-layout.html [4] https://docs.openstack.org/nova/latest/admin/remote-console-access.html
On 9/30/19 12:08 PM, Matt Riedemann wrote:
On 9/30/2019 12:27 PM, Dan Smith wrote:
- Do console proxies need to live in the cells? This is what devstack
does in superconductor mode. I did some digging through nova code, and it looks that way. Testing with novncproxy agrees. This suggests we need to expose a unique proxy endpoint for each cell, and configure all computes to use the right one via e.g. novncproxy_base_url, correct?
I'll punt this to Melanie, as she's the console expert at this point, but I imagine you're right.
Based on the Rocky spec [1] which says:
"instead we will resolve the cell database issue by running console proxies per cell instead of global to a deployment, such that the cell database is local to the console proxy"
Yes it's per-cell. There was stuff in the Rock release notes about this [2] and a lot of confusion around the deprecation of the nova-consoleauth service for which Mel knows the details, but it looks like we really should have something documented about this too, here [3] and/or here [4].
To echo, yes, console proxies need to run per cell. This used to be mentioned in our docs and I looked and found it got removed by the following commit:
https://github.com/openstack/nova/commit/009fd0f35bcb88acc80f12e69d5fb72c0ee...
so, we just need to add back the bit about running console proxies per cell.
-melanie
[1] https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/con...
[2] https://docs.openstack.org/releasenotes/nova/rocky.html [3] https://docs.openstack.org/nova/latest/user/cellsv2-layout.html [4] https://docs.openstack.org/nova/latest/admin/remote-console-access.html
On 9/30/19 8:14 PM, melanie witt wrote:
On 9/30/19 12:08 PM, Matt Riedemann wrote:
On 9/30/2019 12:27 PM, Dan Smith wrote:
- Do console proxies need to live in the cells? This is what devstack
does in superconductor mode. I did some digging through nova code, and it looks that way. Testing with novncproxy agrees. This suggests we need to expose a unique proxy endpoint for each cell, and configure all computes to use the right one via e.g. novncproxy_base_url, correct?
I'll punt this to Melanie, as she's the console expert at this point, but I imagine you're right.
Based on the Rocky spec [1] which says:
"instead we will resolve the cell database issue by running console proxies per cell instead of global to a deployment, such that the cell database is local to the console proxy"
Yes it's per-cell. There was stuff in the Rock release notes about this [2] and a lot of confusion around the deprecation of the nova-consoleauth service for which Mel knows the details, but it looks like we really should have something documented about this too, here [3] and/or here [4].
To echo, yes, console proxies need to run per cell. This used to be mentioned in our docs and I looked and found it got removed by the following commit:
https://github.com/openstack/nova/commit/009fd0f35bcb88acc80f12e69d5fb72c0ee...
so, we just need to add back the bit about running console proxies per cell.
FYI I've proposed a patch to restore the doc about console proxies for review:
https://review.opendev.org/686271
-melanie
[1] https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/con...
[2] https://docs.openstack.org/releasenotes/nova/rocky.html [3] https://docs.openstack.org/nova/latest/user/cellsv2-layout.html [4] https://docs.openstack.org/nova/latest/admin/remote-console-access.html
On Wed, 2 Oct 2019 at 21:11, melanie witt melwittt@gmail.com wrote:
On 9/30/19 8:14 PM, melanie witt wrote:
On 9/30/19 12:08 PM, Matt Riedemann wrote:
On 9/30/2019 12:27 PM, Dan Smith wrote:
- Do console proxies need to live in the cells? This is what devstack
does in superconductor mode. I did some digging through nova code, and it looks that way. Testing with novncproxy agrees. This suggests we need to expose a unique proxy endpoint for each cell, and configure all computes to use the right one via e.g. novncproxy_base_url, correct?
I'll punt this to Melanie, as she's the console expert at this point, but I imagine you're right.
Based on the Rocky spec [1] which says:
"instead we will resolve the cell database issue by running console proxies per cell instead of global to a deployment, such that the cell database is local to the console proxy"
Yes it's per-cell. There was stuff in the Rock release notes about this [2] and a lot of confusion around the deprecation of the nova-consoleauth service for which Mel knows the details, but it looks like we really should have something documented about this too, here [3] and/or here [4].
To echo, yes, console proxies need to run per cell. This used to be mentioned in our docs and I looked and found it got removed by the following commit:
https://github.com/openstack/nova/commit/009fd0f35bcb88acc80f12e69d5fb72c0ee...
so, we just need to add back the bit about running console proxies per cell.
FYI I've proposed a patch to restore the doc about console proxies for review:
Great, thanks. I know it's merged, but I added a comment.
-melanie
[1] https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/con...
[2] https://docs.openstack.org/releasenotes/nova/rocky.html [3] https://docs.openstack.org/nova/latest/user/cellsv2-layout.html [4] https://docs.openstack.org/nova/latest/admin/remote-console-access.html
Thanks all for your responses. Replies to Dan inline.
On Mon, 30 Sep 2019 at 18:27, Dan Smith dms@danplanet.com wrote:
- Is there any benefit to not having a superconductor? Presumably
it's a little more efficient in the single cell case? Also IIUC it only requires a single message queue so is a little simpler?
In a multi-cell case you need it, but you're asking about the case where there's only one (real) cell yeah?
If the deployment is really small, then the overhead of having one is probably measurable and undesirable. I dunno what to tell you about where that cut-off is, unfortunately. However, once you're over a certain number of nodes, that probably shakes out a bit. The superconductor does things that the cell-specific ones won't have to do, so there's about the same amount of total load, just a potentially larger memory footprint for running extra services, which would be measurable at small scales. For a tiny deployment there's also overhead just in the complexity, but one of the goals of v2 has always been to get everyone on the same architecture, so having a "small mode" and a "large mode" brings with it its own complexity.
Thanks for the explanation. We've built in a switch for single or super mode, and single mode keeps us compatible with existing deployments, so I guess we'll keep the switch.
- Do console proxies need to live in the cells? This is what devstack
does in superconductor mode. I did some digging through nova code, and it looks that way. Testing with novncproxy agrees. This suggests we need to expose a unique proxy endpoint for each cell, and configure all computes to use the right one via e.g. novncproxy_base_url, correct?
I'll punt this to Melanie, as she's the console expert at this point, but I imagine you're right.
- Should I upgrade the superconductor or conductor service first?
Superconductor first, although they all kinda have to go around the same time. Superconductor, like the regular conductors, needs to look at the cell database directly, so if you were to upgrade superconductor before the cell database you'd likely have issues. I think probably the ideal would be to upgrade the db schema everywhere (which you can do without rolling code), then upgrade the top-level services (conductor, scheduler, api) and then you could probably get away with doing conductor in the cell along with computes, or whatever. If possible rolling the cell conductors with the top-level services would be ideal.
I should have included my strawman deploy and upgrade flow for context, but I'm still honing it. All DB schema changes will be done up front in both cases.
In terms of ordering, the API-level services (superconductor, API scheduler) are grouped together and will be rolled first - agreeing with what you've said. I think between Ansible's tags and limiting actions to specific hosts, the code can be written to support upgrading all cell conductors together, or at the same time as (well, immediately before) the cell's computes.
The thinking behind upgrading one cell at a time is to limit the blast radius if something goes wrong. You suggest it would be better to roll all cell conductors at the same time though - do you think it's safer to run with the version disparity between conductor and computes rather than super- and cell- conductors?
- Does the cell conductor need access to the API DB?
Technically it should not be allowed to talk to the API DB for "separation of concerns" reasons. However, there are a couple of features that still rely on the cell conductor being able to upcall to the API database, such as the late affinity check. If you can only choose one, then I'd say configure the cell conductors to talk to the API DB, but if there's a knob for "isolate them" it'd be better.
Knobs are easy to make, and difficult to keep working in all positions :) It seems worthwhile in this case.
- What DB configuration should be used in nova.conf when running
online data migrations? I can see some migrations that seem to need the API DB, and others that need a cell DB. If I just give it the API DB, will it use the cell mappings to get to each cell DB, or do I need to run it once for each cell?
The API DB has its own set of migrations, so you obviously need API DB connection info to make that happen. There is no fanout to all the rest of the cells (currently), so you need to run it with a conf file pointing to the cell, for each cell you have. The latest attempt at making this fan out was abanoned in July with no explanation, so it dropped off my radar at least.
That makes sense. The rolling upgrade docs could be a little clearer for multi-cell deployments here.
- After an upgrade, when can we restart services to unpin the compute
RPC version? Looking at the compute RPC API, it looks like the super conductor will remain pinned until all computes have been upgraded. For a cell conductor, it looks like I could restart it to unpin after upgrading all computes in that cell, correct?
Yeah.
- Which services require policy.{yml,json}? I can see policy
referenced in API, conductor and compute.
That's a good question. I would have thought it was just API, so maybe someone else can chime in here, although it's not specific to cells.
Yeah, unrelated to cells, just something I wondered while digging through our nova Ansible role.
Here is the line that made me think policies are required in conductors: https://opendev.org/openstack/nova/src/commit/6d5fdb4ef4dc3e5f40298e751d966c.... I guess this is only required for cell conductors though?
--Dan
Mark Goddard mark@stackhpc.com writes:
The thinking behind upgrading one cell at a time is to limit the blast radius if something goes wrong. You suggest it would be better to roll all cell conductors at the same time though - do you think it's safer to run with the version disparity between conductor and computes rather than super- and cell- conductors?
Yes, the conductors and computes are built to work at different versions. Conductors, not so much. While you can pin the conductor RPC version to *technically* make them talk, they will do things like migrate data to new formats in the cell databases and since they *are* the insulation layer against such changes, older conductors are going to be unhappy if new conductors move data underneath them before they're ready.
Here is the line that made me think policies are required in conductors: https://opendev.org/openstack/nova/src/commit/6d5fdb4ef4dc3e5f40298e751d966c.... I guess this is only required for cell conductors though?
No, actually more likely to be the superconductors I think. However, it could technically be called at the cell level so you probably need to make sure it's there. That might be something left-over from a check that moved to the API and could now be removed (or ignored).
--Dan
On 10/1/2019 5:00 AM, Mark Goddard wrote:
- What DB configuration should be used in nova.conf when running
online data migrations? I can see some migrations that seem to need the API DB, and others that need a cell DB. If I just give it the API DB, will it use the cell mappings to get to each cell DB, or do I need to run it once for each cell?
The API DB has its own set of migrations, so you obviously need API DB connection info to make that happen. There is no fanout to all the rest of the cells (currently), so you need to run it with a conf file pointing to the cell, for each cell you have. The latest attempt at making this fan out was abanoned in July with no explanation, so it dropped off my radar at least.
That makes sense. The rolling upgrade docs could be a little clearer for multi-cell deployments here.
This recently merged, hopefully it helps clarify:
https://review.opendev.org/#/c/671298/
- After an upgrade, when can we restart services to unpin the compute
RPC version? Looking at the compute RPC API, it looks like the super conductor will remain pinned until all computes have been upgraded. For a cell conductor, it looks like I could restart it to unpin after upgrading all computes in that cell, correct?
Yeah.
- Which services require policy.{yml,json}? I can see policy
referenced in API, conductor and compute.
That's a good question. I would have thought it was just API, so maybe someone else can chime in here, although it's not specific to cells.
Yeah, unrelated to cells, just something I wondered while digging through our nova Ansible role.
Here is the line that made me think policies are required in conductors:https://opendev.org/openstack/nova/src/commit/6d5fdb4ef4dc3e5f40298e751d966c.... I guess this is only required for cell conductors though?
That is not the conductor service, it's the API.
On Wed, 2 Oct 2019 at 14:48, Matt Riedemann mriedemos@gmail.com wrote:
On 10/1/2019 5:00 AM, Mark Goddard wrote:
- What DB configuration should be used in nova.conf when running
online data migrations? I can see some migrations that seem to need the API DB, and others that need a cell DB. If I just give it the API DB, will it use the cell mappings to get to each cell DB, or do I need to run it once for each cell?
The API DB has its own set of migrations, so you obviously need API DB connection info to make that happen. There is no fanout to all the rest of the cells (currently), so you need to run it with a conf file pointing to the cell, for each cell you have. The latest attempt at making this fan out was abanoned in July with no explanation, so it dropped off my radar at least.
That makes sense. The rolling upgrade docs could be a little clearer for multi-cell deployments here.
This recently merged, hopefully it helps clarify:
It does help a little for the schema migrations, but the point was about data migrations.
- After an upgrade, when can we restart services to unpin the compute
RPC version? Looking at the compute RPC API, it looks like the super conductor will remain pinned until all computes have been upgraded. For a cell conductor, it looks like I could restart it to unpin after upgrading all computes in that cell, correct?
Yeah.
- Which services require policy.{yml,json}? I can see policy
referenced in API, conductor and compute.
That's a good question. I would have thought it was just API, so maybe someone else can chime in here, although it's not specific to cells.
Yeah, unrelated to cells, just something I wondered while digging through our nova Ansible role.
Here is the line that made me think policies are required in conductors:https://opendev.org/openstack/nova/src/commit/6d5fdb4ef4dc3e5f40298e751d966c.... I guess this is only required for cell conductors though?
That is not the conductor service, it's the API.
My mistake, still learning the flow of communication.
--
Thanks,
Matt
On 10/3/2019 3:24 AM, Mark Goddard wrote:
This recently merged, hopefully it helps clarify:
It does help a little for the schema migrations, but the point was about data migrations.
That's an excellent point. Looking at devstack [1] and grenade [2] we don't necessarily do that properly. For devstack with a fresh install it doesn't really matter but it should matter for grenade since we should be migrating both cell0 and cell1. Grenade does not run in "superconductor" mode so some of the rules might be different there, i.e. grenade's nova.conf has the database pointed at cell1 while devstack has the database config pointed at cell0. Either way we're not properly running the online data migrations per cell DB as far as I can tell. Maybe we just haven't had an online data migration yet that makes that important, but it's definitely wrong. I also don't see anything in the docs for the online_data_migrations command [3] to use the --config-file option to run it against the cell DB config. I can open a bug for that. The upgrade guide should also be updated to mention that like for db sync in https://review.opendev.org/#/c/671298/.
[1] https://github.com/openstack/devstack/blob/1a46c898db9c16173013d95e2bc954992... [2] https://github.com/openstack/grenade/blob/bb14e02a464db2b268930bbba0152862fe... [3] https://docs.openstack.org/nova/latest/cli/nova-manage.html
On 01.10.2019 12:00, Mark Goddard wrote:
Thanks all for your responses. Replies to Dan inline.
On Mon, 30 Sep 2019 at 18:27, Dan Smith dms@danplanet.com wrote:
- Is there any benefit to not having a superconductor? Presumably
it's a little more efficient in the single cell case? Also IIUC it only requires a single message queue so is a little simpler?
In a multi-cell case you need it, but you're asking about the case where there's only one (real) cell yeah?
If the deployment is really small, then the overhead of having one is probably measurable and undesirable. I dunno what to tell you about where that cut-off is, unfortunately. However, once you're over a certain number of nodes, that probably shakes out a bit. The superconductor does things that the cell-specific ones won't have to do, so there's about the same amount of total load, just a potentially larger memory footprint for running extra services, which would be measurable at small scales. For a tiny deployment there's also overhead just in the complexity, but one of the goals of v2 has always been to get everyone on the same architecture, so having a "small mode" and a "large mode" brings with it its own complexity.
Thanks for the explanation. We've built in a switch for single or super mode, and single mode keeps us compatible with existing deployments, so I guess we'll keep the switch.
- Do console proxies need to live in the cells? This is what devstack
does in superconductor mode. I did some digging through nova code, and it looks that way. Testing with novncproxy agrees. This suggests we need to expose a unique proxy endpoint for each cell, and configure all computes to use the right one via e.g. novncproxy_base_url, correct?
I'll punt this to Melanie, as she's the console expert at this point, but I imagine you're right.
- Should I upgrade the superconductor or conductor service first?
Superconductor first, although they all kinda have to go around the same time. Superconductor, like the regular conductors, needs to look at the cell database directly, so if you were to upgrade superconductor before the cell database you'd likely have issues. I think probably the ideal would be to upgrade the db schema everywhere (which you can do without rolling code), then upgrade the top-level services (conductor, scheduler, api) and then you could probably get away with doing conductor in the cell along with computes, or whatever. If possible rolling the cell conductors with the top-level services would be ideal.
I should have included my strawman deploy and upgrade flow for context, but I'm still honing it. All DB schema changes will be done up front in both cases.
In terms of ordering, the API-level services (superconductor, API scheduler) are grouped together and will be rolled first - agreeing with what you've said. I think between Ansible's tags and limiting actions to specific hosts, the code can be written to support upgrading all cell conductors together, or at the same time as (well, immediately before) the cell's computes.
The thinking behind upgrading one cell at a time is to limit the blast radius if something goes wrong. You suggest it would be better to roll all cell conductors at the same time though - do you think it's safer to run with the version disparity between conductor and computes rather than super- and cell- conductors?
I'd say upgrading one cell at a time may be in important consideration for EDGE (DCN) multi-cells deployments, where it may be technically impossible to roll it over all of the remote sites due to reasons.
- Does the cell conductor need access to the API DB?
Technically it should not be allowed to talk to the API DB for "separation of concerns" reasons. However, there are a couple of features that still rely on the cell conductor being able to upcall to the API database, such as the late affinity check. If you can only choose one, then I'd say configure the cell conductors to talk to the API DB, but if there's a knob for "isolate them" it'd be better.
Knobs are easy to make, and difficult to keep working in all positions :) It seems worthwhile in this case.
- What DB configuration should be used in nova.conf when running
online data migrations? I can see some migrations that seem to need the API DB, and others that need a cell DB. If I just give it the API DB, will it use the cell mappings to get to each cell DB, or do I need to run it once for each cell?
The API DB has its own set of migrations, so you obviously need API DB connection info to make that happen. There is no fanout to all the rest of the cells (currently), so you need to run it with a conf file pointing to the cell, for each cell you have. The latest attempt at making this fan out was abanoned in July with no explanation, so it dropped off my radar at least.
That makes sense. The rolling upgrade docs could be a little clearer for multi-cell deployments here.
- After an upgrade, when can we restart services to unpin the compute
RPC version? Looking at the compute RPC API, it looks like the super conductor will remain pinned until all computes have been upgraded. For a cell conductor, it looks like I could restart it to unpin after upgrading all computes in that cell, correct?
Yeah.
- Which services require policy.{yml,json}? I can see policy
referenced in API, conductor and compute.
That's a good question. I would have thought it was just API, so maybe someone else can chime in here, although it's not specific to cells.
Yeah, unrelated to cells, just something I wondered while digging through our nova Ansible role.
Here is the line that made me think policies are required in conductors: https://opendev.org/openstack/nova/src/commit/6d5fdb4ef4dc3e5f40298e751d966c.... I guess this is only required for cell conductors though?
--Dan
On 9/30/2019 11:48 AM, Mark Goddard wrote:
- Do console proxies need to live in the cells? This is what devstack
does in superconductor mode. I did some digging through nova code, and it looks that way. Testing with novncproxy agrees. This suggests we need to expose a unique proxy endpoint for each cell, and configure all computes to use the right one via e.g. novncproxy_base_url, correct?
Unrelated to console proxy but sort of related to this topic, you can also run the metadata-api service globally or per-cell, details are in the docs:
https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#nova-metadat...
CERN runs the metadata API per cell for performance reasons.
participants (6)
-
Belmiro Moreira
-
Bogdan Dobrelya
-
Dan Smith
-
Mark Goddard
-
Matt Riedemann
-
melanie witt