Controversial backport
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi all, we have a Neutron backport proposed in stable/havana that is in grey area in terms of applicability for stable branches. Located at [1]. The controversy is that it changes the default value of a configuration setting, which is generally considered as a no-go for backports. That said, it has additional aspects that may trigger the exception. Basically, the default value of the setting breaks all L3 agents we have in our tree. The failure state is an unstoppable agent resync due to incorrect gateway outside of subnet. To avoid the failure for any L3 agent that we ship, users need to change the value of that setting from default one. It's also not clear what's the value of having a gateway outside the current subnet. As of Juno, this 'feature' is obsolete and marked to be removed in Kilo. There is a change though that there are some third-party agents in the wild that support that 'feature'; in that case, changing the default value will affect them. Though we don't know whether they really exist. All in all, the 'feature' won't be available for them since Kilo, and there is no sign that its removal is controversial, so probably no one is interested in it anyway. So we have a dilemma: we may break the official stable rule of 'not introducing inconmpatible configuration changes' and in that way fix major failure scenario in Neutron; or we may leave everything as-is (=our own agents broken) to stick to the rule. My common sense tells me that we should fix our agents first, and then think about potential third-party L3 agents that possibly rely on that 'feature' being enabled by default. Hence my +2 vote. But your common sense may vary, so I'd like to hear from others on the matter. [1]: https://review.openstack.org/#/c/113129/ /Ihar -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iQEcBAEBCgAGBQJT8wzoAAoJEC5aWaUY1u578HAH/0DB3+IpYWjKbwo9th3Phaf/ EvQEON2tMA177ufdgzO3KsWdx8VJxNzFfHU6Jl8aY9GQ3NYsg+5QYOJPcX0hHg4t g3yfrcUybDZ9roqzHZNYOUddm10Te7a+Q4Ykj2X2QiqEZC/VpxudRYDGU5O5cDpg sNeFltCp/vb/oGfX/E5dxdF5hzGd1sJQvDs+0jXfLghaCmNITvLFELN56JsUFxvr ZuzNxCnyvL7iJSHA+vWiG36jZCF/dR4eYly2CpRaQJ2Fp6ObVvkWCv+U+wtv+5bm TYl8PHpDmmGOlJPaBQn7OXAGc5eQZnkBwXGDxhRl6Fi9JF2fRSV0blSZl4rwpEA= =lxWH -----END PGP SIGNATURE-----
----- Original Message -----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
Hi all,
we have a Neutron backport proposed in stable/havana that is in grey area in terms of applicability for stable branches. Located at [1].
The controversy is that it changes the default value of a configuration setting, which is generally considered as a no-go for backports. That said, it has additional aspects that may trigger the exception.
Basically, the default value of the setting breaks all L3 agents we have in our tree. The failure state is an unstoppable agent resync due to incorrect gateway outside of subnet. To avoid the failure for any L3 agent that we ship, users need to change the value of that setting from default one.
It's also not clear what's the value of having a gateway outside the current subnet. As of Juno, this 'feature' is obsolete and marked to be removed in Kilo.
There is a change though that there are some third-party agents in the wild that support that 'feature'; in that case, changing the default value will affect them. Though we don't know whether they really exist. All in all, the 'feature' won't be available for them since Kilo, and there is no sign that its removal is controversial, so probably no one is interested in it anyway.
So we have a dilemma: we may break the official stable rule of 'not introducing inconmpatible configuration changes' and in that way fix major failure scenario in Neutron; or we may leave everything as-is (=our own agents broken) to stick to the rule.
My common sense tells me that we should fix our agents first, and then think about potential third-party L3 agents that possibly rely on that 'feature' being enabled by default. Hence my +2 vote.
But your common sense may vary, so I'd like to hear from others on the matter.
Some more context: Vishal has a patch on master, currently in review: https://review.openstack.org/#/c/114185/ Which fixes the issue for the L3 agent, but not for the DHCP agent. Personally I think that the backport should be permitted, and the agents should be fixed in the Juno and Kilo cycles.
/Ihar -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
iQEcBAEBCgAGBQJT8wzoAAoJEC5aWaUY1u578HAH/0DB3+IpYWjKbwo9th3Phaf/ EvQEON2tMA177ufdgzO3KsWdx8VJxNzFfHU6Jl8aY9GQ3NYsg+5QYOJPcX0hHg4t g3yfrcUybDZ9roqzHZNYOUddm10Te7a+Q4Ykj2X2QiqEZC/VpxudRYDGU5O5cDpg sNeFltCp/vb/oGfX/E5dxdF5hzGd1sJQvDs+0jXfLghaCmNITvLFELN56JsUFxvr ZuzNxCnyvL7iJSHA+vWiG36jZCF/dR4eYly2CpRaQJ2Fp6ObVvkWCv+U+wtv+5bm TYl8PHpDmmGOlJPaBQn7OXAGc5eQZnkBwXGDxhRl6Fi9JF2fRSV0blSZl4rwpEA= =lxWH -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 19/08/14 10:47, Assaf Muller wrote:
----- Original Message ----- Hi all,
we have a Neutron backport proposed in stable/havana that is in grey area in terms of applicability for stable branches. Located at [1].
The controversy is that it changes the default value of a configuration setting, which is generally considered as a no-go for backports. That said, it has additional aspects that may trigger the exception.
Basically, the default value of the setting breaks all L3 agents we have in our tree. The failure state is an unstoppable agent resync due to incorrect gateway outside of subnet. To avoid the failure for any L3 agent that we ship, users need to change the value of that setting from default one.
It's also not clear what's the value of having a gateway outside the current subnet. As of Juno, this 'feature' is obsolete and marked to be removed in Kilo.
There is a change though that there are some third-party agents in the wild that support that 'feature'; in that case, changing the default value will affect them. Though we don't know whether they really exist. All in all, the 'feature' won't be available for them since Kilo, and there is no sign that its removal is controversial, so probably no one is interested in it anyway.
So we have a dilemma: we may break the official stable rule of 'not introducing inconmpatible configuration changes' and in that way fix major failure scenario in Neutron; or we may leave everything as-is (=our own agents broken) to stick to the rule.
My common sense tells me that we should fix our agents first, and then think about potential third-party L3 agents that possibly rely on that 'feature' being enabled by default. Hence my +2 vote.
But your common sense may vary, so I'd like to hear from others on the matter.
[1]: https://review.openstack.org/#/c/113129/
Some more context: Vishal has a patch on master, currently in review: https://review.openstack.org/#/c/114185/
Which fixes the issue for the L3 agent, but not for the DHCP agent. Personally I think that the backport should be permitted, and the agents should be fixed in the Juno and Kilo cycles.
Thanks. So the feature in question is actually considered as valid, even though exotic. How about enforcing the rule on per-agent base instead of globally? We know that our agents are broken, though we're not sure whether there are other agents in production that are not. So why not just disable the 'feature' for our agents but still allow other third-party agents to run with the current default value, in case they rely on that? /Ihar -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iQEcBAEBCgAGBQJT8xFLAAoJEC5aWaUY1u57lnkIAOQUVXFeGnmLzChcKBR/Bw2D ORDALIBAXOlXbX1UoefJQsW4shZa//fdjXsX1NSaa+p46/OqLVlsuXw2iB/cs4X7 0bdDGL8K8a0fHSVI4Z7ZWnL6DrTm48vZmKv2BwGO+OPgW1TWoN5sw8gV33WevZ8t rQ7JLDxYMXNdB0BCGQmfbUIluhUznVdCLNub26Tw4sa1dtNOrRudOoEy9ylQk9hM w/8QodX4bbpnw+iLw7ox1h7J02XsjkVEpnokF1r4WketxQULE+xNukcekosNy5XJ /seJPjflF09uoxHqScNZC8kVAfGLsNTaaSivX7N+2rGrSBXunZBg0RXIGaTWefA= =9Ygu -----END PGP SIGNATURE-----
I think that only in exceptional cases should we allow changing of default configuration variables. This may break existing setups. I am not in favor of this back port. On 8/19/14, 11:56 AM, "Ihar Hrachyshka" <ihrachys@redhat.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
On 19/08/14 10:47, Assaf Muller wrote:
----- Original Message ----- Hi all,
we have a Neutron backport proposed in stable/havana that is in grey area in terms of applicability for stable branches. Located at [1].
The controversy is that it changes the default value of a configuration setting, which is generally considered as a no-go for backports. That said, it has additional aspects that may trigger the exception.
Basically, the default value of the setting breaks all L3 agents we have in our tree. The failure state is an unstoppable agent resync due to incorrect gateway outside of subnet. To avoid the failure for any L3 agent that we ship, users need to change the value of that setting from default one.
It's also not clear what's the value of having a gateway outside the current subnet. As of Juno, this 'feature' is obsolete and marked to be removed in Kilo.
There is a change though that there are some third-party agents in the wild that support that 'feature'; in that case, changing the default value will affect them. Though we don't know whether they really exist. All in all, the 'feature' won't be available for them since Kilo, and there is no sign that its removal is controversial, so probably no one is interested in it anyway.
So we have a dilemma: we may break the official stable rule of 'not introducing inconmpatible configuration changes' and in that way fix major failure scenario in Neutron; or we may leave everything as-is (=our own agents broken) to stick to the rule.
My common sense tells me that we should fix our agents first, and then think about potential third-party L3 agents that possibly rely on that 'feature' being enabled by default. Hence my +2 vote.
But your common sense may vary, so I'd like to hear from others on the matter.
[1]: https://review.openstack.org/#/c/113129/
Some more context: Vishal has a patch on master, currently in review: https://review.openstack.org/#/c/114185/
Which fixes the issue for the L3 agent, but not for the DHCP agent. Personally I think that the backport should be permitted, and the agents should be fixed in the Juno and Kilo cycles.
Thanks.
So the feature in question is actually considered as valid, even though exotic.
How about enforcing the rule on per-agent base instead of globally? We know that our agents are broken, though we're not sure whether there are other agents in production that are not. So why not just disable the 'feature' for our agents but still allow other third-party agents to run with the current default value, in case they rely on that?
/Ihar -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
iQEcBAEBCgAGBQJT8xFLAAoJEC5aWaUY1u57lnkIAOQUVXFeGnmLzChcKBR/Bw2D ORDALIBAXOlXbX1UoefJQsW4shZa//fdjXsX1NSaa+p46/OqLVlsuXw2iB/cs4X7 0bdDGL8K8a0fHSVI4Z7ZWnL6DrTm48vZmKv2BwGO+OPgW1TWoN5sw8gV33WevZ8t rQ7JLDxYMXNdB0BCGQmfbUIluhUznVdCLNub26Tw4sa1dtNOrRudOoEy9ylQk9hM w/8QodX4bbpnw+iLw7ox1h7J02XsjkVEpnokF1r4WketxQULE+xNukcekosNy5XJ /seJPjflF09uoxHqScNZC8kVAfGLsNTaaSivX7N+2rGrSBXunZBg0RXIGaTWefA= =9Ygu -----END PGP SIGNATURE-----
_______________________________________________ Openstack-stable-maint mailing list Openstack-stable-maint@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-stable-maint
Gary Kotton wrote:
I think that only in exceptional cases should we allow changing of default configuration variables. This may break existing setups. I am not in favor of this back port.
I tend to agree with Gary here. IIUC this is an old bug -- if people encountered it they probably have switched that configuration option to True a long time ago. It's also very easy for downstream consumers to carry the difference if they care (they ship customized config files anyway). Contrast that with breaking existing setups that may rely on that feature... We trade a known evil for a new, unknown one. We also don't mark a config option deprecated in the middle of a stable branch. It's either deprecated at release time, or at the next release time. We can't retroactively deprecate. Some aspects of that patch may still be acceptable though (neutron/db/db_base_plugin_v2.py) and we could document that we recommend people turn that option to True in the next point release releasenotes. -- Thierry Carrez (ttx)
I see another point from the comment from the patch author. According to the comment, To be clear, if you're opposed to this backport, you're in favor of keeping a DoS in any Havana or Icehouse based cloud. Create a free user on any public cloud, and create a script that creates an infinite number of subnets with a gateway not in the subnet. The DHCP agent will then go into an infinite resync cycle, DoSing all other tenants.
From this point of view, this backport sounds completely reasonable to me. Although I agree that the fix itself is reasonable, but I think this DoS scenario is a kind of security topics and we should have a better *process* for such topics. The associated bug 1304181 does not mention such DoS possibility and we should more clear bug description.
I saw a bug report from a user who uses the gateway IP address outside of CIDR range *only once* (I requested a usecase of this option but I never got a reply). The proposed default config value fits almost all use cases and considering the above scenario it sounds reasonable to backport. Thanks, Akihiro On Tue, Aug 19, 2014 at 6:35 PM, Thierry Carrez <thierry@openstack.org> wrote:
Gary Kotton wrote:
I think that only in exceptional cases should we allow changing of default configuration variables. This may break existing setups. I am not in favor of this back port.
I tend to agree with Gary here.
IIUC this is an old bug -- if people encountered it they probably have switched that configuration option to True a long time ago. It's also very easy for downstream consumers to carry the difference if they care (they ship customized config files anyway).
Contrast that with breaking existing setups that may rely on that feature... We trade a known evil for a new, unknown one.
We also don't mark a config option deprecated in the middle of a stable branch. It's either deprecated at release time, or at the next release time. We can't retroactively deprecate.
Some aspects of that patch may still be acceptable though (neutron/db/db_base_plugin_v2.py) and we could document that we recommend people turn that option to True in the next point release releasenotes.
-- Thierry Carrez (ttx)
_______________________________________________ Openstack-stable-maint mailing list Openstack-stable-maint@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-stable-maint
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 19/08/14 11:35, Thierry Carrez wrote:
Gary Kotton wrote:
I think that only in exceptional cases should we allow changing of default configuration variables. This may break existing setups. I am not in favor of this back port.
I tend to agree with Gary here.
IIUC this is an old bug -- if people encountered it they probably have switched that configuration option to True a long time ago. It's also very easy for downstream consumers to carry the difference if they care (they ship customized config files anyway).
And if they haven't encountered the issue yet, and don't know that default value is failing hard, then we leave our users with DoS unfixed, waiting for their users to break the cloud and then debug the issue, finally discovering that we have defaults that are broken and not even documented as such anywhere.
Contrast that with breaking existing setups that may rely on that feature... We trade a known evil for a new, unknown one.
Those setups are beyond our control, we don't even know whether they actually exist. So we trade a known evil for a tiny chance of a new, less evil one (those limitations will be caught by consumers in their testbed, with clear message in the log; and if it's really needed, it's a matter of one line changed in conffile).
We also don't mark a config option deprecated in the middle of a stable branch. It's either deprecated at release time, or at the next release time. We can't retroactively deprecate.
We don't deprecate it in Havana. The patch proposes to change the default value only. If you're concerned about specific description of the setting, we may trim it not to mention the part about its deprecation in later releases.
Some aspects of that patch may still be acceptable though (neutron/db/db_base_plugin_v2.py) and we could document that we recommend people turn that option to True in the next point release releasenotes.
If we don't merge the patch, it's the least we can do for our users. Distributions may also set it in their distro-specific config file (neutron-dist.conf). /Ihar -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iQEcBAEBCgAGBQJT8zmJAAoJEC5aWaUY1u57/lgIAKGJNeZZhNm7NuevmUchHdaZ cf0Tng0Ocfn7J3ZOttZSB9Xw5BSVBNN3nlMEKQQ0/nbLEHnkntt080ctMWjBsDX2 vsMHTBm3IBPihbFyLG0ZRcVeGos5/fqB5vuqmNF7XYjjhi2aQw4kBGLkveGodzyn 3D0JHfN9ZZ9tjj9QqB4StsKN/OzKCehLPImmzSItu5BU3ixlxBBPNio9m8CwuTvl n08OoL3rHWBFkCgzPdY9XGTYMR+Suw3Csm5zfa4Bkx+0RVjt8fYCOpL8QOhHjX3T 2SryXcsmfIvlot6vLOInl7mEINfedC9Yxb48TkVmvAndDhqhWHlnQtIUuEwmo2g= =rX2+ -----END PGP SIGNATURE-----
On 8/19/14, 2:48 PM, "Ihar Hrachyshka" <ihrachys@redhat.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
On 19/08/14 11:35, Thierry Carrez wrote:
Gary Kotton wrote:
I think that only in exceptional cases should we allow changing of default configuration variables. This may break existing setups. I am not in favor of this back port.
I tend to agree with Gary here.
IIUC this is an old bug -- if people encountered it they probably have switched that configuration option to True a long time ago. It's also very easy for downstream consumers to carry the difference if they care (they ship customized config files anyway).
And if they haven't encountered the issue yet, and don't know that default value is failing hard, then we leave our users with DoS unfixed, waiting for their users to break the cloud and then debug the issue, finally discovering that we have defaults that are broken and not even documented as such anywhere.
Where is a DOS attack here? Is this a few extra RPC messages being sent? The bug that this fixes also does not even have the Œimportance¹ assigned. I was under the impression that we only back port Critical and High bugs. I understand that there is an issue, which we all agree can be solved by the user changing a configuration setting. Can anyone point to a customer defining a gateway not on the subnet? I think that is the anomaly here.
Contrast that with breaking existing setups that may rely on that feature... We trade a known evil for a new, unknown one.
Those setups are beyond our control, we don't even know whether they actually exist. So we trade a known evil for a tiny chance of a new, less evil one (those limitations will be caught by consumers in their testbed, with clear message in the log; and if it's really needed, it's a matter of one line changed in conffile).
We also don't mark a config option deprecated in the middle of a stable branch. It's either deprecated at release time, or at the next release time. We can't retroactively deprecate.
We don't deprecate it in Havana. The patch proposes to change the default value only. If you're concerned about specific description of the setting, we may trim it not to mention the part about its deprecation in later releases.
Some aspects of that patch may still be acceptable though (neutron/db/db_base_plugin_v2.py) and we could document that we recommend people turn that option to True in the next point release releasenotes.
If we don't merge the patch, it's the least we can do for our users. Distributions may also set it in their distro-specific config file (neutron-dist.conf).
/Ihar -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
iQEcBAEBCgAGBQJT8zmJAAoJEC5aWaUY1u57/lgIAKGJNeZZhNm7NuevmUchHdaZ cf0Tng0Ocfn7J3ZOttZSB9Xw5BSVBNN3nlMEKQQ0/nbLEHnkntt080ctMWjBsDX2 vsMHTBm3IBPihbFyLG0ZRcVeGos5/fqB5vuqmNF7XYjjhi2aQw4kBGLkveGodzyn 3D0JHfN9ZZ9tjj9QqB4StsKN/OzKCehLPImmzSItu5BU3ixlxBBPNio9m8CwuTvl n08OoL3rHWBFkCgzPdY9XGTYMR+Suw3Csm5zfa4Bkx+0RVjt8fYCOpL8QOhHjX3T 2SryXcsmfIvlot6vLOInl7mEINfedC9Yxb48TkVmvAndDhqhWHlnQtIUuEwmo2g= =rX2+ -----END PGP SIGNATURE-----
_______________________________________________ Openstack-stable-maint mailing list Openstack-stable-maint@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-stable-maint
Gary Kotton wrote:
On 8/19/14, 2:48 PM, "Ihar Hrachyshka" <ihrachys@redhat.com> wrote:
And if they haven't encountered the issue yet, and don't know that default value is failing hard, then we leave our users with DoS unfixed, waiting for their users to break the cloud and then debug the issue, finally discovering that we have defaults that are broken and not even documented as such anywhere.
Where is a DOS attack here? Is this a few extra RPC messages being sent?
If this is a security issue, different rules apply. the first of which is that the Vulnerability Management Team should handle that bug, assess the vulnerability, coordinate the backports and ask for relevant exceptions. You can't just sneak security fixes in without proper announcements (and then use the "security" card to justify exceptions). I added the security flag to that bug so that it gets assessed and handled through the regular channels. -- Thierry Carrez (ttx)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 19/08/14 16:17, Thierry Carrez wrote:
Gary Kotton wrote:
On 8/19/14, 2:48 PM, "Ihar Hrachyshka" <ihrachys@redhat.com> wrote:
And if they haven't encountered the issue yet, and don't know that default value is failing hard, then we leave our users with DoS unfixed, waiting for their users to break the cloud and then debug the issue, finally discovering that we have defaults that are broken and not even documented as such anywhere.
Where is a DOS attack here? Is this a few extra RPC messages being sent?
If this is a security issue, different rules apply. the first of which is that the Vulnerability Management Team should handle that bug, assess the vulnerability, coordinate the backports and ask for relevant exceptions.
You can't just sneak security fixes in without proper announcements (and then use the "security" card to justify exceptions).
I added the security flag to that bug so that it gets assessed and handled through the regular channels.
Fair enough, and thanks! I'm new to the whole process, so I may fail to follow proper procedures sometimes... :) /Ihar -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iQEcBAEBCgAGBQJT810SAAoJEC5aWaUY1u57CS0H/08F+vsjKz85GLRMfLXHFkyp YMyVZr/jgn8g+17PQtT1hHeQYwjIHO9WJyLOD0diui6p+83PaGmvuMUcMsO8bXTZ TKPcOdfDbMmP9+Amm973GtnOdVviVaLUqx1+xGE6Ze/pBHGB50jqWyDjGyOe7lNO B1oTGOWx+Zoyo15189xX0nSpQEvWMVpqGhxvh38gTrwYqJXy1SbNkXeU/CdGZlzB u2DLj+fr7QIggm8CGsZnrIVKmOzdeO17W2oKcsMcQ4QiZh0DCwV7sLmnwTPJeT4Z +G0BnoIJPXlOfZ2j+ce/ttZ0CPzjB37Mg3grYPNQafcIme0ndJC/THiMegVOFWQ= =LcJW -----END PGP SIGNATURE-----
Not fixing this looks to me like we're leaving a form of service denial open to the tenants. Best regards, Miguel Ángel. ----- Original Message -----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
Hi all,
we have a Neutron backport proposed in stable/havana that is in grey area in terms of applicability for stable branches. Located at [1].
The controversy is that it changes the default value of a configuration setting, which is generally considered as a no-go for backports. That said, it has additional aspects that may trigger the exception.
Basically, the default value of the setting breaks all L3 agents we have in our tree. The failure state is an unstoppable agent resync due to incorrect gateway outside of subnet. To avoid the failure for any L3 agent that we ship, users need to change the value of that setting from default one.
It's also not clear what's the value of having a gateway outside the current subnet. As of Juno, this 'feature' is obsolete and marked to be removed in Kilo.
There is a change though that there are some third-party agents in the wild that support that 'feature'; in that case, changing the default value will affect them. Though we don't know whether they really exist. All in all, the 'feature' won't be available for them since Kilo, and there is no sign that its removal is controversial, so probably no one is interested in it anyway.
So we have a dilemma: we may break the official stable rule of 'not introducing inconmpatible configuration changes' and in that way fix major failure scenario in Neutron; or we may leave everything as-is (=our own agents broken) to stick to the rule.
My common sense tells me that we should fix our agents first, and then think about potential third-party L3 agents that possibly rely on that 'feature' being enabled by default. Hence my +2 vote.
But your common sense may vary, so I'd like to hear from others on the matter.
[1]: https://review.openstack.org/#/c/113129/
/Ihar -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
iQEcBAEBCgAGBQJT8wzoAAoJEC5aWaUY1u578HAH/0DB3+IpYWjKbwo9th3Phaf/ EvQEON2tMA177ufdgzO3KsWdx8VJxNzFfHU6Jl8aY9GQ3NYsg+5QYOJPcX0hHg4t g3yfrcUybDZ9roqzHZNYOUddm10Te7a+Q4Ykj2X2QiqEZC/VpxudRYDGU5O5cDpg sNeFltCp/vb/oGfX/E5dxdF5hzGd1sJQvDs+0jXfLghaCmNITvLFELN56JsUFxvr ZuzNxCnyvL7iJSHA+vWiG36jZCF/dR4eYly2CpRaQJ2Fp6ObVvkWCv+U+wtv+5bm TYl8PHpDmmGOlJPaBQn7OXAGc5eQZnkBwXGDxhRl6Fi9JF2fRSV0blSZl4rwpEA= =lxWH -----END PGP SIGNATURE-----
_______________________________________________ Openstack-stable-maint mailing list Openstack-stable-maint@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-stable-maint
participants (6)
-
Akihiro Motoki
-
Assaf Muller
-
Gary Kotton
-
Ihar Hrachyshka
-
Miguel Angel Ajo Pelayo
-
Thierry Carrez