---- On Tue, 29 Dec 2020 09:15:43 -0600 Takashi Kajinami <tkajinam@redhat.com> wrote ----

Hello, For Puppet OpenStack projects I have submitted a series of changes to use policy.yaml instead of policy.json[1]. [1] https://review.opendev.org/q/topic:%22policy-yaml%22+(status:open%20OR%20sta...)

Thanks, Takashi for taking care of it. I have modified the review topic to 'policy-json-to-yaml' so that we track all work together. - https://review.opendev.org/q/topic:%22policy-json-to-yaml%22+(status:open%20...)

One problem I noticed during making these patches is that Gnocchi still uses policy.json.IIUC that policy-in-code is not implemented in gnocchi and the default contents shouldbe migrated to policy.yaml appropriately to keep its functionality. I have submitted a pull request to introduce policy.yaml to replace policy.json to replacepolicy.json by policy.yaml. [2] https://github.com/gnocchixyz/gnocchi/pull/1108 I know that Gnocchi is not a part of OpenStack projects but we should be careful about itbefore we make any changes in oslo.policy because Gnocchi is currently consuming the library.

I agree, also gnoochi capped the oslo.policy with 3.4.0 and we need 3.6.0 for this migration. Matthias already pushed the PR for that https://github.com/gnocchixyz/gnocchi/pull/1097 I hope both of these PRs will be merged soon. -gmann

Thank you,Takashi

On Tue, Dec 29, 2020 at 4:58 PM Dmitriy Rabotyagov <noonedeadpunk@ya.ru> wrote: Hi! Regarding OpenStack-Ansible I was planning to land patches early January. We eventually need to patch every role to change "dest" and "config_type" for placing template, ie. [1] Also we will need to think through removal of old json file for ppl that will perform upgrade, to avoid any possible conflicts and confusions because of the prescence of both files. [1] https://opendev.org/openstack/openstack-ansible-os_glance/src/branch/master/... 26.12.2020, 00:41, "Ghanshyam Mann" <gmann@ghanshyammann.com>:Hello Everyone,

Please find the week's R-16 updates on 'Migrate RBAC Policy Format from JSON to YAML' community-wide goals.

Tracking: https://etherpad.opendev.org/p/migrate-policy-format-from-json-to-yaml

Gerrit Topic: https://review.opendev.org/q/topic:%22policy-json-to-yaml%22+(status:open%20...)

Progress: ======= * Projects completed: 5 * Projects left to merge the patches: 25 * Projects left to push the patches: 2 (horizon and Openstackansible) * Projects do not need any work: 17

Updates: ======= * I have pushed the patches for all the required service projects.

** Because of many services gate is already broken for lower constraints job, these patches might not be green in the test results. I request projects to fix the gate so that we can merge this goal work before m-2.

** There are many project tests where CONF object was not fully initialized before the policy is init. This was working till now as policy init did not use the CONF object but oslo_policy 3.6.0 onwards it needs fully initialized CONF object during init only.

** Aodh work for this goal is blocked because it needs oslo_policy 3.6.0 but gnocchi is capped for oslo_policy 3.4.0 [1] - https://review.opendev.org/c/openstack/aodh/+/768499

* Horizon and Openstackansible work is pending to use/deploy the YAML formatted policy file. I will start exploring this next week or so.

[1] https://github.com/gnocchixyz/gnocchi/blob/e19fda590c7f7f07f1df0ba93177df07d...

Merry Christmas and Happy Holidays!

-gmann

-- Kind Regards,Dmitriy Rabotyagov

---- On Tue, 19 Jan 2021 11:02:44 -0600 Dmitriy Rabotyagov <noonedeadpunk@ya.ru> wrote ----

Hi! I have some follow up questions. On oslo.policy side it looks like it's better to explicitly set policy.yaml path in config and not rely if services have already moved to using yaml files. Or in case policy.json does not exist, oslo will try to load yaml instead?

This was first thought but we can not do that as this will break the existing deployment relying on policy.json. That is why we need to wait for all services to do 1. change the default value of CONF.policy_file to policy.yaml 2. officially deprecate the JSON format policy file support. And once that is done in all openstack services and the operator has moved to policy.yaml then we can change it in oslo.policy safely. Overall what we are trying to achieve is "Convey the JSON->YAML policy file migration properly to the operator and then switch the flag" so that we do not introduce any breaking change and migrate it smoothly.

Another question is more general one and very basic :( I have a feeling that policies are applied without related service reload which means that they're loaded from disk for each incoming request? Is that assumption right? I'm just thinking about the best way to do upgrade and at what point we should be dropping old policy.json and if we can do this before placing policy.yaml or not.

My plan is 1. finish the service side deprecation and default file change in Wallaby 2. give Xena cycle as a buffer for the operator to notice these changes 3. In the Y cycle, we remove the JSON format and file support completly. -gmann 29.12.2020, 20:37, "Ghanshyam Mann" <gmann@ghanshyammann.com>: ---- On Tue, 29 Dec 2020 01:56:22 -0600 Dmitriy Rabotyagov <noonedeadpunk@ya.ru> wrote ----

...

Hi! Regarding OpenStack-Ansible I was planning to land patches early January. We eventually need to patch every role to change "dest" and "config_type" for placing template, ie. [1] Also we will need to think through removal of old json file for ppl that will perform upgrade, to avoid any possible conflicts and confusions because of the prescence of both files. [1] https://opendev.org/openstack/openstack-ansible-os_glance/src/branch/master/...

Thanks, Dmitriy, do let me know if you need help this is a large number of changes. I will be able to push changes for this.

On point of the presence of both files, yes this is a good point. From the service side default value change, I am taking care of this on oslo.policy side[1]. If both files exist and deployment rely on the default value (config option is not overridden ) then oslo policy will pick up the 'policy.json'. With this, we make sure we do not break any upgrade for deployment relying on this default value. In the future, when we decide to remove the support of policy.json then we can remove this fallback logic.

-gmann

[1] https://github.com/openstack/oslo.policy/blob/0a228dea2ee96ec3eabed3361ca225...

...

26.12.2020, 00:41, "Ghanshyam Mann" <gmann@ghanshyammann.com>:Hello Everyone,

Please find the week's R-16 updates on 'Migrate RBAC Policy Format from JSON to YAML' community-wide goals.

Tracking: https://etherpad.opendev.org/p/migrate-policy-format-from-json-to-yaml

Gerrit Topic: https://review.opendev.org/q/topic:%22policy-json-to-yaml%22+(status:open%20...)

Progress: ======= * Projects completed: 5 * Projects left to merge the patches: 25 * Projects left to push the patches: 2 (horizon and Openstackansible) * Projects do not need any work: 17

Updates: ======= * I have pushed the patches for all the required service projects.

** Because of many services gate is already broken for lower constraints job, these patches might not be green in the test results. I request projects to fix the gate so that we can merge this goal work before m-2.

** There are many project tests where CONF object was not fully initialized before the policy is init. This was working till now as policy init did not use the CONF object but oslo_policy 3.6.0 onwards it needs fully initialized CONF object during init only.

** Aodh work for this goal is blocked because it needs oslo_policy 3.6.0 but gnocchi is capped for oslo_policy 3.4.0 [1] - https://review.opendev.org/c/openstack/aodh/+/768499

* Horizon and Openstackansible work is pending to use/deploy the YAML formatted policy file. I will start exploring this next week or so.

[1] https://github.com/gnocchixyz/gnocchi/blob/e19fda590c7f7f07f1df0ba93177df07d...

Merry Christmas and Happy Holidays!

-gmann

-- Kind Regards,Dmitriy Rabotyagov -- Kind Regards,Dmitriy Rabotyagov

On 1/19/21 9:04 PM, Ben Nemec wrote:

There was also a security concern with potentially having multiple policy files and it not being clear which was in use. If someone converted their JSON policy to YAML, but left the JSON one in place, it could result in oslo.policy using the wrong one (or not the one they expect). We decided it was better for each project to make a clean switchover, which allows for things like upgrade checks that oslo.policy couldn't have itself, than to try to handle it all in oslo.policy.

IMO, that's a downstream distro thing. What I did in Debian (and for Victoria already) was having the postinst of each package to rename any existing policy.json into a disabled version. Here's an example with Cinder: if [ -r /etc/cinder/policy.json ] ; then mv /etc/cinder/policy.json /etc/cinder/disabled.policy.json.old fi and then package the yaml file as (example from Nova): /etc/nova/policy.d/00_default_policy.yaml and then setting-up this: policy_dirs = /etc/nova/policy.d The reason I'm doing this way, is that I'm expecting upstream to generate a commented-only yaml file, and final users to drop non-default supplementary files without touching the package default file. So, someone upgrading to Victoria with a non-default policy.json will see its manual tweaks go away, but not completely gone (ie: recoverable from disabled.policy.json.old). Does this seem to be a correct approach? Cheers, Thomas Goirand (zigo)

---- On Tue, 19 Jan 2021 17:25:46 -0600 Thomas Goirand <zigo@debian.org> wrote ----

On 1/19/21 9:04 PM, Ben Nemec wrote:

...

There was also a security concern with potentially having multiple policy files and it not being clear which was in use. If someone converted their JSON policy to YAML, but left the JSON one in place, it could result in oslo.policy using the wrong one (or not the one they expect). We decided it was better for each project to make a clean switchover, which allows for things like upgrade checks that oslo.policy couldn't have itself, than to try to handle it all in oslo.policy.

IMO, that's a downstream distro thing.

What I did in Debian (and for Victoria already) was having the postinst of each package to rename any existing policy.json into a disabled version. Here's an example with Cinder:

if [ -r /etc/cinder/policy.json ] ; then mv /etc/cinder/policy.json /etc/cinder/disabled.policy.json.old fi

and then package the yaml file as (example from Nova): /etc/nova/policy.d/00_default_policy.yaml

and then setting-up this: policy_dirs = /etc/nova/policy.d

The reason I'm doing this way, is that I'm expecting upstream to generate a commented-only yaml file, and final users to drop non-default supplementary files without touching the package default file.

So, someone upgrading to Victoria with a non-default policy.json will see its manual tweaks go away, but not completely gone (ie: recoverable from disabled.policy.json.old).

Does this seem to be a correct approach?

We have two types of default meaning in policy things. 1. default value of config option 'policy_file' which we are changing from 'policy.json' -> 'policy.yaml' and has custom policy rule 2. default value of policy rule which can be in default policy file or non-default (means keeping all the default value of policy rule in the file ). I hope with 'default' '00_default_policy.yaml' you mean the former one. If so then instead of keeping JSON the version also best approach is just using the 'oslopolicy-convert-json-to-yaml' tool to convert it to YAML format in a backward-compatible way. Newly generated YAML file will keep all the custom rule as it is and comment-out the default rule. Let me describe the recommended way for the different scenario: Scenario1: If you do not have any custom rule in the policy file: ------------- Recommendation (Very strong): Do not use any file, just remove it and let policy-in-code serve the purpose. OpenStack does not need a policy file as mandatory and if there is no file present or passed in the config option then it works fine with policy-in-code. Having a policy file with all default rules (uncommented) can cause any issue and this was the the reason why Debian packaging broke with Nova's new policy in the Ussuri release. Scenario2: If you have a policy file with custom rules and in JSON format. ------------- Recommendation: This is the use case we need to fix and this migration is all about. Use the 'oslopolicy-convert-json-to-yaml' tool to convert it to YAML format in a backward-compatible way Or convert it manually but do not keep the old JSON format file in a configured location otherwise Oslo policy will pick that if it finds both JSON and YAML formatted file. Just remove the old JSON file as the same level of permission is present in the converted YAML file. Scenario3: If you have a policy file with custom rules and in YAML format. ------------- This is all good and no work needed for this. [1] https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to... -gmann

Cheers,

Thomas Goirand (zigo)