[oslo][migrator] RFE Configuration mapping tool for upgrade - coordinate teams
Hey teams, ## Summary The main goal of this thread is to coordinate teams to adopt the oslo realease migrator tools and raise prior on this topics. ## History During the openstack submit in 2017 at Sydney a team from the Fujitsu company propose a talk about how to make fast forward upgrade from an openstack release to an another, and present the state of art of the configuration migrator inside openstack and the tools availables to do that. This team describe that for now, we have oslo.config generator tool to show deprecated options in newer release but they are still lacking some mapping configurations. After that the same team propose oslo.config specifications to improve migrators. Where specifications was designed by the following reviews: - https://review.openstack.org/#/c/520043/ - https://review.openstack.org/#/c/526314/ - https://review.openstack.org/#/c/526261/ The main goal of the proposed changes in these specs is to handle configuration changes over releases: - https://specs.openstack.org/openstack/oslo-specs/specs/rocky/handle-config-c... These specifications describe that there should be some helper in oslo.config for automatically adopt new changes and help users manage their configuration files easily. Scenario: Below is the proposed workflow that users can perform on old system to generate new configuration for preparing upgrading to new release: +--------------+ Old configuration +--------> | (N-1 release) | Oslo | | Config +-------> New configuration Namespaces +--------> Migrator | (N release) | | +--------------+ Running on new environment Since these specifications was validated, some patches was submitted by the Fujitsu team to implement them: https://review.openstack.org/#/q/status:open+project:openstack/oslo.config+b... Details: - handle config mapping changes ( https://review.openstack.org/526314 ) - update valid value in choice list for the opt ( https://review.openstack.org/603060 ) - update 'sample_default' instead of 'default' when migrating ( https://review.openstack.org/606211 ) - use difflib to report mismatches in migrator tests ( https://review.openstack.org/606210 ) - replace loop with dictionary lookup ( https://review.openstack.org/606209 ) - replace 'upgrade_value' with 'convert_on_upgrade' ( https://review.openstack.org/606207 ) Since these changes was proposed some peoples from the Fujitsu seems to left the company and also seems to abandon this topic. During the last oslo meeting I had added this topic to the meeting agenda to bring up this topic and to help us to move forward on it. http://eavesdrop.openstack.org/meetings/oslo/2018/oslo.2018-12-17-15.00.log.... http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000856.... ## Current state Some reviews was validated but some others need to be recovered due to the lack of response about the original authors. I've submit some patches on reviews to help us to move on it. I've made some reviews based on the last comments and based on the asked changes but now we need someone with an deeply knowledge on oslo.config to validate the works. ## Others aspects Some tools like puppet/ansible/chef etc... already in use in openstack can already handle migrators/upgrades and manage config changes. - Do we need to introduce an another tool like this one to do this job? - How many people are manually writing OpenStack configs in the first place? - Does it make sense to implement these changes? ## List of similar features on others projects - Tripleo provide a major release upgrade mechanisme ( https://docs.openstack.org/tripleo-docs/latest/upgrade/developer/upgrades/ma... ) - Openstack-Ansible also provide release upgrade mechanisme ( https://docs.openstack.org/openstack-ansible/latest/admin/upgrades/major-upg... ) - OpenStack fuel also propose specifications to upgrade configuration and release ( https://specs.openstack.org/openstack/fuel-specs/specs/7.0/upgrade-major-ope... ) This wiki can be useful to obtain the big picture of the release upgrade management in openstack ( https://wiki.openstack.org/wiki/Upgrade-with-minimal-downtime ) I surely forgot some tools and alternatives, if you see something that can be added here do not hesitate to reply to add it. ## Conclusion I bring up this topic to open a debate so do not hesitate react on this topic by responding at this thread to help us to have a better big picture to make the best choice. Thanks folks. Best regards. -- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
I'll reiterate what I said in the meeting for the broader audience: I'm unclear how much need there is for a tool of this nature. As you mentioned, deployment tools all have config management bits already that will be handling updates and upgrade. The only instance where I could see this being used is if someone were writing their configs by hand, and I'm skeptical that happens very often with a system as complex as OpenStack. When there was a team that was interested in implementing and using this I was fine with adding it, but if the people who wanted the tool are no longer around then we have no one to maintain it or assist with the adoption of some of the new features it adds. Given my concerns above, I don't want to add a large feature that the Oslo team will then have to maintain. We really don't have the capacity to be taking on lower priority work, especially if there's no one saying "yeah, I'll use that". I've left a comment on the first patch in the series[1] (and marked it WIP) to ask if there is still interest in this feature. Of course, if anyone reading this has interest please feel free to indicate that as well. It's unfortunate because I and a number of the other Oslo contributors have put a significant amount of time into the design and implementation of this tool, but if no one's going to use it then I'd rather cut our losses than continue pouring time into it. Thanks. -Ben 1: https://review.openstack.org/#/c/526314 On 12/18/18 9:07 AM, Herve Beraud wrote:
Hey teams,
## Summary
The main goal of this thread is to coordinate teams to adopt the oslo realease migrator tools and raise prior on this topics.
## History
During the openstack submit in 2017 at Sydney a team from the Fujitsu company propose a talk about how to make fast forward upgrade from an openstack release to an another, and present the state of art of the configuration migrator inside openstack and the tools availables to do that.
This team describe that for now, we have oslo.config generator tool to show deprecated options in newer release but they are still lacking some mapping configurations.
After that the same team propose oslo.config specifications to improve migrators.
Where specifications was designed by the following reviews: - https://review.openstack.org/#/c/520043/ - https://review.openstack.org/#/c/526314/ - https://review.openstack.org/#/c/526261/
The main goal of the proposed changes in these specs is to handle configuration changes over releases: - https://specs.openstack.org/openstack/oslo-specs/specs/rocky/handle-config-c...
These specifications describe that there should be some helper in oslo.config for automatically adopt new changes and help users manage their configuration files easily.
Scenario: Below is the proposed workflow that users can perform on old system to generate new configuration for preparing upgrading to new release: +--------------+ Old configuration +--------> | (N-1 release) | Oslo | | Config +-------> New configuration Namespaces +--------> Migrator | (N release) | | +--------------+ Running on new environment
Since these specifications was validated, some patches was submitted by the Fujitsu team to implement them: https://review.openstack.org/#/q/status:open+project:openstack/oslo.config+b...
Details: - handle config mapping changes ( https://review.openstack.org/526314) - update valid value in choice list for the opt ( https://review.openstack.org/603060) - update 'sample_default' instead of 'default' when migrating ( https://review.openstack.org/606211) - use difflib to report mismatches in migrator tests ( https://review.openstack.org/606210) - replace loop with dictionary lookup ( https://review.openstack.org/606209) - replace 'upgrade_value' with 'convert_on_upgrade' ( https://review.openstack.org/606207)
Since these changes was proposed some peoples from the Fujitsu seems to left the company and also seems to abandon this topic.
During the last oslo meeting I had added this topic to the meeting agenda to bring up this topic and to help us to move forward on it. http://eavesdrop.openstack.org/meetings/oslo/2018/oslo.2018-12-17-15.00.log.... http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000856....
## Current state
Some reviews was validated but some others need to be recovered due to the lack of response about the original authors.
I've submit some patches on reviews to help us to move on it.
I've made some reviews based on the last comments and based on the asked changes but now we need someone with an deeply knowledge on oslo.config to validate the works.
## Others aspects
Some tools like puppet/ansible/chef etc... already in use in openstack can already handle migrators/upgrades and manage config changes.
- Do we need to introduce an another tool like this one to do this job? - How many people are manually writing OpenStack configs in the first place? - Does it make sense to implement these changes?
## List of similar features on others projects
- Tripleo provide a major release upgrade mechanisme ( https://docs.openstack.org/tripleo-docs/latest/upgrade/developer/upgrades/ma...) - Openstack-Ansible also provide release upgrade mechanisme ( https://docs.openstack.org/openstack-ansible/latest/admin/upgrades/major-upg...) - OpenStack fuel also propose specifications to upgrade configuration and release ( https://specs.openstack.org/openstack/fuel-specs/specs/7.0/upgrade-major-ope...)
This wiki can be useful to obtain the big picture of the release upgrade management in openstack ( https://wiki.openstack.org/wiki/Upgrade-with-minimal-downtime)
I surely forgot some tools and alternatives, if you see something that can be added here do not hesitate to reply to add it.
## Conclusion
I bring up this topic to open a debate so do not hesitate react on this topic by responding at this thread to help us to have a better big picture to make the best choice.
Thanks folks.
Best regards.
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
Ben Nemec <openstack@nemebean.com> writes:
I'll reiterate what I said in the meeting for the broader audience:
I'm unclear how much need there is for a tool of this nature. As you mentioned, deployment tools all have config management bits already that will be handling updates and upgrade. The only instance where I could see this being used is if someone were writing their configs by hand, and I'm skeptical that happens very often with a system as complex as OpenStack.
When there was a team that was interested in implementing and using this I was fine with adding it, but if the people who wanted the tool are no longer around then we have no one to maintain it or assist with the adoption of some of the new features it adds. Given my concerns above, I don't want to add a large feature that the Oslo team will then have to maintain. We really don't have the capacity to be taking on lower priority work, especially if there's no one saying "yeah, I'll use that".
I've left a comment on the first patch in the series[1] (and marked it WIP) to ask if there is still interest in this feature. Of course, if anyone reading this has interest please feel free to indicate that as well.
It's unfortunate because I and a number of the other Oslo contributors have put a significant amount of time into the design and implementation of this tool, but if no one's going to use it then I'd rather cut our losses than continue pouring time into it.
+1
Thanks.
-Ben
1: https://review.openstack.org/#/c/526314
On 12/18/18 9:07 AM, Herve Beraud wrote:
Hey teams,
## Summary
The main goal of this thread is to coordinate teams to adopt the oslo realease migrator tools and raise prior on this topics.
## History
During the openstack submit in 2017 at Sydney a team from the Fujitsu company propose a talk about how to make fast forward upgrade from an openstack release to an another, and present the state of art of the configuration migrator inside openstack and the tools availables to do that.
This team describe that for now, we have oslo.config generator tool to show deprecated options in newer release but they are still lacking some mapping configurations.
After that the same team propose oslo.config specifications to improve migrators.
Where specifications was designed by the following reviews: - https://review.openstack.org/#/c/520043/ - https://review.openstack.org/#/c/526314/ - https://review.openstack.org/#/c/526261/
The main goal of the proposed changes in these specs is to handle configuration changes over releases: - https://specs.openstack.org/openstack/oslo-specs/specs/rocky/handle-config-c...
These specifications describe that there should be some helper in oslo.config for automatically adopt new changes and help users manage their configuration files easily.
Scenario: Below is the proposed workflow that users can perform on old system to generate new configuration for preparing upgrading to new release: +--------------+ Old configuration +--------> | (N-1 release) | Oslo | | Config +-------> New configuration Namespaces +--------> Migrator | (N release) | | +--------------+ Running on new environment
Since these specifications was validated, some patches was submitted by the Fujitsu team to implement them: https://review.openstack.org/#/q/status:open+project:openstack/oslo.config+b...
Details: - handle config mapping changes ( https://review.openstack.org/526314) - update valid value in choice list for the opt ( https://review.openstack.org/603060) - update 'sample_default' instead of 'default' when migrating ( https://review.openstack.org/606211) - use difflib to report mismatches in migrator tests ( https://review.openstack.org/606210) - replace loop with dictionary lookup ( https://review.openstack.org/606209) - replace 'upgrade_value' with 'convert_on_upgrade' ( https://review.openstack.org/606207)
Since these changes was proposed some peoples from the Fujitsu seems to left the company and also seems to abandon this topic.
During the last oslo meeting I had added this topic to the meeting agenda to bring up this topic and to help us to move forward on it. http://eavesdrop.openstack.org/meetings/oslo/2018/oslo.2018-12-17-15.00.log.... http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000856....
## Current state
Some reviews was validated but some others need to be recovered due to the lack of response about the original authors.
I've submit some patches on reviews to help us to move on it.
I've made some reviews based on the last comments and based on the asked changes but now we need someone with an deeply knowledge on oslo.config to validate the works.
## Others aspects
Some tools like puppet/ansible/chef etc... already in use in openstack can already handle migrators/upgrades and manage config changes.
- Do we need to introduce an another tool like this one to do this job? - How many people are manually writing OpenStack configs in the first place? - Does it make sense to implement these changes?
## List of similar features on others projects
- Tripleo provide a major release upgrade mechanisme ( https://docs.openstack.org/tripleo-docs/latest/upgrade/developer/upgrades/ma...) - Openstack-Ansible also provide release upgrade mechanisme ( https://docs.openstack.org/openstack-ansible/latest/admin/upgrades/major-upg...) - OpenStack fuel also propose specifications to upgrade configuration and release ( https://specs.openstack.org/openstack/fuel-specs/specs/7.0/upgrade-major-ope...)
This wiki can be useful to obtain the big picture of the release upgrade management in openstack ( https://wiki.openstack.org/wiki/Upgrade-with-minimal-downtime)
I surely forgot some tools and alternatives, if you see something that can be added here do not hesitate to reply to add it.
## Conclusion
I bring up this topic to open a debate so do not hesitate react on this topic by responding at this thread to help us to have a better big picture to make the best choice.
Thanks folks.
Best regards.
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
-- Doug
Hi Ben, I am apology that in last month we do not have much time maintaining the code.
but if no one's going to use it then I'd rather cut our losses than continue pouring time into it.
I agree, we will wait for the community to decide the need for the feature. In the near future, we do not have ability to maintain the code. If anyone has interest to continue maintaining the patch, we will support with document, reviewing... in our possibility. Thanks, Phuong. -----Original Message----- From: Ben Nemec [mailto:openstack@nemebean.com] Sent: Thursday, December 20, 2018 12:08 AM To: Herve Beraud; openstack-discuss@lists.openstack.org Subject: Re: [oslo][migrator] RFE Configuration mapping tool for upgrade - coordinate teams I'll reiterate what I said in the meeting for the broader audience: I'm unclear how much need there is for a tool of this nature. As you mentioned, deployment tools all have config management bits already that will be handling updates and upgrade. The only instance where I could see this being used is if someone were writing their configs by hand, and I'm skeptical that happens very often with a system as complex as OpenStack. When there was a team that was interested in implementing and using this I was fine with adding it, but if the people who wanted the tool are no longer around then we have no one to maintain it or assist with the adoption of some of the new features it adds. Given my concerns above, I don't want to add a large feature that the Oslo team will then have to maintain. We really don't have the capacity to be taking on lower priority work, especially if there's no one saying "yeah, I'll use that". I've left a comment on the first patch in the series[1] (and marked it WIP) to ask if there is still interest in this feature. Of course, if anyone reading this has interest please feel free to indicate that as well. It's unfortunate because I and a number of the other Oslo contributors have put a significant amount of time into the design and implementation of this tool, but if no one's going to use it then I'd rather cut our losses than continue pouring time into it. Thanks. -Ben 1: https://review.openstack.org/#/c/526314 On 12/18/18 9:07 AM, Herve Beraud wrote:
Hey teams,
## Summary
The main goal of this thread is to coordinate teams to adopt the oslo realease migrator tools and raise prior on this topics.
## History
During the openstack submit in 2017 at Sydney a team from the Fujitsu company propose a talk about how to make fast forward upgrade from an openstack release to an another, and present the state of art of the configuration migrator inside openstack and the tools availables to do that.
This team describe that for now, we have oslo.config generator tool to show deprecated options in newer release but they are still lacking some mapping configurations.
After that the same team propose oslo.config specifications to improve migrators.
Where specifications was designed by the following reviews: - https://review.openstack.org/#/c/520043/ - https://review.openstack.org/#/c/526314/ - https://review.openstack.org/#/c/526261/
The main goal of the proposed changes in these specs is to handle configuration changes over releases: - https://specs.openstack.org/openstack/oslo-specs/specs/rocky/handle-config-c...
These specifications describe that there should be some helper in oslo.config for automatically adopt new changes and help users manage their configuration files easily.
Scenario: Below is the proposed workflow that users can perform on old system to generate new configuration for preparing upgrading to new release: +--------------+ Old configuration +--------> | (N-1 release) | Oslo | | Config +-------> New configuration Namespaces +--------> Migrator | (N release) | | +--------------+ Running on new environment
Since these specifications was validated, some patches was submitted by the Fujitsu team to implement them: https://review.openstack.org/#/q/status:open+project:openstack/oslo.config+b...
Details: - handle config mapping changes ( https://review.openstack.org/526314) - update valid value in choice list for the opt ( https://review.openstack.org/603060) - update 'sample_default' instead of 'default' when migrating ( https://review.openstack.org/606211) - use difflib to report mismatches in migrator tests ( https://review.openstack.org/606210) - replace loop with dictionary lookup ( https://review.openstack.org/606209) - replace 'upgrade_value' with 'convert_on_upgrade' ( https://review.openstack.org/606207)
Since these changes was proposed some peoples from the Fujitsu seems to left the company and also seems to abandon this topic.
During the last oslo meeting I had added this topic to the meeting agenda to bring up this topic and to help us to move forward on it. http://eavesdrop.openstack.org/meetings/oslo/2018/oslo.2018-12-17-15.00.log.... http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000856....
## Current state
Some reviews was validated but some others need to be recovered due to the lack of response about the original authors.
I've submit some patches on reviews to help us to move on it.
I've made some reviews based on the last comments and based on the asked changes but now we need someone with an deeply knowledge on oslo.config to validate the works.
## Others aspects
Some tools like puppet/ansible/chef etc... already in use in openstack can already handle migrators/upgrades and manage config changes.
- Do we need to introduce an another tool like this one to do this job? - How many people are manually writing OpenStack configs in the first place? - Does it make sense to implement these changes?
## List of similar features on others projects
- Tripleo provide a major release upgrade mechanisme ( https://docs.openstack.org/tripleo-docs/latest/upgrade/developer/upgrades/ma...) - Openstack-Ansible also provide release upgrade mechanisme ( https://docs.openstack.org/openstack-ansible/latest/admin/upgrades/major-upg...) - OpenStack fuel also propose specifications to upgrade configuration and release ( https://specs.openstack.org/openstack/fuel-specs/specs/7.0/upgrade-major-ope...)
This wiki can be useful to obtain the big picture of the release upgrade management in openstack ( https://wiki.openstack.org/wiki/Upgrade-with-minimal-downtime)
I surely forgot some tools and alternatives, if you see something that can be added here do not hesitate to reply to add it.
## Conclusion
I bring up this topic to open a debate so do not hesitate react on this topic by responding at this thread to help us to have a better big picture to make the best choice.
Thanks folks.
Best regards.
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
Le jeu. 20 déc. 2018 à 09:26, Nguyen Hung, Phuong <phuongnh@vn.fujitsu.com> a écrit :
Hi Ben,
I am apology that in last month we do not have much time maintaining the code.
but if no one's going to use it then I'd rather cut our losses than continue pouring time into it.
I agree, we will wait for the community to decide the need for the feature. In the near future, we do not have ability to maintain the code. If anyone has interest to continue maintaining the patch, we will support with document, reviewing... in our possibility.
I can help you to maintain the code if needed. Personaly I doesn't need this feature so I agree Ben and Doug point of view. We need to measure how many this feature is useful and if it make sense to support and maintain more code in the future related to this feature without any usages behind that.
Thanks, Phuong.
-----Original Message----- From: Ben Nemec [mailto:openstack@nemebean.com] Sent: Thursday, December 20, 2018 12:08 AM To: Herve Beraud; openstack-discuss@lists.openstack.org Subject: Re: [oslo][migrator] RFE Configuration mapping tool for upgrade - coordinate teams
I'll reiterate what I said in the meeting for the broader audience:
I'm unclear how much need there is for a tool of this nature. As you mentioned, deployment tools all have config management bits already that will be handling updates and upgrade. The only instance where I could see this being used is if someone were writing their configs by hand, and I'm skeptical that happens very often with a system as complex as OpenStack.
When there was a team that was interested in implementing and using this I was fine with adding it, but if the people who wanted the tool are no longer around then we have no one to maintain it or assist with the adoption of some of the new features it adds. Given my concerns above, I don't want to add a large feature that the Oslo team will then have to maintain. We really don't have the capacity to be taking on lower priority work, especially if there's no one saying "yeah, I'll use that".
I've left a comment on the first patch in the series[1] (and marked it WIP) to ask if there is still interest in this feature. Of course, if anyone reading this has interest please feel free to indicate that as well.
It's unfortunate because I and a number of the other Oslo contributors have put a significant amount of time into the design and implementation of this tool, but if no one's going to use it then I'd rather cut our losses than continue pouring time into it.
Thanks.
-Ben
1: https://review.openstack.org/#/c/526314
On 12/18/18 9:07 AM, Herve Beraud wrote:
Hey teams,
## Summary
The main goal of this thread is to coordinate teams to adopt the oslo realease migrator tools and raise prior on this topics.
## History
During the openstack submit in 2017 at Sydney a team from the Fujitsu company propose a talk about how to make fast forward upgrade from an openstack release to an another, and present the state of art of the configuration migrator inside openstack and the tools availables to do that.
This team describe that for now, we have oslo.config generator tool to show deprecated options in newer release but they are still lacking some mapping configurations.
After that the same team propose oslo.config specifications to improve migrators.
Where specifications was designed by the following reviews: - https://review.openstack.org/#/c/520043/ - https://review.openstack.org/#/c/526314/ - https://review.openstack.org/#/c/526261/
The main goal of the proposed changes in these specs is to handle configuration changes over releases: -
https://specs.openstack.org/openstack/oslo-specs/specs/rocky/handle-config-c...
These specifications describe that there should be some helper in oslo.config for automatically adopt new changes and help users manage their configuration files easily.
Scenario: Below is the proposed workflow that users can perform on old system to generate new configuration for preparing upgrading to new release: +--------------+ Old configuration +--------> | (N-1 release) | Oslo | | Config +-------> New configuration Namespaces +--------> Migrator | (N release) | | +--------------+ Running on new environment
Since these specifications was validated, some patches was submitted by the Fujitsu team to implement them:
https://review.openstack.org/#/q/status:open+project:openstack/oslo.config+b...
Details: - handle config mapping changes (
https://review.openstack.org/526314)
- update valid value in choice list for the opt ( https://review.openstack.org/603060) - update 'sample_default' instead of 'default' when migrating ( https://review.openstack.org/606211) - use difflib to report mismatches in migrator tests ( https://review.openstack.org/606210) - replace loop with dictionary lookup ( https://review.openstack.org/606209) - replace 'upgrade_value' with 'convert_on_upgrade' ( https://review.openstack.org/606207)
Since these changes was proposed some peoples from the Fujitsu seems to left the company and also seems to abandon this topic.
During the last oslo meeting I had added this topic to the meeting agenda to bring up this topic and to help us to move forward on it.
http://eavesdrop.openstack.org/meetings/oslo/2018/oslo.2018-12-17-15.00.log....
http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000856....
## Current state
Some reviews was validated but some others need to be recovered due to
lack of response about the original authors.
I've submit some patches on reviews to help us to move on it.
I've made some reviews based on the last comments and based on the asked changes but now we need someone with an deeply knowledge on oslo.config to validate the works.
## Others aspects
Some tools like puppet/ansible/chef etc... already in use in openstack can already handle migrators/upgrades and manage config changes.
- Do we need to introduce an another tool like this one to do this job? - How many people are manually writing OpenStack configs in the first
the place?
- Does it make sense to implement these changes?
## List of similar features on others projects
- Tripleo provide a major release upgrade mechanisme (
https://docs.openstack.org/tripleo-docs/latest/upgrade/developer/upgrades/ma... )
- Openstack-Ansible also provide release upgrade mechanisme (
https://docs.openstack.org/openstack-ansible/latest/admin/upgrades/major-upg... )
- OpenStack fuel also propose specifications to upgrade configuration and release (
https://specs.openstack.org/openstack/fuel-specs/specs/7.0/upgrade-major-ope... )
This wiki can be useful to obtain the big picture of the release upgrade management in openstack ( https://wiki.openstack.org/wiki/Upgrade-with-minimal-downtime)
I surely forgot some tools and alternatives, if you see something that can be added here do not hesitate to reply to add it.
## Conclusion
I bring up this topic to open a debate so do not hesitate react on this topic by responding at this thread to help us to have a better big picture to make the best choice.
Thanks folks.
Best regards.
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
On 12/20/18 4:41 AM, Herve Beraud wrote:
Le jeu. 20 déc. 2018 à 09:26, Nguyen Hung, Phuong <phuongnh@vn.fujitsu.com <mailto:phuongnh@vn.fujitsu.com>> a écrit :
Hi Ben,
I am apology that in last month we do not have much time maintaining the code.
> but if no one's going to use it then I'd rather cut our > losses than continue pouring time into it.
I agree, we will wait for the community to decide the need for the feature. In the near future, we do not have ability to maintain the code. If anyone has interest to continue maintaining the patch, we will support with document, reviewing... in our possibility.
I can help you to maintain the code if needed.
Personaly I doesn't need this feature so I agree Ben and Doug point of view.
We need to measure how many this feature is useful and if it make sense to support and maintain more code in the future related to this feature without any usages behind that.
We discussed this again in the Oslo meeting this week, and to share with the wider audience here's what I propose: Since the team that initially proposed the feature and that we expected to help maintain it are no longer able to do so, and it's not clear to the Oslo team that there is sufficient demand for a rather complex feature like this, I suggest that we either WIP or abandon the current patch series. Gerrit never forgets, so if at some point there are contributors (new or old) who have a vested interest in the feature we can always resurrect it. If you have any thoughts about this plan please let me know. Otherwise I will act on it sometime in the near-ish future. In the meantime, if anyone is desperate for Oslo work to do here are a few things that have been lingering on my todo list: * We have a unit test in oslo.utils (test_excutils) that is still using mox. That needs to be migrated to mock. * oslo.cookiecutter has a number of things that are out of date (doc layout, lack of reno, coverage job). Since it's unlikely we've reached peak Oslo library we should update that so there aren't a bunch of post-creation changes needed like there were with oslo.upgradecheck (and I'm guessing oslo.limit). * The config validator still needs support for dynamic groups, if oslo.config is your thing. * There are 326 bugs open across Oslo projects. Help wanted. :-) Thanks. -Ben
Hi Ben,
I suggest that we either WIP or abandon the current patch series. ... If you have any thoughts about this plan please let me know. Otherwise I will act on it sometime in the near-ish future.
Thanks for your consideration. I am agree with you, please help me to abandon them because I am not privileged with those patches. Regards, Phuong. -----Original Message----- From: Ben Nemec [mailto:openstack@nemebean.com] Sent: Thursday, January 10, 2019 6:12 AM To: Herve Beraud; Nguyen, Hung Phuong Cc: openstack-discuss@lists.openstack.org Subject: Re: [oslo][migrator] RFE Configuration mapping tool for upgrade - coordinate teams On 12/20/18 4:41 AM, Herve Beraud wrote:
Le jeu. 20 déc. 2018 à 09:26, Nguyen Hung, Phuong <phuongnh@vn.fujitsu.com <mailto:phuongnh@vn.fujitsu.com>> a écrit :
Hi Ben,
I am apology that in last month we do not have much time maintaining the code.
> but if no one's going to use it then I'd rather cut our > losses than continue pouring time into it.
I agree, we will wait for the community to decide the need for the feature. In the near future, we do not have ability to maintain the code. If anyone has interest to continue maintaining the patch, we will support with document, reviewing... in our possibility.
I can help you to maintain the code if needed.
Personaly I doesn't need this feature so I agree Ben and Doug point of view.
We need to measure how many this feature is useful and if it make sense to support and maintain more code in the future related to this feature without any usages behind that.
We discussed this again in the Oslo meeting this week, and to share with the wider audience here's what I propose: Since the team that initially proposed the feature and that we expected to help maintain it are no longer able to do so, and it's not clear to the Oslo team that there is sufficient demand for a rather complex feature like this, I suggest that we either WIP or abandon the current patch series. Gerrit never forgets, so if at some point there are contributors (new or old) who have a vested interest in the feature we can always resurrect it. If you have any thoughts about this plan please let me know. Otherwise I will act on it sometime in the near-ish future. In the meantime, if anyone is desperate for Oslo work to do here are a few things that have been lingering on my todo list: * We have a unit test in oslo.utils (test_excutils) that is still using mox. That needs to be migrated to mock. * oslo.cookiecutter has a number of things that are out of date (doc layout, lack of reno, coverage job). Since it's unlikely we've reached peak Oslo library we should update that so there aren't a bunch of post-creation changes needed like there were with oslo.upgradecheck (and I'm guessing oslo.limit). * The config validator still needs support for dynamic groups, if oslo.config is your thing. * There are 326 bugs open across Oslo projects. Help wanted. :-) Thanks. -Ben
"Nguyen Hung, Phuong" <phuongnh@vn.fujitsu.com> writes:
Hi Ben,
I suggest that we either WIP or abandon the current patch series. ... If you have any thoughts about this plan please let me know. Otherwise I will act on it sometime in the near-ish future.
Thanks for your consideration. I am agree with you, please help me to abandon them because I am not privileged with those patches.
Regards, Phuong.
+1 for abandoning them, at least for now. As Ben points out, gerrit will still have copies. Doug
-----Original Message----- From: Ben Nemec [mailto:openstack@nemebean.com] Sent: Thursday, January 10, 2019 6:12 AM To: Herve Beraud; Nguyen, Hung Phuong Cc: openstack-discuss@lists.openstack.org Subject: Re: [oslo][migrator] RFE Configuration mapping tool for upgrade - coordinate teams
On 12/20/18 4:41 AM, Herve Beraud wrote:
Le jeu. 20 déc. 2018 à 09:26, Nguyen Hung, Phuong <phuongnh@vn.fujitsu.com <mailto:phuongnh@vn.fujitsu.com>> a écrit :
Hi Ben,
I am apology that in last month we do not have much time maintaining the code.
> but if no one's going to use it then I'd rather cut our > losses than continue pouring time into it.
I agree, we will wait for the community to decide the need for the feature. In the near future, we do not have ability to maintain the code. If anyone has interest to continue maintaining the patch, we will support with document, reviewing... in our possibility.
I can help you to maintain the code if needed.
Personaly I doesn't need this feature so I agree Ben and Doug point of view.
We need to measure how many this feature is useful and if it make sense to support and maintain more code in the future related to this feature without any usages behind that.
We discussed this again in the Oslo meeting this week, and to share with the wider audience here's what I propose:
Since the team that initially proposed the feature and that we expected to help maintain it are no longer able to do so, and it's not clear to the Oslo team that there is sufficient demand for a rather complex feature like this, I suggest that we either WIP or abandon the current patch series. Gerrit never forgets, so if at some point there are contributors (new or old) who have a vested interest in the feature we can always resurrect it.
If you have any thoughts about this plan please let me know. Otherwise I will act on it sometime in the near-ish future.
In the meantime, if anyone is desperate for Oslo work to do here are a few things that have been lingering on my todo list:
* We have a unit test in oslo.utils (test_excutils) that is still using mox. That needs to be migrated to mock. * oslo.cookiecutter has a number of things that are out of date (doc layout, lack of reno, coverage job). Since it's unlikely we've reached peak Oslo library we should update that so there aren't a bunch of post-creation changes needed like there were with oslo.upgradecheck (and I'm guessing oslo.limit). * The config validator still needs support for dynamic groups, if oslo.config is your thing. * There are 326 bugs open across Oslo projects. Help wanted. :-)
Thanks.
-Ben
-- Doug
Make sense so +1 Le jeu. 10 janv. 2019 14:27, Doug Hellmann <doug@doughellmann.com> a écrit :
"Nguyen Hung, Phuong" <phuongnh@vn.fujitsu.com> writes:
Hi Ben,
I suggest that we either WIP or abandon the current patch series. ... If you have any thoughts about this plan please let me know. Otherwise I will act on it sometime in the near-ish future.
Thanks for your consideration. I am agree with you, please help me to abandon them because I am not privileged with those patches.
Regards, Phuong.
+1 for abandoning them, at least for now. As Ben points out, gerrit will still have copies.
Doug
-----Original Message----- From: Ben Nemec [mailto:openstack@nemebean.com] Sent: Thursday, January 10, 2019 6:12 AM To: Herve Beraud; Nguyen, Hung Phuong Cc: openstack-discuss@lists.openstack.org Subject: Re: [oslo][migrator] RFE Configuration mapping tool for upgrade
- coordinate teams
On 12/20/18 4:41 AM, Herve Beraud wrote:
Le jeu. 20 déc. 2018 à 09:26, Nguyen Hung, Phuong <phuongnh@vn.fujitsu.com <mailto:phuongnh@vn.fujitsu.com>> a écrit :
Hi Ben,
I am apology that in last month we do not have much time maintaining the code.
> but if no one's going to use it then I'd rather cut our > losses than continue pouring time into it.
I agree, we will wait for the community to decide the need for the feature. In the near future, we do not have ability to maintain the code. If anyone has interest to continue maintaining the patch, we will support with document, reviewing... in our possibility.
I can help you to maintain the code if needed.
Personaly I doesn't need this feature so I agree Ben and Doug point of
view.
We need to measure how many this feature is useful and if it make sense to support and maintain more code in the future related to this feature without any usages behind that.
We discussed this again in the Oslo meeting this week, and to share with the wider audience here's what I propose:
Since the team that initially proposed the feature and that we expected to help maintain it are no longer able to do so, and it's not clear to the Oslo team that there is sufficient demand for a rather complex feature like this, I suggest that we either WIP or abandon the current patch series. Gerrit never forgets, so if at some point there are contributors (new or old) who have a vested interest in the feature we can always resurrect it.
If you have any thoughts about this plan please let me know. Otherwise I will act on it sometime in the near-ish future.
In the meantime, if anyone is desperate for Oslo work to do here are a few things that have been lingering on my todo list:
* We have a unit test in oslo.utils (test_excutils) that is still using mox. That needs to be migrated to mock. * oslo.cookiecutter has a number of things that are out of date (doc layout, lack of reno, coverage job). Since it's unlikely we've reached peak Oslo library we should update that so there aren't a bunch of post-creation changes needed like there were with oslo.upgradecheck (and I'm guessing oslo.limit). * The config validator still needs support for dynamic groups, if oslo.config is your thing. * There are 326 bugs open across Oslo projects. Help wanted. :-)
Thanks.
-Ben
-- Doug
Thanks for the quick feedback everyone. I've abandoned the patch series, although I did pull out one change that seemed to be a valid bugfix independent of the migrator work: https://review.openstack.org/#/c/607690/ On 1/10/19 9:15 AM, Herve Beraud wrote:
Make sense so +1
Le jeu. 10 janv. 2019 14:27, Doug Hellmann <doug@doughellmann.com <mailto:doug@doughellmann.com>> a écrit :
"Nguyen Hung, Phuong" <phuongnh@vn.fujitsu.com <mailto:phuongnh@vn.fujitsu.com>> writes:
> Hi Ben, > >> I suggest that we either WIP or abandon the current >> patch series. > ... >> If you have any thoughts about this plan please let me know. Otherwise I >> will act on it sometime in the near-ish future. > > Thanks for your consideration. I am agree with you, please help me to abandon them because I am not privileged with those patches. > > Regards, > Phuong.
+1 for abandoning them, at least for now. As Ben points out, gerrit will still have copies.
Doug
> > -----Original Message----- > From: Ben Nemec [mailto:openstack@nemebean.com <mailto:openstack@nemebean.com>] > Sent: Thursday, January 10, 2019 6:12 AM > To: Herve Beraud; Nguyen, Hung Phuong > Cc: openstack-discuss@lists.openstack.org <mailto:openstack-discuss@lists.openstack.org> > Subject: Re: [oslo][migrator] RFE Configuration mapping tool for upgrade - coordinate teams > > > > On 12/20/18 4:41 AM, Herve Beraud wrote: >> >> >> Le jeu. 20 déc. 2018 à 09:26, Nguyen Hung, Phuong >> <phuongnh@vn.fujitsu.com <mailto:phuongnh@vn.fujitsu.com> <mailto:phuongnh@vn.fujitsu.com <mailto:phuongnh@vn.fujitsu.com>>> a écrit : >> >> Hi Ben, >> >> I am apology that in last month we do not have much time maintaining >> the code. >> >> > but if no one's going to use it then I'd rather cut our >> > losses than continue pouring time into it. >> >> I agree, we will wait for the community to decide the need for the >> feature. >> In the near future, we do not have ability to maintain the code. If >> anyone >> has interest to continue maintaining the patch, we will support with >> document, >> reviewing... in our possibility. >> >> >> I can help you to maintain the code if needed. >> >> Personaly I doesn't need this feature so I agree Ben and Doug point of view. >> >> We need to measure how many this feature is useful and if it make sense >> to support and maintain more code in the future related to this feature >> without any usages behind that. > > We discussed this again in the Oslo meeting this week, and to share with > the wider audience here's what I propose: > > Since the team that initially proposed the feature and that we expected > to help maintain it are no longer able to do so, and it's not clear to > the Oslo team that there is sufficient demand for a rather complex > feature like this, I suggest that we either WIP or abandon the current > patch series. Gerrit never forgets, so if at some point there are > contributors (new or old) who have a vested interest in the feature we > can always resurrect it. > > If you have any thoughts about this plan please let me know. Otherwise I > will act on it sometime in the near-ish future. > > In the meantime, if anyone is desperate for Oslo work to do here are a > few things that have been lingering on my todo list: > > * We have a unit test in oslo.utils (test_excutils) that is still using > mox. That needs to be migrated to mock. > * oslo.cookiecutter has a number of things that are out of date (doc > layout, lack of reno, coverage job). Since it's unlikely we've reached > peak Oslo library we should update that so there aren't a bunch of > post-creation changes needed like there were with oslo.upgradecheck (and > I'm guessing oslo.limit). > * The config validator still needs support for dynamic groups, if > oslo.config is your thing. > * There are 326 bugs open across Oslo projects. Help wanted. :-) > > Thanks. > > -Ben >
-- Doug
participants (4)
-
Ben Nemec
-
Doug Hellmann
-
Herve Beraud
-
Nguyen Hung, Phuong