Re: [OpenStack-I18n] [OpenStack-Infra] [I18n] Regarding Zanata upgrade plan to 3.9.6 with Xenial: Help is needed
Hello, I would like to summarize the status for Zanata (translation platform) upgrade here from various but a little bit scattered results, and also ask some questions to infra team. - I18n team completed tests with current Zanata (3.7.3) with Xenial [1], but found one error mentioned in [2]. It seems that more memory size allocation for Zanata with Java 8 is needed. Could you upgrade the memory size for translate-dev server first to test again? - I remember that newer version of openstackid needs to be tested with translate-dev. So I have just uploaded this patch: https://review.openstack.org/#/c/419667/ . I18n team needs more tests, but I think it is a good time to change to openstackid-dev for such testing. Please ping me after openstackid-dev test with translate-dev is completed with no error :) - On last I18n team meeting, I18n team recognized that the backup would be so important. Is there more disks for such backup on translate-dev and translate.o.o server? And the approach implementing like [3] looks quite a good idea I think. Thanks, Frank! - Can I have root access to translate-dev and translate server? I really appreciate kind and continuous help for such Zanata upgrade, but sometimes I feel that considering priority like in [5] would be also a good idea. (Or somewhere on i18n-specs, which is not yet revised with items [6]). Since O cycle is rather short and also StringFreeze period is important for I18n team with translations, it seems that applying new Zanata version into production (I mean, translate.openstack.org) would be difficult. But, I really want to upgrade Zanata 3.9.6 with Xenial in translate-dev.openstack.org during O cycle and hope that all tests will be fine. Then I would like to have some discussions during PTG event including this topic :) [1] https://etherpad.openstack.org/p/i18n-zanata-test-for-translate-dev [2] http://lists.openstack.org/pipermail/openstack-i18n/2017-January/002643.html [3] https://review.openstack.org/#/c/419389/ [4] http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/s... [5] http://specs.openstack.org/openstack-infra/infra-specs/#priority-efforts [6] http://specs.openstack.org/openstack/i18n-specs/ With many thanks, /Ian Clark Boylan wrote on 12/8/2016 7:56 AM:
----- Original Nachricht ---- Von: Clark Boylan <cboylan@sapwetik.org> An: "Ian Y. Choi" <ianyrchoi@gmail.com>, OpenStack Infra <openstack-infra@lists.openstack.org> Datum: 03.12.2016 00:37 Betreff: Re: [OpenStack-Infra] [I18n] Regarding Zanata upgrade plan to 3.9.6 with Xenial: Help is needed
On Tue, Nov 22, 2016, at 03:56 PM, Clark Boylan wrote:
On Wed, Nov 9, 2016, at 09:49 AM, Ian Y. Choi wrote:
Hello,
I18n team currently uses Zanata in translate.openstack.org so frequently as an OpenStack translation platform. I really appreciate great help from all contributors, especially infra team members and Zanata development team :
http://specs.openstack.org/openstack-infra/infra-specs/specs/migrate_to_zana ta.html
.
Thanks to [1], the current deployed Zanata version is 3.7.3. During about one year, there were some discussions for Zanata enhancements in many i18n IRC meetings, and especially Austin & Barcelona summits [2]. Zanata development team members kindly listen to such enhancements, and they are actively upgrading Zanata with fixing bugs and incorporating cool features, which are very helpful for translators.
One main obstacle for upgrading Zanata (as far as I know) was that newer versions of Zanata requires Java 8, but it is not default on Trusty. To upgrade Zanata, upgrading from Trusty to Xenial for translate.openstack.org is needed since default-jre-headless on Trusty is Java 7 and Xenial is Java 8.
Recently [3] has been merged, so I really hope that i18n team will see newer version of Zanata soon :)
I18n team discussed Zanata upgrade with pleia2 and clarkb during i18n Barcelona meetup (See number 4 in [4]), and the following is my thoughts on appropriate procedures to support Zanata latest version (Currently 3.9.6 - [5]):
1. Xenial OS test for translate-dev.openstack.org : IMO after tests from infra team, [6] will be changed from "# Node-OS: trusty" to "# Node-OS: xenial"
2. Using openstackid instead of openstackid-dev for translators to test translate-dev.o.o : I uploaded a patch [7].
3. Uploading a patch on openstack-infra/puppet-zanata for Zanata 3.9.6 : [8] is a reference for previous upgrade from Zanata 3.7.2 to 3.7.3
4. translate-dev.o.o with Zanata 3.9.6 will be ready => I18n translators will test it :)
5. If there will be no error for Zanata 3.9,6, then node upgrade from Trusty to Xenial and Zanata upgrade to 3.9.6 is needed for translate.openstack.org : changing [9] line and also proposing a patch similar to [10] will be needed later.
Since newer version of Zanata is what translators are expecting a lot :) , I would like to kindly ask infra team members for the help of reviewing and accomplishing such steps with higher priority.
@clarkb, would the proposed procedures cover all the things regarding Zanata upgrade? If yes, would you share a sort of scheduled plans? IMO Zanata development team will kindly help following schedules if there will be some issues which are dependent to Zanata itself.
[1] https://review.openstack.org/#/c/240383/ [2] https://etherpad.openstack.org/p/I18n-Zanata-enhancement [3] https://review.openstack.org/#/c/384239/ [4] https://etherpad.openstack.org/p/barcelona-i18n-meetup [5] https://github.com/zanata/zanata-platform/releases [6]
http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/s ite.pp#n1282
[7] https://review.openstack.org/#/c/393405/ [8] https://review.openstack.org/#/c/239617/1/manifests/init.pp [9]
[10] https://review.openstack.org/#/c/232313/ Just a quick status update on this. I have updated puppet things so that we can deploy translate-dev01.openstack.org on Xenial then CNAME
http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/s ite.pp#n1257 translate-dev.openstack.org to it. (This is part of how we are trying to deploy services in the future to avoid being tied to a single instance in our puppetry). This includes running the noop puppet apply test against Xenial for these nodes on puppet changes as well.
All of that has gone relatively well and I just tried to deploy on Xenial and have run into a few problems. The puppet output [11] shows that the service isn't installed properly on Ubuntu Xenial. Looking at the puppet-wildfly module [12] it doesn't appear to support systemd on debuntu yet. Due to the upcoming thanksgiving holiday I doubt I will get around to updating and testing that this week, so would be great if someone else is able to work with the puppet-wildfly upstream to fix this.
One other thing that came up is do we need to copy any files from the old server to the new server? Specifically it looks like portions of `/home/wildfly/zanata` may need to be preserved. Or is everything important in the database?
In any case once the wildfly puppetry is working I think the next step is to upgrade zanata to 3.9.6, change the openid server, and have the team test it. Once that looks good its on to the production server.
[11] http://paste.openstack.org/show/590149/ [12]
https://github.com/biemond/biemond-wildfly/blob/master/manifests/service.pp# L35-L46
Time for an update, and this time it is good news. I managed to hack around the systemd issue by adding a puppet exec between installing the init script and starting the service that runs a systemctl daemon-reload [13]. This is necessary to make systemd see the init script allowing it to be used to start the service.
With that sorted I have created a new server called translate-dev01.openstack.org and updated DNS to have translate-dev.openstack.org point to it. This server is running on Ubuntu Xenial with Java 8 and using the old version of Zanata (3.7.3). Now would be a good time to test that this works so that we don't have to do the entire upgrade migration in one go when we do production. Once we are happy with Zanata 3.7.3 on Ubuntu Xenial and Java 8 we can upgrade translate-dev.openstack.org to the version of Zanata we want, then test again. And once that is done we can redeploy and upgrade the production server on Xenial as well.
Let me know how your testing goes and we can take it from there.
Hi Clark,
thanks for the work. Regarding the systemd issue just to ensure: you are using Wildfly 10 with wildfly puppet module v1.0.0? It seems the systemd stuff is already implemented there and to use with:
class { '::wildfly': ... init_system => 'systemd', } This actually looks brand new. I will have to look at it more closely. I ended up hacking around it locally though. The shipped sys v init script works fine with systemd you just have to inject a daemon-reload since
On Wed, Dec 7, 2016, at 01:08 AM, Frank Kloeker wrote: puppet doesn't know how to do that when dealing with systemd.
Clark
On Thu, Jan 12, 2017, at 02:36 PM, Ian Y. Choi wrote:
- I18n team completed tests with current Zanata (3.7.3) with Xenial [1], but found one error mentioned in [2]. It seems that more memory size allocation for Zanata with Java 8 is needed. Could you upgrade the memory size for translate-dev server first to test again?
How much memory do we think will be necessary? The current instance is an 8GB RAM instance. As long as we have quota for it, there should not be a problem redeploying on a bigger instance.
- I remember that newer version of openstackid needs to be tested with translate-dev. So I have just uploaded this patch: https://review.openstack.org/#/c/419667/ . I18n team needs more tests, but I think it is a good time to change to openstackid-dev for such testing. Please ping me after openstackid-dev test with translate-dev is completed with no error :)
- On last I18n team meeting, I18n team recognized that the backup would be so important. Is there more disks for such backup on translate-dev and translate.o.o server? And the approach implementing like [3] looks quite a good idea I think. Thanks, Frank!
I have commented on this change. I would prefer we not backup the dev server and only backup the production server. We don't treat our dev servers as permanent or reliable installs as they may need to be reloaded for various reasons to aid development and testing. Also we should set up backups similar to how we do the other services (I left a comment on this change with a link to an example).
- Can I have root access to translate-dev and translate server?
This is something that can be discussed with the infra team, typically we would give access to someone assisting with implementation to the -dev server while keeping the production server as infra-root only. I will make sure fungi sees this. Hope this helps, Clark
On 2017-01-13 13:31:54 -0800 (-0800), Clark Boylan wrote:
On Thu, Jan 12, 2017, at 02:36 PM, Ian Y. Choi wrote: [...]
- Can I have root access to translate-dev and translate server?
This is something that can be discussed with the infra team, typically we would give access to someone assisting with implementation to the -dev server while keeping the production server as infra-root only. I will make sure fungi sees this.
Echoing Clark, as I really have nothing more to add... Ideally the dev server should be identical enough to production under normal circumstances that root access to dev is sufficient to test theories and confirm issues. If you need logs or other similar artifacts from the production instance, we have a dozen root admins scattered around the globe who should be available to get those for you on demand. If this arrangement is still inconvenient, then we can work to improve dev/production symmetry or safely increase debugging data availability for production as needed. -- Jeremy Stanley
Clark Boylan wrote on 1/14/2017 6:31 AM:
On Thu, Jan 12, 2017, at 02:36 PM, Ian Y. Choi wrote:
- I18n team completed tests with current Zanata (3.7.3) with Xenial [1], but found one error mentioned in [2]. It seems that more memory size allocation for Zanata with Java 8 is needed. Could you upgrade the memory size for translate-dev server first to test again? How much memory do we think will be necessary? The current instance is an 8GB RAM instance. As long as we have quota for it, there should not be a problem redeploying on a bigger instance. Regarding this, I have recently seen another Zanata server: https://translate.zanata.org/ already uses Zanata 3.9.6.
Alex, could you share how much memory is being used for this server? (server memory size, jvm heap size, and so on). I think the server will use Java 8 (OS will be Fedora, right?). It might provide good estimation of how much memory is needed for translate-dev and also translate (in future) with us I think.
- I remember that newer version of openstackid needs to be tested with translate-dev. So I have just uploaded this patch: https://review.openstack.org/#/c/419667/ . I18n team needs more tests, but I think it is a good time to change to openstackid-dev for such testing. Please ping me after openstackid-dev test with translate-dev is completed with no error :)
- On last I18n team meeting, I18n team recognized that the backup would be so important. Is there more disks for such backup on translate-dev and translate.o.o server? And the approach implementing like [3] looks quite a good idea I think. Thanks, Frank! I have commented on this change. I would prefer we not backup the dev server and only backup the production server. We don't treat our dev servers as permanent or reliable installs as they may need to be reloaded for various reasons to aid development and testing. Also we should set up backups similar to how we do the other services (I left a comment on this change with a link to an example). Thanks for kind comments, Clark! I have just read and I agree with your comment.
- Can I have root access to translate-dev and translate server? This is something that can be discussed with the infra team, typically we would give access to someone assisting with implementation to the -dev server while keeping the production server as infra-root only. I will make sure fungi sees this.
I agree with your thoughts and also fungi's comments mentioned in another thread :) One additional thought I just want add for the consideration of this is that at least it would be much better if at least one active I18n team member who assists the implementation, on-going upgrade and operational issues (e.g., Zanata login is not working) has the access to dev and production servers with providing good insights on Zanata infra side with I18n team members. Also, I expect more aligned communication between I18n team and Infra team. To accomplish this, I would at least one I18n team member will have good insight on such as understanding how Zanata deployment has implemented previously and seeing & analyzing server logs. This one I18n member to tho such role should not be me, but unfortunately, it is difficult for me to find another relevant I18n team member now. Now I would like to think more on it and also discuss with I18n team members, and then ask again if my thoughts will come up (Or I may need more discussion with infra team members during PTG). With many thanks, /Ian
Hope this helps, Clark
How much memory do we think will be necessary? The current instance is an 8GB RAM instance. As long as we have quota for it, there should not be a problem redeploying on a bigger instance.
Can we look at 10GB? and run the same test again to make sure its running alright. I assume mysql DB is running on separate server?
I have commented on this change. I would prefer we not backup the dev server and only backup the production server. We don't treat our dev servers as permanent or reliable installs as they may need to be reloaded for various reasons to aid development and testing. Also we should set up backups similar to how we do the other services (I left a comment on this change with a link to an example).
Agree. I would suggest if possible, sync DB data from prod to dev weekly. This would help on allowing us to debug if there's any issue in prod. This is something that can be discussed with the infra team, typically
we would give access to someone assisting with implementation to the -dev server while keeping the production server as infra-root only. I will make sure fungi sees this.
Again, if we can constantly sync prod to dev instance, we will only need root access to dev server. --------------------------------------------- Alex Eng Senior Software Engineer Globalisation Tools Engineering DID: +61 3514 8262 <callto:+61+3514+8262> Mobile: +614 2335 3457 <callto:+614+2335+3457> Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office: +61 7 3514 8100 <callto:+61+7+3514+8100> Fax: +61 7 3514 8199 <callto:+61+7+3514+8199> Website: www.redhat.com On Sat, Jan 14, 2017 at 7:31 AM, Clark Boylan <cboylan@sapwetik.org> wrote:
On Thu, Jan 12, 2017, at 02:36 PM, Ian Y. Choi wrote:
- I18n team completed tests with current Zanata (3.7.3) with Xenial [1], but found one error mentioned in [2]. It seems that more memory size allocation for Zanata with Java 8 is needed. Could you upgrade the memory size for translate-dev server first to test again?
How much memory do we think will be necessary? The current instance is an 8GB RAM instance. As long as we have quota for it, there should not be a problem redeploying on a bigger instance.
- I remember that newer version of openstackid needs to be tested with translate-dev. So I have just uploaded this patch: https://review.openstack.org/#/c/419667/ . I18n team needs more tests, but I think it is a good time to change to openstackid-dev for such testing. Please ping me after openstackid-dev test with translate-dev is completed with no error :)
- On last I18n team meeting, I18n team recognized that the backup would be so important. Is there more disks for such backup on translate-dev and translate.o.o server? And the approach implementing like [3] looks quite a good idea I think. Thanks, Frank!
I have commented on this change. I would prefer we not backup the dev server and only backup the production server. We don't treat our dev servers as permanent or reliable installs as they may need to be reloaded for various reasons to aid development and testing. Also we should set up backups similar to how we do the other services (I left a comment on this change with a link to an example).
- Can I have root access to translate-dev and translate server?
This is something that can be discussed with the infra team, typically we would give access to someone assisting with implementation to the -dev server while keeping the production server as infra-root only. I will make sure fungi sees this.
Hope this helps, Clark
Hi, Before increasing the memory size, there's few info we need to find out: - JVM memory allocation (need the Zanata 3.9.6 url to get access to the info) - Ian, do you have the log file (stack trace) of when the error occur? Also, which Zanata you were using to test 3.9.6? https://translate-dev.openstack.org is not upgraded yet. I suspect its the JVM allocation which might need some tweak. But its best it I can access to the 3.9.6-dev and try to replicate the error from there. --------------------------------------------- Alex Eng Senior Software Engineer Globalisation Tools Engineering DID: +61 3514 8262 <callto:+61+3514+8262> Mobile: +614 2335 3457 <callto:+614+2335+3457> Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office: +61 7 3514 8100 <callto:+61+7+3514+8100> Fax: +61 7 3514 8199 <callto:+61+7+3514+8199> Website: www.redhat.com On Mon, Jan 16, 2017 at 8:01 AM, Alex Eng <aeng@redhat.com> wrote:
How much memory do we think will be necessary? The current instance is
an 8GB RAM instance. As long as we have quota for it, there should not be a problem redeploying on a bigger instance.
Can we look at 10GB? and run the same test again to make sure its running alright. I assume mysql DB is running on separate server?
I have commented on this change. I would prefer we not backup the dev server and only backup the production server. We don't treat our dev servers as permanent or reliable installs as they may need to be reloaded for various reasons to aid development and testing. Also we should set up backups similar to how we do the other services (I left a comment on this change with a link to an example).
Agree. I would suggest if possible, sync DB data from prod to dev weekly. This would help on allowing us to debug if there's any issue in prod.
This is something that can be discussed with the infra team, typically
we would give access to someone assisting with implementation to the -dev server while keeping the production server as infra-root only. I will make sure fungi sees this.
Again, if we can constantly sync prod to dev instance, we will only need root access to dev server.
---------------------------------------------
Alex Eng Senior Software Engineer Globalisation Tools Engineering DID: +61 3514 8262 <callto:+61+3514+8262> Mobile: +614 2335 3457 <callto:+614+2335+3457>
Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office: +61 7 3514 8100 <callto:+61+7+3514+8100> Fax: +61 7 3514 8199 <callto:+61+7+3514+8199> Website: www.redhat.com
On Sat, Jan 14, 2017 at 7:31 AM, Clark Boylan <cboylan@sapwetik.org> wrote:
On Thu, Jan 12, 2017, at 02:36 PM, Ian Y. Choi wrote:
- I18n team completed tests with current Zanata (3.7.3) with Xenial [1], but found one error mentioned in [2]. It seems that more memory size allocation for Zanata with Java 8 is needed. Could you upgrade the memory size for translate-dev server first to test again?
How much memory do we think will be necessary? The current instance is an 8GB RAM instance. As long as we have quota for it, there should not be a problem redeploying on a bigger instance.
- I remember that newer version of openstackid needs to be tested with translate-dev. So I have just uploaded this patch: https://review.openstack.org/#/c/419667/ . I18n team needs more tests, but I think it is a good time to change to openstackid-dev for such testing. Please ping me after openstackid-dev test with translate-dev is completed with no error :)
- On last I18n team meeting, I18n team recognized that the backup would be so important. Is there more disks for such backup on translate-dev and translate.o.o server? And the approach implementing like [3] looks quite a good idea I think. Thanks, Frank!
I have commented on this change. I would prefer we not backup the dev server and only backup the production server. We don't treat our dev servers as permanent or reliable installs as they may need to be reloaded for various reasons to aid development and testing. Also we should set up backups similar to how we do the other services (I left a comment on this change with a link to an example).
- Can I have root access to translate-dev and translate server?
This is something that can be discussed with the infra team, typically we would give access to someone assisting with implementation to the -dev server while keeping the production server as infra-root only. I will make sure fungi sees this.
Hope this helps, Clark
Hi, After looking at the stack trace (log) and checking with our team members, we agree that the memory issue might be due to Zanata and the way it handle large project in copy version. There are few factors which contribute to this: - mysql driver - way of Zanata handling large project during copy version Obviously we didn't use the same large scale project for testing before, that is why the issue wasn't visible. To keep things going, I suggest test copy version on a smaller scale project, while we try to implement the solution in Zanata. Let me know what you think. --------------------------------------------- Alex Eng Senior Software Engineer Globalisation Tools Engineering DID: +61 3514 8262 <callto:+61+3514+8262> Mobile: +614 2335 3457 <callto:+614+2335+3457> Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office: +61 7 3514 8100 <callto:+61+7+3514+8100> Fax: +61 7 3514 8199 <callto:+61+7+3514+8199> Website: www.redhat.com On Mon, Jan 16, 2017 at 9:25 AM, Alex Eng <aeng@redhat.com> wrote:
Hi,
Before increasing the memory size, there's few info we need to find out:
- JVM memory allocation (need the Zanata 3.9.6 url to get access to the info) - Ian, do you have the log file (stack trace) of when the error occur? Also, which Zanata you were using to test 3.9.6? https://translate-dev. openstack.org is not upgraded yet.
I suspect its the JVM allocation which might need some tweak. But its best it I can access to the 3.9.6-dev and try to replicate the error from there.
---------------------------------------------
Alex Eng Senior Software Engineer Globalisation Tools Engineering DID: +61 3514 8262 <callto:+61+3514+8262> Mobile: +614 2335 3457 <callto:+614+2335+3457>
Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office: +61 7 3514 8100 <callto:+61+7+3514+8100> Fax: +61 7 3514 8199 <callto:+61+7+3514+8199> Website: www.redhat.com
On Mon, Jan 16, 2017 at 8:01 AM, Alex Eng <aeng@redhat.com> wrote:
How much memory do we think will be necessary? The current instance is
an 8GB RAM instance. As long as we have quota for it, there should not be a problem redeploying on a bigger instance.
Can we look at 10GB? and run the same test again to make sure its running alright. I assume mysql DB is running on separate server?
I have commented on this change. I would prefer we not backup the dev server and only backup the production server. We don't treat our dev servers as permanent or reliable installs as they may need to be reloaded for various reasons to aid development and testing. Also we should set up backups similar to how we do the other services (I left a comment on this change with a link to an example).
Agree. I would suggest if possible, sync DB data from prod to dev weekly. This would help on allowing us to debug if there's any issue in prod.
This is something that can be discussed with the infra team, typically
we would give access to someone assisting with implementation to the -dev server while keeping the production server as infra-root only. I will make sure fungi sees this.
Again, if we can constantly sync prod to dev instance, we will only need root access to dev server.
---------------------------------------------
Alex Eng Senior Software Engineer Globalisation Tools Engineering DID: +61 3514 8262 <callto:+61+3514+8262> Mobile: +614 2335 3457 <callto:+614+2335+3457>
Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office: +61 7 3514 8100 <callto:+61+7+3514+8100> Fax: +61 7 3514 8199 <callto:+61+7+3514+8199> Website: www.redhat.com
On Sat, Jan 14, 2017 at 7:31 AM, Clark Boylan <cboylan@sapwetik.org> wrote:
On Thu, Jan 12, 2017, at 02:36 PM, Ian Y. Choi wrote:
- I18n team completed tests with current Zanata (3.7.3) with Xenial [1], but found one error mentioned in [2]. It seems that more memory size allocation for Zanata with Java 8 is needed. Could you upgrade the memory size for translate-dev server first to test again?
How much memory do we think will be necessary? The current instance is an 8GB RAM instance. As long as we have quota for it, there should not be a problem redeploying on a bigger instance.
- I remember that newer version of openstackid needs to be tested with translate-dev. So I have just uploaded this patch: https://review.openstack.org/#/c/419667/ . I18n team needs more tests, but I think it is a good time to change to openstackid-dev for such testing. Please ping me after openstackid-dev test with translate-dev is completed with no error :)
- On last I18n team meeting, I18n team recognized that the backup would be so important. Is there more disks for such backup on translate-dev and translate.o.o server? And the approach implementing like [3] looks quite a good idea I think. Thanks, Frank!
I have commented on this change. I would prefer we not backup the dev server and only backup the production server. We don't treat our dev servers as permanent or reliable installs as they may need to be reloaded for various reasons to aid development and testing. Also we should set up backups similar to how we do the other services (I left a comment on this change with a link to an example).
- Can I have root access to translate-dev and translate server?
This is something that can be discussed with the infra team, typically we would give access to someone assisting with implementation to the -dev server while keeping the production server as infra-root only. I will make sure fungi sees this.
Hope this helps, Clark
Hello, I already talked with Alex through IRC: Testing copy version with a smaller scale project was completely fine. As ianw shared: http://lists.openstack.org/pipermail/openstack-infra/2017-January/005058.htm... and http://lists.openstack.org/pipermail/openstack-i18n/2017-January/002670.html , gc and database status might be suspected which contribute to too slow Zanata. Some additional thoughts from me are: - Different mysql driver from Trusty: https://review.openstack.org/#/c/406398/1/manifests/init.pp might contribute, after seeing Alex's comment on "mysql driver". - Current master version of openstack-manuals in translate.o.o (not translate-dev.o.o) has 17,423,952 words, which is larger than the number of words for versions: test-version-creation2, test-version-creation, and stable-newton. If it is really the bug from Zanata, then future execution of creating a new version: "stable-ocata" after Ocata documentation release will contribute to production server downtime. @clarkb, then might it be better to also test with larger memory size? Or testing openstackid-dev for a while would be a good idea now? One correction: I thought http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/s... enumerates the root admins in translate.openstack.org and also translate-dev.openstack.org, but it was my complete misinterpretation on the file. After deeper investigation, I have found that the list is for Zanata instance admins : https://review.openstack.org/#/c/172528/ . With many thanks, /Ian Alex Eng wrote on 1/16/2017 1:11 PM:
Hi,
After looking at the stack trace (log) and checking with our team members, we agree that the memory issue might be due to Zanata and the way it handle large project in copy version. There are few factors which contribute to this: - mysql driver - way of Zanata handling large project during copy version
Obviously we didn't use the same large scale project for testing before, that is why the issue wasn't visible. To keep things going, I suggest test copy version on a smaller scale project, while we try to implement the solution in Zanata.
Let me know what you think.
---------------------------------------------
Alex Eng Senior Software Engineer Globalisation Tools Engineering DID:+61 3514 8262 <callto:+61+3514+8262> Mobile:+614 2335 3457 <callto:+614+2335+3457>
Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office:+61 7 3514 8100 <callto:+61+7+3514+8100> Fax:+61 7 3514 8199 <callto:+61+7+3514+8199> Website:www.redhat.com <http://www.redhat.com/>
On Mon, Jan 16, 2017 at 9:25 AM, Alex Eng <aeng@redhat.com <mailto:aeng@redhat.com>> wrote:
Hi,
Before increasing the memory size, there's few info we need to find out:
- JVM memory allocation (need the Zanata 3.9.6 url to get access to the info) - Ian, do you have the log file (stack trace) of when the error occur? Also, which Zanata you were using to test 3.9.6? https://translate-dev.openstack.org <https://translate-dev.openstack.org> is not upgraded yet.
I suspect its the JVM allocation which might need some tweak. But its best it I can access to the 3.9.6-dev and try to replicate the error from there.
---------------------------------------------
Alex Eng Senior Software Engineer Globalisation Tools Engineering DID:+61 3514 8262 <callto:+61+3514+8262> Mobile:+614 2335 3457 <callto:+614+2335+3457>
Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office:+61 7 3514 8100 <callto:+61+7+3514+8100> Fax:+61 7 3514 8199 <callto:+61+7+3514+8199> Website:www.redhat.com <http://www.redhat.com/>
On Mon, Jan 16, 2017 at 8:01 AM, Alex Eng <aeng@redhat.com <mailto:aeng@redhat.com>> wrote:
How much memory do we think will be necessary? The current instance is an 8GB RAM instance. As long as we have quota for it, there should not be a problem redeploying on a bigger instance.
Can we look at 10GB? and run the same test again to make sure its running alright. I assume mysql DB is running on separate server?
I have commented on this change. I would prefer we not backup the dev server and only backup the production server. We don't treat our dev servers as permanent or reliable installs as they may need to be reloaded for various reasons to aid development and testing. Also we should set up backups similar to how we do the other services (I left a comment on this change with a link to an example).
Agree. I would suggest if possible, sync DB data from prod to dev weekly. This would help on allowing us to debug if there's any issue in prod.
This is something that can be discussed with the infra team, typically we would give access to someone assisting with implementation to the -dev server while keeping the production server as infra-root only. I will make sure fungi sees this.
Again, if we can constantly sync prod to dev instance, we will only need root access to dev server.
---------------------------------------------
Alex Eng Senior Software Engineer Globalisation Tools Engineering DID:+61 3514 8262 <callto:+61+3514+8262> Mobile:+614 2335 3457 <callto:+614+2335+3457>
Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office:+61 7 3514 8100 <callto:+61+7+3514+8100> Fax:+61 7 3514 8199 <callto:+61+7+3514+8199> Website:www.redhat.com <http://www.redhat.com/>
On Sat, Jan 14, 2017 at 7:31 AM, Clark Boylan <cboylan@sapwetik.org <mailto:cboylan@sapwetik.org>> wrote:
On Thu, Jan 12, 2017, at 02:36 PM, Ian Y. Choi wrote: > - I18n team completed tests with current Zanata (3.7.3) with Xenial [1], > but found one error > mentioned in [2]. It seems that more memory size allocation for > Zanata with Java 8 is needed. > Could you upgrade the memory size for translate-dev server first to > test again?
How much memory do we think will be necessary? The current instance is an 8GB RAM instance. As long as we have quota for it, there should not be a problem redeploying on a bigger instance.
> - I remember that newer version of openstackid needs to be tested with > translate-dev. > So I have just uploaded this patch: > https://review.openstack.org/#/c/419667/ <https://review.openstack.org/#/c/419667/> . > I18n team needs more tests, but I think it is a good time to change > to openstackid-dev for such testing. > Please ping me after openstackid-dev test with translate-dev is > completed with no error :) > > - On last I18n team meeting, I18n team recognized that the backup would > be so important. > Is there more disks for such backup on translate-dev and > translate.o.o server? > And the approach implementing like [3] looks quite a good idea I > think. Thanks, Frank!
I have commented on this change. I would prefer we not backup the dev server and only backup the production server. We don't treat our dev servers as permanent or reliable installs as they may need to be reloaded for various reasons to aid development and testing. Also we should set up backups similar to how we do the other services (I left a comment on this change with a link to an example).
> - Can I have root access to translate-dev and translate server?
This is something that can be discussed with the infra team, typically we would give access to someone assisting with implementation to the -dev server while keeping the production server as infra-root only. I will make sure fungi sees this.
Hope this helps, Clark
On Mon, Jan 16, 2017, at 09:37 AM, Ian Y. Choi wrote:
Hello,
I already talked with Alex through IRC: Testing copy version with a smaller scale project was completely fine.
As ianw shared: http://lists.openstack.org/pipermail/openstack-infra/2017-January/005058.htm... and http://lists.openstack.org/pipermail/openstack-i18n/2017-January/002670.html , gc and database status might be suspected which contribute to too slow Zanata.
Some additional thoughts from me are:
- Different mysql driver from Trusty: https://review.openstack.org/#/c/406398/1/manifests/init.pp might contribute, after seeing Alex's comment on "mysql driver". - Current master version of openstack-manuals in translate.o.o (not translate-dev.o.o) has 17,423,952 words, which is larger than the number of words for versions: test-version-creation2, test-version-creation, and stable-newton. If it is really the bug from Zanata, then future execution of creating a new version: "stable-ocata" after Ocata documentation release will contribute to production server downtime.
@clarkb, then might it be better to also test with larger memory size? Or testing openstackid-dev for a while would be a good idea now?
One correction: I thought http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/s... enumerates the root admins in translate.openstack.org and also translate-dev.openstack.org, but it was my complete misinterpretation on the file. After deeper investigation, I have found that the list is for Zanata instance admins : https://review.openstack.org/#/c/172528/ .
I'd like to make sure we get back on the same page with everything going on as I feel like we have all gone in about 5 different directions the last couple days. Clarifications: We have not upgraded to 3.9.6 yet. We are merely running the same version of Zanata as in production (3.7.3) on top of Xenial and Java 8 instead of Trusty and Java 7. This is because it is easier to separate the OS upgrade from the application upgrade and will allow us to upgrade the OS for translate.openstack.org first, then upgrade Zanata. The MySQL instance is running in a Trove instance and is not collocated with Zanata on the same instance. The Zanata wildfly instance can use 4096 Megabytes of memory (according to the Xmx value) on an 8GB memory VM. TODO: There have been a list of things that have been suggested that we try and I would like use to decide whcih are most important and start there. Increase memory on translate-dev.openstack.org host. I think the next biggest VM size is 16GB of ram. That doubles system memory and we can double Xmx to 8192MB as well. Increase size of Trove database instance. It is currently 2GB RAM instance, we can double that to 4GB. Upgrade Zanata to 3.9.6 on translate-dev.openstack.org if you are happy with calling this issue and Zanata bug and go from there. Then once at least some of this is done and we are happy with it we can upgrade the production instance. What are the things that people feel are important here? Personally I think that if we only have performance issues when hitting what is probably a Zanata bug then I wouldn't bother increasing the resources available to the dev instance and instead continue focusing on testing of the upgrade. My suggestion would be to upgrade to 3.9.6 and if it looks good do the production instance as well. But feedback is much appreciated as I know that others have been putting a bit of effort into this and may have better ideas than me :) Clark
My suggestion would be to upgrade to 3.9.6 and if it
looks good do the production instance as well. +1 --------------------------------------------- Alex Eng Senior Software Engineer Globalisation Tools Engineering DID: +61 3514 8262 <callto:+61+3514+8262> Mobile: +614 2335 3457 <callto:+614+2335+3457> Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office: +61 7 3514 8100 <callto:+61+7+3514+8100> Fax: +61 7 3514 8199 <callto:+61+7+3514+8199> Website: www.redhat.com
Hello, Thanks a lot for so wonderful clarification with detail explanation! I agree that it is time to upgrade Zanata to a newer version with Xenial on translate-dev server. Just two things I would like to say now a little bit more are: - I am not too much familiar with database backups for mysql on Trove, but [1] is related with database backup? Also, I was informed that Reik, Frank's colleague, proposed a patch [2] for translate (production) server backup. It would be much better if it is also considered during the overall upgrade procedure not to lose database data on production. - When I have just added some sentences on [3], I have found that Zanata 4.0 is now released and also it has more enhancements: personally, shortcut key for review approval & reject is so cool...! Then in my opinion, upgrading to Zanata 4.0 not 3.9.6 would be fine. (Doesn't it, Alex?) [1] http://docs.openstack.org/infra/system-config/sysadmin.html#backups [2] https://review.openstack.org/#/c/420735/ [3] https://etherpad.openstack.org/p/I18n-Zanata-enhancement With many thanks, /Ian Clark Boylan wrote on 1/17/2017 4:34 AM:
On Mon, Jan 16, 2017, at 09:37 AM, Ian Y. Choi wrote:
Hello,
I already talked with Alex through IRC: Testing copy version with a smaller scale project was completely fine.
As ianw shared: http://lists.openstack.org/pipermail/openstack-infra/2017-January/005058.htm... and http://lists.openstack.org/pipermail/openstack-i18n/2017-January/002670.html , gc and database status might be suspected which contribute to too slow Zanata.
Some additional thoughts from me are:
- Different mysql driver from Trusty: https://review.openstack.org/#/c/406398/1/manifests/init.pp might contribute, after seeing Alex's comment on "mysql driver". - Current master version of openstack-manuals in translate.o.o (not translate-dev.o.o) has 17,423,952 words, which is larger than the number of words for versions: test-version-creation2, test-version-creation, and stable-newton. If it is really the bug from Zanata, then future execution of creating a new version: "stable-ocata" after Ocata documentation release will contribute to production server downtime.
@clarkb, then might it be better to also test with larger memory size? Or testing openstackid-dev for a while would be a good idea now?
One correction: I thought http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/s... enumerates the root admins in translate.openstack.org and also translate-dev.openstack.org, but it was my complete misinterpretation on the file. After deeper investigation, I have found that the list is for Zanata instance admins : https://review.openstack.org/#/c/172528/ . I'd like to make sure we get back on the same page with everything going on as I feel like we have all gone in about 5 different directions the last couple days.
Clarifications:
We have not upgraded to 3.9.6 yet. We are merely running the same version of Zanata as in production (3.7.3) on top of Xenial and Java 8 instead of Trusty and Java 7. This is because it is easier to separate the OS upgrade from the application upgrade and will allow us to upgrade the OS for translate.openstack.org first, then upgrade Zanata.
The MySQL instance is running in a Trove instance and is not collocated with Zanata on the same instance.
The Zanata wildfly instance can use 4096 Megabytes of memory (according to the Xmx value) on an 8GB memory VM.
TODO:
There have been a list of things that have been suggested that we try and I would like use to decide whcih are most important and start there.
Increase memory on translate-dev.openstack.org host. I think the next biggest VM size is 16GB of ram. That doubles system memory and we can double Xmx to 8192MB as well.
Increase size of Trove database instance. It is currently 2GB RAM instance, we can double that to 4GB.
Upgrade Zanata to 3.9.6 on translate-dev.openstack.org if you are happy with calling this issue and Zanata bug and go from there.
Then once at least some of this is done and we are happy with it we can upgrade the production instance.
What are the things that people feel are important here? Personally I think that if we only have performance issues when hitting what is probably a Zanata bug then I wouldn't bother increasing the resources available to the dev instance and instead continue focusing on testing of the upgrade. My suggestion would be to upgrade to 3.9.6 and if it looks good do the production instance as well.
But feedback is much appreciated as I know that others have been putting a bit of effort into this and may have better ideas than me :)
Clark
Yes. 4.0 has some major backend/feature changes But it requires Wildfly 10 (i believe openstack is running on wildfly 9) and some configuration changes. I would happy to help out if needed :) --------------------------------------------- Alex Eng Senior Software Engineer Globalisation Tools Engineering DID: +61 3514 8262 <callto:+61+3514+8262> Mobile: +614 2335 3457 <callto:+614+2335+3457> Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office: +61 7 3514 8100 <callto:+61+7+3514+8100> Fax: +61 7 3514 8199 <callto:+61+7+3514+8199> Website: www.redhat.com
On Tue, Jan 17, 2017, at 01:55 PM, Alex Eng wrote:
Yes. 4.0 has some major backend/feature changes
But it requires Wildfly 10 (i believe openstack is running on wildfly 9) and some configuration changes. I would happy to help out if needed :)
Yup we are running wildfly 9. That seems to be set in openstack-infra/system-config/manifests/site.pp. Maybe it is easier for now to do the jump to Xenial + java 8, then 3.9.6 for the features, then figure out 4.0? Are the feature changes in 4.0 worth skipping 3.9.6 for? Clark
I think at the moment, its best to upgrade to 3.9.6. We can worry about 4.0 later. --------------------------------------------- Alex Eng Senior Software Engineer Globalisation Tools Engineering DID: +61 3514 8262 <callto:+61+3514+8262> Mobile: +614 2335 3457 <callto:+614+2335+3457> Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office: +61 7 3514 8100 <callto:+61+7+3514+8100> Fax: +61 7 3514 8199 <callto:+61+7+3514+8199> Website: www.redhat.com On Wed, Jan 18, 2017 at 10:45 AM, Clark Boylan <cboylan@sapwetik.org> wrote:
On Tue, Jan 17, 2017, at 01:55 PM, Alex Eng wrote:
Yes. 4.0 has some major backend/feature changes
But it requires Wildfly 10 (i believe openstack is running on wildfly 9) and some configuration changes. I would happy to help out if needed :)
Yup we are running wildfly 9. That seems to be set in openstack-infra/system-config/manifests/site.pp.
Maybe it is easier for now to do the jump to Xenial + java 8, then 3.9.6 for the features, then figure out 4.0? Are the feature changes in 4.0 worth skipping 3.9.6 for?
Clark
On Tue, Jan 17, 2017, at 09:11 PM, Alex Eng wrote:
I think at the moment, its best to upgrade to 3.9.6. We can worry about 4.0 later.
Sounds good, I have gone ahead and pushed up https://review.openstack.org/422124 which should upgrade translate dev for us. Ian, can you please check this review and +1 it when you are ready to upgrade? One thing I noticed when writing this change is that the puppet-zanata manifest includes info for wildfly hibernate and wildfly mojarra packages but those packages are for wildfly 8 and we are running 9 [1]. Checking in the sourceforge downloads dir I don't see any wildfly 9 equivalents. Does this mean that the wildfly 8 packages are fine to use or maybe they are no longer necessary and we can clean this up? [1] https://git.openstack.org/cgit/openstack-infra/puppet-zanata/tree/manifests/... Clark
On 19/01/17 04:27, Clark Boylan wrote:
One thing I noticed when writing this change is that the puppet-zanata manifest includes info for wildfly hibernate and wildfly mojarra packages but those packages are for wildfly 8 and we are running 9 [1]. Checking in the sourceforge downloads dir I don't see any wildfly 9 equivalents. Does this mean that the wildfly 8 packages are fine to use or maybe they are no longer necessary and we can clean this up?
I had the same concerns when I updated from Wildfly 8 to Wildfly 9. They were certainly still required for 9, no matter what the download page said. -- Steve <dstufft> computers seemed like a good idea in 1946 and look at where we are now
participants (5)
-
Alex Eng
-
Clark Boylan
-
Ian Y. Choi
-
Jeremy Stanley
-
Steve Kowalik