Re: [OpenStack-I18n] [OpenStack-Infra] [I18n] Regarding Zanata upgrade plan to 3.9.6 with Xenial: Help is needed
Hello, I would like to summarize the status for Zanata (translation platform) upgrade here from various but a little bit scattered results, and also ask some questions to infra team. - I18n team completed tests with current Zanata (3.7.3) with Xenial [1], but found one error mentioned in [2]. It seems that more memory size allocation for Zanata with Java 8 is needed. Could you upgrade the memory size for translate-dev server first to test again? - I remember that newer version of openstackid needs to be tested with translate-dev. So I have just uploaded this patch: https://review.openstack.org/#/c/419667/ . I18n team needs more tests, but I think it is a good time to change to openstackid-dev for such testing. Please ping me after openstackid-dev test with translate-dev is completed with no error :) - On last I18n team meeting, I18n team recognized that the backup would be so important. Is there more disks for such backup on translate-dev and translate.o.o server? And the approach implementing like [3] looks quite a good idea I think. Thanks, Frank! - Can I have root access to translate-dev and translate server? I really appreciate kind and continuous help for such Zanata upgrade, but sometimes I feel that considering priority like in [5] would be also a good idea. (Or somewhere on i18n-specs, which is not yet revised with items [6]). Since O cycle is rather short and also StringFreeze period is important for I18n team with translations, it seems that applying new Zanata version into production (I mean, translate.openstack.org) would be difficult. But, I really want to upgrade Zanata 3.9.6 with Xenial in translate-dev.openstack.org during O cycle and hope that all tests will be fine. Then I would like to have some discussions during PTG event including this topic :) [1] https://etherpad.openstack.org/p/i18n-zanata-test-for-translate-dev [2] http://lists.openstack.org/pipermail/openstack-i18n/2017-January/002643.html [3] https://review.openstack.org/#/c/419389/ [4] http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/s... [5] http://specs.openstack.org/openstack-infra/infra-specs/#priority-efforts [6] http://specs.openstack.org/openstack/i18n-specs/ With many thanks, /Ian Clark Boylan wrote on 12/8/2016 7:56 AM:
On Thu, Jan 12, 2017, at 02:36 PM, Ian Y. Choi wrote:
How much memory do we think will be necessary? The current instance is an 8GB RAM instance. As long as we have quota for it, there should not be a problem redeploying on a bigger instance.
I have commented on this change. I would prefer we not backup the dev server and only backup the production server. We don't treat our dev servers as permanent or reliable installs as they may need to be reloaded for various reasons to aid development and testing. Also we should set up backups similar to how we do the other services (I left a comment on this change with a link to an example).
- Can I have root access to translate-dev and translate server?
This is something that can be discussed with the infra team, typically we would give access to someone assisting with implementation to the -dev server while keeping the production server as infra-root only. I will make sure fungi sees this. Hope this helps, Clark
On 2017-01-13 13:31:54 -0800 (-0800), Clark Boylan wrote:
Echoing Clark, as I really have nothing more to add... Ideally the dev server should be identical enough to production under normal circumstances that root access to dev is sufficient to test theories and confirm issues. If you need logs or other similar artifacts from the production instance, we have a dozen root admins scattered around the globe who should be available to get those for you on demand. If this arrangement is still inconvenient, then we can work to improve dev/production symmetry or safely increase debugging data availability for production as needed. -- Jeremy Stanley
Clark Boylan wrote on 1/14/2017 6:31 AM:
Alex, could you share how much memory is being used for this server? (server memory size, jvm heap size, and so on). I think the server will use Java 8 (OS will be Fedora, right?). It might provide good estimation of how much memory is needed for translate-dev and also translate (in future) with us I think.
I agree with your thoughts and also fungi's comments mentioned in another thread :) One additional thought I just want add for the consideration of this is that at least it would be much better if at least one active I18n team member who assists the implementation, on-going upgrade and operational issues (e.g., Zanata login is not working) has the access to dev and production servers with providing good insights on Zanata infra side with I18n team members. Also, I expect more aligned communication between I18n team and Infra team. To accomplish this, I would at least one I18n team member will have good insight on such as understanding how Zanata deployment has implemented previously and seeing & analyzing server logs. This one I18n member to tho such role should not be me, but unfortunately, it is difficult for me to find another relevant I18n team member now. Now I would like to think more on it and also discuss with I18n team members, and then ask again if my thoughts will come up (Or I may need more discussion with infra team members during PTG). With many thanks, /Ian
Hope this helps, Clark
Can we look at 10GB? and run the same test again to make sure its running alright. I assume mysql DB is running on separate server?
Agree. I would suggest if possible, sync DB data from prod to dev weekly. This would help on allowing us to debug if there's any issue in prod. This is something that can be discussed with the infra team, typically
Again, if we can constantly sync prod to dev instance, we will only need root access to dev server. --------------------------------------------- Alex Eng Senior Software Engineer Globalisation Tools Engineering DID: +61 3514 8262 <callto:+61+3514+8262> Mobile: +614 2335 3457 <callto:+614+2335+3457> Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office: +61 7 3514 8100 <callto:+61+7+3514+8100> Fax: +61 7 3514 8199 <callto:+61+7+3514+8199> Website: www.redhat.com On Sat, Jan 14, 2017 at 7:31 AM, Clark Boylan <cboylan@sapwetik.org> wrote:
Hi, Before increasing the memory size, there's few info we need to find out: - JVM memory allocation (need the Zanata 3.9.6 url to get access to the info) - Ian, do you have the log file (stack trace) of when the error occur? Also, which Zanata you were using to test 3.9.6? https://translate-dev.openstack.org is not upgraded yet. I suspect its the JVM allocation which might need some tweak. But its best it I can access to the 3.9.6-dev and try to replicate the error from there. --------------------------------------------- Alex Eng Senior Software Engineer Globalisation Tools Engineering DID: +61 3514 8262 <callto:+61+3514+8262> Mobile: +614 2335 3457 <callto:+614+2335+3457> Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office: +61 7 3514 8100 <callto:+61+7+3514+8100> Fax: +61 7 3514 8199 <callto:+61+7+3514+8199> Website: www.redhat.com On Mon, Jan 16, 2017 at 8:01 AM, Alex Eng <aeng@redhat.com> wrote:
Hi, After looking at the stack trace (log) and checking with our team members, we agree that the memory issue might be due to Zanata and the way it handle large project in copy version. There are few factors which contribute to this: - mysql driver - way of Zanata handling large project during copy version Obviously we didn't use the same large scale project for testing before, that is why the issue wasn't visible. To keep things going, I suggest test copy version on a smaller scale project, while we try to implement the solution in Zanata. Let me know what you think. --------------------------------------------- Alex Eng Senior Software Engineer Globalisation Tools Engineering DID: +61 3514 8262 <callto:+61+3514+8262> Mobile: +614 2335 3457 <callto:+614+2335+3457> Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office: +61 7 3514 8100 <callto:+61+7+3514+8100> Fax: +61 7 3514 8199 <callto:+61+7+3514+8199> Website: www.redhat.com On Mon, Jan 16, 2017 at 9:25 AM, Alex Eng <aeng@redhat.com> wrote:
Hello, I already talked with Alex through IRC: Testing copy version with a smaller scale project was completely fine. As ianw shared: http://lists.openstack.org/pipermail/openstack-infra/2017-January/005058.htm... and http://lists.openstack.org/pipermail/openstack-i18n/2017-January/002670.html , gc and database status might be suspected which contribute to too slow Zanata. Some additional thoughts from me are: - Different mysql driver from Trusty: https://review.openstack.org/#/c/406398/1/manifests/init.pp might contribute, after seeing Alex's comment on "mysql driver". - Current master version of openstack-manuals in translate.o.o (not translate-dev.o.o) has 17,423,952 words, which is larger than the number of words for versions: test-version-creation2, test-version-creation, and stable-newton. If it is really the bug from Zanata, then future execution of creating a new version: "stable-ocata" after Ocata documentation release will contribute to production server downtime. @clarkb, then might it be better to also test with larger memory size? Or testing openstackid-dev for a while would be a good idea now? One correction: I thought http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/s... enumerates the root admins in translate.openstack.org and also translate-dev.openstack.org, but it was my complete misinterpretation on the file. After deeper investigation, I have found that the list is for Zanata instance admins : https://review.openstack.org/#/c/172528/ . With many thanks, /Ian Alex Eng wrote on 1/16/2017 1:11 PM:
On Mon, Jan 16, 2017, at 09:37 AM, Ian Y. Choi wrote:
I'd like to make sure we get back on the same page with everything going on as I feel like we have all gone in about 5 different directions the last couple days. Clarifications: We have not upgraded to 3.9.6 yet. We are merely running the same version of Zanata as in production (3.7.3) on top of Xenial and Java 8 instead of Trusty and Java 7. This is because it is easier to separate the OS upgrade from the application upgrade and will allow us to upgrade the OS for translate.openstack.org first, then upgrade Zanata. The MySQL instance is running in a Trove instance and is not collocated with Zanata on the same instance. The Zanata wildfly instance can use 4096 Megabytes of memory (according to the Xmx value) on an 8GB memory VM. TODO: There have been a list of things that have been suggested that we try and I would like use to decide whcih are most important and start there. Increase memory on translate-dev.openstack.org host. I think the next biggest VM size is 16GB of ram. That doubles system memory and we can double Xmx to 8192MB as well. Increase size of Trove database instance. It is currently 2GB RAM instance, we can double that to 4GB. Upgrade Zanata to 3.9.6 on translate-dev.openstack.org if you are happy with calling this issue and Zanata bug and go from there. Then once at least some of this is done and we are happy with it we can upgrade the production instance. What are the things that people feel are important here? Personally I think that if we only have performance issues when hitting what is probably a Zanata bug then I wouldn't bother increasing the resources available to the dev instance and instead continue focusing on testing of the upgrade. My suggestion would be to upgrade to 3.9.6 and if it looks good do the production instance as well. But feedback is much appreciated as I know that others have been putting a bit of effort into this and may have better ideas than me :) Clark
My suggestion would be to upgrade to 3.9.6 and if it
looks good do the production instance as well. +1 --------------------------------------------- Alex Eng Senior Software Engineer Globalisation Tools Engineering DID: +61 3514 8262 <callto:+61+3514+8262> Mobile: +614 2335 3457 <callto:+614+2335+3457> Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office: +61 7 3514 8100 <callto:+61+7+3514+8100> Fax: +61 7 3514 8199 <callto:+61+7+3514+8199> Website: www.redhat.com
Hello, Thanks a lot for so wonderful clarification with detail explanation! I agree that it is time to upgrade Zanata to a newer version with Xenial on translate-dev server. Just two things I would like to say now a little bit more are: - I am not too much familiar with database backups for mysql on Trove, but [1] is related with database backup? Also, I was informed that Reik, Frank's colleague, proposed a patch [2] for translate (production) server backup. It would be much better if it is also considered during the overall upgrade procedure not to lose database data on production. - When I have just added some sentences on [3], I have found that Zanata 4.0 is now released and also it has more enhancements: personally, shortcut key for review approval & reject is so cool...! Then in my opinion, upgrading to Zanata 4.0 not 3.9.6 would be fine. (Doesn't it, Alex?) [1] http://docs.openstack.org/infra/system-config/sysadmin.html#backups [2] https://review.openstack.org/#/c/420735/ [3] https://etherpad.openstack.org/p/I18n-Zanata-enhancement With many thanks, /Ian Clark Boylan wrote on 1/17/2017 4:34 AM:
Yes. 4.0 has some major backend/feature changes But it requires Wildfly 10 (i believe openstack is running on wildfly 9) and some configuration changes. I would happy to help out if needed :) --------------------------------------------- Alex Eng Senior Software Engineer Globalisation Tools Engineering DID: +61 3514 8262 <callto:+61+3514+8262> Mobile: +614 2335 3457 <callto:+614+2335+3457> Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office: +61 7 3514 8100 <callto:+61+7+3514+8100> Fax: +61 7 3514 8199 <callto:+61+7+3514+8199> Website: www.redhat.com
On Tue, Jan 17, 2017, at 01:55 PM, Alex Eng wrote:
Yup we are running wildfly 9. That seems to be set in openstack-infra/system-config/manifests/site.pp. Maybe it is easier for now to do the jump to Xenial + java 8, then 3.9.6 for the features, then figure out 4.0? Are the feature changes in 4.0 worth skipping 3.9.6 for? Clark
I think at the moment, its best to upgrade to 3.9.6. We can worry about 4.0 later. --------------------------------------------- Alex Eng Senior Software Engineer Globalisation Tools Engineering DID: +61 3514 8262 <callto:+61+3514+8262> Mobile: +614 2335 3457 <callto:+614+2335+3457> Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office: +61 7 3514 8100 <callto:+61+7+3514+8100> Fax: +61 7 3514 8199 <callto:+61+7+3514+8199> Website: www.redhat.com On Wed, Jan 18, 2017 at 10:45 AM, Clark Boylan <cboylan@sapwetik.org> wrote:
On Tue, Jan 17, 2017, at 09:11 PM, Alex Eng wrote:
Sounds good, I have gone ahead and pushed up https://review.openstack.org/422124 which should upgrade translate dev for us. Ian, can you please check this review and +1 it when you are ready to upgrade? One thing I noticed when writing this change is that the puppet-zanata manifest includes info for wildfly hibernate and wildfly mojarra packages but those packages are for wildfly 8 and we are running 9 [1]. Checking in the sourceforge downloads dir I don't see any wildfly 9 equivalents. Does this mean that the wildfly 8 packages are fine to use or maybe they are no longer necessary and we can clean this up? [1] https://git.openstack.org/cgit/openstack-infra/puppet-zanata/tree/manifests/... Clark
On 19/01/17 04:27, Clark Boylan wrote:
I had the same concerns when I updated from Wildfly 8 to Wildfly 9. They were certainly still required for 9, no matter what the download page said. -- Steve <dstufft> computers seemed like a good idea in 1946 and look at where we are now
participants (5)
-
Alex Eng
-
Clark Boylan
-
Ian Y. Choi
-
Jeremy Stanley
-
Steve Kowalik