Hello, I already talked with Alex through IRC: Testing copy version with a smaller scale project was completely fine. As ianw shared: http://lists.openstack.org/pipermail/openstack-infra/2017-January/005058.htm... and http://lists.openstack.org/pipermail/openstack-i18n/2017-January/002670.html , gc and database status might be suspected which contribute to too slow Zanata. Some additional thoughts from me are: - Different mysql driver from Trusty: https://review.openstack.org/#/c/406398/1/manifests/init.pp might contribute, after seeing Alex's comment on "mysql driver". - Current master version of openstack-manuals in translate.o.o (not translate-dev.o.o) has 17,423,952 words, which is larger than the number of words for versions: test-version-creation2, test-version-creation, and stable-newton. If it is really the bug from Zanata, then future execution of creating a new version: "stable-ocata" after Ocata documentation release will contribute to production server downtime. @clarkb, then might it be better to also test with larger memory size? Or testing openstackid-dev for a while would be a good idea now? One correction: I thought http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/s... enumerates the root admins in translate.openstack.org and also translate-dev.openstack.org, but it was my complete misinterpretation on the file. After deeper investigation, I have found that the list is for Zanata instance admins : https://review.openstack.org/#/c/172528/ . With many thanks, /Ian Alex Eng wrote on 1/16/2017 1:11 PM:
Hi,
After looking at the stack trace (log) and checking with our team members, we agree that the memory issue might be due to Zanata and the way it handle large project in copy version. There are few factors which contribute to this: - mysql driver - way of Zanata handling large project during copy version
Obviously we didn't use the same large scale project for testing before, that is why the issue wasn't visible. To keep things going, I suggest test copy version on a smaller scale project, while we try to implement the solution in Zanata.
Let me know what you think.
---------------------------------------------
Alex Eng Senior Software Engineer Globalisation Tools Engineering DID:+61 3514 8262 <callto:+61+3514+8262> Mobile:+614 2335 3457 <callto:+614+2335+3457>
Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office:+61 7 3514 8100 <callto:+61+7+3514+8100> Fax:+61 7 3514 8199 <callto:+61+7+3514+8199> Website:www.redhat.com <http://www.redhat.com/>
On Mon, Jan 16, 2017 at 9:25 AM, Alex Eng <aeng@redhat.com <mailto:aeng@redhat.com>> wrote:
Hi,
Before increasing the memory size, there's few info we need to find out:
- JVM memory allocation (need the Zanata 3.9.6 url to get access to the info) - Ian, do you have the log file (stack trace) of when the error occur? Also, which Zanata you were using to test 3.9.6? https://translate-dev.openstack.org <https://translate-dev.openstack.org> is not upgraded yet.
I suspect its the JVM allocation which might need some tweak. But its best it I can access to the 3.9.6-dev and try to replicate the error from there.
---------------------------------------------
Alex Eng Senior Software Engineer Globalisation Tools Engineering DID:+61 3514 8262 <callto:+61+3514+8262> Mobile:+614 2335 3457 <callto:+614+2335+3457>
Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office:+61 7 3514 8100 <callto:+61+7+3514+8100> Fax:+61 7 3514 8199 <callto:+61+7+3514+8199> Website:www.redhat.com <http://www.redhat.com/>
On Mon, Jan 16, 2017 at 8:01 AM, Alex Eng <aeng@redhat.com <mailto:aeng@redhat.com>> wrote:
How much memory do we think will be necessary? The current instance is an 8GB RAM instance. As long as we have quota for it, there should not be a problem redeploying on a bigger instance.
Can we look at 10GB? and run the same test again to make sure its running alright. I assume mysql DB is running on separate server?
I have commented on this change. I would prefer we not backup the dev server and only backup the production server. We don't treat our dev servers as permanent or reliable installs as they may need to be reloaded for various reasons to aid development and testing. Also we should set up backups similar to how we do the other services (I left a comment on this change with a link to an example).
Agree. I would suggest if possible, sync DB data from prod to dev weekly. This would help on allowing us to debug if there's any issue in prod.
This is something that can be discussed with the infra team, typically we would give access to someone assisting with implementation to the -dev server while keeping the production server as infra-root only. I will make sure fungi sees this.
Again, if we can constantly sync prod to dev instance, we will only need root access to dev server.
---------------------------------------------
Alex Eng Senior Software Engineer Globalisation Tools Engineering DID:+61 3514 8262 <callto:+61+3514+8262> Mobile:+614 2335 3457 <callto:+614+2335+3457>
Red Hat, Asia-Pacific Pty Ltd Level 1, 193 North Quay Brisbane 4000 Office:+61 7 3514 8100 <callto:+61+7+3514+8100> Fax:+61 7 3514 8199 <callto:+61+7+3514+8199> Website:www.redhat.com <http://www.redhat.com/>
On Sat, Jan 14, 2017 at 7:31 AM, Clark Boylan <cboylan@sapwetik.org <mailto:cboylan@sapwetik.org>> wrote:
On Thu, Jan 12, 2017, at 02:36 PM, Ian Y. Choi wrote: > - I18n team completed tests with current Zanata (3.7.3) with Xenial [1], > but found one error > mentioned in [2]. It seems that more memory size allocation for > Zanata with Java 8 is needed. > Could you upgrade the memory size for translate-dev server first to > test again?
How much memory do we think will be necessary? The current instance is an 8GB RAM instance. As long as we have quota for it, there should not be a problem redeploying on a bigger instance.
> - I remember that newer version of openstackid needs to be tested with > translate-dev. > So I have just uploaded this patch: > https://review.openstack.org/#/c/419667/ <https://review.openstack.org/#/c/419667/> . > I18n team needs more tests, but I think it is a good time to change > to openstackid-dev for such testing. > Please ping me after openstackid-dev test with translate-dev is > completed with no error :) > > - On last I18n team meeting, I18n team recognized that the backup would > be so important. > Is there more disks for such backup on translate-dev and > translate.o.o server? > And the approach implementing like [3] looks quite a good idea I > think. Thanks, Frank!
I have commented on this change. I would prefer we not backup the dev server and only backup the production server. We don't treat our dev servers as permanent or reliable installs as they may need to be reloaded for various reasons to aid development and testing. Also we should set up backups similar to how we do the other services (I left a comment on this change with a link to an example).
> - Can I have root access to translate-dev and translate server?
This is something that can be discussed with the infra team, typically we would give access to someone assisting with implementation to the -dev server while keeping the production server as infra-root only. I will make sure fungi sees this.
Hope this helps, Clark