Open Stack

Hello,

I already talked with Alex through IRC: Testing copy version with a 
smaller scale project was completely fine.

As ianw shared: 
http://lists.openstack.org/pipermail/openstack-infra/2017-January/005058.html 
and
http://lists.openstack.org/pipermail/openstack-i18n/2017-January/002670.html 
,
gc and database status might be suspected which contribute to too slow 
Zanata.

Some additional thoughts from me are:

- Different mysql driver from Trusty: 
https://review.openstack.org/#/c/406398/1/manifests/init.pp
   might contribute, after seeing Alex's comment on "mysql driver".
- Current master version of openstack-manuals in translate.o.o (not 
translate-dev.o.o) has 17,423,952 words, which is larger
   than the number of words for versions: test-version-creation2, 
test-version-creation, and stable-newton.
   If it is really the bug from Zanata, then future execution of 
creating a new version: "stable-ocata" after Ocata documentation release
   will contribute to production server downtime.

@clarkb, then might it be better to also test with larger memory size?
     Or testing openstackid-dev for a while would be a good idea now?


One correction: I thought 
http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n1392
enumerates the root admins in translate.openstack.org and also 
translate-dev.openstack.org, but it was my complete
misinterpretation on the file. After deeper investigation, I have found 
that the list is for Zanata instance admins
: https://review.openstack.org/#/c/172528/ .



With many thanks,

/Ian

Alex Eng wrote on 1/16/2017 1:11 PM:
> Hi,
>
> After looking at the stack trace (log) and checking with our team 
> members, we agree that the memory issue might be due to Zanata and the 
> way it handle large project in copy version.
> There are few factors which contribute to this:
> - mysql driver
> - way of Zanata handling large project during copy version
>
> Obviously we didn't use the same large scale project for testing 
> before, that is why the issue wasn't visible.
> To keep things going, I suggest test copy version on a smaller scale 
> project, while we try to implement the solution in Zanata.
>
> Let me know what you think.
>
>
>
> ---------------------------------------------
>
> Alex Eng
> Senior Software Engineer
> Globalisation Tools Engineering
> DID:+61 3514 8262 <callto:+61+3514+8262>
> Mobile:+614 2335 3457 <callto:+614+2335+3457>
>
> Red Hat, Asia-Pacific Pty Ltd
> Level 1, 193 North Quay
> Brisbane 4000
> Office:+61 7 3514 8100 <callto:+61+7+3514+8100>
> Fax:+61 7 3514 8199 <callto:+61+7+3514+8199>
> Website:www.redhat.com <http://www.redhat.com/>
>
> On Mon, Jan 16, 2017 at 9:25 AM, Alex Eng <aeng at redhat.com 
> <mailto:aeng at redhat.com>> wrote:
>
>     Hi,
>
>     Before increasing the memory size, there's few info we need to
>     find out:
>
>     - JVM memory allocation (need the Zanata 3.9.6 url to get access
>     to the info)
>     - Ian, do you have the log file (stack trace) of when the error
>     occur? Also, which Zanata you were using to test 3.9.6?
>     https://translate-dev.openstack.org
>     <https://translate-dev.openstack.org> is not upgraded yet.
>
>     I suspect its the JVM allocation which might need some tweak. But
>     its best it I can access to the 3.9.6-dev and try to replicate the
>     error from there.
>
>
>
>
>
>     ---------------------------------------------
>
>     Alex Eng
>     Senior Software Engineer
>     Globalisation Tools Engineering
>     DID:+61 3514 8262 <callto:+61+3514+8262>
>     Mobile:+614 2335 3457 <callto:+614+2335+3457>
>
>     Red Hat, Asia-Pacific Pty Ltd
>     Level 1, 193 North Quay
>     Brisbane 4000
>     Office:+61 7 3514 8100 <callto:+61+7+3514+8100>
>     Fax:+61 7 3514 8199 <callto:+61+7+3514+8199>
>     Website:www.redhat.com <http://www.redhat.com/>
>
>     On Mon, Jan 16, 2017 at 8:01 AM, Alex Eng <aeng at redhat.com
>     <mailto:aeng at redhat.com>> wrote:
>
>             How much memory do we think will be necessary? The current
>             instance is
>             an 8GB RAM instance. As long as we have quota for it,
>             there should not
>             be a problem redeploying on a bigger instance.
>
>
>         Can we look at 10GB? and run the same test again to make sure
>         its running alright.
>         I assume mysql DB is running on separate server?
>
>
>             I have commented on this change. I would prefer we not
>             backup the dev
>             server and only backup the production server. We don't
>             treat our dev
>             servers as permanent or reliable installs as they may need
>             to be
>             reloaded for various reasons to aid development and
>             testing. Also we
>             should set up backups similar to how we do the other
>             services (I left a
>             comment on this change with a link to an example).
>
>
>         Agree. I would suggest if possible, sync DB data from prod to
>         dev weekly.
>         This would help on allowing us to debug if there's any issue
>         in prod.
>
>             This is something that can be discussed with the infra
>             team, typically
>             we would give access to someone assisting with
>             implementation to the
>             -dev server while keeping the production server as
>             infra-root only. I
>             will make sure fungi sees this. 
>
>
>         Again, if we can constantly sync prod to dev instance, we will
>         only need root access to dev server.
>
>
>         ---------------------------------------------
>
>         Alex Eng
>         Senior Software Engineer
>         Globalisation Tools Engineering
>         DID:+61 3514 8262 <callto:+61+3514+8262>
>         Mobile:+614 2335 3457 <callto:+614+2335+3457>
>
>         Red Hat, Asia-Pacific Pty Ltd
>         Level 1, 193 North Quay
>         Brisbane 4000
>         Office:+61 7 3514 8100 <callto:+61+7+3514+8100>
>         Fax:+61 7 3514 8199 <callto:+61+7+3514+8199>
>         Website:www.redhat.com <http://www.redhat.com/>
>
>         On Sat, Jan 14, 2017 at 7:31 AM, Clark Boylan
>         <cboylan at sapwetik.org <mailto:cboylan at sapwetik.org>> wrote:
>
>             On Thu, Jan 12, 2017, at 02:36 PM, Ian Y. Choi wrote:
>             > - I18n team completed tests with current Zanata (3.7.3)
>             with Xenial [1],
>             > but found one error
>             >    mentioned in [2]. It seems that more memory size
>             allocation for
>             > Zanata with Java 8 is needed.
>             >    Could you upgrade the memory size for translate-dev
>             server first to
>             > test again?
>
>             How much memory do we think will be necessary? The current
>             instance is
>             an 8GB RAM instance. As long as we have quota for it,
>             there should not
>             be a problem redeploying on a bigger instance.
>
>             > - I remember that newer version of openstackid needs to
>             be tested with
>             > translate-dev.
>             >    So I have just uploaded this patch:
>             > https://review.openstack.org/#/c/419667/
>             <https://review.openstack.org/#/c/419667/> .
>             >    I18n team needs more tests, but I think it is a good
>             time to change
>             > to openstackid-dev for such testing.
>             >    Please ping me after openstackid-dev test with
>             translate-dev is
>             > completed with no error :)
>             >
>             > - On last I18n team meeting, I18n team recognized that
>             the backup would
>             > be so important.
>             >    Is there more disks for such backup on translate-dev and
>             > translate.o.o server?
>             >    And the approach implementing like [3] looks quite a
>             good idea I
>             > think. Thanks, Frank!
>
>             I have commented on this change. I would prefer we not
>             backup the dev
>             server and only backup the production server. We don't
>             treat our dev
>             servers as permanent or reliable installs as they may need
>             to be
>             reloaded for various reasons to aid development and
>             testing. Also we
>             should set up backups similar to how we do the other
>             services (I left a
>             comment on this change with a link to an example).
>
>             > - Can I have root access to translate-dev and translate
>             server?
>
>             This is something that can be discussed with the infra
>             team, typically
>             we would give access to someone assisting with
>             implementation to the
>             -dev server while keeping the production server as
>             infra-root only. I
>             will make sure fungi sees this.
>
>             Hope this helps,
>             Clark
>
>
>
>