<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Guys,<br>
<br>
I want to pay your attention that we need to not only fix snapshots
generation issue, but also prevent caused by it unexpected services
failures (see details in a duplicate [0] of original [1] bug), which
would become a challenge for not experienced users (for example
he/she won't be able to authenticate in GUI or CLI some time after
snapshot generation is started). I get this issue on our bare-metal
lab (10 slaves) each time I have an environment which is running
more than 2 days.<br>
<br>
Links usage for files copying doesn't help in such case, because
tarball is still saved on /var partition. Also, if I want to
workaround this issue, I have to perform a lot of actions: s
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
hrink 'os-varlog' volume (because it's the biggest [2] one) in order
to increase unallocated disk space, resize its FS, create new
volume, create FS, mount it to /var/www/nailgun/dump and update
fstab. Not easy way to make "Generate Diagnostic Snapshot
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
" button work, right?<br>
<br>
So, if we are going to address this diagnostic snapshot issue in
8.0, I want to remind you about b) option :)<br>
<br>
<blockquote type="cite">b) Make the snapshot location share the
diskspace of /var/log?</blockquote>
<br>
Thanks!<br>
<br>
[0] <a class="moz-txt-link-freetext" href="https://bugs.launchpad.net/fuel/+bug/1530131">https://bugs.launchpad.net/fuel/+bug/1530131</a><br>
[1] <a class="moz-txt-link-freetext" href="https://bugs.launchpad.net/fuel/+bug/1529182">https://bugs.launchpad.net/fuel/+bug/1529182</a><br>
[2] <a class="moz-txt-link-freetext" href="http://paste.openstack.org/show/484895/">http://paste.openstack.org/show/484895/</a><br>
<br>
<div class="moz-cite-prefix">On 18.01.16 13:20, Maciej Kwiek wrote:<br>
</div>
<blockquote
cite="mid:CADfAwYMFbWJ9ufc-waeQkxOxb-pGLcDUDARe1dj19iwXrg7EXQ@mail.gmail.com"
type="cite">
<div dir="ltr">Igor: It seems that <span style="font-size:12.8px">fqdn
-> </span><span style="font-size:12.8px">ipaddr will
indeed be resolved. Please share your feedback in review: <a
moz-do-not-send="true"
href="https://review.openstack.org/#/c/266964/3"><a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/266964/3">https://review.openstack.org/#/c/266964/3</a></a></span></div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Jan 15, 2016 at 4:25 PM, Igor
Kalnitsky <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:ikalnitsky@mirantis.com" target="_blank">ikalnitsky@mirantis.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Sheena -<br>
<br>
What do you mean by *targeted*? Shotgun's designed to be a
*targeted*<br>
solution. If someone wants more *precise* targets - it's
easy to<br>
specify them in Nailgun's settings.yaml.<br>
<span class="HOEnZb"><font color="#888888"><br>
- Igor<br>
</font></span>
<div class="HOEnZb">
<div class="h5"><br>
On Fri, Jan 15, 2016 at 5:02 PM, Sheena Gregson <<a
moz-do-not-send="true"
href="mailto:sgregson@mirantis.com"><a class="moz-txt-link-abbreviated" href="mailto:sgregson@mirantis.com">sgregson@mirantis.com</a></a>>
wrote:<br>
> I’ve also seen the request multiple times to be
able to provide more<br>
> targeted snapshots which might also (partially)
solve this problem as it<br>
> would require significantly less disk space to grab
logs from a subset of<br>
> nodes for a specific window of time, instead of the
more robust grab-all<br>
> solution we have now.<br>
><br>
><br>
><br>
> From: Maciej Kwiek [mailto:<a
moz-do-not-send="true"
href="mailto:mkwiek@mirantis.com"><a class="moz-txt-link-abbreviated" href="mailto:mkwiek@mirantis.com">mkwiek@mirantis.com</a></a>]<br>
> Sent: Thursday, January 14, 2016 5:59 AM<br>
> To: OpenStack Development Mailing List (not for
usage questions)<br>
> <<a moz-do-not-send="true"
href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
> Subject: Re: [openstack-dev] [Fuel] Diagnostic
snapshot generation is broken<br>
> due to lack of disk space<br>
><br>
><br>
><br>
> Igor,<br>
><br>
><br>
><br>
> I will investigate this, thanks!<br>
><br>
><br>
><br>
> Artem,<br>
><br>
><br>
><br>
> I guess that if we have an untrusted user on master
node, he could just put<br>
> something he wants to be in the snapshot in
/var/log without having to time<br>
> the attack carefully with tar execution.<br>
><br>
><br>
><br>
> I want to use links for directories, this saves me
the trouble of creating<br>
> hardlinks for every single file in the directory.
Although with how<br>
> exclusion is currently implemented it can cause
deleting log files from<br>
> original directories, need to check this out.<br>
><br>
><br>
><br>
> About your PS: whole /var/log on master node (not
in container) is currently<br>
> downloaded, I think we shouldn't change this as we
plan to drop containers<br>
> in 9.0.<br>
><br>
><br>
><br>
> Cheers,<br>
><br>
> Maciej<br>
><br>
><br>
><br>
> On Thu, Jan 14, 2016 at 12:32 PM, Artem Panchenko
<<a moz-do-not-send="true"
href="mailto:apanchenko@mirantis.com">apanchenko@mirantis.com</a>><br>
> wrote:<br>
><br>
> Hi,<br>
><br>
> using symlinks is a bit dangerous, here is a quote
from the man you<br>
> mentioned [0]:<br>
><br>
>> The `--dereference' option is unsafe if an
untrusted user can modify<br>
>> directories while tar is running.<br>
><br>
> Hard links usage is much safer, because you can't
use them for directories.<br>
> But at the same time implementation in shotgun
would be more complicated<br>
> than with symlinks.<br>
><br>
> Anyway, in order to determine what linking to use
we need to decide where<br>
> (/var/log or another partition) diagnostic snapshot
will be stored.<br>
><br>
> p.s.<br>
><br>
>>This doesn't really give us much right now,
because most of the logs are<br>
>> fetched from master node via ssh due to shotgun
being run in mcollective<br>
>> container<br>
><br>
><br>
><br>
> AFAIK '/var/log/docker-logs/' is available from
mcollective container and<br>
> mounted to /var/log/:<br>
><br>
> [root@fuel-lab-cz5557 ~]# dockerctl shell
mcollective mount -l | grep<br>
> os-varlog<br>
> /dev/mapper/os-varlog on /var/log type ext4<br>
> (rw,relatime,stripe=128,data=ordered)<br>
><br>
> From my experience '/var/log/docker-logs/remote'
folder is most ' heavy'<br>
> thing in snapshot.<br>
><br>
> [0] <a moz-do-not-send="true"
href="http://www.gnu.org/software/tar/manual/html_node/dereference.html"
rel="noreferrer" target="_blank">http://www.gnu.org/software/tar/manual/html_node/dereference.html</a><br>
><br>
> Thanks!<br>
><br>
><br>
><br>
> On 14.01.16 13:00, Igor Kalnitsky wrote:<br>
><br>
> I took a glance on Maciej's patch and it adds a
switch to tar command<br>
><br>
> to make it follow symbolic links<br>
><br>
> Yeah, that should work. Except one thing - we
previously had fqdn -><br>
><br>
> ipaddr links in snapshots. So now they will be
resolved into full<br>
><br>
> copy?<br>
><br>
><br>
><br>
> I meant that symlinks also give us the benefit of
not using additional<br>
><br>
> space (just as hardlinks do) while being able to
link to files from<br>
><br>
> different filesystems.<br>
><br>
> I'm sorry, I got you wrong. :)<br>
><br>
><br>
><br>
> - Igor<br>
><br>
><br>
><br>
> On Thu, Jan 14, 2016 at 12:34 PM, Maciej Kwiek <<a
moz-do-not-send="true"
href="mailto:mkwiek@mirantis.com"><a class="moz-txt-link-abbreviated" href="mailto:mkwiek@mirantis.com">mkwiek@mirantis.com</a></a>>
wrote:<br>
><br>
> Igor,<br>
><br>
><br>
><br>
> I meant that symlinks also give us the benefit of
not using additional space<br>
><br>
> (just as hardlinks do) while being able to link to
files from different<br>
><br>
> filesystems.<br>
><br>
><br>
><br>
> Also, as Barłomiej pointed out the `h` switch for
tar should do the trick<br>
><br>
> [1].<br>
><br>
><br>
><br>
> Cheers,<br>
><br>
> Maciej<br>
><br>
><br>
><br>
> [1] <a moz-do-not-send="true"
href="http://www.gnu.org/software/tar/manual/html_node/dereference.html"
rel="noreferrer" target="_blank">http://www.gnu.org/software/tar/manual/html_node/dereference.html</a><br>
><br>
><br>
><br>
> On Thu, Jan 14, 2016 at 11:22 AM, Bartlomiej
Piotrowski<br>
><br>
> <<a moz-do-not-send="true"
href="mailto:bpiotrowski@mirantis.com">bpiotrowski@mirantis.com</a>>
wrote:<br>
><br>
> Igor,<br>
><br>
><br>
><br>
> I took a glance on Maciej's patch and it adds a
switch to tar command to<br>
><br>
> make it follow symbolic links, so it looks good to
me.<br>
><br>
><br>
><br>
> Bartłomiej<br>
><br>
><br>
><br>
> On Thu, Jan 14, 2016 at 10:39 AM, Igor Kalnitsky
<<a moz-do-not-send="true"
href="mailto:ikalnitsky@mirantis.com">ikalnitsky@mirantis.com</a>><br>
><br>
> wrote:<br>
><br>
> Hey Maceij -<br>
><br>
><br>
><br>
> About hardlinks - wouldn't it be better to use
symlinks?<br>
><br>
> This way we don't occupy more space than necessary<br>
><br>
> AFAIK, hardlinks won't occupy much space. They are
the links, after all.<br>
><br>
> :)<br>
><br>
><br>
><br>
> As for symlinks, I'm afraid shotgun (and fabric
underneath) won't<br>
><br>
> resolve them and links are get to snapshot As Is.
That means if there<br>
><br>
> will be no content in the snapshot they are
pointing to, they are<br>
><br>
> simply useless. Needs to be checked, though.<br>
><br>
><br>
><br>
> - Igor<br>
><br>
><br>
><br>
> On Thu, Jan 14, 2016 at 10:31 AM, Maciej Kwiek <<a
moz-do-not-send="true"
href="mailto:mkwiek@mirantis.com"><a class="moz-txt-link-abbreviated" href="mailto:mkwiek@mirantis.com">mkwiek@mirantis.com</a></a>><br>
><br>
> wrote:<br>
><br>
> Thanks for your insight guys!<br>
><br>
><br>
><br>
> I agree with Oleg, I will see what I can do to make
this work this way.<br>
><br>
><br>
><br>
> About hardlinks - wouldn't it be better to use
symlinks? This way we<br>
><br>
> don't<br>
><br>
> occupy more space than necessary, and we can link
to files and<br>
><br>
> directories<br>
><br>
> that are in other block device than /var. Please
see [1] review for a<br>
><br>
> proposed change that introduces symlinks.<br>
><br>
><br>
><br>
> This doesn't really give us much right now, because
most of the logs<br>
><br>
> are<br>
><br>
> fetched from master node via ssh due to shotgun
being run in<br>
><br>
> mcollective<br>
><br>
> container, but it's something! When we remove
containers, this will<br>
><br>
> prove<br>
><br>
> more useful.<br>
><br>
><br>
><br>
> Regards,<br>
><br>
> Maciej Kwiek<br>
><br>
><br>
><br>
> [1] <a moz-do-not-send="true"
href="https://review.openstack.org/#/c/266964/"
rel="noreferrer" target="_blank">https://review.openstack.org/#/c/266964/</a><br>
><br>
><br>
><br>
> On Tue, Jan 12, 2016 at 1:51 PM, Oleg Gelbukh <<a
moz-do-not-send="true"
href="mailto:ogelbukh@mirantis.com"><a class="moz-txt-link-abbreviated" href="mailto:ogelbukh@mirantis.com">ogelbukh@mirantis.com</a></a>><br>
><br>
> wrote:<br>
><br>
> I think we need to find a way to:<br>
><br>
><br>
><br>
> 1) verify the size of snapshot without actually
making it and compare<br>
><br>
> to<br>
><br>
> the available disk space beforehand.<br>
><br>
> 2) refuse to create snapshot if space is
insufficient and notify user<br>
><br>
> (otherwise it breaks Admin node as we have seen)<br>
><br>
> 3) provide a way to prioritize elements of the
snapshot and exclude<br>
><br>
> them<br>
><br>
> based on the priorities or user choice.<br>
><br>
><br>
><br>
> This will allow for better and safer UX with the
snapshot.<br>
><br>
><br>
><br>
> --<br>
><br>
> Best regards,<br>
><br>
> Oleg Gelbukh<br>
><br>
><br>
><br>
> On Tue, Jan 12, 2016 at 1:47 PM, Maciej Kwiek <<a
moz-do-not-send="true"
href="mailto:mkwiek@mirantis.com"><a class="moz-txt-link-abbreviated" href="mailto:mkwiek@mirantis.com">mkwiek@mirantis.com</a></a>><br>
><br>
> wrote:<br>
><br>
> Hi!<br>
><br>
><br>
><br>
> I need some advice on how to tackle this issue.
There is a bug [1]<br>
><br>
> describing the problem with creating a diagnostic
snapshot. The issue<br>
><br>
> is<br>
><br>
> that /var/log has 100GB available, while /var
(where diagnostic<br>
><br>
> snapshot is<br>
><br>
> being generated -
/var/www/nailgun/dump/fuel-snapshot according to<br>
><br>
> [2]) has<br>
><br>
> 10GB available, so dumping the logs can be an issue
when logs size<br>
><br>
> exceed<br>
><br>
> free space in /var.<br>
><br>
><br>
><br>
> There are several things we could do, but I am
unsure on which course<br>
><br>
> to<br>
><br>
> take. Should we<br>
><br>
> a) Allocate more disk space for /var/www (or for
whole /var)?<br>
><br>
> b) Make the snapshot location share the diskspace
of /var/log?<br>
><br>
> c) Something else? What?<br>
><br>
><br>
><br>
> Please share your thoughts on this.<br>
><br>
><br>
><br>
> Cheers,<br>
><br>
> Maciej Kwiek<br>
><br>
><br>
><br>
> [1] <a moz-do-not-send="true"
href="https://bugs.launchpad.net/fuel/+bug/1529182"
rel="noreferrer" target="_blank">https://bugs.launchpad.net/fuel/+bug/1529182</a><br>
><br>
> [2]<br>
><br>
><br>
><br>
> <a moz-do-not-send="true"
href="https://github.com/openstack/fuel-web/blob/2855a9ba925c146b4802ab3cd2185f1dce2d8a6a/nailgun/nailgun/settings.yaml#L717"
rel="noreferrer" target="_blank">https://github.com/openstack/fuel-web/blob/2855a9ba925c146b4802ab3cd2185f1dce2d8a6a/nailgun/nailgun/settings.yaml#L717</a><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
>
__________________________________________________________________________<br>
><br>
> OpenStack Development Mailing List (not for usage
questions)<br>
><br>
> Unsubscribe:<br>
><br>
> <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
><br>
> <a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
>
__________________________________________________________________________<br>
><br>
> OpenStack Development Mailing List (not for usage
questions)<br>
><br>
> Unsubscribe:<br>
><br>
> <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
><br>
> <a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
>
__________________________________________________________________________<br>
><br>
> OpenStack Development Mailing List (not for usage
questions)<br>
><br>
> Unsubscribe:<br>
><br>
> <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
><br>
> <a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
><br>
><br>
>
__________________________________________________________________________<br>
><br>
> OpenStack Development Mailing List (not for usage
questions)<br>
><br>
> Unsubscribe:<br>
><br>
> <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
><br>
> <a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
><br>
><br>
>
__________________________________________________________________________<br>
><br>
> OpenStack Development Mailing List (not for usage
questions)<br>
><br>
> Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
><br>
> <a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
><br>
><br>
>
__________________________________________________________________________<br>
><br>
> OpenStack Development Mailing List (not for usage
questions)<br>
><br>
> Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
><br>
> <a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
>
__________________________________________________________________________<br>
><br>
> OpenStack Development Mailing List (not for usage
questions)<br>
><br>
> Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
><br>
> <a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
> --<br>
><br>
> Artem Panchenko<br>
><br>
> QA Engineer<br>
><br>
><br>
>
__________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage
questions)<br>
> Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
><br>
>
__________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage
questions)<br>
> Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage
questions)<br>
Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Artem Panchenko
QA Engineer</pre>
</body>
</html>