<div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 25, 2016 at 7:32 AM, Dmitry Guryanov <span dir="ltr"><<a href="mailto:dguryanov@mirantis.com" target="_blank">dguryanov@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div style="font-size:large">Here is the bug which I'm trying to fix - <a href="https://bugs.launchpad.net/fuel/+bug/1538587" target="_blank">https://bugs.launchpad.net/fuel/+bug/1538587</a>.</div><div style="font-size:large"><br></div><div style="font-size:large">In VMs (set up with fuel-virtualbox) kernel panic occurs every time you delete node, stack trace shows error in ext4 driver [1].</div><div style="font-size:large">The same as in the bug.</div><div style="font-size:large"><br></div><div style="font-size:large">Here is a patch - <a href="https://review.openstack.org/297669" target="_blank">https://review.openstack.org/297669</a> . I've checked it with virtual box VMs and it works fine.</div><div style="font-size:large"><br></div><div style="font-size:large">I propose also don't reboot nodes in case of kernel panic, so that we'll catch possible errors, but maybe it's too dangerous before release.</div><div style="font-size:large"><br></div></div></blockquote><div><br></div><div>The panic is in there to prevent controllers from staying active with a bad disk. If the file system on a controller goes RO, the node stays in the cluster and causes errors with the openstack deployment. The node erase code tries to disable this prior to erasing the disk so if it's not working we need to fix that, not remove it.</div><div><br></div><div>Thanks,</div><div>-Alex</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div style="font-size:large"></div><div style="font-size:large">[1]</div><div style="font-size:large"><div>[13607.545119] EXT4-fs error (device dm-0) in ext4_reserve_inode_write:4928: IO failure</div><div>[13608.157968] EXT4-fs error (device dm-0) in ext4_reserve_inode_write:4928: IO failure</div><div>[13608.780695] EXT4-fs error (device dm-0) in ext4_reserve_inode_write:4928: IO failure</div><div>[13609.471245] Aborting journal on device dm-0-8.</div><div>[13609.478549] EXT4-fs error (device dm-0) in ext4_dirty_inode:5047: IO failure</div><div>[13610.069244] EXT4-fs error (device dm-0) in ext4_dirty_inode:5047: IO failure</div><div>[13610.698915] Kernel panic - not syncing: EXT4-fs (device dm-0): panic forced after error</div><div>[13610.698915] </div><div>[13611.060673] CPU: 0 PID: 8676 Comm: systemd-udevd Not tainted 3.13.0-83-generic #127-Ubuntu</div><div>[13611.236566] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006</div><div>[13611.887198] 00000000fffffffb ffff88003b6e9a08 ffffffff81725992 ffffffff81a77878</div><div>[13612.527154] ffff88003b6e9a80 ffffffff8171e80b ffffffff00000010 ffff88003b6e9a90</div><div>[13613.037061] ffff88003b6e9a30 ffff88003b6e9a50 ffff8800367f2ad0 0000000000000040</div><div>[13613.717119] Call Trace:</div><div>[13613.927162] [<ffffffff81725992>] dump_stack+0x45/0x56</div><div>[13614.306858] [<ffffffff8171e80b>] panic+0xc8/0x1e1</div><div>[13614.767154] [<ffffffff8125e7c6>] ext4_handle_error.part.187+0xa6/0xb0</div><div>[13615.187201] [<ffffffff8125eddb>] __ext4_std_error+0x7b/0x100</div><div>[13615.627960] [<ffffffff81244c64>] ext4_reserve_inode_write+0x44/0xa0</div><div>[13616.007943] [<ffffffff81247f80>] ? ext4_dirty_inode+0x40/0x60</div><div>[13616.448084] [<ffffffff81244d04>] ext4_mark_inode_dirty+0x44/0x1f0</div><div>[13616.917611] [<ffffffff8126f7f9>] ? __ext4_journal_start_sb+0x69/0xe0</div><div>[13617.367730] [<ffffffff81247f80>] ext4_dirty_inode+0x40/0x60</div><div>[13617.747567] [<ffffffff811e858a>] __mark_inode_dirty+0x10a/0x2d0</div><div>[13618.088060] [<ffffffff811d94e1>] update_time+0x81/0xd0</div><div>[13618.467965] [<ffffffff811d96f0>] file_update_time+0x80/0xd0</div><div>[13618.977649] [<ffffffff811511f0>] __generic_file_aio_write+0x180/0x3d0</div><div>[13619.467993] [<ffffffff81151498>] generic_file_aio_write+0x58/0xa0</div><div>[13619.978080] [<ffffffff8123c712>] ext4_file_write+0xa2/0x3f0</div><div>[13620.467624] [<ffffffff81158066>] ? free_hot_cold_page_list+0x46/0xa0</div><div>[13621.038045] [<ffffffff8115d400>] ? release_pages+0x80/0x210</div><div>[13621.408080] [<ffffffff811bdf5a>] do_sync_write+0x5a/0x90</div><div>[13621.818155] [<ffffffff810e52f6>] do_acct_process+0x4e6/0x5c0</div><div>[13622.278005] [<ffffffff810e5a91>] acct_process+0x71/0xa0</div><div>[13622.597617] [<ffffffff8106a3cf>] do_exit+0x80f/0xa50</div><div>[13622.968015] [<ffffffff811c041e>] ? ____fput+0xe/0x10</div><div>[13623.337738] [<ffffffff8106a68f>] do_group_exit+0x3f/0xa0</div><div>[13623.738020] [<ffffffff8106a704>] SyS_exit_group+0x14/0x20</div><div>[13624.137447] [<ffffffff8173659d>] system_call_fastpath+0x1a/0x1f</div><div>[13624.518044] Rebooting in 10 seconds..</div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Tue, Mar 22, 2016 at 1:07 PM, Dmitry Guryanov <span dir="ltr"><<a href="mailto:dguryanov@mirantis.com" target="_blank">dguryanov@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hello,<div><br></div><div>Here is a start of the discussion - <a href="http://lists.openstack.org/pipermail/openstack-dev/2015-December/083021.html" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2015-December/083021.html</a> . I've subscribed to this mailing list later, so can reply there.</div><div><br></div><div>Currently we clear node's disks in two places. The first one is before reboot into bootstrap image [0] and the second - just before provisioning in fuel-agent [1].</div><div><br></div><div>There are two problems, which should be solved with erasing first megabyte of disk data: node should not boot from hdd after reboot and new partitioning scheme should overwrite the previous one.</div><div><br></div><div>The first problem could be solved with zeroing first 512 bytes of each disk (not partition). Even 446 to be precise, because last 66 bytes are partition scheme, see <a href="https://wiki.archlinux.org/index.php/Master_Boot_Record" target="_blank">https://wiki.archlinux.org/index.php/Master_Boot_Record</a> .</div><div><br></div><div>The second problem should be solved only after reboot into bootstrap. Because if we bring a new node to the cluster from some other place and boot it with bootstrap image it will possibly have disks with some partitions, md devices and lvm volumes. So all these entities should be correctly cleared before provisioning, not before reboot. And fuel-agent does it in [1].<br></div><div><br></div><div>I propose to remove erasing first 1M of each partiton, because it can lead to errors in FS kernel drivers and kernel panic. An existing workaround, that in case of kernel panic we do reboot is bad because it may occur just after clearing first partition of the first disk and after reboot bios will read MBR of the second disk and boot from it instead of network. Let's just clear first 446 bytes of each disk.</div><div><br></div><div><br clear="all"><div>[0] <a href="https://github.com/openstack/fuel-astute/blob/master/mcagents/erase_node.rb#L162-L174" target="_blank">https://github.com/openstack/fuel-astute/blob/master/mcagents/erase_node.rb#L162-L174</a><br></div><div>[1] <a href="https://github.com/openstack/fuel-agent/blob/master/fuel_agent/manager.py#L194-L221" target="_blank">https://github.com/openstack/fuel-agent/blob/master/fuel_agent/manager.py#L194-L221</a><span><font color="#888888"><br></font></span></div><span><font color="#888888"><div><br></div><div><br></div>-- <br><div><div dir="ltr">Dmitry Guryanov<br></div></div>
</font></span></div></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br><div><div dir="ltr">Dmitry Guryanov<br></div></div>
</font></span></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div>