<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">We upgraded our cloud from Essex -> Folsom yesterday and had a major loss of data that I thought I'd share.<div><br></div><div>With Essex the flag remove_used_base_images had a default of False, with Folsom it was changed to True. We hadn't explicitly set this so we had whatever the default was.</div><div><br></div><div>After the upgrade which went relatively smoothly (a lot easier than diablo -> essex) almost all our base images were deleted by the image cache clean up.</div><div>I can't explain how this happened. We lost a total of about 70 images that affected ~200 running instances.</div><div><br></div><div>We have since disabled this flag until we can find out what went wrong. I can see it in the logs and if this flag is enabled it would delete a lot of in use base files still.</div><div><br></div><div>We have an nfs mounted /var/lib/nova/instances directory where the _base dir is located so I'm wondering if this had something to do with it? </div><div>Is the image cache cleanup meant to work in a shared instance storage environment?</div><div><br></div><div><br></div><div>We also came across an issue where some compute nodes were reporting bogus resource stats. Eg:</div><div><br></div><div><div style="margin: 0px; font-size: 12px; ">2012-11-13 05:04:38 INFO nova.compute.manager [-] Updating host status</div><div style="margin: 0px; font-size: 12px; ">2012-11-13 05:06:14 AUDIT nova.compute.resource_tracker [-] Free ram (MB): -739665</div><div style="margin: 0px; font-size: 12px; ">2012-11-13 05:06:14 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 12654</div><div style="margin: 0px; font-size: 12px; ">2012-11-13 05:06:14 AUDIT nova.compute.resource_tracker [-] Free VCPUS: -188</div><div style="margin: 0px; font-size: 12px; ">2012-11-13 05:06:14 INFO nova.compute.resource_tracker [-] Compute_service record updated for np-rcc6</div></div><div style="margin: 0px; font-size: 12px; "><br></div><div style="margin: 0px; font-size: 12px; ">This happened to be addressed by the following bug, it turns out it does a regex for the db filter.</div><div style="margin: 0px; font-size: 12px; "><div style="margin: 0px; "><a href="https://bugs.launchpad.net/nova/+bug/1060363">https://bugs.launchpad.net/nova/+bug/1060363</a></div><div style="margin: 0px; "><br></div></div><div style="margin: 0px; font-size: 12px; ">So a compute node of np-rcc5 would also pull in np-rcc50, np-rcc51.. and so on and so on. </div><div style="margin: 0px; font-size: 12px; "><br></div><div style="margin: 0px; font-size: 12px; "><br></div><div style="margin: 0px; font-size: 12px; ">All in all apart from our huge data loss the upgrade went pretty well. </div><div style="margin: 0px; font-size: 12px; "><br></div><div style="margin: 0px; font-size: 12px; ">The main issues we have now are usability issues with the dashboard:</div><div style="margin: 0px; font-size: 12px; ">Pagination doesn't work</div><div style="margin: 0px; font-size: 12px; ">The green notification boxes that appear top right get in the way of the links behind them</div><div style="margin: 0px; font-size: 12px; ">The new containers view is confusing and you can no longer see how much data in a specific container like you used to.</div><div style="margin: 0px; font-size: 12px; ">The launch instance box sometimes gets the bottom cut off making it useless</div><div style="margin: 0px; font-size: 12px; ">Same with if you have lots of security groups in terms of the launch instance box</div><div style="margin: 0px; font-size: 12px; "><br></div><div style="margin: 0px; font-size: 12px; ">I should also add we have moved to a using nova cells, this went pretty smoothly and we're awaiting eagerly for the cells code to hit trunk so we can contribute our enhancements to cells.</div><div style="margin: 0px; font-size: 12px; "><br></div><div style="margin: 0px; font-size: 12px; "><br></div><div style="margin: 0px; font-size: 12px; ">Cheers,</div><div style="margin: 0px; font-size: 12px; ">Sam</div><div style="margin: 0px; font-size: 12px; "><br></div><div style="margin: 0px; font-size: 12px; "><br></div><div style="margin: 0px; font-size: 12px; "><br></div><div style="margin: 0px; font-size: 12px; "><br></div><div><br></div></body></html>