<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Mar 14, 2013 at 10:00 AM, Michael Still <span dir="ltr"><<a href="mailto:mikal@stillhq.com" target="_blank">mikal@stillhq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>On Thu, Mar 14, 2013 at 10:51 AM, Joe Topjian <<a href="mailto:joe.topjian@cybera.ca" target="_blank">joe.topjian@cybera.ca</a>> wrote:<br>
> <a href="https://bugs.launchpad.net/nova/+bug/1126375" target="_blank">https://bugs.launchpad.net/nova/+bug/1126375</a><br>
><br>
> Admittedly, the bug report does not explain the scenario in detail, but I<br>
> noted "No matter how many precautions are taken, some scenarios will still<br>
> slip by." which I still firmly believe. My intention was to push for the<br>
> cleanup to be turned off by default before discussing the possible ways it<br>
> would't work as expected. I felt that if by simply describing the scenario,<br>
> that single scenario would be accounted for but thought would not go into<br>
> any other ways it could happen (I feel this is what happened with the NeCTAR<br>
> incident).<br>
><br>
> I fully admit to being difficult with this, but it's something I believe<br>
> strongly in. I have never run into another service or package that has a<br>
> task enabled by default which deletes (rather than archives or recycles)<br>
> data. I am all for these types of cleanup tasks, but feel they must be<br>
> opt-in.<br>
<br>
</div>I vetoed that review and I'd do it again. Nothing you have said has<br>
convinced me that the cleaner should be turned off by default. What<br>
went wrong with Nectar is that they deployed code without testing it<br>
in their environment. We've already discussed my feelings about that.<br></blockquote><div><br></div><div>I could have tested for weeks in a lab and never come across the scenario that I described where I lost images. </div>
<div><br></div><div>Operators do test things, but unfortunately production environments tends to bring out the most random events you'd never think of. </div>
<div><br></div><div>It's not always possible to create an exact lab of a production environment. If I'm running a 10,000 node cloud, do I need a 10,000 node lab to be sure of my tests? What about a 1,000 node cloud or 500 node? Or 5? How can I be sure that my tests are one-to-one transferrable to my production environment? Similarly, how can I simulate production workloads in a lab? How can I simulate end-user decisions that sometimes seem totally off the wall or random?</div>
<div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Frankly, I think its much worse to disable it and have compute nodes<br>
fill their disks, than to have automated cleanup. The whole point of<br>
cloud infrastructure is to manage machines so you don't have to.<br>
Performing a manual cleanup on 10,000 compute nodes is not something<br>
we should force operators to do.<br></blockquote><div><br></div><div>I think it should be the operator who ultimately makes the choice on how they manage their disks. I think it's great that a cleanup process exists, but I strongly feel it should be opt-in. No offence, but you don't understand my environment. Why should you make the decision on what should be running by default?</div>
<div><br></div><div>If it is off by default, then the operator has not lost anything. If it's on by default, there is the possibility that the operator will lose something that they did not want lost.<br></div>
<div><br></div><div>I agree, if it is off by default, then the operator does risk running out of disk space. However, that problem and resolution is much more familiar to an operator than a missing _base image and resulting broken instances.</div>
<div><br></div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">A simple example of something which cleans caches would be squid. I'm<br>
sure I can find other examples trivially. We can't ship software with<br>
an unbounded disk cache -- its guaranteed to hurt users.<br></blockquote><div><br></div><div>If Squid acted the same as _base, then entire websites (or whatever else was being cached) would be broke when the cache was pruned. </div>
<div><br></div><div>I tend to think of this cleanup process as more of a mailbox issue. It's the difference between getting a call from a user saying "my mailbox is full" versus "20 important emails just disappeared".</div>
<div><br></div><div>The potential end-users visibility and downtime of the cleanup process is very high. </div>
<div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
If you find bugs, report the actual bug. The devs aren't clairvoyant,<br>
and we can only fix things we're told about. I've now spent a year<br>
talking to ops people and trying to get them to help us help them<br>
(tagging bugs with the ops tag for example). I've spent the majority<br>
of my development time trying to make things easier for ops folks. I<br>
am very offended that you think we're deliberately trying to make your<br>
life harder, and to be honest it makes me wonder why I bother spending<br>
time trying to help.<br></blockquote><div><br></div><div>No one is clairvoyant - not devs, operators, or end users. It's for that reason that potentially dangerous processes need to be handled with caution. I think it's a good indication that a certain feature or process needs to be handled cautiously when you're unable to foresee what kinds of damage it can do. That's all I'm trying to say.</div>
<div><br></div><div>I am not saying you're deliberately making life harder. I understand that this discussion of a single topic is getting more attention than many other great contributions you've made to OpenStack and that's unfortunate.</div>
<div><br></div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<span><font color="#888888"><br>
Michael<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br>Joe Topjian<div>Systems Administrator</div><div>Cybera Inc.</div><div><br></div><div><a href="http://www.cybera.ca" target="_blank">www.cybera.ca</a></div>
<div><br></div><div><font color="#666666"><span>Cybera</span><span> is a not-for-profit organization that works to spur and support innovation, for the economic benefit of Alberta, through the use of cyberinfrastructure.</span></font></div>
</div></div>