<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=iso-8859-1"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cordia New";
panose-1:2 11 3 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cordia New";
panose-1:2 11 3 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:42338997;
mso-list-type:hybrid;
mso-list-template-ids:-401678016 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:148206272;
mso-list-type:hybrid;
mso-list-template-ids:-37569252 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l2
{mso-list-id:209464756;
mso-list-type:hybrid;
mso-list-template-ids:-50287206 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l2:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l2:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l2:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l2:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l2:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l2:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l2:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l2:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l3
{mso-list-id:335570662;
mso-list-type:hybrid;
mso-list-template-ids:-1545727698 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l3:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l3:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l3:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l3:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l3:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l3:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l3:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l3:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l3:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l4
{mso-list-id:618952972;
mso-list-type:hybrid;
mso-list-template-ids:907192688 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l4:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l4:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l4:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l4:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l4:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l4:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l4:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l4:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l4:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l5
{mso-list-id:1144204375;
mso-list-type:hybrid;
mso-list-template-ids:1656276620 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l5:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l5:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l5:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l5:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l5:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l5:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l5:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l5:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l5:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l6
{mso-list-id:1175803769;
mso-list-type:hybrid;
mso-list-template-ids:-389488252 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l6:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l6:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l6:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l6:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l6:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l6:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l6:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l6:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l6:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l7
{mso-list-id:1212041427;
mso-list-type:hybrid;
mso-list-template-ids:2014339034 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l7:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l7:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l7:level3
{mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l7:level4
{mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l7:level5
{mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l7:level6
{mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l7:level7
{mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l7:level8
{mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l7:level9
{mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l8
{mso-list-id:1688676922;
mso-list-type:hybrid;
mso-list-template-ids:1627674688 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l8:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l8:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l8:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l8:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l8:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l8:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l8:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l8:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l8:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l9
{mso-list-id:1925065435;
mso-list-type:hybrid;
mso-list-template-ids:-1129384052 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l9:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l9:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l9:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l9:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l9:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l9:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l9:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l9:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l9:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='color:#1F497D'>Forwarding to the main list, since the sub-team lists are going away.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> nova-upgrades-bounces+john.garbutt=eu.citrix.com@lists.launchpad.net [mailto:nova-upgrades-bounces+john.garbutt=eu.citrix.com@lists.launchpad.net] <b>On Behalf Of </b>John Garbutt<br><b>Sent:</b> 07 December 2011 11:57<br><b>To:</b> 'Hookway, Ray'; nova-upgrades@lists.launchpad.net<br><b>Subject:</b> Re: [Nova-upgrades] Upgrades<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='color:#1F497D'>Hi Ray,<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>No problem, thanks for the feedback. I think we mostly agree. Here are a few more ideas, and some updates on what we have got working at Citrix. Sorry this is such a long email (again).<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>From what I remember “upgrades between each milestone release” were suggested as desirable in the design summit, but I certainly agree getting upgrades between major releases is more important. Following on from that, I think hotfixing is also important (the ability to easily add security fixes, etc).<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>I don’t think we will be able to achieve a rolling upgrade form Diablo to Essex, but we can try. I think we should aim at changes to Essex so that Essex->F can be a rolling upgrade. We can possibly look at changing Diablo stable to enable Diablo->Essex, but I would class that as much lower priority. I know our PM suggested that Essex->G would be nice, but let’s not run before we can walk.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>I guess we need to scope which components we are considering. I would suggest we include all core projects, except swift (I think they have their own story already?). So the list becomes:<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l6 level1 lfo2'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Keystone<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l6 level1 lfo2'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Glance<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l6 level1 lfo2'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Nova<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l6 level1 lfo2'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Horizon<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Aims for the upgrade:<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l5 level1 lfo4'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>No API downtime (or maybe a small period of read-only access / degraded service?)<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l5 level1 lfo4'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>No instance downtime (apart from live migration)<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>This might not be possible, but should at least be the long term aim. Perhaps we should sart <o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>What about this for a list of scenarios:<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l4 level1 lfo6'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Hotfix:<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l4 level2 lfo6'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>No change to Rabbit or SQL scheme<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l4 level2 lfo6'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Consider a security issue in any of the components<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l4 level1 lfo6'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Hypervisor upgrade<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l4 level2 lfo6'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>we can probably assume some level of (live) migration<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l4 level2 lfo6'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>but we need nova to play nicely with the hypervisor during upgrade<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l4 level1 lfo6'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>RabbitMQ or MySQL upgrade (but no scheme change)<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l4 level2 lfo6'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Not sure, but maybe RabbitMQ’s mirrored queues can help reduce downtime?<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l4 level1 lfo6'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Full upgrade:<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l4 level2 lfo6'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Get new functionality and bug fixes<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l4 level2 lfo6'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>New Rabbit message format (possibly)<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l4 level2 lfo6'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>MySQL scheme changes<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l4 level2 lfo6'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Replace all components<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l4 level1 lfo6'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Rollback<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l4 level2 lfo6'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>restore their system to its previous state if upgrade fails<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l4 level1 lfo6'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>?Config changes<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l4 level2 lfo6'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Not sure this is our problem?<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l4 level2 lfo6'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>but that would cover:<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:108.0pt;text-indent:-18.0pt;mso-list:l4 level3 lfo6'><![if !supportLists]><span style='font-family:Wingdings;color:#1F497D'><span style='mso-list:Ignore'>§<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Networking mode changes<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:108.0pt;text-indent:-18.0pt;mso-list:l4 level3 lfo6'><![if !supportLists]><span style='font-family:Wingdings;color:#1F497D'><span style='mso-list:Ignore'>§<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Scheduling config changes<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l4 level2 lfo6'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>I would suggest just creating a new zone to achieve this kind of change, if nova does not support it, at least for the moment<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>At Citrix we have done some work around the Hotfix scenario (but with a non-rolling upgrade). We were using our virtual appliance style packing (for more details see: </span><a href="http://www.citrix.com/tv/#videos/3837">http://www.citrix.com/tv/#videos/3837</a>). <span style='color:#1F497D'>We now have a Jenkins test that is able to do the following:<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l9 level1 lfo8'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Install an OpenStack cloud on XenServer using a few instances of our ‘OpenStack VPX’<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l9 level1 lfo8'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Check the system is working (upload and launch an instance)<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l9 level1 lfo8'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Turn off the old servers<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l9 level1 lfo8'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Create some new instances of the VPX<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l9 level1 lfo8'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Migrate the following services onto the new VPXes:<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l9 level2 lfo8'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Keystone (both), Glance (both), Nova-API, Nova-Scheduler, Nova-Network (in flat mode), Nova-Compute, Dashboard<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l9 level1 lfo8'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Obviously we have <o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>So key things we have done to make this possible:<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l2 level1 lfo10'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Use the --host flag to assign a GUID to a worker, so the worker is not tied to the hostname of the machine it is running on<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l2 level1 lfo10'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>We don’t use a load balancer in front of the API nodes yet, so we had to hack a few things:<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l2 level2 lfo10'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>glance images have they keystone hostname hardcoded in the DB, so we have to re-write that to point to the new location.<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l2 level2 lfo10'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Keystone endpoints also need updating, for keystone, nova, glance<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l2 level1 lfo10'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Whilst not yet tested automatically in our setup, Horizon can be upgraded using Citrix NetScalers graceful shutdown mode. This is where we can send new traffic only to the new machines, and users are given some time to complete their existing sessions on the old servers.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Other problems we can see coming, but have not yet fixed/resolved:<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l3 level1 lfo12'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Graceful Service shutdown<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>We need to be able to tell all services to stop getting messages from the service queue<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Then also then need to complete all their currently pending tasks (like wait for the download from Glance to finish)<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>The work on improving the Compute Worker State Machine’s resilience, may reduce the impact of not being able to shutdown <o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Ideally we should have config files that trigger the service to correctly pick up the new configuration in a graceful way<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l3 level1 lfo12'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Ability to disable Compute nodes<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>It would be nice to be able to disable a compute node<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Then move the VM instances to another host (using the scheduler)<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>In network HA, you will need to nominate a new network node to start forwarding the traffic somehow?<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Perform the required upgrade procedures (hypervisor upgrade, and OpenStack upgrade)<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Ability to enable the node again, and let the scheduler start picking it again<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l3 level1 lfo12'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Volume<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>EwanM has fixed a big around the use of the --host flag (<a href="https://bugs.launchpad.net/nova/+bug/898290">https://bugs.launchpad.net/nova/+bug/898290</a>)<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>iscsi target needs to persist across upgrades<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>hopefully are work on supporting XenServer’s storage manager will reduce problem in this area<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l3 level1 lfo12'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Networking<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>We haven’t looked at Quantum yet.<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Hopefully the network node, when started on a new node, will be able to regenerate it’s configuration from a database, not yet sure this works<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>We need a way to configure an alias ip for the network node’s gateway address, so it is the same across the upgrade, so you can setup a new machine next to the old network node, and move the traffic across to the new node with minimal downtime<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>CloudStack.org have some funky technology for making this hand over nice and smooth<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Network HA at least reduces the impact of any network node upgrade, but we need to look at if this is works for VLAN as well as Flat DHCP<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l3 level1 lfo12'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Database issues<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>The obvious sticking points…<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>How can we have old and new services both accessing either the new or old database scheme?<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Should all new nodes be able to talk to an old database (horrid code bloat) or should all old nodes be able to talk to the new database (restricts the possible changes to the DB schema)?<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Is it safe to run the db migrate script while other are also access the database? <o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l3 level1 lfo12'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Message Queue issues<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Should the rabbit queue names include the message protocol version<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Should new nodes all be able to talk to old rabbit protocol, while there are still “old nodes” in the cloud?<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Should the new protocol by backwards compatible to old nodes are able to talk to new nodes without noticing?<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Should we move to versioning the rabbit protocol in the same way as the REST APIs so that they can interoperate more easily?<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l3 level1 lfo12'><![if !supportLists]><span style='font-family:Symbol;color:#1F497D'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Deployment advice<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>We need to document the best way to configure the system to enable a graceful upgrade<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Things like setting the --host flag to a GUID<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Ensure all API nodes are behind a load balancer for public and private endpoints<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Use the above addresses to configure keystone, and all the flags like the glance url for nova compute<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l3 level2 lfo12'><![if !supportLists]><span style='font-family:"Courier New";color:#1F497D'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Plus all the other things I have forgotten or not yet stumbled across<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Some of the above need policy decisions (Vish?) and tests to ensure they are enforced. Most of them need some code changes / “features” adding into nova.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>John<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> nova-upgrades-bounces+john.garbutt=eu.citrix.com@lists.launchpad.net [mailto:nova-upgrades-bounces+john.garbutt=eu.citrix.com@lists.launchpad.net] <b>On Behalf Of </b>Hookway, Ray<br><b>Sent:</b> 06 December 2011 17:15<br><b>To:</b> nova-upgrades@lists.launchpad.net<br><b>Subject:</b> Re: [Nova-upgrades] Upgrades<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='color:#1F497D'>Some comments on John’s message are below in </span><span style='color:red'>red</span><span style='color:#1F497D'>. John, thanks for giving this a kick!<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Here are some further thoughts on how to proceed.<o:p></o:p></span></p><p class=MsoNormal style='margin-left:18.0pt'><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l7 level1 lfo14'><![if !supportLists]><span style='color:#1F497D'><span style='mso-list:Ignore'>1.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Agree on goals. (The goals below are a great start. Need to capture the overall objective of transparent upgrades.)<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l7 level1 lfo14'><![if !supportLists]><span style='color:#1F497D'><span style='mso-list:Ignore'>2.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Define some upgrade scenarios that are felt to be representative of the kinds of upgrades that we will want to make.<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l7 level2 lfo14'><![if !supportLists]><span style='color:#1F497D'><span style='mso-list:Ignore'>a.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>API changes<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l7 level2 lfo14'><![if !supportLists]><span style='color:#1F497D'><span style='mso-list:Ignore'>b.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Changes to the network configuration<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l7 level2 lfo14'><![if !supportLists]><span style='color:#1F497D'><span style='mso-list:Ignore'>c.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Upgrades of system components like RabbitMQ, Database<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l7 level2 lfo14'><![if !supportLists]><span style='color:#1F497D'><span style='mso-list:Ignore'>d.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Hypervisor upgrades<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l7 level2 lfo14'><![if !supportLists]><span style='color:#1F497D'><span style='mso-list:Ignore'>e.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Scheduler changes (possibly including changes to changes to data on which scheduling decisions are based)<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l7 level1 lfo14'><![if !supportLists]><span style='color:#1F497D'><span style='mso-list:Ignore'>3.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Define generic upgrade approach. Should include things like sequencing of components. Require that future milestones support this approach to upgrade from previously released version.<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l7 level1 lfo14'><![if !supportLists]><span style='color:#1F497D'><span style='mso-list:Ignore'>4.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Message versioning<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l7 level2 lfo14'><![if !supportLists]><span style='color:#1F497D'><span style='mso-list:Ignore'>a.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Include upgrades to RabbitMQ and supporting libraries<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l7 level1 lfo14'><![if !supportLists]><span style='color:#1F497D'><span style='mso-list:Ignore'>5.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Database versioning<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l7 level2 lfo14'><![if !supportLists]><span style='color:#1F497D'><span style='mso-list:Ignore'>a.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Include upgrades to MySQL and sqlalchemy<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l7 level2 lfo14'><![if !supportLists]><span style='color:#1F497D'><span style='mso-list:Ignore'>b.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Include changes to things like caching<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l7 level1 lfo14'><![if !supportLists]><span style='color:#1F497D'><span style='mso-list:Ignore'>6.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Network upgrade paths<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l7 level1 lfo14'><![if !supportLists]><span style='color:#1F497D'><span style='mso-list:Ignore'>7.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Coordination with other development groups<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l7 level1 lfo14'><![if !supportLists]><span style='color:#1F497D'><span style='mso-list:Ignore'>8.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='color:#1F497D'>Testing – how do we ensure that upgrades are in fact transparent<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>I welcome comments on the above. I think it’s important to pin down items 1 and 2 in order to evaluate work on the other items. I will draft a “goals” document that we can then discuss it. Goals have been previously discussed in the blueprints mentioned below and at the design summit. My objective is to capture the goals in a single place. I’m also going to start working on item 2. Do we have any volunteers for the other items? If people have been working on, for example, database versioning, what progress have you made? What issues have you encountered?<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>-Ray Hookway (rjh)<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> nova-upgrades-bounces+ray.hookway=hp.com@lists.launchpad.net [mailto:nova-upgrades-bounces+ray.hookway=hp.com@lists.launchpad.net] <b>On Behalf Of </b>John Garbutt<br><b>Sent:</b> Tuesday, November 08, 2011 1:20 PM<br><b>To:</b> 'nova-upgrades@lists.launchpad.net'<br><b>Subject:</b> [Nova-upgrades] Upgrades<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Hi,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I just wanted to introduce the (rough) blueprint I drafted before the summit:<o:p></o:p></p><p class=MsoNormal><a href="https://blueprints.launchpad.net/nova/+spec/upgrade-with-minimal-downtime">https://blueprints.launchpad.net/nova/+spec/upgrade-with-minimal-downtime</a><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>There is also a related blueprint from Matt Dietz:<o:p></o:p></p><p class=MsoNormal><a href="https://blueprints.launchpad.net/nova/+spec/deployability-improvements">https://blueprints.launchpad.net/nova/+spec/deployability-improvements</a><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>There are a few questions I am wondering about:<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l1 level1 lfo16'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>What are people working on right now? Let’s talk to stop any duplicate effort!<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'><span style='color:red'>Defining upgrade sequence and basic approach. My objectives align with yours below.<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l1 level1 lfo16'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Are there any meetings scheduled yet?<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'><span style='color:red'>Not yet – we need to get going.<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l1 level1 lfo16'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Where are we aiming in the Essex timeframe?<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'><span style='color:red'>Would like to have a transparent upgrade path from Diablo to Essesx. Will take commitment from other workgroups (e.g., database).<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l1 level1 lfo16'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>What requirements/issues do we need to raise with other working groups? (Database clean-up, etc)<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'><span style='color:red'>Need to review this on a component by component basis.<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>To start the discussion, here is an idea of end goal I was imagining in in the blueprint:<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo18'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>API endpoints (and dashboard) always available during upgrade<o:p></o:p></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l0 level2 lfo18'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Using load balancer graceful shutdown<o:p></o:p></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l0 level2 lfo18'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>No API messages or tasks lost<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo18'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Minimal loss of instance connectivity<o:p></o:p></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l0 level2 lfo18'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Use an IP alias for transparent gateway changes (consider keepalived and conntrackd)<o:p></o:p></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l0 level2 lfo18'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>New style Network HA to reduce the effected number of VMs<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo18'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Minimal loss of volume connectivity<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo18'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Rolling Upgrades of OpenStack components<o:p></o:p></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l0 level2 lfo18'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Different versions can co-exist within a single zone<o:p></o:p></p><p class=MsoNormal style='margin-left:72.0pt'><span style='color:red'>This is the key to transparent upgrades<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:108.0pt;text-indent:-18.0pt;mso-list:l0 level3 lfo18'><![if !supportLists]><span lang=FR style='font-family:Wingdings'><span style='mso-list:Ignore'>§<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]> <span lang=FR>Glance API versions, Message Queue formats, Database Scheme changes, etc.<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l0 level2 lfo18'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Upgrade each component and/or host in turn to stop large amounts of downtime<o:p></o:p></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l0 level2 lfo18'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Ability to migrate the database scheme with minimal disruption<o:p></o:p></p><p class=MsoListParagraph style='margin-left:108.0pt;text-indent:-18.0pt;mso-list:l0 level3 lfo18'><![if !supportLists]><span style='font-family:Wingdings'><span style='mso-list:Ignore'>§<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Ideally without having to stop connections to the database<o:p></o:p></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l0 level2 lfo18'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Support side-by-side upgrades to try and minimize the downtime <span style='color:red'>– is this different from “Different versions can co-exist”?</span><o:p></o:p></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo18'><![if !supportLists]><span lang=FR style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=FR>Transparent Hypervisor Upgrades<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l0 level2 lfo18'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>(where possible) live migrate instances to another hypervisor before upgrade<o:p></o:p></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l0 level2 lfo18'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>In the worst case, consider suspending instances across upgrade<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo18'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Other upgrades<o:p></o:p></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l0 level2 lfo18'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>MySQL, RabbitMQ and other supporting systems<o:p></o:p></p><p class=MsoNormal style='margin-left:72.0pt'><span style='color:red'>Need to determine what it takes to do this. Versioning of messages?<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo18'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Support rolling back to the previous version<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo18'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Support upgrades between each milestone release, and between each major release<o:p></o:p></p><p class=MsoNormal style='margin-left:54.0pt'><span style='color:red'>Not clear to me that transparent upgrades are needed between milestone releases<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo18'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Gating trunk on the ability to upgrade from the previous milestone and previous release<o:p></o:p></p><p class=MsoNormal style='margin-left:54.0pt'><span style='color:red'>This is really important. Releases that can’t be upgraded transparently are not deployable.<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Right now there are quite a few things we need to support all this:<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l8 level1 lfo20'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Graceful service shutdown<o:p></o:p></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l8 level2 lfo20'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Service stops listening to Rabbit queues<o:p></o:p></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l8 level2 lfo20'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Service then completes all current work<o:p></o:p></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l8 level2 lfo20'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Only then does it stop<o:p></o:p></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l8 level2 lfo20'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Prevents getting into an inconsistent state, and minimizes the risk of looking like you have lost a message queue message<o:p></o:p></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l8 level2 lfo20'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Alternatively, ensure all the services will recover correctly when they are started again on a new machine<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l8 level1 lfo20'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]> Allow different versions of nova-compute, nova-scheduler, glance, swift to co-exist<o:p></o:p></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l8 level2 lfo20'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Need to define how the Database Scheme / Database layer can evolve between versions<o:p></o:p></p><p class=MsoListParagraph style='margin-left:108.0pt;text-indent:-18.0pt;mso-list:l8 level3 lfo20'><![if !supportLists]><span style='font-family:Wingdings'><span style='mso-list:Ignore'>§<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Should we upgrade the database before adding any new components?<o:p></o:p></p><p class=MsoListParagraph style='margin-left:108.0pt;text-indent:-18.0pt;mso-list:l8 level3 lfo20'><![if !supportLists]><span style='font-family:Wingdings'><span style='mso-list:Ignore'>§<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Should we add all the new components before we upgrade the database?<o:p></o:p></p><p class=MsoListParagraph style='margin-left:72.0pt;text-indent:-18.0pt;mso-list:l8 level2 lfo20'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>o<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Need to define the Message Queue Message formats, and maybe version them<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>An interim step could well be to support upgrades where we support different zones using different versions. So during the upgrade you will lose just a zone at a time, and not the whole cloud. <span style='color:red'>Would like to be able to upgrade a single zone transparently.<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks,<o:p></o:p></p><p class=MsoNormal>John<o:p></o:p></p></div></div></div></body></html>