<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:211578935;
mso-list-type:hybrid;
mso-list-template-ids:4728660 -184806412 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:DengXian;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1
{mso-list-id:1369643112;
mso-list-template-ids:229664932;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2
{mso-list-id:2048871635;
mso-list-template-ids:-293201972;}
@list l2:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l2:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi Gibi, Dan, Alex and dear Nova cores<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Nova Spec: <a href="https://review.opendev.org/#/c/689070/">
https://review.opendev.org/#/c/689070/</a><o:p></o:p></p>
<p class="MsoNormal">Cinder Spec: <a href="https://review.opendev.org/#/c/684556/">
https://review.opendev.org/#/c/684556/</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regarding the volume local cache spec, after several rounds of discussion in cinder weekly meeting, we continued discussed in cinder virtual mid-cycle ptg yesterday, and now cinder team is close to get agreement, and major ideas are:<o:p></o:p></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3">Cinder sets “cacheable” property in volume type<o:p></o:p>
<ul style="margin-top:0in" type="circle">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo3">Cinder should guarantee “cacheable” is set correctly, that means: should not set with “multiattach”, should not set for a backend that cannot be cached (NFS, RBD)<o:p></o:p></li></ul>
</li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3">Os-brick prevents to attach a volume with cache mode that not safe in cloud environment<o:p></o:p>
<ul style="margin-top:0in" type="circle">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo3">e.g. prevent write-back cache mode. So that all the operations, like live migration/snapshot/consistent group/volume backup can work as usual.<o:p></o:p></li></ul>
</li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3">Nova schedules the VM with “cacheable” volume to servers with cache capability.<o:p></o:p>
<ul style="margin-top:0in" type="circle">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo3">If no such available server, just go ahead without cache, but don’t fail it.<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo3">Schedule based on flavor (need more design here with Nova expert)<o:p></o:p></li></ul>
</li></ul>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Could you please comment on cinder spec if any before cinder team getting this hammered out? Thank you so much.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Below is the performance test I did personally, using Optane SSD p4800x 750G as the cache. Setting fio with block size = 4k, iodepth=1 which is typical for latency measurement. The data may be slightly different on different environment,
just FYI:<o:p></o:p></p>
<p class="MsoNormal">In rand read test:<o:p></o:p></p>
<p class="MsoNormal"><img border="0" width="794" height="299" style="width:8.2708in;height:3.1145in" id="Picture_x0020_1" src="cid:image004.png@01D5D107.23D05610"><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In rand write test:<o:p></o:p></p>
<p class="MsoNormal"><img border="0" width="785" height="318" style="width:8.177in;height:3.3125in" id="Picture_x0020_2" src="cid:image006.png@01D5D107.23D05610"><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In 70% rand read + 30% rand write mix test:<o:p></o:p></p>
<p class="MsoNormal"><img border="0" width="810" height="316" style="width:8.4375in;height:3.2916in" id="Picture_x0020_3" src="cid:image009.png@01D5D107.23D05610"><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Ceph in above chart means the baseline test, as shown in path
<span lang="ZH-CN" style="font-family:DengXian">①</span><span lang="ZH-CN"> </span>
below.<o:p></o:p></p>
<p class="MsoNormal">Write-Through and Write-Back means open-cas cached, as shown in path<span lang="ZH-CN" style="font-family:DengXian">②</span>.<o:p></o:p></p>
<p class="MsoNormal"><img border="0" width="544" height="288" style="width:5.6666in;height:3.0in" id="Picture_x0020_4" src="cid:image010.png@01D5D0FD.83839680" alt="cid:image010.png@01D5D0FD.83839680"><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So even write-through mode, we still have lots of performance gains.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Some concerns maybe:<o:p></o:p></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3">In real world, the cache hit rate may cannot be so high (95% and more).<o:p></o:p>
<ul style="margin-top:0in" type="circle">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo3">It depends on how big the cache device is. But normally a fast SSD with x TB size or a persistent memory with 512G size is big enough for hot data cache.<o:p></o:p></li></ul>
</li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3">What will happen when backend storage is RDMA<o:p></o:p>
<ul style="margin-top:0in" type="circle">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo3">Pure RDMA network link latency would be as low as ~10us. If plus the disk io to the storage system, the final latency would be dozens of microseconds. So it’s not suitable for an fast
ssd(itself ~10us) to cache for RDMA volume, but we can use persistent memory (with latency about hundreds of nanosecond) to do the cache. I will do the measurement after Chinese new year.<o:p></o:p></li></ul>
</li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3">It’s a pain that ceph cannot be supported<o:p></o:p>
<ul style="margin-top:0in" type="circle">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo3">The major concern to mount ceph volume to host OS is the security considerations, right? It is a performance / security tradeoff for operators. In private cloud which OpenStack maybe
mostly used, trust host OS would may be not a big problem, anyway, other backends rather than ceph are still doing in this way. I’m not going to change ceph to mount to host OS in this spec, but it’s not difficult for customer to switch to this way, like they
customized other things.<o:p></o:p></li></ul>
</li></ul>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regards<o:p></o:p></p>
<p class="MsoNormal">LiangFang<o:p></o:p></p>
</div>
</body>
</html>