答复: 答复: [largescale-sig]scaling story

25 Oct 2024

      ...
<o:p> </o:p></span></p><p class=3DMsoNormal><span =
class=3D--l><span lang=3DEN-US>Hello =
Marc,<o:p></o:p></span></span></p><p class=3DMsoNormal><span =
class=3D--l><span lang=3DEN-US>    Sorry for late =
response.<o:p></o:p></span></span></p><p class=3DMsoNormal =
...
         </span><b><span =
lang=3DEN-US>RabbitMQ: </span></b><span class=3D--l><span =
lang=3DEN-US>yeah, the number of available file descriptors is =
significantly higher for the rabbitmq process, but in the test of =
concurrent create 3000 virtual machines case, we find =
nova/cinder/neutron components connections to rabbitmq is up to 10K on =
...
<o:p> </o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US =
...
<o:p> </o:p></span></p><div><div =
...
<span class=3D--l>)<br><br>A example:<br><br></span></span><span =
class=3D--l><span lang=3DEN-US style=3D'font-family:"Courier =
New"'>URL=3D</span></span><span lang=3DEN-US><a =
0	*H÷
 010	+0	*H÷
 $
åContent-Type: multipart/alternative;
	boundary="----=_NextPart_000_01ED_01DB2710.4F8B4990"

This is a multipart message in MIME format.

------=_NextPart_000_01ED_01DB2710.4F8B4990
Content-Type: text/plain;
	charset="gb2312"
Content-Transfer-Encoding: 8bit

Hello Marc,

    Sorry for late response.

Database: One DB connection consumes about 100KB memory, in actual testing,
the max connections is up to 15K, and including the buffer pool, the
database consumes up to 20G of memory.

         RabbitMQ: yeah, the number of available file descriptors is
significantly higher for the rabbitmq process, but in the test of concurrent
create 3000 virtual machines case, we find nova/cinder/neutron components
connections to rabbitmq is up to 10K on the management webpage, so we set
the max rabbitmq connection to 20K.

·¢ŒþÈË: Marc Schoechlin [mailto:ms@256bit.org]
·¢ËÍÊ±Œä: 2024Äê10ÔÂ22ÈÕ 2:29
ÊÕŒþÈË: Alex Song (ËÎÎÄÆœ) <songwenping@inspur.com>;
openstack-discuss@lists.openstack.org
Ö÷Ìâ: Re: ŽðžŽ: [largescale-sig]scaling story

Hello Alex,

Am 21.10.24 um 08:06 schrieb Alex Song (ËÎÎÄÆœ):

Database settings

10k DB connections consume up to 21G of memory, which only accounts for 10%
of the server's memory in our env and will not cause OOM risk.

I think that's a lot and I would size it differently if it were my setup.
The calculation only refers to the working memory that the Connections
roughly use. Not included in this calculation is, for example, the buffer
pool of the database itself.
In most cases, there are also processes that could use a lot of memory
outside the database itself.

RabbitMQ

The maximum number of RabbitMQ connections is 20000, which is obtained by
test in the 3000 node environment.

That is interesting. What was the limiting factor? Usually the number of
available file descriptors is significantly higher for the RabbitMQ process.
(see /proc/<pid>/limits)

Incidentally, I have had good experiences with RabbitMQ Perftest during
performance tests these days.
(https://perftest.rabbitmq.com/)

A example:

URL= <amqp://openstack:mypassword@10.10.21.12:5672>
"amqp://openstack:mypassword@10.10.21.12:5672"
docker run -it --net host --rm pivotalrabbitmq/ <perf-test:latest>
perf-test:latest  \
          --queue-pattern-from 1 --queue-pattern-to 500 \
          --producers 500 --consumers 15 \
          --variable-size 1000:30 \
          --variable-size 10000:20 \
          --variable-size 5000:45  \
          --quorum-queue --queue perftest  \
          --uri "$URL"

Respectful regards
Marc

------=_NextPart_000_01ED_01DB2710.4F8B4990
Content-Type: text/html;
	charset="gb2312"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" =
xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta =
http-equiv=3DContent-Type content=3D"text/html; charset=3Dgb2312"><meta =
name=3DGenerator content=3D"Microsoft Word 15 (filtered =
medium)"><style><!--
/* Font Definitions */
@font-face
	{font-family:=CB=CE=CC=E5;
	panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:"\@=CB=CE=CC=E5";
	panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
	{font-family:=CE=A2=C8=ED=D1=C5=BA=DA;
	panose-1:2 11 5 3 2 2 4 2 2 4;}
@font-face
	{font-family:"\@=CE=A2=C8=ED=D1=C5=BA=DA";
	panose-1:2 11 5 3 2 2 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0cm;
	margin-bottom:.0001pt;
	font-size:12.0pt;
	font-family:=CB=CE=CC=E5;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
p
	{mso-style-priority:99;
	mso-margin-top-alt:auto;
	margin-right:0cm;
	mso-margin-bottom-alt:auto;
	margin-left:0cm;
	font-size:12.0pt;
	font-family:=CB=CE=CC=E5;}
span.--l
	{mso-style-name:--l;}
span.--r
	{mso-style-name:--r;}
span.EmailStyle20
	{mso-style-type:personal;
	font-family:"Calibri",sans-serif;
	color:#1F497D;}
span.EmailStyle21
	{mso-style-type:personal-reply;
	font-family:"Calibri",sans-serif;
	color:#1F497D;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-size:10.0pt;}
@page WordSection1
	{size:612.0pt 792.0pt;
	margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]--></head><body lang=3DZH-CN link=3Dblue =
vlink=3Dpurple><div class=3DWordSection1><p class=3DMsoNormal><span =
lang=3DEN-US =
style=3D'font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D'=
style=3D'text-indent:21.0pt'><b><span lang=3DEN-US>Database: =
</span></b><span class=3D--l><span lang=3DEN-US>One DB connection =
consumes about 100KB memory, in actual testing, the max connections is =
up to 15K, and including the buffer pool, the database consumes up to =
20G of memory. </span></span><span lang=3DEN-US><o:p></o:p></span></p><p =
class=3DMsoNormal><span lang=3DEN-US =
style=3D'font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D'=
the management webpage, so we set the max rabbitmq connection to =
20K.</span></span><span lang=3DEN-US><o:p></o:p></span></p><p =
class=3DMsoNormal><span lang=3DEN-US =
style=3D'font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D'=
style=3D'font-size:10.5pt;font-family:"Calibri",sans-serif;color:#1F497D'=
style=3D'border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm =
0cm 0cm'><p class=3DMsoNormal><b><span =
style=3D'font-size:11.0pt;font-family:"=CE=A2=C8=ED=D1=C5=BA=DA",sans-ser=
if'>=B7=A2=BC=FE=C8=CB<span lang=3DEN-US>:</span></span></b><span =
lang=3DEN-US =
style=3D'font-size:11.0pt;font-family:"=CE=A2=C8=ED=D1=C5=BA=DA",sans-ser=
if'> Marc Schoechlin [mailto:ms@256bit.org] <br></span><b><span =
style=3D'font-size:11.0pt;font-family:"=CE=A2=C8=ED=D1=C5=BA=DA",sans-ser=
if'>=B7=A2=CB=CD=CA=B1=BC=E4<span lang=3DEN-US>:</span></span></b><span =
lang=3DEN-US =
style=3D'font-size:11.0pt;font-family:"=CE=A2=C8=ED=D1=C5=BA=DA",sans-ser=
if'> 2024</span><span =
style=3D'font-size:11.0pt;font-family:"=CE=A2=C8=ED=D1=C5=BA=DA",sans-ser=
if'>=C4=EA<span lang=3DEN-US>10</span>=D4=C2<span =
lang=3DEN-US>22</span>=C8=D5<span lang=3DEN-US> =
2:29<br></span><b>=CA=D5=BC=FE=C8=CB<span lang=3DEN-US>:</span></b><span =
lang=3DEN-US> Alex Song (</span>=CB=CE=CE=C4=C6=BD<span lang=3DEN-US>) =
<songwenping@inspur.com>; =
openstack-discuss@lists.openstack.org<br></span><b>=D6=F7=CC=E2<span =
lang=3DEN-US>:</span></b><span lang=3DEN-US> Re: =
</span>=B4=F0=B8=B4<span lang=3DEN-US>: [largescale-sig]scaling =
story<o:p></o:p></span></span></p></div></div><p class=3DMsoNormal><span =
lang=3DEN-US><o:p> </o:p></span></p><p><span lang=3DEN-US>Hello =
Alex,<o:p></o:p></span></p><div><p class=3DMsoNormal><span =
lang=3DEN-US>Am 21.10.24 um 08:06 schrieb Alex Song =
(</span>=CB=CE=CE=C4=C6=BD<span =
lang=3DEN-US>):<o:p></o:p></span></p></div><blockquote =
style=3D'margin-top:5.0pt;margin-bottom:5.0pt'><p><b><span =
lang=3DEN-US>Database settings</span></b><span =
lang=3DEN-US><o:p></o:p></span></p><p class=3DMsoNormal><span =
lang=3DEN-US>10k DB connections consume up to 21G of memory, which only =
accounts for 10% of the server's memory in our env and will not cause =
OOM risk.<o:p></o:p></span></p></blockquote><p class=3DMsoNormal><span =
lang=3DEN-US>I think that's a lot and I would size it differently if it =
were my setup.<br>The calculation only refers to the working memory that =
the Connections roughly use. Not included in this calculation is, for =
example, the buffer pool of the database itself.<br>In most cases, there =
are also processes that could use a lot of memory outside the database =
itself.<br><br><o:p></o:p></span></p><blockquote =
style=3D'margin-top:5.0pt;margin-bottom:5.0pt'><p =
class=3DMsoNormal><span lang=3DEN-US> <o:p></o:p></span></p><p =
class=3DMsoNormal><b><span lang=3DEN-US>RabbitMQ</span></b><span =
lang=3DEN-US><o:p></o:p></span></p><p class=3DMsoNormal><span =
lang=3DEN-US> <o:p></o:p></span></p><p class=3DMsoNormal><span =
lang=3DEN-US>The maximum number of RabbitMQ connections is 20000, which =
is obtained by test in the 3000 node =
environment.<o:p></o:p></span></p></blockquote><p =
class=3DMsoNormal><span class=3D--l><span lang=3DEN-US>That is =
interesting. What was the limiting factor? Usually the number of =
available file descriptors is significantly higher for the RabbitMQ =
process.<br>(see /proc/<pid>/limits)<br><br>Incidentally, I have =
had good experiences with RabbitMQ Perftest during performance tests =
these days.<br>(</span></span><span lang=3DEN-US><a =
href=3D"https://perftest.rabbitmq.com/">https://perftest.rabbitmq.com/</a=
href=3D"amqp://openstack:mypassword@10.10.21.12:5672"><span =
style=3D'font-family:"Courier =
New"'>"amqp://openstack:mypassword@10.10.21.12:5672"</span></a>=
</span><span class=3D--l><span lang=3DEN-US =
style=3D'font-family:"Courier New"'><br>docker run -it --net host --rm =
pivotalrabbitmq/</span></span><span lang=3DEN-US><a =
href=3D"perf-test:latest"><span style=3D'font-family:"Courier =
New"'>perf-test:latest</span></a></span><span class=3D--l><span =
lang=3DEN-US style=3D'font-family:"Courier New"'>  =
\<br>          =
--queue-pattern-f|rom 1 --queue-pattern-to 500 =
\<br>          --producers =
500 --consumers 15 =
\<br>          =
--variable-size 1000:30 =
\<br>          =
--variable-size 10000:20 =
\<br>          =
--variable-size 5000:45  =
\<br>          =
--quorum-queue --queue perftest  =
\<br>          --uri =
"$URL"</span><span lang=3DEN-US><br><br></span></span><span =
lang=3DEN-US><o:p></o:p></span></p><p><span class=3D--l><span =
lang=3DEN-US>Respectful regards <br>Marc<br><br></span></span><span =
lang=3DEN-US><o:p></o:p></span></p></div></body></html>
------=_NextPart_000_01ED_01DB2710.4F8B4990--
 0É0± xðáwIÊèë`{0
	*H÷
0Y10
	&ò,dcom10
	&ò,dlangchao10
	&ò,dhome10U	INSPUR-CA0
170109092830Z
340511122004Z0Y10
	&ò,dcom10
	&ò,dlangchao10
	&ò,dhome10U	INSPUR-CA0"0
	*H÷
0
«ä5ïc$Œ©æ'µ¯Þ6>úUKÛdÔ²Áe9Î~{BîÒLgD÷*wvVÊŠ/DýUj_xá\m/
óž=kÐ·éGÙœQ€ýx~Wùgk ÛÜøãÔ7É6NçÏ*?n°Ê²mhùè{Ã¯ôÌÆ	7üF-Î<@ÃÓÍ¬WçÅyåLZrF
 6~føÈ×T~$0d¡ýL|zšøW=ötÚ%ýq,¥Ã~Ÿ"ÀýŸÑö2T,QÕÔ,dºÂÅ^§ÈôïåJ)ëVvp
Ó£00	+7CA0U0Uÿ0ÿ0U^YŠŽLX`Nöµ¥9Š2Á5j0	+70#	+7&a°$öz(¶o§K0
	*H÷
daòYÙ~×ì×NÑ3ŽËlP±i!¿ýÅòsÝLºP;I4hžØþ¡L¶äŠJêà·Õ:1àEÕs®9UÛ:8Mh{Ä²ìºŒ£·ÚŒó×÷C_ŸhoÏúÆËYM&UØ³?9êèr·ìgP_÷mÇ#YÖpîw0º¹«0$p³jºnº¥¯/Ôä¬÷Ú
ŠÖ5uþQÝ¶h(y¶ènnã%E,øú&Zâ"dKØ³ãi5VúP9±œ/ñ
õkØŸsá/Xðï»¯€µ³O÷Í-ó¿Mè³3Óž{G(ªŸßŒG0³0 ~d¢LÅ££j¿d¢0
	*H÷
0Y10
	&ò,dcom10
	&ò,dlangchao10
	&ò,dhome10U	INSPUR-CA0
241015081309Z
291014081309Z0¯10
	&ò,dcom10
	&ò,dlangchao10
	&ò,dhome1-0+U$æµåæµªæœ®æ°æ®ææ¯æéå¬åž10U	å®æå¹³1%0#	*H÷
	songwenping@inspur.com0"0
	*H÷
0
ÓÀ3²\ž¹JÇUê²	x| ËÃê«T\]µmK¿§'ËXÙ*çö Z;9ìÒhmçIæF©[.« õ9ò¯¶:üœ:Õ0ËÃ°7w-bŽô¿ýËÙßéèã&Æ®µ³«Îá1q×\Ï
ÜÀosô5vÓI[£Ìª:Pè¥~%Ú³%¡øþzæ]go+or{Ø
Üðë(o»#»ÓÌÞ2£ÿìöŒw? úE¡Àþí±à ±Œë±÷0cÃ1,«oã3?FpBšn'QÃ]÷y{èÖ#dŒ±£00=	+700.&+7ò©×z©=÷Ø\Jý&§Mda0)U%"0 ++
+7
0U 05	+7
(0&0
+0
+0
+7
0D	*H÷
	7050*H÷
0*H÷
0+0
*H÷
0IUB0@ &
+7 songwenping@inspur.comsongwenping@inspur.com0UÝµ>}6ÞúÏÀoÄÌ¹P0U#0^YŠŽLX`Nöµ¥9Š2Á5j0U00ÿ ü ùºldap:///CN=INSPUR-CA,CN=JTCA2012,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=home,DC=langchao,DC=com?certificateRevocationList?base?objectClass=cRLDistributionPoint:http://JTCA2012.home.langchao.com/CertEnroll/INSPUR-CA.crl0,+00±+0€ldap:///CN=INSPUR-CA,CN=AIA,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=home,DC=langchao,DC=com?cACertificate?base?objectClass=certificationAuthority0d+0Xhttp://JTCA2012.home.langchao.com/CertEnroll/JTCA2012.home.langchao.com_INSPUR-CA(1).crt0S	+7F0D B
+7 42S-1-5-21-1606980848-706699826-1801674531-2274525320
	*H÷
sMIú;E}auÅÚ'õÛ<ÿ!b27óu"ãìïwªBÉî9¶zÎ
|Ý»H+4šÏçÉxè
§!Ó·-ÏâoU
ŽlÍààV«"Ý5¢lzÝdcŽäøw^¯uâ+ßkäÃZì§q×rKéOæþä},öõ}:«óàuSºËuÏ±³ûËõ._À0é[¢'m°@eV>Ì-Ú÷8Š$a>á^iiªßçüÏø(Ç;bÌºkMÝé
ÌrðÂm1MvhL£ÈC|?100p0Y10
	&ò,dcom10
	&ò,dlangchao10
	&ò,dhome10U	INSPUR-CA~d¢LÅ££j¿d¢0	+ ø0	*H÷
	1	*H÷
0	*H÷
	1
241025110134Z0#	*H÷
	1ôÓâÙ"2Ë}SçÅø
:ò0	+71r0p0Y10
	&ò,dcom10
	&ò,dlangchao10
	&ò,dhome10U	INSPUR-CA~d¢LÅ££j¿d¢0*H÷
	1r p0Y10
	&ò,dcom10
	&ò,dlangchao10
	&ò,dhome10U	INSPUR-CA~d¢LÅ££j¿d¢0	*H÷
	100	`He*0	`He0
*H÷
0	`He0*H÷
0
*H÷
@0+0	`He0	`He0	`He0
	*H÷
/GIÓE¯bxb¥£U5:bMØéÂÌQ$]Ãx«áXîLOË4	œaPÑíYìCUøUOõž^±nV«_]9m1€iñÂ}p;Ô¥¶¥æÔŸi¶Æ·á§,ÿÖ6DaöÓ6;:jçŸÍ{ã¥Æ+tö*(gÀ1f°Ê²:ÇHä©xqŸC°@ètÍáTãurÐæ6*ÙÓ1Pr ÓGÐGÞ¥µ¶«Ø~e¿4 }j/ç-öIJÔår@î±ÙßÂ{0I.g,ÀÞÔ²2sM§

答复: 答复: [largescale-sig]scaling story

Alex Song (宋文平)