<div dir="ltr">Sorry for the spam, silence for a week then there's no shutting me up :)<div><br></div><div>Apart from registering the osd bootstrap key, I now have a running 3 node cluster</div><div>
<p class=""><span class="">2015-03-03 07:01:40.484418 mon.0 [INF] pgmap v20: 64 pgs: 64 active+clean; 0 bytes data, 101 MB used, 1498 GB / 1498 GB avail</span></p></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 2, 2015 at 10:57 PM, Stuart Fox <span dir="ltr"><<a href="mailto:stuart@demonware.net" target="_blank">stuart@demonware.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Update 2<div><br></div><div>Running</div><div>
<p><b><span> </span><span>ceph auth del client.bootstrap-osd</span></b></p><p><span>Then</span></p><p><span>
</span></p><p><span><b>ceph auth add client.bootstrap-osd -i /var/lib/ceph/bootstrap-osd/ceph.keyring</b></span></p><p>Allows me to run puppet and get an osd working</p></div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Mon, Mar 2, 2015 at 10:37 PM, Stuart Fox <span dir="ltr"><<a href="mailto:stuart@demonware.net" target="_blank">stuart@demonware.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Sorry again ..... its been a trying week at work and I've only just had time to pick this up<div><br></div><div>So, progress. I can now get the mons up and running. Took some fiddling with, the $cluster param breaks just about everything so had to stick with the default cluster name. I can live with that! I havent had to patch anything this time round.</div><div><br></div><div>Next problem seems to be registering the osd_bootstrap key, the error is </div><div><br></div><div>
<p><span>Notice: /Stage[main]/Cloud::Roles::Ofceph/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/returns: + ceph --name mon. --keyring /var/lib/ceph/mon/ceph-ceph-001/keyring auth add client.bootstrap-osd --in-file=/var/lib/ceph/bootstrap-osd/ceph.keyring</span></p>
<p><span>Notice: /Stage[main]/Cloud::Roles::Ofceph/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/returns: libust[28614/28614]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305)</span></p>
<p><span>Notice: /Stage[main]/Cloud::Roles::Ofceph/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/returns: Error EINVAL: entity client.bootstrap-osd exists but key does not match</span></p>
<p><span>Error: /bin/true # comment to satisfy puppet syntax requirements</span></p>
<p><span>set -ex</span></p>
<p><span>ceph --name 'mon.' --keyring '/var/lib/ceph/mon/ceph-ceph-001/keyring' auth add client.bootstrap-osd --in-file=/var/lib/ceph/bootstrap-osd/ceph.keyring returned 22 instead of one of [0]</span></p>
<p><span>Error: /Stage[main]/Openfish::Roles::Ofceph/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/returns: change from notrun to 0 failed: /bin/true # comment to satisfy puppet syntax requirements</span></p>
<p><span>set -ex</span></p>
<p><span>ceph --name 'mon.' --keyring '/var/lib/ceph/mon/ceph-ceph-001/keyring' auth add client.bootstrap-osd --in-file=/var/lib/ceph/bootstrap-osd/ceph.keyring returned 22 instead of one of [0]</span></p><p><br></p><p>I havent gotten as far as the osd's or gateways yet</p><p>Help greatly appreciated!</p><p><br></p></div></div><div class="gmail_extra"><div><div><br><div class="gmail_quote">On Tue, Feb 24, 2015 at 10:31 AM, David Gurtner <span dir="ltr"><<a href="mailto:aldavud@crimson.ch" target="_blank">aldavud@crimson.ch</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi again<br>
<br>
Just to test if the modules generally work on Trusty I set up a three<br>
node cluster with shared mon/osd using the configuration below and<br>
that worked. I was using the master branch of the module and had<br>
firewall disabled.<br>
<br>
Cheers,<br>
David<br>
<br>
<br>
$admin_key = 'AQCTg71RsNIHORAAW+O6FCMZWBjmVfMIPk3MhQ=='<br>
$mon_key = 'AQDesGZSsC7KJBAAw+W/Z4eGSQGAIbxWjxjvfw=='<br>
$bootstrap_osd_key = 'AQABsWZSgEDmJhAAkAGSOOAJwrMHrM5Pz5On1A=='<br>
$fsid = '066F558C-6789-4A93-AAF1-5AF1BA01A3AD'<br>
<br>
node /first|second|third/ {<br>
class { 'ceph::repo': }<br>
class { 'ceph':<br>
fsid => $fsid,<br>
mon_initial_members => 'first,second,third',<br>
mon_host => '10.11.12.2,10.11.12.3,10.11.12.4',<br>
}<br>
ceph::mon { $::hostname:<br>
key => $mon_key,<br>
}<br>
Ceph::Key {<br>
inject => true,<br>
inject_as_id => 'mon.',<br>
inject_keyring => "/var/lib/ceph/mon/ceph-${::hostname}/keyring",<br>
}<br>
ceph::key { 'client.admin':<br>
secret => $admin_key,<br>
cap_mon => 'allow *',<br>
cap_osd => 'allow *',<br>
cap_mds => 'allow',<br>
}<br>
ceph::key { 'client.bootstrap-osd':<br>
secret => $bootstrap_osd_key,<br>
keyring_path => '/var/lib/ceph/bootstrap-osd/ceph.keyring',<br>
cap_mon => 'allow profile bootstrap-osd',<br>
}<br>
ceph::osd { '/dev/sdb': }<br>
<div><div>}<br>
<br>
<br>
On Mon, Feb 23, 2015 at 11:39 PM, David Gurtner <<a href="mailto:aldavud@crimson.ch" target="_blank">aldavud@crimson.ch</a>> wrote:<br>
> Hi Stuart<br>
><br>
> osds can only come up once at least the initial mon services are<br>
> running and formed a quorum, so lets concentrate at getting the mons<br>
> up first.<br>
> If using cephx authentication all mon services need to be started at<br>
> the same time (a similar time), translated to puppet this means all<br>
> the puppet runs on the mons need to be started within a certain time<br>
> frame (before the timeout of the mons trying to connect each other).<br>
> The reason being that the mons first need to form a quorum before most<br>
> of the further actions can happen.<br>
><br>
> Can you provide some additional information about your setup? Specifically:<br>
> - what's your exact puppet modules configuration<br>
> - what version are you running?<br>
> - please be aware that running on Ubuntu 14.04 is currently not<br>
> getting integration tested (currently only Ubuntu 12.04 and CentOS 6<br>
> are tested, support for CentOS 7 is in the works).<br>
> - how many mons are you running<br>
> - are the osds running on the same hosts as the mons<br>
> - do you use a single network, or do you use separate cluster/public networks<br>
> - do you have a firewall running, if yes, can you share your firewall<br>
> configuration (iptables -vnL)<br>
> - what exactly did you need to patch?<br>
><br>
> Cheers,<br>
> David<br>
><br>
> On Mon, Feb 23, 2015 at 8:16 PM, Stuart Fox <<a href="mailto:stuart@demonware.net" target="_blank">stuart@demonware.net</a>> wrote:<br>
>> Sorry to ask a question and run, I had a busy weekend. I'll be back on the<br>
>> ceph POC later today.<br>
>><br>
>> I'll paste some outputs from the puppet runs shortly but essentially, no mon<br>
>> or osd comes up.<br>
>> I had to patch several manifests to explicitly reference /etc/ceph/ceph.conf<br>
>> and given that the puppet module calls a reasonable amount of shell code,<br>
>> getting debug info is not easy. The puppet runs exit correctly but no<br>
>> service comes up.<br>
>> I'm using an almost cut'n'paste from USECASES.md, even down to keeping the<br>
>> keys the same.<br>
>><br>
>> BR,<br>
>> Stuart<br>
>><br>
>> On Sun, Feb 22, 2015 at 6:57 AM, David Moreau Simard <<a href="mailto:dmsimard@iweb.com" target="_blank">dmsimard@iweb.com</a>><br>
>> wrote:<br>
>>><br>
>>> Hey Stuart,<br>
>>><br>
>>> You might want to look at (and use, if you want) the built-in roles and<br>
>>> profiles layer [1] which gives you a good idea of how the module is used.<br>
>>><br>
>>> It leverages ceph::profile::params to pass values [2] to other classes<br>
>>> such as ceph::profile::osd [3].<br>
>>><br>
>>> Another good place to look at how to use the module would be the<br>
>>> integration tests [4].<br>
>>> Since the module is integration tested, this means we actually use the<br>
>>> module to deploy a virtual Ceph cluster and we test that the cluster works -<br>
>>> see for example the tests for ceph::profile::osd [4].<br>
>>><br>
>>> I'm dmsimard on #puppet-openstack and #openstack-operators if you need a<br>
>>> hand to get this to work.<br>
>>> This is still a relatively new module in comparison to the likes of<br>
>>> puppet-nova and such. Feedback and contributions are appreciated!<br>
>>><br>
>>> [1]<br>
>>> <a href="https://github.com/stackforge/puppet-ceph/tree/master/manifests/profile" target="_blank">https://github.com/stackforge/puppet-ceph/tree/master/manifests/profile</a><br>
>>> [2]<br>
>>> <a href="https://github.com/stackforge/puppet-ceph/blob/master/manifests/profile/params.pp" target="_blank">https://github.com/stackforge/puppet-ceph/blob/master/manifests/profile/params.pp</a><br>
>>> [3]<br>
>>> <a href="https://github.com/stackforge/puppet-ceph/blob/master/manifests/profile/osd.pp" target="_blank">https://github.com/stackforge/puppet-ceph/blob/master/manifests/profile/osd.pp</a><br>
>>> [4]<br>
>>> <a href="https://github.com/stackforge/puppet-ceph/blob/master/spec/system/ceph_profile_osd_spec.rb" target="_blank">https://github.com/stackforge/puppet-ceph/blob/master/spec/system/ceph_profile_osd_spec.rb</a><br>
>>> --<br>
>>> David Moreau Simard<br>
>>><br>
>>> From: Andrew Woodward <<a href="mailto:xarses@gmail.com" target="_blank">xarses@gmail.com</a>><br>
>>> Date: Saturday, February 21, 2015 at 4:39 PM<br>
>>> To: Stuart Fox <<a href="mailto:stuart@demonware.net" target="_blank">stuart@demonware.net</a>><br>
>>> Cc: "<a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.openstack.org</a>"<br>
>>> <<a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.openstack.org</a>>, "<a href="mailto:puppet-openstack@puppetlabs.com" target="_blank">puppet-openstack@puppetlabs.com</a>"<br>
>>> <<a href="mailto:puppet-openstack@puppetlabs.com" target="_blank">puppet-openstack@puppetlabs.com</a>><br>
>>> Subject: Re: [Openstack-operators] Ceph puppet module<br>
>>><br>
>>> [+puppet-openstack ml]<br>
>>><br>
>>> Stuart,<br>
>>><br>
>>> Please review the USECASES.md such as [1]. You should find usable examples<br>
>>> there. If you are still having problems please reach out with a more detail<br>
>>> about what you configuration you are attempting to deploy. The module<br>
>>> authors can be found in #puppet-openstack and are usually lurking on the<br>
>>> puppet-openstack mailing list (CC'd).<br>
>>><br>
>>><br>
>>> [1]<a href="https://github.com/stackforge/puppet-ceph/blob/master/USECASES.md#i-want-to-operate-a-production-cluster" target="_blank">https://github.com/stackforge/puppet-ceph/blob/master/USECASES.md#i-want-to-operate-a-production-cluster</a><br>
>>><br>
>>><br>
>>><br>
>>> On Sat, Feb 21, 2015 at 12:04 AM, Stuart Fox <<a href="mailto:stuart@demonware.net" target="_blank">stuart@demonware.net</a>> wrote:<br>
>>>><br>
>>>> Hey all<br>
>>>><br>
>>>> Im having a complete nightmare trying to get ceph deployed using the<br>
>>>> <a href="https://github.com/stackforge/puppet-ceph" target="_blank">https://github.com/stackforge/puppet-ceph</a> module. Scant documentation isn't<br>
>>>> helping my cause!<br>
>>>><br>
>>>> Is anybody else using this module? Im attempting to deploy ceph as a<br>
>>>> cinder backend in Juno on Ubuntu 14.04 although I haven't gotten as far as<br>
>>>> integration yet.<br>
>>>><br>
>>>> Does anyone have working example's I could look at?<br>
>>>><br>
>>>> --<br>
>>>> BR,<br>
>>>> Stuart<br>
>>>><br>
>>>> _______________________________________________<br>
>>>> OpenStack-operators mailing list<br>
>>>> <a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
>>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
>>>><br>
>>><br>
>>><br>
>>><br>
>>> --<br>
>>> Andrew<br>
>>> Mirantis<br>
>>> Fuel community ambassador<br>
>>> Ceph community<br>
>><br>
>><br>
>><br>
>><br>
>> --<br>
>> BR,<br>
>> Stuart<br>
>><br>
>> _______________________________________________<br>
>> OpenStack-operators mailing list<br>
>> <a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
>><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div><div dir="ltr">BR,<div>Stuart</div></div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br><div><div dir="ltr">BR,<div>Stuart</div></div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">BR,<div>Stuart</div></div></div>
</div>