[Openstack-operators] Ceph puppet module

Stuart Fox stuart at demonware.net
Tue Mar 3 06:57:12 UTC 2015 

Update 2

Running

* ceph auth del client.bootstrap-osd*

Then

*ceph auth add client.bootstrap-osd   -i
/var/lib/ceph/bootstrap-osd/ceph.keyring*

Allows me to run puppet and get an osd working

On Mon, Mar 2, 2015 at 10:37 PM, Stuart Fox <stuart at demonware.net> wrote:

> Sorry again ..... its been a trying week at work and I've only just had
> time to pick this up
>
> So, progress. I can now get the mons up and running. Took some fiddling
> with, the $cluster param breaks just about everything so had to stick with
> the default cluster name. I can live with that! I havent had to patch
> anything this time round.
>
> Next problem seems to be registering the osd_bootstrap key, the error is
>
> Notice:
> /Stage[main]/Cloud::Roles::Ofceph/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/returns:
> + ceph --name mon. --keyring /var/lib/ceph/mon/ceph-ceph-001/keyring auth
> add client.bootstrap-osd --in-file=/var/lib/ceph/bootstrap-osd/ceph.keyring
>
> Notice:
> /Stage[main]/Cloud::Roles::Ofceph/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/returns:
> libust[28614/28614]: Warning: HOME environment variable not set. Disabling
> LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305)
>
> Notice:
> /Stage[main]/Cloud::Roles::Ofceph/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/returns:
> Error EINVAL: entity client.bootstrap-osd exists but key does not match
>
> Error: /bin/true # comment to satisfy puppet syntax requirements
>
> set -ex
>
> ceph   --name 'mon.'   --keyring
> '/var/lib/ceph/mon/ceph-ceph-001/keyring'  auth add client.bootstrap-osd
> --in-file=/var/lib/ceph/bootstrap-osd/ceph.keyring returned 22 instead of
> one of [0]
>
> Error:
> /Stage[main]/Openfish::Roles::Ofceph/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/returns:
> change from notrun to 0 failed: /bin/true # comment to satisfy puppet
> syntax requirements
>
> set -ex
>
> ceph   --name 'mon.'   --keyring
> '/var/lib/ceph/mon/ceph-ceph-001/keyring'  auth add client.bootstrap-osd
> --in-file=/var/lib/ceph/bootstrap-osd/ceph.keyring returned 22 instead of
> one of [0]
>
>
> I havent gotten as far as the osd's or gateways yet
>
> Help greatly appreciated!
>
>
>
> On Tue, Feb 24, 2015 at 10:31 AM, David Gurtner <aldavud at crimson.ch>
> wrote:
>
>> Hi again
>>
>> Just to test if the modules generally work on Trusty I set up a three
>> node cluster with shared mon/osd using the configuration below and
>> that worked. I was using the master branch of the module and had
>> firewall disabled.
>>
>> Cheers,
>> David
>>
>>
>> $admin_key = 'AQCTg71RsNIHORAAW+O6FCMZWBjmVfMIPk3MhQ=='
>> $mon_key = 'AQDesGZSsC7KJBAAw+W/Z4eGSQGAIbxWjxjvfw=='
>> $bootstrap_osd_key = 'AQABsWZSgEDmJhAAkAGSOOAJwrMHrM5Pz5On1A=='
>> $fsid = '066F558C-6789-4A93-AAF1-5AF1BA01A3AD'
>>
>> node /first|second|third/ {
>>   class { 'ceph::repo': }
>>   class { 'ceph':
>>     fsid                      => $fsid,
>>     mon_initial_members       => 'first,second,third',
>>     mon_host                  => '10.11.12.2,10.11.12.3,10.11.12.4',
>>   }
>>   ceph::mon { $::hostname:
>>     key => $mon_key,
>>   }
>>   Ceph::Key {
>>     inject         => true,
>>     inject_as_id   => 'mon.',
>>     inject_keyring => "/var/lib/ceph/mon/ceph-${::hostname}/keyring",
>>   }
>>   ceph::key { 'client.admin':
>>     secret  => $admin_key,
>>     cap_mon => 'allow *',
>>     cap_osd => 'allow *',
>>     cap_mds => 'allow',
>>   }
>>   ceph::key { 'client.bootstrap-osd':
>>     secret       => $bootstrap_osd_key,
>>     keyring_path => '/var/lib/ceph/bootstrap-osd/ceph.keyring',
>>     cap_mon      => 'allow profile bootstrap-osd',
>>   }
>>   ceph::osd { '/dev/sdb': }
>> }
>>
>>
>> On Mon, Feb 23, 2015 at 11:39 PM, David Gurtner <aldavud at crimson.ch>
>> wrote:
>> > Hi Stuart
>> >
>> > osds can only come up once at least the initial mon services are
>> > running and formed a quorum, so lets concentrate at getting the mons
>> > up first.
>> > If using cephx authentication all mon services need to be started at
>> > the same time (a similar time), translated to puppet this means all
>> > the puppet runs on the mons need to be started within a certain time
>> > frame (before the timeout of the mons trying to connect each other).
>> > The reason being that the mons first need to form a quorum before most
>> > of the further actions can happen.
>> >
>> > Can you provide some additional information about your setup?
>> Specifically:
>> > - what's your exact puppet modules configuration
>> > - what version are you running?
>> > - please be aware that running on Ubuntu 14.04 is currently not
>> > getting integration tested (currently only Ubuntu 12.04 and CentOS 6
>> > are tested, support for CentOS 7 is in the works).
>> > - how many mons are you running
>> > - are the osds running on the same hosts as the mons
>> > - do you use a single network, or do you use separate cluster/public
>> networks
>> > - do you have a firewall running, if yes, can you share your firewall
>> > configuration (iptables -vnL)
>> > - what exactly did you need to patch?
>> >
>> > Cheers,
>> > David
>> >
>> > On Mon, Feb 23, 2015 at 8:16 PM, Stuart Fox <stuart at demonware.net>
>> wrote:
>> >> Sorry to ask a question and run, I had a busy weekend. I'll be back on
>> the
>> >> ceph POC later today.
>> >>
>> >> I'll paste some outputs from the puppet runs shortly but essentially,
>> no mon
>> >> or osd comes up.
>> >> I had to patch several manifests to explicitly reference
>> /etc/ceph/ceph.conf
>> >> and given that the puppet module calls a reasonable amount of shell
>> code,
>> >> getting debug info is not easy. The puppet runs exit correctly but no
>> >> service comes up.
>> >> I'm using an almost cut'n'paste from USECASES.md, even down to keeping
>> the
>> >> keys the same.
>> >>
>> >> BR,
>> >> Stuart
>> >>
>> >> On Sun, Feb 22, 2015 at 6:57 AM, David Moreau Simard <
>> dmsimard at iweb.com>
>> >> wrote:
>> >>>
>> >>> Hey Stuart,
>> >>>
>> >>> You might want to look at (and use, if you want) the built-in roles
>> and
>> >>> profiles layer [1] which gives you a good idea of how the module is
>> used.
>> >>>
>> >>> It leverages ceph::profile::params to pass values [2] to other classes
>> >>> such as ceph::profile::osd [3].
>> >>>
>> >>> Another good place to look at how to use the module would be the
>> >>> integration tests [4].
>> >>> Since the module is integration tested, this means we actually use the
>> >>> module to deploy a virtual Ceph cluster and we test that the cluster
>> works -
>> >>> see for example the tests for ceph::profile::osd [4].
>> >>>
>> >>> I'm dmsimard on #puppet-openstack and #openstack-operators if you
>> need a
>> >>> hand to get this to work.
>> >>> This is still a relatively new module in comparison to the likes of
>> >>> puppet-nova and such. Feedback and contributions are appreciated!
>> >>>
>> >>> [1]
>> >>>
>> https://github.com/stackforge/puppet-ceph/tree/master/manifests/profile
>> >>> [2]
>> >>>
>> https://github.com/stackforge/puppet-ceph/blob/master/manifests/profile/params.pp
>> >>> [3]
>> >>>
>> https://github.com/stackforge/puppet-ceph/blob/master/manifests/profile/osd.pp
>> >>> [4]
>> >>>
>> https://github.com/stackforge/puppet-ceph/blob/master/spec/system/ceph_profile_osd_spec.rb
>> >>> --
>> >>> David Moreau Simard
>> >>>
>> >>> From: Andrew Woodward <xarses at gmail.com>
>> >>> Date: Saturday, February 21, 2015 at 4:39 PM
>> >>> To: Stuart Fox <stuart at demonware.net>
>> >>> Cc: "openstack-operators at lists.openstack.org"
>> >>> <openstack-operators at lists.openstack.org>, "
>> puppet-openstack at puppetlabs.com"
>> >>> <puppet-openstack at puppetlabs.com>
>> >>> Subject: Re: [Openstack-operators] Ceph puppet module
>> >>>
>> >>> [+puppet-openstack ml]
>> >>>
>> >>> Stuart,
>> >>>
>> >>> Please review the USECASES.md such as [1]. You should find usable
>> examples
>> >>> there. If you are still having problems please reach out with a more
>> detail
>> >>> about what you configuration you are attempting to deploy. The module
>> >>> authors can be found in #puppet-openstack and are usually lurking on
>> the
>> >>> puppet-openstack mailing list (CC'd).
>> >>>
>> >>>
>> >>> [1]
>> https://github.com/stackforge/puppet-ceph/blob/master/USECASES.md#i-want-to-operate-a-production-cluster
>> >>>
>> >>>
>> >>>
>> >>> On Sat, Feb 21, 2015 at 12:04 AM, Stuart Fox <stuart at demonware.net>
>> wrote:
>> >>>>
>> >>>> Hey all
>> >>>>
>> >>>> Im having a complete nightmare trying to get ceph deployed using the
>> >>>> https://github.com/stackforge/puppet-ceph module. Scant
>> documentation isn't
>> >>>> helping my cause!
>> >>>>
>> >>>> Is anybody else using this module? Im attempting to deploy ceph as a
>> >>>> cinder backend in Juno on Ubuntu 14.04 although I haven't gotten as
>> far as
>> >>>> integration yet.
>> >>>>
>> >>>> Does anyone have working example's I could look at?
>> >>>>
>> >>>> --
>> >>>> BR,
>> >>>> Stuart
>> >>>>
>> >>>> _______________________________________________
>> >>>> OpenStack-operators mailing list
>> >>>> OpenStack-operators at lists.openstack.org
>> >>>>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Andrew
>> >>> Mirantis
>> >>> Fuel community ambassador
>> >>> Ceph community
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> BR,
>> >> Stuart
>> >>
>> >> _______________________________________________
>> >> OpenStack-operators mailing list
>> >> OpenStack-operators at lists.openstack.org
>> >>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> >>
>>
>
>
>
> --
> BR,
> Stuart
>



-- 
BR,
Stuart
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20150302/8849af92/attachment.html>

More information about the OpenStack-operators mailing list