[Openstack-operators] Ceph puppet module
Stuart Fox
stuart at demonware.net
Tue Mar 3 07:03:51 UTC 2015
Sorry for the spam, silence for a week then there's no shutting me up :)
Apart from registering the osd bootstrap key, I now have a running 3 node
cluster
2015-03-03 07:01:40.484418 mon.0 [INF] pgmap v20: 64 pgs: 64 active+clean;
0 bytes data, 101 MB used, 1498 GB / 1498 GB avail
On Mon, Mar 2, 2015 at 10:57 PM, Stuart Fox <stuart at demonware.net> wrote:
> Update 2
>
> Running
>
> * ceph auth del client.bootstrap-osd*
>
> Then
>
> *ceph auth add client.bootstrap-osd -i
> /var/lib/ceph/bootstrap-osd/ceph.keyring*
>
> Allows me to run puppet and get an osd working
>
> On Mon, Mar 2, 2015 at 10:37 PM, Stuart Fox <stuart at demonware.net> wrote:
>
>> Sorry again ..... its been a trying week at work and I've only just had
>> time to pick this up
>>
>> So, progress. I can now get the mons up and running. Took some fiddling
>> with, the $cluster param breaks just about everything so had to stick with
>> the default cluster name. I can live with that! I havent had to patch
>> anything this time round.
>>
>> Next problem seems to be registering the osd_bootstrap key, the error is
>>
>> Notice:
>> /Stage[main]/Cloud::Roles::Ofceph/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/returns:
>> + ceph --name mon. --keyring /var/lib/ceph/mon/ceph-ceph-001/keyring auth
>> add client.bootstrap-osd --in-file=/var/lib/ceph/bootstrap-osd/ceph.keyring
>>
>> Notice:
>> /Stage[main]/Cloud::Roles::Ofceph/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/returns:
>> libust[28614/28614]: Warning: HOME environment variable not set. Disabling
>> LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305)
>>
>> Notice:
>> /Stage[main]/Cloud::Roles::Ofceph/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/returns:
>> Error EINVAL: entity client.bootstrap-osd exists but key does not match
>>
>> Error: /bin/true # comment to satisfy puppet syntax requirements
>>
>> set -ex
>>
>> ceph --name 'mon.' --keyring
>> '/var/lib/ceph/mon/ceph-ceph-001/keyring' auth add client.bootstrap-osd
>> --in-file=/var/lib/ceph/bootstrap-osd/ceph.keyring returned 22 instead of
>> one of [0]
>>
>> Error:
>> /Stage[main]/Openfish::Roles::Ofceph/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/returns:
>> change from notrun to 0 failed: /bin/true # comment to satisfy puppet
>> syntax requirements
>>
>> set -ex
>>
>> ceph --name 'mon.' --keyring
>> '/var/lib/ceph/mon/ceph-ceph-001/keyring' auth add client.bootstrap-osd
>> --in-file=/var/lib/ceph/bootstrap-osd/ceph.keyring returned 22 instead of
>> one of [0]
>>
>>
>> I havent gotten as far as the osd's or gateways yet
>>
>> Help greatly appreciated!
>>
>>
>>
>> On Tue, Feb 24, 2015 at 10:31 AM, David Gurtner <aldavud at crimson.ch>
>> wrote:
>>
>>> Hi again
>>>
>>> Just to test if the modules generally work on Trusty I set up a three
>>> node cluster with shared mon/osd using the configuration below and
>>> that worked. I was using the master branch of the module and had
>>> firewall disabled.
>>>
>>> Cheers,
>>> David
>>>
>>>
>>> $admin_key = 'AQCTg71RsNIHORAAW+O6FCMZWBjmVfMIPk3MhQ=='
>>> $mon_key = 'AQDesGZSsC7KJBAAw+W/Z4eGSQGAIbxWjxjvfw=='
>>> $bootstrap_osd_key = 'AQABsWZSgEDmJhAAkAGSOOAJwrMHrM5Pz5On1A=='
>>> $fsid = '066F558C-6789-4A93-AAF1-5AF1BA01A3AD'
>>>
>>> node /first|second|third/ {
>>> class { 'ceph::repo': }
>>> class { 'ceph':
>>> fsid => $fsid,
>>> mon_initial_members => 'first,second,third',
>>> mon_host => '10.11.12.2,10.11.12.3,10.11.12.4',
>>> }
>>> ceph::mon { $::hostname:
>>> key => $mon_key,
>>> }
>>> Ceph::Key {
>>> inject => true,
>>> inject_as_id => 'mon.',
>>> inject_keyring => "/var/lib/ceph/mon/ceph-${::hostname}/keyring",
>>> }
>>> ceph::key { 'client.admin':
>>> secret => $admin_key,
>>> cap_mon => 'allow *',
>>> cap_osd => 'allow *',
>>> cap_mds => 'allow',
>>> }
>>> ceph::key { 'client.bootstrap-osd':
>>> secret => $bootstrap_osd_key,
>>> keyring_path => '/var/lib/ceph/bootstrap-osd/ceph.keyring',
>>> cap_mon => 'allow profile bootstrap-osd',
>>> }
>>> ceph::osd { '/dev/sdb': }
>>> }
>>>
>>>
>>> On Mon, Feb 23, 2015 at 11:39 PM, David Gurtner <aldavud at crimson.ch>
>>> wrote:
>>> > Hi Stuart
>>> >
>>> > osds can only come up once at least the initial mon services are
>>> > running and formed a quorum, so lets concentrate at getting the mons
>>> > up first.
>>> > If using cephx authentication all mon services need to be started at
>>> > the same time (a similar time), translated to puppet this means all
>>> > the puppet runs on the mons need to be started within a certain time
>>> > frame (before the timeout of the mons trying to connect each other).
>>> > The reason being that the mons first need to form a quorum before most
>>> > of the further actions can happen.
>>> >
>>> > Can you provide some additional information about your setup?
>>> Specifically:
>>> > - what's your exact puppet modules configuration
>>> > - what version are you running?
>>> > - please be aware that running on Ubuntu 14.04 is currently not
>>> > getting integration tested (currently only Ubuntu 12.04 and CentOS 6
>>> > are tested, support for CentOS 7 is in the works).
>>> > - how many mons are you running
>>> > - are the osds running on the same hosts as the mons
>>> > - do you use a single network, or do you use separate cluster/public
>>> networks
>>> > - do you have a firewall running, if yes, can you share your firewall
>>> > configuration (iptables -vnL)
>>> > - what exactly did you need to patch?
>>> >
>>> > Cheers,
>>> > David
>>> >
>>> > On Mon, Feb 23, 2015 at 8:16 PM, Stuart Fox <stuart at demonware.net>
>>> wrote:
>>> >> Sorry to ask a question and run, I had a busy weekend. I'll be back
>>> on the
>>> >> ceph POC later today.
>>> >>
>>> >> I'll paste some outputs from the puppet runs shortly but essentially,
>>> no mon
>>> >> or osd comes up.
>>> >> I had to patch several manifests to explicitly reference
>>> /etc/ceph/ceph.conf
>>> >> and given that the puppet module calls a reasonable amount of shell
>>> code,
>>> >> getting debug info is not easy. The puppet runs exit correctly but no
>>> >> service comes up.
>>> >> I'm using an almost cut'n'paste from USECASES.md, even down to
>>> keeping the
>>> >> keys the same.
>>> >>
>>> >> BR,
>>> >> Stuart
>>> >>
>>> >> On Sun, Feb 22, 2015 at 6:57 AM, David Moreau Simard <
>>> dmsimard at iweb.com>
>>> >> wrote:
>>> >>>
>>> >>> Hey Stuart,
>>> >>>
>>> >>> You might want to look at (and use, if you want) the built-in roles
>>> and
>>> >>> profiles layer [1] which gives you a good idea of how the module is
>>> used.
>>> >>>
>>> >>> It leverages ceph::profile::params to pass values [2] to other
>>> classes
>>> >>> such as ceph::profile::osd [3].
>>> >>>
>>> >>> Another good place to look at how to use the module would be the
>>> >>> integration tests [4].
>>> >>> Since the module is integration tested, this means we actually use
>>> the
>>> >>> module to deploy a virtual Ceph cluster and we test that the cluster
>>> works -
>>> >>> see for example the tests for ceph::profile::osd [4].
>>> >>>
>>> >>> I'm dmsimard on #puppet-openstack and #openstack-operators if you
>>> need a
>>> >>> hand to get this to work.
>>> >>> This is still a relatively new module in comparison to the likes of
>>> >>> puppet-nova and such. Feedback and contributions are appreciated!
>>> >>>
>>> >>> [1]
>>> >>>
>>> https://github.com/stackforge/puppet-ceph/tree/master/manifests/profile
>>> >>> [2]
>>> >>>
>>> https://github.com/stackforge/puppet-ceph/blob/master/manifests/profile/params.pp
>>> >>> [3]
>>> >>>
>>> https://github.com/stackforge/puppet-ceph/blob/master/manifests/profile/osd.pp
>>> >>> [4]
>>> >>>
>>> https://github.com/stackforge/puppet-ceph/blob/master/spec/system/ceph_profile_osd_spec.rb
>>> >>> --
>>> >>> David Moreau Simard
>>> >>>
>>> >>> From: Andrew Woodward <xarses at gmail.com>
>>> >>> Date: Saturday, February 21, 2015 at 4:39 PM
>>> >>> To: Stuart Fox <stuart at demonware.net>
>>> >>> Cc: "openstack-operators at lists.openstack.org"
>>> >>> <openstack-operators at lists.openstack.org>, "
>>> puppet-openstack at puppetlabs.com"
>>> >>> <puppet-openstack at puppetlabs.com>
>>> >>> Subject: Re: [Openstack-operators] Ceph puppet module
>>> >>>
>>> >>> [+puppet-openstack ml]
>>> >>>
>>> >>> Stuart,
>>> >>>
>>> >>> Please review the USECASES.md such as [1]. You should find usable
>>> examples
>>> >>> there. If you are still having problems please reach out with a more
>>> detail
>>> >>> about what you configuration you are attempting to deploy. The module
>>> >>> authors can be found in #puppet-openstack and are usually lurking on
>>> the
>>> >>> puppet-openstack mailing list (CC'd).
>>> >>>
>>> >>>
>>> >>> [1]
>>> https://github.com/stackforge/puppet-ceph/blob/master/USECASES.md#i-want-to-operate-a-production-cluster
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Sat, Feb 21, 2015 at 12:04 AM, Stuart Fox <stuart at demonware.net>
>>> wrote:
>>> >>>>
>>> >>>> Hey all
>>> >>>>
>>> >>>> Im having a complete nightmare trying to get ceph deployed using the
>>> >>>> https://github.com/stackforge/puppet-ceph module. Scant
>>> documentation isn't
>>> >>>> helping my cause!
>>> >>>>
>>> >>>> Is anybody else using this module? Im attempting to deploy ceph as a
>>> >>>> cinder backend in Juno on Ubuntu 14.04 although I haven't gotten as
>>> far as
>>> >>>> integration yet.
>>> >>>>
>>> >>>> Does anyone have working example's I could look at?
>>> >>>>
>>> >>>> --
>>> >>>> BR,
>>> >>>> Stuart
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> OpenStack-operators mailing list
>>> >>>> OpenStack-operators at lists.openstack.org
>>> >>>>
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Andrew
>>> >>> Mirantis
>>> >>> Fuel community ambassador
>>> >>> Ceph community
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> BR,
>>> >> Stuart
>>> >>
>>> >> _______________________________________________
>>> >> OpenStack-operators mailing list
>>> >> OpenStack-operators at lists.openstack.org
>>> >>
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>> >>
>>>
>>
>>
>>
>> --
>> BR,
>> Stuart
>>
>
>
>
> --
> BR,
> Stuart
>
--
BR,
Stuart
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20150302/1204f5f9/attachment.html>
More information about the OpenStack-operators
mailing list