[openstack-dev] [fuel] librarian-puppet integration, need help with build tasks for fuel-library

Alex Schultz aschultz at mirantis.com
Mon Jul 20 15:31:55 UTC 2015


Vladimir,

Thanks.  Can you point me to the error for perestroika? I'd be happy to
take a look as well. I spent most of Friday throwing various options at the
CI system to try and figure out how to get the spec to work with the CI
fuel-library package building so perhaps there's a different way to handle
this in the spec.

-Alex

On Mon, Jul 20, 2015 at 10:02 AM, Vladimir Kozhukalov <
vkozhukalov at mirantis.com> wrote:

> Alex,
>
> As I've just found out this package available here [1] is not actually
> build with your patch (instead it is from previous successful build). Looks
> like "perestroika" can not build this package due to some environment
> related issues. I've poked Dmitry Burmistrov to check it out.
> However, your patch is OK, make system can build this package and ISO
> passes BVT tests.
>
>
> [1]
> http://172.18.160.74/osci/review/CR-202763/mos/7.0/fuel/base/centos6/Packages/fuel-library7.0-7.0.0-1.mos6888.git.bac86fe.noarch.rpm
>
> Vladimir Kozhukalov
>
> On Mon, Jul 20, 2015 at 4:04 PM, Vladimir Kozhukalov <
> vkozhukalov at mirantis.com> wrote:
>
>> Alex,
>>
>> Yes, spec is much better place to introduce this. BTW, "perestroika"
>> builds new package for every commit and patch set and publishes them via
>> HTTP. Please check here
>> http://172.18.160.74/osci/review/CR-202763/mos/7.0/fuel/base/centos6/Packages/fuel-library7.0-7.0.0-1.mos6888.git.bac86fe.noarch.rpm
>> if the package contains all necessary upstream modules.
>>
>> Vladimir Kozhukalov
>>
>> On Sat, Jul 18, 2015 at 3:28 AM, Alex Schultz <aschultz at mirantis.com>
>> wrote:
>>
>>> Not until we start using it then any ci that tests with that module will
>>> validate the modules inclusion. You can check the output of the jobs as we
>>> are printing what modules are managed by librarian.
>>>
>>> -Alex
>>> On Jul 17, 2015 6:17 PM, "Andrew Woodward" <xarses at gmail.com> wrote:
>>>
>>>> Fantastic, do we have some way to validate that the module was pulled
>>>> in properly as part of fuel-library CI?
>>>>
>>>> On Fri, Jul 17, 2015 at 2:48 PM Alex Schultz <aschultz at mirantis.com>
>>>> wrote:
>>>>
>>>>> Hey All,
>>>>>
>>>>> I've figured it out without having to modify the fuel-main build code.
>>>>> I've updated the fuel-library spec with a build action that invokes the
>>>>> script to pull down external modules.  Please take some time to review the
>>>>> two reviews out there for this change to see if there are any issues with
>>>>> the way it is implemented.
>>>>>
>>>>> https://review.openstack.org/#/c/202763/
>>>>> https://review.openstack.org/#/c/202767/
>>>>>
>>>>> This is a first step towards being able to pull in unmodified external
>>>>> puppet modules.
>>>>>
>>>>> Thanks,
>>>>> -Alex
>>>>>
>>>>> On Fri, Jul 17, 2015 at 4:23 PM, Andrew Woodward <
>>>>> awoodward at mirantis.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 17, 2015 at 12:24 PM Vladimir Kozhukalov <
>>>>>> vkozhukalov at mirantis.com> wrote:
>>>>>>
>>>>>>> Alex,
>>>>>>>
>>>>>>> Great that you did this. Now I think I can prepare fuel-main patch
>>>>>>> to invoke this script right before building fuel-library package. I'll add
>>>>>>> you to review it. Is it ok if I do this monday morning?
>>>>>>>
>>>>>>
>>>>>> Keep in minde we agreeded to a deadline to get this sorted and in
>>>>>> shape to land by EOD monday or we will have to retain the old, and crappy
>>>>>> fork method. If possible please work out how this needs to work as early as
>>>>>> possible so Alex can continue.
>>>>>>
>>>>>>
>>>>>>> Vladimir Kozhukalov
>>>>>>>
>>>>>>> On Fri, Jul 17, 2015 at 5:51 PM, Alex Schultz <aschultz at mirantis.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hey Vladimir,
>>>>>>>>
>>>>>>>> On Fri, Jul 17, 2015 at 7:33 AM, Vladimir Kozhukalov <
>>>>>>>> vkozhukalov at mirantis.com> wrote:
>>>>>>>>
>>>>>>>>> Alex,
>>>>>>>>>
>>>>>>>>> Gathering upstream modules certainly should be implemented as a
>>>>>>>>> separate script so as to make it possible to use it wherever we need this
>>>>>>>>> (tests, builds, etc.) According to builds there are two things
>>>>>>>>>
>>>>>>>>> 1) We have so called "perestroika" package build system (Dmitry
>>>>>>>>> Burmistrov is a main contributor here). By the end of next week we are
>>>>>>>>> going to switch building all the packages to "perestroika". And in order to
>>>>>>>>> gather upstream modules right before building fuel-library package, we need
>>>>>>>>> to change perestroika build scripts.
>>>>>>>>>
>>>>>>>>> 2) Currently we build packages using make system and you are right
>>>>>>>>> about the place where you need to make changes.
>>>>>>>>> https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82
>>>>>>>>> If you create shell script, I'll help you to add it to make code.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> I have updated my review[0] to extract the update logic to it's own
>>>>>>>> bash script that can be invoked by the build scripts.  Let me know what
>>>>>>>> would be the best way to wedge this in there.  I think for the
>>>>>>>> "perestroika" this would also be needed for the fuel-library build, so if
>>>>>>>> you point me at that I can see if I can help make that change as well.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> -Alex
>>>>>>>>
>>>>>>>> [0] https://review.openstack.org/#/c/202763/
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Vladimir Kozhukalov
>>>>>>>>>
>>>>>>>>> On Fri, Jul 17, 2015 at 2:56 PM, Aleksandr Didenko <
>>>>>>>>> adidenko at mirantis.com> wrote:
>>>>>>>>>
>>>>>>>>>> I believe build_repo function is the best way to do this [0]. So
>>>>>>>>>> for fuel-library we'll need to run a shell script right from the repo
>>>>>>>>>> before 'touch $$@'. We can make it either conditional ( test -f
>>>>>>>>>> ./path/additional_build_script.sh && bash ./path/additional_build_script.sh
>>>>>>>>>> ) or as additional parameter to function and add it in fuel-library call [1]
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Alex
>>>>>>>>>>
>>>>>>>>>> [0]
>>>>>>>>>> https://github.com/stackforge/fuel-main/blob/master/repos.mk#L16-L37
>>>>>>>>>> [1]
>>>>>>>>>> https://github.com/stackforge/fuel-main/blob/master/repos.mk#L45
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jul 17, 2015 at 2:37 PM, Alex Schultz <
>>>>>>>>>> aschultz at mirantis.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hey Alex,
>>>>>>>>>>>
>>>>>>>>>>> On Jul 17, 2015 4:32 AM, "Aleksandr Didenko" <
>>>>>>>>>>> adidenko at mirantis.com> wrote:
>>>>>>>>>>> >
>>>>>>>>>>> > Hi,
>>>>>>>>>>> >
>>>>>>>>>>> > I think that we should provide a separate script that will
>>>>>>>>>>> fetch the upstream modules into fuel-library/deployment/puppet/ directory.
>>>>>>>>>>> It will allow us to have everything in a single place and use this script
>>>>>>>>>>> in ISO build process and CI jobs.
>>>>>>>>>>> >
>>>>>>>>>>>
>>>>>>>>>>> Right. That is what I'm going for. The issue I need help with is
>>>>>>>>>>> the best way to execute this as part of the build process.  From what i
>>>>>>>>>>> understand of the build process is that we are using git archive for all
>>>>>>>>>>> pieces so I'm not sure how to wedge in an extra script execution to do the
>>>>>>>>>>> module fetch.  The creation of the script isn't the issue, the issue is how
>>>>>>>>>>> can I properly run it as part of the build process.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> > Regards,
>>>>>>>>>>> > Alex
>>>>>>>>>>> >
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> -Alex
>>>>>>>>>>>
>>>>>>>>>>> > On Thu, Jul 16, 2015 at 11:17 PM, Alex Schultz <
>>>>>>>>>>> aschultz at mirantis.com> wrote:
>>>>>>>>>>> >>
>>>>>>>>>>> >> Hello everyone,
>>>>>>>>>>> >>
>>>>>>>>>>> >> I have committed the initial configuration required to start
>>>>>>>>>>> leveraging librarian-puppet as part of the way we pull in upstream puppet
>>>>>>>>>>> modules[0]. Additionally, I have also committed a change that would pull in
>>>>>>>>>>> the openstack-ironic module[1].  The one piece that is missing from this
>>>>>>>>>>> being a complete solution is the ability to run librarian-puppet as part of
>>>>>>>>>>> our build process for the fuel-library.  I've looked into the fuel-main
>>>>>>>>>>> build scripts and I think it's over my head to figure this out just by
>>>>>>>>>>> looking. Can anyone explain to me or assist me in how I could go about
>>>>>>>>>>> modifying the existing build system to be able to run librarian-puppet to
>>>>>>>>>>> prepare the source for the package?  In my initial investigation, it looks
>>>>>>>>>>> like it would be a modification of the fuel-main/packages/
>>>>>>>>>>> module.mk[3] file.  I basically need to do the
>>>>>>>>>>> prepare_library[3] function from the 202763 review[0] after we've pulled
>>>>>>>>>>> all the sources together to fetch the upstream modules.
>>>>>>>>>>> >>
>>>>>>>>>>> >>
>>>>>>>>>>> >> Thanks,
>>>>>>>>>>> >> -Alex
>>>>>>>>>>> >>
>>>>>>>>>>> >> [0] https://review.openstack.org/202763
>>>>>>>>>>> >> [1] https://review.openstack.org/202767
>>>>>>>>>>> >> [2]
>>>>>>>>>>> https://github.com/stackforge/fuel-main/blob/master/packages/module.mk#L63-L82
>>>>>>>>>>> >> [3]
>>>>>>>>>>> https://review.openstack.org/#/c/202763/1/utils/jenkins/fuel_noop_tests.rb
>>>>>>>>>>> >>
>>>>>>>>>>> >>
>>>>>>>>>>> __________________________________________________________________________
>>>>>>>>>>> >> OpenStack Development Mailing List (not for usage questions)
>>>>>>>>>>> >> Unsubscribe:
>>>>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>>>>> >>
>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>>> >>
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> __________________________________________________________________________
>>>>>>>>>>> > OpenStack Development Mailing List (not for usage questions)
>>>>>>>>>>> > Unsubscribe:
>>>>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>>>>> >
>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>>> >
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> __________________________________________________________________________
>>>>>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>>>>>> Unsubscribe:
>>>>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> __________________________________________________________________________
>>>>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>>>>> Unsubscribe:
>>>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> __________________________________________________________________________
>>>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>>>> Unsubscribe:
>>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> __________________________________________________________________________
>>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>>> Unsubscribe:
>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> __________________________________________________________________________
>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>> Unsubscribe:
>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>> --
>>>>>> --
>>>>>> Andrew Woodward
>>>>>> Mirantis
>>>>>> Fuel Community Ambassador
>>>>>> Ceph Community
>>>>>>
>>>>>>
>>>>>> __________________________________________________________________________
>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>> Unsubscribe:
>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>>>>
>>>>> __________________________________________________________________________
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>> --
>>>>
>>>> --
>>>>
>>>> Andrew Woodward
>>>>
>>>> Mirantis
>>>>
>>>> Fuel Community Ambassador
>>>>
>>>> Ceph Community
>>>>
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150720/abbdd5cc/attachment.html>


More information about the OpenStack-dev mailing list