Open Stack

On 27 July 2016 at 17:52, Paul Belanger <pabelanger at redhat.com> wrote:
> On Wed, Jul 27, 2016 at 02:54:00PM +0100, Derek Higgins wrote:
>> On 21 July 2016 at 23:04, Paul Belanger <pabelanger at redhat.com> wrote:
>> > Greetings,
>> >
>> > I write today to see how I can remove this server from tripleo-test-cloud-rh2. I
>> > have an open patch[1] currently to migrate tripleo-ci to use our AFS mirrors for
>> > centos and epel.  However, I'm still struggling to see what else you are using
>> > the local mirror for.
>> >
>> > From what I see, there appears to be some puppet modules in the mirror?
>> >
>> > The reason I am doing this work, is to help bring tripleo inline with
>> > openstack-infra tooling.  There shouldn't be the need for a project to maintain
>> > its own infrastructure outside of openstack-infra.  If so, I see that as some
>> > sort of a failure between the project and openstack-infra.   And with that in
>> > mind, I am here to help fix that.
>> >
>> > For the most part, I think we have everything currently in place to migrate away
>> > from your locally mirror. I just need some help figuring what else is left and
>> > then delete it.
>>
>> Hi Paul,
>>     The mirror server hosts 3 sets of data used in CI long with a cron
>> a job aimed at promoting trunk repositories,
>> The first you've already mentioned, there is a list of puppet modules
>> hosted here, we soon hope to move to packaged puppet modules so the
>> need for this will go away.
>>
> Ya, I was looking at an open review to rework this. If we moved these puppet
> modules to tarballs over git repos, I think we could mirror them pretty easy
> into our AFS mirrors.  Them being git repos requires more work because some
> policies around git repos.

We wont need to do anything here, the patch to move away from git
repos will be instead using the rdo packaged puppet modules, so we
wont need anything from infra for this, we just end up using the rdo
repository like we do for all other openstack projects.

>
>> The second is a mirror of the centos cloud images, these are updated
>> hourly by the centos-cloud-images cronjob[1], I guess these could be
>> easily replaced with the AFS server
>>
> So 2 things here.
>
> 1) I've reached out to CentOS asking to enable rsync support on
> http://cloud.centos.org/ if they do that, I can easily enable rsync for it.

Great

>
> 2) What about moving away from the centos diskimage-builder element and switch
> to centos-minimal element. I have an open review for this, but need help on
> actually testing this.  It moves away from using the cloud image, and instead
> uses yumdownloader to prebuild the images.

Its possible, but I think out of scope for a general ci thread, its
more of a tripleo decision so maybe needs its own thread to get a
wider audience.

>
>> Then we come to the parts where it will probably be more tricky to
>> move away from our own server
>>
>> o cached images - our nightly periodic jobs run tripleo ci with
>> master/HEAD for all openstack projects (using the most recent rdo
>> trunk repository), if the jobs pass then we upload the overcloud-full
>> and ipa images to the mirror server along with logging what jobs
>> passed, this happens at the end of toci_instack.sh[2], nothing else
>> happens at this point the files are just uploaded nothing starts using
>> them yet.
>>
> I suggest we move this to tarballs.o.o for now, this is what other projects are
> doing.  I believe we are also considering moving this process into AFS too.

Ok, its an option worth looking at if we could make it work.

>
>> o promote script - hourly we then run the promote script[3], this
>> script is whats responsible for the promotion of the master rdo
>> repository that is used by tripleo ci (and devs), it checks to see if
>> images have been updated to the mirror server by the periodic jobs,
>> and if all of the jobs we care about (currently
>> periodic-tripleo-ci-centos-7-ovb-ha
>> periodic-tripleo-ci-centos-7-ovb-nonha[4]) passed then it does 2
>> things
>>   1. updates the current-tripleo link on the mirror server[5]
>>   2. updates the current-tripleo link on the rdo trunk server[6]
>> By doing this we ensure that the the current-tripleo link on the rdo
>> trunk server is always pointing to something that has passed tripleo
>> ci jobs, and that tripleo ci is using cached images that were built
>> using this repository
>>
> Okay, I think we need to dive more into this. It might be possible to make this
> a post job or use mirror-update.openstack.org

A post job might work if it could find the status of all the perioc
jobs, I'm not
familiar with mirror-update.openstack.org what does it do?

>
>> We've had to run this promote script on the mirror server as the
>> individual jobs run independently and in oder to make the promote
>> decision we needed somewhere that is aware of the status of all the
>> jobs
>>
>> Hope this answers your questions,
>> Derek.
>>
>> [1] - http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/mirror-server/mirror-server.pp#n40
>> [2] - http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/toci_instack.sh#n198
>> [3] - http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/mirror-server/promote.sh
>> [4] - http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/mirror-server/mirror-server.pp#n51
>> [5] - http://8.43.87.241/builds/current-tripleo/
>> [6] - http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master-tripleo/
>>
>> >
>> > [1] https://review.openstack.org/#/c/326143/
>> >
>> > __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev