Open Stack

On Wed, Jul 27, 2016 at 02:54:00PM +0100, Derek Higgins wrote:
> On 21 July 2016 at 23:04, Paul Belanger <pabelanger at redhat.com> wrote:
> > Greetings,
> >
> > I write today to see how I can remove this server from tripleo-test-cloud-rh2. I
> > have an open patch[1] currently to migrate tripleo-ci to use our AFS mirrors for
> > centos and epel.  However, I'm still struggling to see what else you are using
> > the local mirror for.
> >
> > From what I see, there appears to be some puppet modules in the mirror?
> >
> > The reason I am doing this work, is to help bring tripleo inline with
> > openstack-infra tooling.  There shouldn't be the need for a project to maintain
> > its own infrastructure outside of openstack-infra.  If so, I see that as some
> > sort of a failure between the project and openstack-infra.   And with that in
> > mind, I am here to help fix that.
> >
> > For the most part, I think we have everything currently in place to migrate away
> > from your locally mirror. I just need some help figuring what else is left and
> > then delete it.
> 
> Hi Paul,
>     The mirror server hosts 3 sets of data used in CI long with a cron
> a job aimed at promoting trunk repositories,
> The first you've already mentioned, there is a list of puppet modules
> hosted here, we soon hope to move to packaged puppet modules so the
> need for this will go away.
> 
Ya, I was looking at an open review to rework this. If we moved these puppet
modules to tarballs over git repos, I think we could mirror them pretty easy
into our AFS mirrors.  Them being git repos requires more work because some
policies around git repos.

> The second is a mirror of the centos cloud images, these are updated
> hourly by the centos-cloud-images cronjob[1], I guess these could be
> easily replaced with the AFS server
> 
So 2 things here.

1) I've reached out to CentOS asking to enable rsync support on
http://cloud.centos.org/ if they do that, I can easily enable rsync for it.

2) What about moving away from the centos diskimage-builder element and switch
to centos-minimal element. I have an open review for this, but need help on
actually testing this.  It moves away from using the cloud image, and instead
uses yumdownloader to prebuild the images.

> Then we come to the parts where it will probably be more tricky to
> move away from our own server
> 
> o cached images - our nightly periodic jobs run tripleo ci with
> master/HEAD for all openstack projects (using the most recent rdo
> trunk repository), if the jobs pass then we upload the overcloud-full
> and ipa images to the mirror server along with logging what jobs
> passed, this happens at the end of toci_instack.sh[2], nothing else
> happens at this point the files are just uploaded nothing starts using
> them yet.
> 
I suggest we move this to tarballs.o.o for now, this is what other projects are
doing.  I believe we are also considering moving this process into AFS too.

> o promote script - hourly we then run the promote script[3], this
> script is whats responsible for the promotion of the master rdo
> repository that is used by tripleo ci (and devs), it checks to see if
> images have been updated to the mirror server by the periodic jobs,
> and if all of the jobs we care about (currently
> periodic-tripleo-ci-centos-7-ovb-ha
> periodic-tripleo-ci-centos-7-ovb-nonha[4]) passed then it does 2
> things
>   1. updates the current-tripleo link on the mirror server[5]
>   2. updates the current-tripleo link on the rdo trunk server[6]
> By doing this we ensure that the the current-tripleo link on the rdo
> trunk server is always pointing to something that has passed tripleo
> ci jobs, and that tripleo ci is using cached images that were built
> using this repository
> 
Okay, I think we need to dive more into this. It might be possible to make this
a post job or use mirror-update.openstack.org

> We've had to run this promote script on the mirror server as the
> individual jobs run independently and in oder to make the promote
> decision we needed somewhere that is aware of the status of all the
> jobs
> 
> Hope this answers your questions,
> Derek.
> 
> [1] - http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/mirror-server/mirror-server.pp#n40
> [2] - http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/toci_instack.sh#n198
> [3] - http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/mirror-server/promote.sh
> [4] - http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/mirror-server/mirror-server.pp#n51
> [5] - http://8.43.87.241/builds/current-tripleo/
> [6] - http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master-tripleo/
> 
> >
> > [1] https://review.openstack.org/#/c/326143/
> >
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev