[tripleo] Copying tripleo container images to quay.io
Hello, I just wrote a tool in go that is right now copying every 2 hours all the containers from rdo registry to quay.io. Right now it's copying all branches: master, victoria, ussuri, train, stein, queens and rocky. You can see the containers in https://quay.io/tripleo{release} for example https://quay.io/tripleomaster The tool basically searches for container build jobs with success status, parse the containers that was built, copy these containers to quay.io and tag it with current-tripleo and the built hash. The code it's based on skopeo, and I did not use it, because there are some other stuff required on quay.io side that requires the use of quay.io api, like create an already public repository, parsing the tripleo container build job, tagging, etc. Also I wanted to play with go :) Right now the code is under review at https://review.rdoproject.org/r/#/c/31133/ and I got it running on our toolbox, and he's an log example (without the --debug flag) time="2020-11-19T12:06:17Z" level=info msg="Copying image centos-binary-horizon:4fad79713786f77292e59fa1c036f588 in tripleoussuri namespace" time="2020-11-19T12:06:18Z" level=info msg="Tagging current-tripleo to sha256:db39e7d43d4c8eec82f61acd4956d7f165b595d515f270f8dabe0c9b009c95f2" "Updated" time="2020-11-19T12:06:19Z" level=info msg="Copying image centos-binary-ceilometer-base:4fad79713786f77292e59fa1c036f588 in tripleoussuri namespace" time="2020-11-19T12:06:20Z" level=info msg="Tagging current-tripleo to sha256:5eee849a9f74d0107cbfd2f456f701be37256ea711079cffcc189d140e8f3176" "Updated" time="2020-11-19T12:06:21Z" level=info msg="Copying image centos-binary-gnocchi-base:4fad79713786f77292e59fa1c036f588 in tripleoussuri namespace" time="2020-11-19T12:06:22Z" level=info msg="Tagging current-tripleo to sha256:5b46ea43d41e56dd81005a234631905a8ad594ef8432ae78542392114956f85e" "Updated" If you want to play around with this, feel free to do so, and any feedback is welcome :) Kind regards, -- Arx Cruz Software Engineer Red Hat EMEA <https://www.redhat.com> arxcruz@redhat.com @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>
Quick question here (maybe I don't have the whole context). Isn't it possible to use tools like skopeo to do this image sync and avoid adding code that will be eventually more complex to maintain? Cheers, Carlos. On Thu, Nov 19, 2020 at 3:05 PM Arx Cruz <arxcruz@redhat.com> wrote:
Hello,
I just wrote a tool in go that is right now copying every 2 hours all the containers from rdo registry to quay.io.
Right now it's copying all branches: master, victoria, ussuri, train, stein, queens and rocky. You can see the containers in https://quay.io/tripleo{release} for example https://quay.io/tripleomaster
The tool basically searches for container build jobs with success status, parse the containers that was built, copy these containers to quay.io and tag it with current-tripleo and the built hash.
The code it's based on skopeo, and I did not use it, because there are some other stuff required on quay.io side that requires the use of quay.io api, like create an already public repository, parsing the tripleo container build job, tagging, etc. Also I wanted to play with go :)
Right now the code is under review at https://review.rdoproject.org/r/#/c/31133/ and I got it running on our toolbox, and he's an log example (without the --debug flag)
time="2020-11-19T12:06:17Z" level=info msg="Copying image centos-binary-horizon:4fad79713786f77292e59fa1c036f588 in tripleoussuri namespace" time="2020-11-19T12:06:18Z" level=info msg="Tagging current-tripleo to sha256:db39e7d43d4c8eec82f61acd4956d7f165b595d515f270f8dabe0c9b009c95f2" "Updated"
time="2020-11-19T12:06:19Z" level=info msg="Copying image centos-binary-ceilometer-base:4fad79713786f77292e59fa1c036f588 in tripleoussuri namespace" time="2020-11-19T12:06:20Z" level=info msg="Tagging current-tripleo to sha256:5eee849a9f74d0107cbfd2f456f701be37256ea711079cffcc189d140e8f3176" "Updated"
time="2020-11-19T12:06:21Z" level=info msg="Copying image centos-binary-gnocchi-base:4fad79713786f77292e59fa1c036f588 in tripleoussuri namespace" time="2020-11-19T12:06:22Z" level=info msg="Tagging current-tripleo to sha256:5b46ea43d41e56dd81005a234631905a8ad594ef8432ae78542392114956f85e" "Updated"
If you want to play around with this, feel free to do so, and any feedback is welcome :)
Kind regards,
--
Arx Cruz
Software Engineer
Red Hat EMEA <https://www.redhat.com>
arxcruz@redhat.com @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>
I mean in terms of just adding the bits you are missing? Like the parsing and tagging? On Thu, Nov 19, 2020 at 3:16 PM Carlos Camacho Gonzalez <ccamacho@redhat.com> wrote:
Quick question here (maybe I don't have the whole context).
Isn't it possible to use tools like skopeo to do this image sync and avoid adding code that will be eventually more complex to maintain?
Cheers, Carlos.
On Thu, Nov 19, 2020 at 3:05 PM Arx Cruz <arxcruz@redhat.com> wrote:
Hello,
I just wrote a tool in go that is right now copying every 2 hours all the containers from rdo registry to quay.io.
Right now it's copying all branches: master, victoria, ussuri, train, stein, queens and rocky. You can see the containers in https://quay.io/tripleo{release} for example https://quay.io/tripleomaster
The tool basically searches for container build jobs with success status, parse the containers that was built, copy these containers to quay.io and tag it with current-tripleo and the built hash.
The code it's based on skopeo, and I did not use it, because there are some other stuff required on quay.io side that requires the use of quay.io api, like create an already public repository, parsing the tripleo container build job, tagging, etc. Also I wanted to play with go :)
Right now the code is under review at https://review.rdoproject.org/r/#/c/31133/ and I got it running on our toolbox, and he's an log example (without the --debug flag)
time="2020-11-19T12:06:17Z" level=info msg="Copying image centos-binary-horizon:4fad79713786f77292e59fa1c036f588 in tripleoussuri namespace" time="2020-11-19T12:06:18Z" level=info msg="Tagging current-tripleo to sha256:db39e7d43d4c8eec82f61acd4956d7f165b595d515f270f8dabe0c9b009c95f2" "Updated"
time="2020-11-19T12:06:19Z" level=info msg="Copying image centos-binary-ceilometer-base:4fad79713786f77292e59fa1c036f588 in tripleoussuri namespace" time="2020-11-19T12:06:20Z" level=info msg="Tagging current-tripleo to sha256:5eee849a9f74d0107cbfd2f456f701be37256ea711079cffcc189d140e8f3176" "Updated"
time="2020-11-19T12:06:21Z" level=info msg="Copying image centos-binary-gnocchi-base:4fad79713786f77292e59fa1c036f588 in tripleoussuri namespace" time="2020-11-19T12:06:22Z" level=info msg="Tagging current-tripleo to sha256:5b46ea43d41e56dd81005a234631905a8ad594ef8432ae78542392114956f85e" "Updated"
If you want to play around with this, feel free to do so, and any feedback is welcome :)
Kind regards,
--
Arx Cruz
Software Engineer
Red Hat EMEA <https://www.redhat.com>
arxcruz@redhat.com @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>
top-post.. Arx has also now posted the log where this sync / copy can be viewed. If you end up hitting an issue when pulling containers from quay, this log would be a good place to poke at. http://38.102.83.131/copy-quay.txt Thanks all and thanks Arx!!! On Thu, Nov 19, 2020 at 7:32 AM Carlos Camacho Gonzalez <ccamacho@redhat.com> wrote:
I mean in terms of just adding the bits you are missing? Like the parsing and tagging?
On Thu, Nov 19, 2020 at 3:16 PM Carlos Camacho Gonzalez < ccamacho@redhat.com> wrote:
Quick question here (maybe I don't have the whole context).
Isn't it possible to use tools like skopeo to do this image sync and avoid adding code that will be eventually more complex to maintain?
Cheers, Carlos.
On Thu, Nov 19, 2020 at 3:05 PM Arx Cruz <arxcruz@redhat.com> wrote:
Hello,
I just wrote a tool in go that is right now copying every 2 hours all the containers from rdo registry to quay.io.
Right now it's copying all branches: master, victoria, ussuri, train, stein, queens and rocky. You can see the containers in https://quay.io/tripleo{release} for example https://quay.io/tripleomaster
The tool basically searches for container build jobs with success status, parse the containers that was built, copy these containers to quay.io and tag it with current-tripleo and the built hash.
The code it's based on skopeo, and I did not use it, because there are some other stuff required on quay.io side that requires the use of quay.io api, like create an already public repository, parsing the tripleo container build job, tagging, etc. Also I wanted to play with go :)
Right now the code is under review at https://review.rdoproject.org/r/#/c/31133/ and I got it running on our toolbox, and he's an log example (without the --debug flag)
time="2020-11-19T12:06:17Z" level=info msg="Copying image centos-binary-horizon:4fad79713786f77292e59fa1c036f588 in tripleoussuri namespace" time="2020-11-19T12:06:18Z" level=info msg="Tagging current-tripleo to sha256:db39e7d43d4c8eec82f61acd4956d7f165b595d515f270f8dabe0c9b009c95f2" "Updated"
time="2020-11-19T12:06:19Z" level=info msg="Copying image centos-binary-ceilometer-base:4fad79713786f77292e59fa1c036f588 in tripleoussuri namespace" time="2020-11-19T12:06:20Z" level=info msg="Tagging current-tripleo to sha256:5eee849a9f74d0107cbfd2f456f701be37256ea711079cffcc189d140e8f3176" "Updated"
time="2020-11-19T12:06:21Z" level=info msg="Copying image centos-binary-gnocchi-base:4fad79713786f77292e59fa1c036f588 in tripleoussuri namespace" time="2020-11-19T12:06:22Z" level=info msg="Tagging current-tripleo to sha256:5b46ea43d41e56dd81005a234631905a8ad594ef8432ae78542392114956f85e" "Updated"
If you want to play around with this, feel free to do so, and any feedback is welcome :)
Kind regards,
--
Arx Cruz
Software Engineer
Red Hat EMEA <https://www.redhat.com>
arxcruz@redhat.com @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>
Note that skopeo will shortly have the layers diffs [0] based "copy" operations [1],[2],[3]. It also deduplicates things and uses local cache to not repeat redundant data transfers. Would be great if TripleO container registry could start using that super cool feature as early adoption. We could make a build with these patches and try that for the subject tooling for TCIB distribution over quay.io. IIUC, the process is like: * as input we have full layers from the source registry, built with TCIB; * skopeo uses its super powers to redo it on fly into layers diff * either skopeo as well, or maybe even tripleoclient prepare image CLI pushes those optimized layers of the images to the target quay.io registry. * ??? * profit! [0] https://github.com/containers/image/pull/902 [1] https://github.com/containers/image/pull/1084 [2] https://github.com/containers/storage/pull/775 [3] https://github.com/containers/skopeo/pull/1111 On 11/19/20 3:25 PM, Carlos Camacho Gonzalez wrote:
I mean in terms of just adding the bits you are missing? Like the parsing and tagging?
On Thu, Nov 19, 2020 at 3:16 PM Carlos Camacho Gonzalez <ccamacho@redhat.com <mailto:ccamacho@redhat.com>> wrote:
Quick question here (maybe I don't have the whole context).
Isn't it possible to use tools like skopeo to do this image sync and avoid adding code that will be eventually more complex to maintain?
Cheers, Carlos.
On Thu, Nov 19, 2020 at 3:05 PM Arx Cruz <arxcruz@redhat.com <mailto:arxcruz@redhat.com>> wrote:
Hello,
I just wrote a tool in go that is right now copying every 2 hours all the containers from rdo registry to quay.io <http://quay.io>.
Right now it's copying all branches: master, victoria, ussuri, train, stein, queens and rocky. You can see the containers in https://quay.io/tripleo{release} <https://quay.io/tripleo%7Brelease%7D> for example https://quay.io/tripleomaster
The tool basically searches for container build jobs with success status, parse the containers that was built, copy these containers to quay.io <http://quay.io> and tag it with current-tripleo and the built hash.
The code it's based on skopeo, and I did not use it, because there are some other stuff required on quay.io <http://quay.io> side that requires the use of quay.io <http://quay.io> api, like create an already public repository, parsing the tripleo container build job, tagging, etc. Also I wanted to play with go :)
Right now the code is under review at https://review.rdoproject.org/r/#/c/31133/ and I got it running on our toolbox, and he's an log example (without the --debug flag)
time="2020-11-19T12:06:17Z" level=info msg="Copying image centos-binary-horizon:4fad79713786f77292e59fa1c036f588 in tripleoussuri namespace" time="2020-11-19T12:06:18Z" level=info msg="Tagging current-tripleo to sha256:db39e7d43d4c8eec82f61acd4956d7f165b595d515f270f8dabe0c9b009c95f2" "Updated"
time="2020-11-19T12:06:19Z" level=info msg="Copying image centos-binary-ceilometer-base:4fad79713786f77292e59fa1c036f588 in tripleoussuri namespace" time="2020-11-19T12:06:20Z" level=info msg="Tagging current-tripleo to sha256:5eee849a9f74d0107cbfd2f456f701be37256ea711079cffcc189d140e8f3176" "Updated"
time="2020-11-19T12:06:21Z" level=info msg="Copying image centos-binary-gnocchi-base:4fad79713786f77292e59fa1c036f588 in tripleoussuri namespace" time="2020-11-19T12:06:22Z" level=info msg="Tagging current-tripleo to sha256:5b46ea43d41e56dd81005a234631905a8ad594ef8432ae78542392114956f85e" "Updated"
If you want to play around with this, feel free to do so, and any feedback is welcome :)
Kind regards,
--
Arx Cruz
Software Engineer
Red Hat EMEA <https://www.redhat.com>
arxcruz@redhat.com <mailto:arxcruz@redhat.com>
@RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>
-- Best regards, Bogdan Dobrelya, Irc #bogdando
participants (4)
-
Arx Cruz
-
Bogdan Dobrelya
-
Carlos Camacho Gonzalez
-
Wesley Hayutin