Open Stack

Excerpts from John Dickinson's message of 2017-01-03 12:00:07 -0800:
> 
> On 3 Jan 2017, at 10:38, Doug Hellmann wrote:
> 
> > Excerpts from John Dickinson's message of 2017-01-03 09:02:19 -0800:
> >>
> >> On 2 Jan 2017, at 13:06, Davanum Srinivas wrote:
> >>
> >>> Folks,
> >>>
> >>> Short Story :
> >>> [1] has merged in devstack, it adds support for a python 3.5 based
> >>> up/down devstack test that just starts services and brings them down.
> >>> see [2] for a test run.
> >>>
> >>> Need help from swift folks:
> >>> Swift still needs work i have gotten as far as [3] UnpicklingError on
> >>> ring data using [4][5][6][7]. Can someone from the swift team pick
> >>> this up?
> >>> Once you get this working, please add "swift" to the white list in [8]
> >>> and remove the disable_service for swift services in [9]
> >>
> >> IIRC the issue is the differences between pickle, json, and arrays in py2 vs py3 (short summary: you can't deserialize in py3 stuff that was serialized in py2 without first changing the py2 code).
> >
> > Is that right? It seems like it would be the other way around. There
> > are newer pickle protocols in 3 that aren't available at all for 2.
> >
> > There are also some new options to the load() function in 3 to do
> > things like fix imports for standard library modules that were
> > renamed and set the right default string encoding to make it possible
> > to change the *3* code to be able to more easily load a pickle
> > created by 2 [1].  Is that what you meant?
> 
> Nah, it's not the pickle protocol. It's the different way (some versions of) py27 [de]serializes arrays vs how py3 does it. The following breaks for py2.7.10 and works for py2.7.12 (.11 is untested).
> 
> python3 -c 'import array, pickle, os, sys; pickle.dump(array.array("I", [0, 0, 0]), os.fdopen(1, "wb"), protocol=2)' | python -c 'import pickle, os, sys; print(pickle.load(os.fdopen(0, "rb")))'

o_O

Sigh.

> 
> So maybe there are ways to always ensure doing the right thing through some combination of try/excepts, "if six.PY3" blocks, plus lots of docs for ops to make sure upgrades go smoothly, but the real solution is to not use pickle. This is doubly true when considering that the data structure will be used by non-python code, too.

"Friends don't let their friends use pickle for portable persistence."

> 
> This is one of the things to put on the list for py3, and it certainly is not the last.

It would be good to start building up the list in the "Common Pitfalls"
section of https://wiki.openstack.org/wiki/Python3

Doug

> 
> --John
> 
> >
> > Doug
> >
> > [1] https://docs.python.org/3.5/library/pickle.html#pickle.load
> >
> >>
> >> We're tracking this at https://bugs.launchpad.net/swift/+bug/1644387 and have a patch at https://review.openstack.org/#/c/401397/, but I can't guarantee that services will start as soon as that patch lands. (i.e. it's necessary, but might not be sufficient)
> >>
> >> --John
> >>
> >>>
> >>> Other teams:
> >>> Please consider adding DSVM jobs with USE_PYTHON3=True for your
> >>> projects. This will hopefully help us get to our Pike goal for Python
> >>> 3.x [10]
> >>>
> >>> Please stop by #openstack-python3 channel to chat.
> >>>
> >>> Thanks,
> >>> Dims
> >>>
> >>> [1] https://review.openstack.org/#/c/414176/
> >>> [2] http://logs.openstack.org/76/414176/11/check/gate-devstack-dsvm-py35-updown-ubuntu-xenial-nv/
> >>> [3] http://logs.openstack.org/00/412500/7/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/f5a7fe7/logs/screen-s-proxy.txt.gz
> >>> [4] https://review.openstack.org/#/c/412500/
> >>> [5] https://review.openstack.org/#/c/414727/
> >>> [6] https://review.openstack.org/#/c/416064/
> >>> [7] https://review.openstack.org/#/c/416084/
> >>> [8] https://review.openstack.org/#/c/414176/11/inc/python
> >>> [9] https://review.openstack.org/#/c/413775/5/jenkins/jobs/devstack-gate.yaml
> >>> [10] https://review.openstack.org/#/c/349069/
> >>>
> >>> -- 
> >>> Davanum Srinivas :: https://twitter.com/dims
> >>>
> >>> __________________________________________________________________________
> >>> OpenStack Development Mailing List (not for usage questions)
> >>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev