Hi Rob,

We looked a little into this and it seems like the byte string interface will preserve NAN. We're going to dig deeper into the serialization code for a fix.

Anyway, here is a code snippet for using the byte string interface. The byte string can be pickled, so a workaround for flex would be to add an extra conversion step to byte string before pickling.

from __future__ import print_function

import pickle
from scitbx.array_family import flex

# NAN in flex.double
a = flex.random_double(5)
n = float('nan')
a[3] = n

# pickle test
p = pickle.dumps(a)
ap = pickle.loads(p)

# byte string test
b = a.copy_to_byte_str()
ab = flex.double_from_byte_str(b)

# pickled byte string test
pb = pickle.dumps(b)
apb = flex.double_from_byte_str(pickle.loads(pb))

# check arrays
print('Original:', list(a))
print('Pickle:  ', list(ap), 'Oh no!')
print('Bytes:   ', list(ab))
print('Bytes2:  ', list(apb))


--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709
Fax: (510) 486-5909


On Mon, Jun 10, 2019 at 7:10 AM Robert Oeffner <rdo20@cam.ac.uk> wrote:
Hi all,

It seems that a flex.double array containing "nan" values doesn't
survive pickling as in:

oeffner@grove:~$ cctbx.python
Python 2.7.15 (default, Sep 11 2018, 23:48:45)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-18)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>>
 >>> from cctbx.array_family import flex
 >>> import pickle
 >>> flex_nans = flex.double(3, float('nan'))
 >>> list(flex_nans)
[nan, nan, nan]
 >>> list(pickle.loads( pickle.dumps(flex_nans)))
[0.0, 0.0, 0.0]
 >>>
 >>>

If the flex double is cast into a python list then it does survive
pickling as in:

 >>>
 >>> list(pickle.loads( pickle.dumps(list( flex_nans ))))
[nan, nan, nan]
 >>>

I would like to use nan values to indicate missing or invalid data
values. Would anyone have a suggestion how to go about this?


Many thanks,


Rob


--
Robert Oeffner, Ph.D.
Research Associate, The Read Group
Department of Haematology,
Cambridge Institute for Medical Research
University of Cambridge
Cambridge Biomedical Campus
The Keith Peters Building
Hills Road
Cambridge CB2 0XY

www.cimr.cam.ac.uk/investigators/read/index.html
tel: +44(0)1223 763234

_______________________________________________
cctbxbb mailing list
cctbxbb@phenix-online.org
http://phenix-online.org/mailman/listinfo/cctbxbb