[cctbxbb] problem when using Python to convert from numpy array to flex.int array

Tue May 3 19:44:07 PDT 2016

Graeme and others,

Converting directly to flex would be nice, but the compression-related libraries (lz4, bitshuffle) don't know about flex and turn data into numpy arrays.

It still seems to me like the following simple script *should* really work, but from what I've seen of the tests, they really only test conversion from flex to numpy, not fresh creation of a numpy array and converting into flex. What's a bit strange is that I was able to get much of my image data converted, but it's not consistent!

import numpy
from scitbx.array_family import flex
a=numpy.ndarray(shape=[3072,3072],dtype=numpy.uint16)
flex.int(a)

Jun

Jun Aishima | Post Doc - MX | Australian Synchrotron 
p: (03) 8540 4114 | f: (03) 8540 4200 
jun.aishima at synchrotron.org.au | www.synchrotron.org.au
800 Blackburn Road, Clayton, Victoria 3168

-----Original Message-----
From: cctbxbb-bounces at phenix-online.org [mailto:cctbxbb-bounces at phenix-online.org] On Behalf Of Graeme.Winter at diamond.ac.uk
Sent: Tuesday, 3 May 2016 5:06 PM
To: cctbxbb at phenix-online.org
Subject: Re: [cctbxbb] problem when using Python to convert from numpy array to flex.int array

Hi Jun

OK, got it. Should be a way of efficiently casting an image over 0mq over to a flex array directly without the intermediate numpy step

Not sure what the 0mq interface will be like though (modulo with catching up with email, I think it is one of the unread things in my inbox) but if it comes across compressed should be able to work with it just fine.

Will investigate this end, meantimes developing algorithms with dials / cctbx from flex arrays should work out just fine :)

Cheerio Graeme

-----Original Message-----
From: cctbxbb-bounces at phenix-online.org [mailto:cctbxbb-bounces at phenix-online.org] On Behalf Of Jun Aishima
Sent: 03 May 2016 07:29
To: cctbx mailing list
Subject: Re: [cctbxbb] problem when using Python to convert from numpy array to flex.int array

Graeme,

I'm actually trying to make the ADSC do more Eiger-like things. Unfortunately, that means I won't be loading the image on the receiving end, but am trying to reconstruct and analyse an image that arrived via zmq.

Jun

Jun Aishima | Post Doc - MX | Australian Synchrotron 
p: (03) 8540 4114 | f: (03) 8540 4200 
jun.aishima at synchrotron.org.au | www.synchrotron.org.au
800 Blackburn Road, Clayton, Victoria 3168

-----Original Message-----
From: cctbxbb-bounces at phenix-online.org [mailto:cctbxbb-bounces at phenix-online.org] On Behalf Of Graeme.Winter at diamond.ac.uk
Sent: Tuesday, 3 May 2016 3:57 PM
To: cctbxbb at phenix-online.org
Subject: Re: [cctbxbb] problem when using Python to convert from numpy array to flex.int array

Hi Jun

Short version

from dxtbx import load
im = load('test.cbf')
raw = im.get_raw_data()
print raw.focus()

i.e. raw is flex array of pixel data, should work with most/all synchrotron detectors

best wishes Graeme

On 3 May 2016, at 06:11, Jun Aishima <jun.aishima at synchrotron.org.au<mailto:jun.aishima at synchrotron.org.au>> wrote:

Nick and Peter,

It will take me a while to get back into C++ to figure out what is happening with the flex.int() constructor! However, I like your suggestion of using dxtbx – I didn’t know it had methods to do something like this, so I’ll see what I can find. I will definitely send you the image off-board anyways, just so you have an idea what I am seeing.

By the way, Aaron had answered me, and by then I had managed to reproduce the problem entirely using phenix.python and only numpy to create an empty array, and then attempting to convert that into flex:

import numpy
from scitbx.array_family import flex
a=numpy.ndarray(shape=[3072,3072],dtype=numpy.uint16)
flex.int(a)

Jun

Jun Aishima | Post Doc - MX | Australian Synchrotron
p: (03) 8540 4114 | f: (03) 8540 4200
jun.aishima at synchrotron.org.au<mailto:jun.aishima at synchrotron.org.au> | www.synchrotron.org.au<x-msg://153/www.synchrotron.org.au>
800 Blackburn Road, Clayton, Victoria 3168

From: cctbxbb-bounces at phenix-online.org<mailto:cctbxbb-bounces at phenix-online.org> [mailto:cctbxbb-bounces at phenix-online.org] On Behalf Of Nicholas Sauter
Sent: Tuesday, 3 May 2016 3:04 PM
To: cctbx mailing list
Subject: Re: [cctbxbb] problem when using Python to convert from numpy array to flex.int array

Jun,

You'll have to drill down a bit further to assess what exactly is the argument being passed (file.data[3071]) to the constructor of flex.int<http://flex.int/>.  Otherwise you can send the raw image file & I'll have a look.  Why aren't you using dxtbx to read the raw data directly to a scitbx flex type?

Nick

Nicholas K. Sauter, Ph. D.
Computer Staff Scientist, Molecular Biophysics and Integrated Bioimaging Division Lawrence Berkeley National Laboratory
1 Cyclotron Rd., Bldg. 33R0345
Berkeley, CA 94720
(510) 486-5713

On Mon, May 2, 2016 at 8:34 PM, Jun Aishima <jun.aishima at synchrotron.org.au<mailto:jun.aishima at synchrotron.org.au>> wrote:
I have used the bootstrap method to build cctbx with my own Python (in a virtual environment). I setup my virtual environment, then source $CCTBX/build/setpaths.sh (bash).

I am currently trying to turn a detector image (uint16) read in with fabio, which results in a numpy ndarray, and then turn that into a scitbx flex array. Since I’ve had problems doing this in my system, I’ve written a little script that reproduces the problem:

**********
import fabio
file=fabio.open('testcrystal_1_001.img') # adsc detector image file.data[0] from scitbx.array_family import flex
flex.int<http://flex.int/>(file.data[3071]) # this is a 3072x3072 image. For some reason, row 0 causes no problems.
**********
At the end of this email is the stack trace I get.

Has anybody had similar issues? I first saw this when I was trying to convert the entire image into a flex array, but now I’ve seen it with only one row [3071], while still working for [0:1500], so it doesn’t seem like a simple memory issue. Any help would be appreciated.

Jun Aishima

**********

show_stack(1): <stdin>(1) <module>
libc backtrace (23 frames, most recent call last):
  /home/aishimaj/spotter/bin/python() [0x414199]
  /lib64/libc.so.6(__libc_start_main+0xfd) [0x366d21ed5d]
  /home/aishimaj/spotter/bin/python(Py_Main+0xb22) [0x4150b2]
  /home/aishimaj/spotter/bin/python(PyRun_AnyFileExFlags+0x6b) [0x4ce82b]
  /home/aishimaj/spotter/bin/python(PyRun_InteractiveLoopFlags+0x4e) [0x4ce01e]
  /home/aishimaj/spotter/bin/python(PyRun_InteractiveOneFlags+0x1ff) [0x4cdd9f]
  /home/aishimaj/spotter/bin/python(PyEval_EvalCode+0x32) [0x4ad362]
  /home/aishimaj/spotter/bin/python(PyEval_EvalCodeEx+0x877) [0x4ad257]
  /home/aishimaj/spotter/bin/python(PyEval_EvalFrameEx+0x6a0b) [0x4aa8bb]
  /home/aishimaj/spotter/bin/python(PyObject_Call+0x47) [0x419ba7]
  /home/aishimaj/spotter/bin/python() [0x46df9f]
  /home/aishimaj/spotter/bin/python() [0x47623f]
  /home/aishimaj/spotter/bin/python(PyObject_Call+0x47) [0x419ba7]
  /home/aishimaj/spotter/bin/python() [0x42296f]
  /home/aishimaj/spotter/bin/python(PyObject_Call+0x47) [0x419ba7]
  /beamline/phenix/bootstrap/build/lib/libboost_python.so(+0x2d088) [0x7f5561101088]
  /beamline/phenix/bootstrap/build/lib/libboost_python.so(boost::python::handle_exception_impl(boost::function0<void>)+0x43) [0x7f5561112013]
  /beamline/phenix/bootstrap/build/lib/libboost_python.so(+0x2fe08) [0x7f5561103e08]
  /beamline/phenix/bootstrap/build/lib/libboost_python.so(boost::python::objects::function::call(_object*, _object*) const+0xef) [0x7f55611031ff]
  /beamline/phenix/bootstrap/build/lib/scitbx_array_family_flex_ext.so(boost::python::objects::signature_py_function_impl<boost::python::detail::caller<scitbx::af::versa<int, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > >* (*)(boost::python::numeric::array const&), boost::python::detail::constructor_policy<boost::python::default_call_policies>, boost::mpl::vector2<scitbx::af::versa<int, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > >*, boost::python::numeric::array const&> >, boost::mpl::v_item<void, boost::mpl::v_item<boost::python::api::object, boost::mpl::v_mask<boost::mpl::vector2<scitbx::af::versa<int, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > >*, boost::python::numeric::array const&>, 1>, 1>, 1> >::operator()(_object*, _object*)+0x52) [0x7f555ed68452]
  /beamline/phenix/bootstrap/build/lib/scitbx_array_family_flex_ext.so(scitbx::af::boost_python::flex_int_from_numpy_array(boost::python::numeric::array const&)+0x29) [0x7f555f039f79]
  /beamline/phenix/bootstrap/build/lib/scitbx_array_family_flex_ext.so(scitbx::af::versa<int, scitbx::af::flex_grid<scitbx::af::small<long, 10ul> > > scitbx::af::boost_python::versa_flex_from_numpy_array<int>(boost::python::numeric::array const&)+0x1c87) [0x7f555f0415c7]
  /lib64/libc.so.6() [0x366d2326a0]
Segmentation fault (Python and libc call stacks above)
                This crash may be due to a problem in any imported
                Python module, including modules which are not part
                of the cctbx project. To disable the traps leading
                to this message, define these environment variables
                (e.g. assign the value 1):
                    BOOST_ADAPTBX_FPE_DEFAULT
                    BOOST_ADAPTBX_SIGNALS_DEFAULT
                This will NOT solve the problem, just mask it, but
                may allow you to proceed in case it is not critical.

Jun Aishima | Post Doc - MX | Australian Synchrotron
p: (03) 8540 4114 | f: (03) 8540 4200
jun.aishima at synchrotron.org.au<mailto:jun.aishima at synchrotron.org.au> | www.synchrotron.org.au<http://www.synchrotron.org.au/>
800 Blackburn Road, Clayton, Victoria 3168

_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
http://phenix-online.org/mailman/listinfo/cctbxbb

--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org
http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org
http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org
http://phenix-online.org/mailman/listinfo/cctbxbb