Search results for query "look through"

527 messages

Re: [cctbxbb] Python 3 status

by Nicholas Sauter

As I indicated the syntax conversion process has already begun. Please continually do "git pull" to keep your sources up to date. As I go through the code I'm keeping a "blog" about the syntax needed to be cross-compatible between Python 2 & 3, and it is here: https://docs.google.com/document/d/1ao8WbehWj5cyb03iTtJmpB4oJX3MqJFyGDAu2AP… NKS Nicholas K. Sauter, Ph. D. Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713 On Tue, Sep 4, 2018 at 12:07 PM, James Holton <jmholton(a)lbl.gov> wrote: > It has been brought to my attention that little attempt at humor could > have been considered offensive. So clearly that that attempt failed. My > sincere apologies to anyone who was offended. > > In all seriousness, I hope it is neither humorous nor offensive that I do > appreciate all the hard work that goes into maintaining CCTBX, and we all > know that 2.7->3.0 is coming. I am at best a minor contributor, but I think > we can all identify with the crisis of time management between maintaining > what you have and preparing for the future. None of us are lazy or bored. > Quite the opposite. Graeme began this thread asking for "any thoughts", and > my two thoughts are: > 1) machine learning has in no way abandoned python 2.7 > 2) my limited sense of the community is one of reticence when it comes to > python3. > > I think that reticence comes from being already very busy maintaining one > copy of your code, and knowing that maintaining two copies is going to be > at least twice the work. Am I wrong in thinking there will be a period > where CCTBX supports both python2.7 and python3 ? Once maintenance load is > doubled, how much is left for new developments? These are not easy > questions. My advise is one of caution, but one voice in the crowd does not > speak for the whole. Graeme is right in thinking that discussion is in > order, and I agree that face-to-face discussion is much less prone to > offense than digital channels. > > Back into lurk mode now... > > -James > > > On 9/4/2018 10:04 AM, James Holton wrote: > >> It is most certainly not true that machine learning libraries require >> Python3. Everything we're doing at SLAC is using 2.7. I should admit this >> is mostly tensorflow-based stuff, but the install of the 2.7 version is >> quite painless. On my CentOS 7 system it is: >> >> yum install gcc gcc-c++ python-pip python-devel atlas atlas-devel >> gcc-gfortran openssl-devel libffi-devel >> pip install --upgrade numpy scipy wheel cryptography >> pip install --upgrade six scikit-learn protobuf >> pip install --upgrade scikit-image >> pip install --upgrade h5py >> pip install --upgrade tensorflow-gpu >> pip install --upgrade keras >> >> The 3.0 versions are available, and if you believe what you read on the >> internet they are all the rage and only loosers are still not using >> Python3. And c'mon! Use Python3 already! But for all the developers I >> know who are actually doing real work the 2.7->3.0 transition is something >> they keep meaning to do and will get to someday soon. Like when things >> "calm down". >> >> I'm still waiting for that to happen too... >> >> -James Holton >> MAD Scientist >> >> On 9/3/2018 11:50 PM, Graeme.Winter(a)Diamond.ac.uk wrote: >> >>> Hi Billy >>> >>> Sounds good - part of the reason I was asking is that we have students >>> here at Diamond using CCTBX / DIALS as a library and also wanting to use >>> machine learning libraries - however these all seem to be written in >>> Python3 so at the moment we’re in the position of dumping out files in one >>> Python and loading in another to work. We’d anticipated this being a >>> temporary state of affairs as we thought the Python3 migration was more >>> urgent on the LCLS side - turns out this was out-of-date information! >>> >>> Anyhow, good to know, please shout if you’d like some help with the >>> porting >>> >>> Cheers Graeme >>> >>> >>> On 4 Sep 2018, at 06:47, Billy Poon <BKPoon(a)lbl.gov<mailto:BKPoon@ >>> lbl.gov>> wrote: >>> >>> Hi Graeme, >>> >>> I'll be starting some of the Python 3 work probably next week. In >>> Cambridge, I'll be looking into building some dependencies as conda >>> packages on Windows since it'll be easier to work with Rob in person. >>> >>> -- >>> Billy K. Poon >>> Research Scientist, Molecular Biophysics and Integrated Bioimaging >>> Lawrence Berkeley National Laboratory >>> 1 Cyclotron Road, M/S 33R0345 >>> Berkeley, CA 94720 >>> Tel: (510) 486-5709 >>> Fax: (510) 486-5909 >>> Web: https://phenix-online.org<https://phenix-online.org/> >>> >>> >>> On Mon, Sep 3, 2018 at 11:03 AM Graeme.Winter(a)Diamond.ac.uk<mailto: >>> Graeme.Winter(a)Diamond.ac.uk> <Graeme.Winter(a)diamond.ac.uk<mailto: >>> Graeme.Winter(a)diamond.ac.uk>> wrote: >>> Nick, >>> >>> Fair enough, no problem >>> >>> I had a memory that there was a push from lcls via new psana? Anyhow, no >>> worries >>> >>> Cheers Graeme >>> >>> >>> ________________________________ >>> From: cctbxbb-bounces(a)phenix-online.org<mailto:cctbxbb-bounces@phe >>> nix-online.org> <cctbxbb-bounces(a)phenix-online.org<mailto: >>> cctbxbb-bounces(a)phenix-online.org>> on behalf of Nicholas Sauter < >>> nksauter(a)lbl.gov<mailto:[email protected]>> >>> Sent: 03 September 2018 18:57:03 >>> To: cctbx mailing list >>> Subject: Re: [cctbxbb] Python 3 status >>> >>> I've just begun syntax conversion (1% done). I expect Billy and I will >>> work on it over the next few months. I don't believe it is a high priority >>> discu <https://maps.google.com/?q=priority+discu&entry=gmail&source=g>ssion >>> topic at present. >>> NKS >>> >>> Nicholas K. Sauter, Ph. D. >>> Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division >>> Lawrence Berkeley National Laboratory >>> 1 Cyclotron Rd., Bldg. 33R0345 >>> Berkeley, CA 94720 >>> (510) 486-5713 >>> >>> On Mon, Sep 3, 2018 at 12:35 AM, Graeme.Winter(a)Diamond.ac.uk<mailto: >>> Graeme.Winter(a)Diamond.ac.uk><mailto:[email protected]<mailto: >>> Graeme.Winter(a)Diamond.ac.uk>> <Graeme.Winter(a)diamond.ac.uk<mailto: >>> Graeme.Winter(a)diamond.ac.uk><mailto:[email protected]<mailto: >>> Graeme.Winter(a)diamond.ac.uk>>> wrote: >>> Howdy, >>> >>> OK, so I am guessing that the radio silence on this thread indicates >>> that I’ve probably not missed much, and this is not really being actively >>> pursued at this time in core CCTBX, so I will assume there is no interest >>> in discussing progress in person later in the month - it’ll keep. Obviously >>> getting the PHENIX and DIALS major releases out of the door as been / is >>> the priority. >>> >>> best wishes Graeme >>> >>> >>> On 29 Aug 2018, at 12:47, Graeme.Winter(a)Diamond.ac.uk<mailto: >>>> Graeme.Winter(a)Diamond.ac.uk><mailto:[email protected]<mailto: >>>> Graeme.Winter(a)Diamond.ac.uk>> <Graeme.Winter(a)diamond.ac.uk<mailto: >>>> Graeme.Winter(a)diamond.ac.uk><mailto:[email protected]<mailto: >>>> Graeme.Winter(a)diamond.ac.uk>>> wrote: >>>> >>>> Hi Folks, >>>> >>>> Are there any updates on the progress to Python 3 compatibility? I know >>>> there is a project here >>>> >>>> https://github.com/cctbx/cctbx_project/projects/2 >>>> >>>> but I have lost the thread of activities since I have been out of the >>>> office much of the summer. At the very least I know no progress has been >>>> made from this side of the pond as we’ve put this work down for now. >>>> >>>> i would guess that a lot is on hold while the PHENIX release is done - >>>> however I thought I should check in case I have missed anything? >>>> >>>> Part of why I ask is that there is a PHENIX meeting in Cambridge real >>>> soon now (mid September) and I wonder if it is worth folks from DIALS and >>>> PHENIX getting together at that time to discuss this? >>>> >>>> Any thoughts? >>>> >>>> Thanks & best wishes Graeme >>>> >>>> -- >>>> This e-mail and any attachments may contain confidential, copyright and >>>> or privileged material, and are for the use of the intended addressee only. >>>> If you are not the intended addressee or an authorised recipient of the >>>> addressee please notify us of receipt by returning the e-mail and do not >>>> use, copy, retain, distribute or disclose the information in or attached to >>>> the e-mail. >>>> Any opinions expressed within this e-mail are those of the individual >>>> and not necessarily of Diamond Light Source Ltd. >>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any >>>> attachments are free from viruses and we cannot accept liability for any >>>> damage which you may sustain as a result of software viruses which may be >>>> transmitted in or with the message. >>>> Diamond Light Source Limited (company no. 4375679). Registered in >>>> England and Wales with its registered office at Diamond House, Harwell >>>> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom >>>> >>>> _______________________________________________ >>>> cctbxbb mailing list >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto: >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]>> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >>>> >>> >>> _______________________________________________ >>> cctbxbb mailing list >>> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto: >>> cctbxbb(a)phenix-online.org<mailto:[email protected]>> >>> http://phenix-online.org/mailman/listinfo/cctbxbb >>> >>> >>> _______________________________________________ >>> cctbxbb mailing list >>> cctbxbb(a)phenix-online.org<mailto:[email protected]> >>> http://phenix-online.org/mailman/listinfo/cctbxbb >>> _______________________________________________ >>> cctbxbb mailing list >>> cctbxbb(a)phenix-online.org<mailto:[email protected]> >>> http://phenix-online.org/mailman/listinfo/cctbxbb >>> >>> >>> _______________________________________________ >>> cctbxbb mailing list >>> cctbxbb(a)phenix-online.org >>> http://phenix-online.org/mailman/listinfo/cctbxbb >>> >> >> > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb >

7 years, 5 months

Re: [cctbxbb] some thoughts on cctbx and pip

by Billy Poon

Hi Gergely, Let me build the test package tomorrow. All the gritty details for building with conda is being finalized and the official documentation will be updated to describe the steps. It would be too confusing to keep changing the documentation as the process evolves. You do have the general process, though, which is summarized as follows. 1) Installing dependencies. The cctbx_dependencies metapackage was an initial approach for managing the CCTBX dependencies, but after contacting the conda-forge folks, they recommended using the --only-deps flag. So when the CCTBX conda package is available, you'll be able to get a set of dependencies with, conda install -c conda-forge --only-deps cctbx By default, the bootstrap.py file will automatically install a set of dependencies in the "conda_base" directory (and a conda installation if one is not found). It just uses standard conda environment files located in libtbx/auto_build/conda_envs, so you do not need to install cctbx_dependencies as a separate step. The environment files avoid channel issues by explicitly defining the channel to pull the packages from and the cctbx channel just stores copies of packages from conda-forge. There were issues earlier where the conda-forge packages would sometimes be moved to a different label. The --use-conda flag also accepts a path to $CONDA_PREFIX if you want to use a specific environment for testing. 2) Building. The bootstrap.py file handles that with SCons. 3) Running. After building, there should be a setpaths.sh (and .csh) file that adds build/bin to your path. The build/bin directory has our dispatchers, which are just scripts that set up the environment variables for you. This prevents other programs from loading our libraries, whose versions may conflict. You should see that there is a "python" dispatcher, which is a convenience for developers. Otherwise, you can use libtbx.python, which will be able to import CCTBX modules. The running part is where the conda package for CCTBX will be different than this build. Since our Python files and extensions modules will be in the "site-packages" directory for Python, the PYTHONPATH variable will not be needed (and conda suggests that that variable not be set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so LD_LIBRARY_PATH is not needed. In an active environment, PATH will already be modified. And then our LIBTBX_BUILD directory can be set to sys.prefix. So with the conda package, you would only need to activate your conda environment and CCTBX should integrate with other conda packages. What is the conflict with hdf5? That's something that should be fixed. Thanks! -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona <gkatona(a)gmail.com> wrote: > Dear Billy, > > Thank you for this update and for your efforts! I found a solution and > indeed most things already work in anaconda3. The steps I took (even > if these instructions will have short expiry date): > > Modified .condarc with: > channels: > - cctbx > - conda-forge > - defaults > > run > conda conda install cctbx_dependencies python=3.6 > > I expect this will work without python=3.6 in the near future. > > Then compiling cctbx with anaconda3 python went without any problem > when using these flags: > python bootstrap.py --use-conda --python3 --nproc=12 > > Then I probably did the unorthodox thing and sourced these directories: > > setenv LIBTBX_BUILD /home/gergely/cctbx/build > setenv PATH ${PATH}:/home/gergely/cctbx/build/bin > setenv PYTHONPATH > > /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH > setenv LD_LIBRARY_PATH > > /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH > > There is probably a better way to put this into conda environment. > With these steps I could run one of my scripts depending on cctbx. The > only problem I found is that hdf5 library had conflict with another > package in conda, but I does not find this as a showstopper. > > Best wishes, > > Gergely > > > > > > > On Fri, Dec 13, 2019 at 7:23 PM Billy Poon <BKPoon(a)lbl.gov> wrote: > > > > Hi Gergely, > > > > It's still a work in progress. I'm sorting out some Windows issues right > now. I can probably build a test package on a separate channel for people > that want to test it (let's say next week?). I'll provide instructions, but > basically, the test conda package will be in its own separate channel and > the dependencies will be pulled from the conda-forge channel. I want most > things to be working correctly on Python 2.7, 3.6, 3.7, and 3.8 on all 3 > platforms. > > > > Thanks! > > > > -- > > Billy K. Poon > > Research Scientist, Molecular Biophysics and Integrated Bioimaging > > Lawrence Berkeley National Laboratory > > 1 Cyclotron Road, M/S 33R0345 > > Berkeley, CA 94720 > > Tel: (510) 486-5709 > > Fax: (510) 486-5909 > > Web: https://phenix-online.org > > > > > > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona <gkatona(a)gmail.com> > wrote: > >> > >> Dear Billy, > >> > >> This sounds very promising and exciting. I am not sure if cctbx is > >> already functional as a conda package in anaconda3 (Linux) or this is > >> still work in progress. My technical expertise does not allow me to > >> tell the difference. What I tried: > >> > >> Fresh install of anaconda3. Adding - cctbx and - conda-forge to > >> .condarc . Installing pyside2 with conda. Running conda install > >> conda_dependencies . I get a lot package version conflict, and I > >> cannot import cctbx or iotbx to anaconda python. Am I following the > >> right instructions? Or it is too early to expect that cctbx works when > >> installed through conda? > >> > >> Best wishes, > >> > >> Gergely > >> > >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon <BKPoon(a)lbl.gov> wrote: > >> > > >> > Hi all, > >> > > >> > For a brief update, I have submitted a recipe for cctbxlite to > conda-forge (https://github.com/conda-forge/staged-recipes/pull/10021) > and support for Python 3.7 and 3.8 is being added ( > https://github.com/cctbx/cctbx_project/pull/409). With some fixes for > Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms > (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8. > Some additional changes will be needed to get Windows to work with Python 3 > and for tests to pass with Boost 1.70.0. That will enable the conda-forge > recipe to build for all platforms and for all supported versions of Python. > >> > > >> > Currently, the conda-forge recipe will install into the conda python > and cctbx imports can be done without sourcing the environment scripts. It > looks like a lot of the environment variables being set in the dispatchers > can be removed since the Python files and C++ extensions are in the right > places. I'll update the libtbx_env file so that commands that load the > environment can work correctly. > >> > > >> > -- > >> > Billy K. Poon > >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging > >> > Lawrence Berkeley National Laboratory > >> > 1 Cyclotron Road, M/S 33R0345 > >> > Berkeley, CA 94720 > >> > Tel: (510) 486-5709 > >> > Fax: (510) 486-5909 > >> > Web: https://phenix-online.org > >> > > >> > > >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll <tic20(a)cam.ac.uk> > wrote: > >> >> > >> >> Hi Luc, > >> >> > >> >> That sounds promising. From there, I’d need to work out how to make > a fully-packaged installer (basically a modified wheel file) for the > ChimeraX ToolShed - the aim is for the end user to not have to worry about > any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would > need to be set dynamically before first import - but doesn’t seem > insurmountable. > >> >> > >> >> Thanks, > >> >> > >> >> Tristan > >> >> > >> >> > >> >> > >> >> Tristan Croll > >> >> Research Fellow > >> >> Cambridge Institute for Medical Research > >> >> University of Cambridge CB2 0XY > >> >> > >> >> > >> >> > >> >> > >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis <luc_j_bourhis(a)mac.com> > wrote: > >> >> > > >> >> > Hi Tristan, > >> >> > > >> >> > cctbx could be built to use your ChimeraX python, now that cctbx > is moving to Python 3. The option —with-python is there for that with the > bootstrap script. The specific environment setup boil down to setting two > environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux, PATH > on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a framework > such as ChimeraX, that should not be difficult to ensure those two > variables are set. > >> >> > > >> >> > Best wishes, > >> >> > > >> >> > Luc > >> >> > > >> >> > > >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll <tic20(a)cam.ac.uk> wrote: > >> >> >> > >> >> >> To add my two cents on this: probably the second-most common > question I've had about ISOLDE's implementation is, "why didn't you use > CCTBX?". The honest answer to that is, "I didn't know how." > >> >> >> > >> >> >> Still don't, really - although the current developments are > rather promising. The problem I've faced is that CCTBX was designed as its > own self-contained Python (2.7, until very recently) environment, with its > own interpreter and a lot of very specific environment setup. Meanwhile I'm > developing ISOLDE in ChimeraX, which is *also* its own self-contained > Python (3.7) environment. To plug one into the other in that form... well, > I don't think I'm a good enough programmer to really know where to start. > >> >> >> > >> >> >> The move to Conda and a more modular CCTBX architecture should > make a lot more possible in that direction. Pip would be even better for me > personally (ChimeraX can install directly from the PyPI, but doesn't > interact with Conda) - but I understand pretty well the substantial > challenge that would amount to (not least being that the PyPI imposes a > limit - around 100MB from memory? - on the size of an individual package). > >> >> >> > >> >> >> Best regards, > >> >> >> > >> >> >> Tristan > >> >> >> > >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote: > >> >> >>> Hi Graeme, > >> >> >>> Yes, I know. But “black" is a program doing a very particular > task > >> >> >>> (code formatting from the top of my head). Requiring to use a > wrapper > >> >> >>> for python itself is another level. But ok, I think I am > mellowing to > >> >> >>> the idea after all! Talking with people around me, and > extrapolating, > >> >> >>> I would bet that, right now, a great majority of people > interested by > >> >> >>> cctbx in pip have already used the cctbx, so they know about the > >> >> >>> Python wrapper, and they would not be too sanguine about that. My > >> >> >>> concern is for the future, when pip will be the first time some > people > >> >> >>> use cctbx. Big fat warning notices on PyPI page and a better > error > >> >> >>> message when cctbx fails because LIBTBX_BUILD is not set would be > >> >> >>> needed but that could be all right. > >> >> >>> If we do a pip installer, we should aim at a minimal install: > cctbx, > >> >> >>> iotbx and their dependencies, and that’s it. > >> >> >>> Best wishes, > >> >> >>> Luc > >> >> >>>> On 23 Aug 2019, at 07:17, Graeme.Winter(a)Diamond.ac.uk < > Graeme.Winter(a)diamond.ac.uk> wrote: > >> >> >>>> Without discussing the merits of this or whether we _choose_ to > make the move to supporting PIP, I am certain it would be _possible_ - many > other packages make dispatcher scripts when you pip install them e.g. > >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat $(which black) > >> >> >>>> /Library/Frameworks/Python.framework/Versions/3.6/bin/black > >> >> >>>> > #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 > >> >> >>>> # -*- coding: utf-8 -*- > >> >> >>>> import re > >> >> >>>> import sys > >> >> >>>> from black import main > >> >> >>>> if __name__ == '__main__': > >> >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', > sys.argv[0]) > >> >> >>>> sys.exit(main()) > >> >> >>>> So we _could_ work around the absence of LIBTBX_BUILD etc. in > the system. Whether or not we elect to do the work is a different question, > and it seems clear that here are very mixed opinions on this. > >> >> >>>> Best wishes Graeme > >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis <luc_j_bourhis(a)mac.com > <mailto:[email protected]>> wrote: > >> >> >>>> Hi, > >> >> >>>> Even if we managed to ship our the boost dynamic libraries with > pip, it would still not be pip-like, as we would still need our python > wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work > with the standard python exe. LD_LIBRARY_PATH, we could get around that by > changing the way we compile, using -Wl,-R, which is the runtime equivalent > of build time -L. That’s a significant change that would need to be tested. > But there is no way around setting LIBTBX_BUILD right now. Leaving that to > the user is horrible. Perhaps there is a way to hack libtbx/env_config.py > so that we can hardwire LIBTBX_BUILD in there when pip installs? > >> >> >>>> Best wishes, > >> >> >>>> Luc > >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis <luc_j_bourhis(a)mac.com > <mailto:[email protected]>> wrote: > >> >> >>>> Hi, > >> >> >>>> I did look into that many years ago, and even toyed with > building a pip installer. What stopped me is the exact conclusion you > reached too: the user would not have the pip experience he expects. You are > right that it is a lot of effort but is it worth it? Considering that > remark, I don’t think so. Now, Conda was created specifically to go beyond > pip pure-python-only support. Since cctbx has garnered support for Conda, > the best avenue imho is to go the extra length to have a package on > Anaconda.org<http://anaconda.org/>, and then to advertise it hard to > every potential user out there. > >> >> >>>> Best wishes, > >> >> >>>> Luc > >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster <asbrewster(a)lbl.gov > <mailto:[email protected]>> wrote: > >> >> >>>> Hi, to avoid clouding Dorothee's documentation email thread, > which I think is a highly useful enterprise, here's some thoughts about > putting cctbx into pip. Pip doesn't install non-python dependencies well. > I don't think boost is available as a package on pip (at least the package > version we use). wxPython4 isn't portable through pip ( > https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython…). > MPI libraries are system dependent. If cctbx were a pure python package, > pip would be fine, but cctbx is not. > >> >> >>>> All that said, we could build a manylinux1 version of cctbx and > upload it to PyPi (I'm just learning about this). For a pip package to be > portable (which is a requirement for cctbx), it needs to conform to PEP513, > the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For > example, numpy is built according to this standard (see > https://pypi.org/project/numpy/#files, where you'll see the manylinux1 > wheel). Note, the manylinux1 standard is built with Centos 5.11 which we > no longer support. > >> >> >>>> There is also a manylinux2010 standard, which is based on > Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a > more attainable target (note though by default C++11 is not supported on > Centos 6). > >> >> >>>> If we built a manylinuxX version of cctbx and uploaded it to > PyPi, the user would need all the non-python dependencies. There's no way > to specify these in pip. For example, cctbx requires boost 1.63 or > better. The user will need to have it in a place their python can find it, > or we could package it ourselves and supply it, similar to how the pip h5py > package now comes with an hd5f library, or how the pip numpy package > includes an openblas library. We'd have to do the same for any packages we > depend on that aren't on pip using the manylinux standards, such as > wxPython4. > >> >> >>>> Further, we need to think about how dials and other cctbx-based > packages interact. If pip install cctbx is set up, how does pip install > dials work, such that any dials shared libraries can find the cctbx > libraries? Can shared libraries from one pip package link against > libraries in another pip package? Would each package need to supply its > own boost? Possibly this is well understood in the pip field, but not by > me :) > >> >> >>>> Finally, there's the option of providing a source pip package. > This would require the full compiler toolchain for any given platform > (macOS, linux, windows). These are likely available for developers, but > not for general users. > >> >> >>>> Anyway, these are some of the obstacles. Not saying it isn't > possible, it's just a lot of effort. > >> >> >>>> Thanks, > >> >> >>>> -Aaron > >> >> >>>> _______________________________________________ > >> >> >>>> cctbxbb mailing list > >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]> > >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> >>>> _______________________________________________ > >> >> >>>> cctbxbb mailing list > >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]> > >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> >>>> _______________________________________________ > >> >> >>>> cctbxbb mailing list > >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]> > >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> >>>> -- > >> >> >>>> This e-mail and any attachments may contain confidential, > copyright and or privileged material, and are for the use of the intended > addressee only. If you are not the intended addressee or an authorised > recipient of the addressee please notify us of receipt by returning the > e-mail and do not use, copy, retain, distribute or disclose the information > in or attached to the e-mail. > >> >> >>>> Any opinions expressed within this e-mail are those of the > individual and not necessarily of Diamond Light Source Ltd. > >> >> >>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or > any attachments are free from viruses and we cannot accept liability for > any damage which you may sustain as a result of software viruses which may > be transmitted in or with the message. > >> >> >>>> Diamond Light Source Limited (company no. 4375679). Registered > in England and Wales with its registered office at Diamond House, Harwell > Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom > >> >> >>>> _______________________________________________ > >> >> >>>> cctbxbb mailing list > >> >> >>>> cctbxbb(a)phenix-online.org > >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> >>> _______________________________________________ > >> >> >>> cctbxbb mailing list > >> >> >>> cctbxbb(a)phenix-online.org > >> >> >>> http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> >> > >> >> >> > >> >> >> _______________________________________________ > >> >> >> cctbxbb mailing list > >> >> >> cctbxbb(a)phenix-online.org > >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> > > >> >> > > >> >> > _______________________________________________ > >> >> > cctbxbb mailing list > >> >> > cctbxbb(a)phenix-online.org > >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> > >> >> > >> >> _______________________________________________ > >> >> cctbxbb mailing list > >> >> cctbxbb(a)phenix-online.org > >> >> http://phenix-online.org/mailman/listinfo/cctbxbb > >> > > >> > _______________________________________________ > >> > cctbxbb mailing list > >> > cctbxbb(a)phenix-online.org > >> > http://phenix-online.org/mailman/listinfo/cctbxbb > >> > >> > >> > >> -- > >> Gergely Katona, PhD > >> Associate Professor > >> Department of Chemistry and Molecular Biology, University of Gothenburg > >> Box 462, 40530 Göteborg, Sweden > >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 > >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se > >> > >> _______________________________________________ > >> cctbxbb mailing list > >> cctbxbb(a)phenix-online.org > >> http://phenix-online.org/mailman/listinfo/cctbxbb > > > > _______________________________________________ > > cctbxbb mailing list > > cctbxbb(a)phenix-online.org > > http://phenix-online.org/mailman/listinfo/cctbxbb > > > > -- > Gergely Katona, PhD > Associate Professor > Department of Chemistry and Molecular Biology, University of Gothenburg > Box 462, 40530 Göteborg, Sweden > Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 > Web: http://katonalab.eu, Email: gergely.katona(a)gu.se > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb >

6 years, 1 month

Re: [cctbxbb] some thoughts on cctbx and pip

by Gergely Katona

Dear Billy, Thank you for the detailed explanation, I look forward to do further testing! I aim to pool all modules under the same environment, if there are conflicts then I just try to reshuffle the order of imports. So far this did not cause problems for me even when I was using system python, but of course it is not the most prudent thing to do. With anaconda everything is much more standardized and isolated already and without being superuser I can have my familiar environment at any synchrotron based cluster. It is great that cctbx will be an integral part of this ecosystem and this was also the last thing holding me back from adopting python3. About hdf5, pymc3 requires it and importing causes a kernel restart with the following error messages, curiously if I import h5py first this can be avoided. Best wishes, Gergely import pymc3 as pm Warning! ***HDF5 library version mismatched error*** The HDF5 header files used to compile this application do not match the version used by the HDF5 library to which this application is linked. Data corruption or segmentation faults may occur if the application continues. This can happen when an application was compiled by one version of HDF5 but linked with a different version of static or shared HDF5 library. You should recompile the application or check your shared library related settings such as 'LD_LIBRARY_PATH'. You can, at your own risk, disable this warning by setting the environment variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'. Setting it to 2 or higher will suppress the warning messages totally. Headers are 1.10.4, library is 1.10.5 SUMMARY OF THE HDF5 CONFIGURATION ================================= General Information: ------------------- HDF5 Version: 1.10.5 Configured on: Tue Oct 22 12:02:13 UTC 2019 Configured by: conda@16247e67ecd5 Host system: x86_64-conda_cos6-linux-gnu Uname information: Linux 16247e67ecd5 4.15.0-1059-azure #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Byte sex: little-endian Installation point: /home/gergely/anaconda3 Compiling Options: ------------------ Build Mode: production Debugging Symbols: no Asserts: no Profiling: no Optimization Level: high Linking Options: ---------------- Libraries: static, shared Statically Linked Executables: LDFLAGS: -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib -Wl,-rpath-link,/home/gergely/anaconda3/lib -L/home/gergely/anaconda3/lib H5_LDFLAGS: AM_LDFLAGS: -L/home/gergely/anaconda3/lib Extra libraries: -lrt -lpthread -lz -ldl -lm Archiver: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar AR_FLAGS: cr Ranlib: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib Languages: ---------- C: yes C Compiler: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/home/gergely/anaconda3/include H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API AM_CPPFLAGS: -I/home/gergely/anaconda3/include C Flags: -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/home/gergely/anaconda3/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5 -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix H5 C Flags: -std=c99 -pedantic -Wall -Wextra -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings -finline-functions -s -Wno-inline -Wno-aggregate-return -Wno-missing-format-attribute -Wno-missing-noreturn -O AM C Flags: Shared C Library: yes Static C Library: yes Fortran: yes Fortran Compiler: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran Fortran Flags: H5 Fortran Flags: -pedantic -Wall -Wextra -Wunderflow -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2 AM Fortran Flags: Shared Fortran Library: yes Static Fortran Library: yes C++: yes C++ Compiler: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++ C++ Flags: -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/home/gergely/anaconda3/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5 -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix H5 C++ Flags: -pedantic -Wall -W -Wundef -Wshadow -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor -Wctor-dtor-privacy -Wabi -finline-functions -s -O AM C++ Flags: Shared C++ Library: yes Static C++ Library: yes Java: no Features: --------- Parallel HDF5: no Parallel Filtered Dataset Writes: no Large Parallel I/O: no High-level library: yes Threadsafety: yes Default API mapping: v110 With deprecated public symbols: yes I/O filters (external): deflate(zlib) MPE: no Direct VFD: no dmalloc: no Packages w/ extra debug output: none API tracing: no Using memory checker: yes Memory allocation sanity checks: no Function stack tracing: no Strict file format checks: no Optimization instrumentation: no On Tue, Dec 17, 2019 at 8:35 AM Billy Poon <BKPoon(a)lbl.gov> wrote: > > Hi Gergely, > > Let me build the test package tomorrow. All the gritty details for building with conda is being finalized and the official documentation will be updated to describe the steps. It would be too confusing to keep changing the documentation as the process evolves. You do have the general process, though, which is summarized as follows. > > 1) Installing dependencies. The cctbx_dependencies metapackage was an initial approach for managing the CCTBX dependencies, but after contacting the conda-forge folks, they recommended using the --only-deps flag. So when the CCTBX conda package is available, you'll be able to get a set of dependencies with, > > conda install -c conda-forge --only-deps cctbx > > By default, the bootstrap.py file will automatically install a set of dependencies in the "conda_base" directory (and a conda installation if one is not found). It just uses standard conda environment files located in libtbx/auto_build/conda_envs, so you do not need to install cctbx_dependencies as a separate step. The environment files avoid channel issues by explicitly defining the channel to pull the packages from and the cctbx channel just stores copies of packages from conda-forge. There were issues earlier where the conda-forge packages would sometimes be moved to a different label. The --use-conda flag also accepts a path to $CONDA_PREFIX if you want to use a specific environment for testing. > > 2) Building. The bootstrap.py file handles that with SCons. > > 3) Running. After building, there should be a setpaths.sh (and .csh) file that adds build/bin to your path. The build/bin directory has our dispatchers, which are just scripts that set up the environment variables for you. This prevents other programs from loading our libraries, whose versions may conflict. You should see that there is a "python" dispatcher, which is a convenience for developers. Otherwise, you can use libtbx.python, which will be able to import CCTBX modules. > > The running part is where the conda package for CCTBX will be different than this build. Since our Python files and extensions modules will be in the "site-packages" directory for Python, the PYTHONPATH variable will not be needed (and conda suggests that that variable not be set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so LD_LIBRARY_PATH is not needed. In an active environment, PATH will already be modified. And then our LIBTBX_BUILD directory can be set to sys.prefix. > > So with the conda package, you would only need to activate your conda environment and CCTBX should integrate with other conda packages. What is the conflict with hdf5? That's something that should be fixed. Thanks! > > -- > Billy K. Poon > Research Scientist, Molecular Biophysics and Integrated Bioimaging > Lawrence Berkeley National Laboratory > 1 Cyclotron Road, M/S 33R0345 > Berkeley, CA 94720 > Tel: (510) 486-5709 > Fax: (510) 486-5909 > Web: https://phenix-online.org > > > On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona <gkatona(a)gmail.com> wrote: >> >> Dear Billy, >> >> Thank you for this update and for your efforts! I found a solution and >> indeed most things already work in anaconda3. The steps I took (even >> if these instructions will have short expiry date): >> >> Modified .condarc with: >> channels: >> - cctbx >> - conda-forge >> - defaults >> >> run >> conda conda install cctbx_dependencies python=3.6 >> >> I expect this will work without python=3.6 in the near future. >> >> Then compiling cctbx with anaconda3 python went without any problem >> when using these flags: >> python bootstrap.py --use-conda --python3 --nproc=12 >> >> Then I probably did the unorthodox thing and sourced these directories: >> >> setenv LIBTBX_BUILD /home/gergely/cctbx/build >> setenv PATH ${PATH}:/home/gergely/cctbx/build/bin >> setenv PYTHONPATH >> /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH >> setenv LD_LIBRARY_PATH >> /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH >> >> There is probably a better way to put this into conda environment. >> With these steps I could run one of my scripts depending on cctbx. The >> only problem I found is that hdf5 library had conflict with another >> package in conda, but I does not find this as a showstopper. >> >> Best wishes, >> >> Gergely >> >> >> >> >> >> >> On Fri, Dec 13, 2019 at 7:23 PM Billy Poon <BKPoon(a)lbl.gov> wrote: >> > >> > Hi Gergely, >> > >> > It's still a work in progress. I'm sorting out some Windows issues right now. I can probably build a test package on a separate channel for people that want to test it (let's say next week?). I'll provide instructions, but basically, the test conda package will be in its own separate channel and the dependencies will be pulled from the conda-forge channel. I want most things to be working correctly on Python 2.7, 3.6, 3.7, and 3.8 on all 3 platforms. >> > >> > Thanks! >> > >> > -- >> > Billy K. Poon >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging >> > Lawrence Berkeley National Laboratory >> > 1 Cyclotron Road, M/S 33R0345 >> > Berkeley, CA 94720 >> > Tel: (510) 486-5709 >> > Fax: (510) 486-5909 >> > Web: https://phenix-online.org >> > >> > >> > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona <gkatona(a)gmail.com> wrote: >> >> >> >> Dear Billy, >> >> >> >> This sounds very promising and exciting. I am not sure if cctbx is >> >> already functional as a conda package in anaconda3 (Linux) or this is >> >> still work in progress. My technical expertise does not allow me to >> >> tell the difference. What I tried: >> >> >> >> Fresh install of anaconda3. Adding - cctbx and - conda-forge to >> >> .condarc . Installing pyside2 with conda. Running conda install >> >> conda_dependencies . I get a lot package version conflict, and I >> >> cannot import cctbx or iotbx to anaconda python. Am I following the >> >> right instructions? Or it is too early to expect that cctbx works when >> >> installed through conda? >> >> >> >> Best wishes, >> >> >> >> Gergely >> >> >> >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon <BKPoon(a)lbl.gov> wrote: >> >> > >> >> > Hi all, >> >> > >> >> > For a brief update, I have submitted a recipe for cctbxlite to conda-forge (https://github.com/conda-forge/staged-recipes/pull/10021) and support for Python 3.7 and 3.8 is being added (https://github.com/cctbx/cctbx_project/pull/409). With some fixes for Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8. Some additional changes will be needed to get Windows to work with Python 3 and for tests to pass with Boost 1.70.0. That will enable the conda-forge recipe to build for all platforms and for all supported versions of Python. >> >> > >> >> > Currently, the conda-forge recipe will install into the conda python and cctbx imports can be done without sourcing the environment scripts. It looks like a lot of the environment variables being set in the dispatchers can be removed since the Python files and C++ extensions are in the right places. I'll update the libtbx_env file so that commands that load the environment can work correctly. >> >> > >> >> > -- >> >> > Billy K. Poon >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging >> >> > Lawrence Berkeley National Laboratory >> >> > 1 Cyclotron Road, M/S 33R0345 >> >> > Berkeley, CA 94720 >> >> > Tel: (510) 486-5709 >> >> > Fax: (510) 486-5909 >> >> > Web: https://phenix-online.org >> >> > >> >> > >> >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll <tic20(a)cam.ac.uk> wrote: >> >> >> >> >> >> Hi Luc, >> >> >> >> >> >> That sounds promising. From there, I’d need to work out how to make a fully-packaged installer (basically a modified wheel file) for the ChimeraX ToolShed - the aim is for the end user to not have to worry about any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would need to be set dynamically before first import - but doesn’t seem insurmountable. >> >> >> >> >> >> Thanks, >> >> >> >> >> >> Tristan >> >> >> >> >> >> >> >> >> >> >> >> Tristan Croll >> >> >> Research Fellow >> >> >> Cambridge Institute for Medical Research >> >> >> University of Cambridge CB2 0XY >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis <luc_j_bourhis(a)mac.com> wrote: >> >> >> > >> >> >> > Hi Tristan, >> >> >> > >> >> >> > cctbx could be built to use your ChimeraX python, now that cctbx is moving to Python 3. The option —with-python is there for that with the bootstrap script. The specific environment setup boil down to setting two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a framework such as ChimeraX, that should not be difficult to ensure those two variables are set. >> >> >> > >> >> >> > Best wishes, >> >> >> > >> >> >> > Luc >> >> >> > >> >> >> > >> >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll <tic20(a)cam.ac.uk> wrote: >> >> >> >> >> >> >> >> To add my two cents on this: probably the second-most common question I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". The honest answer to that is, "I didn't know how." >> >> >> >> >> >> >> >> Still don't, really - although the current developments are rather promising. The problem I've faced is that CCTBX was designed as its own self-contained Python (2.7, until very recently) environment, with its own interpreter and a lot of very specific environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is *also* its own self-contained Python (3.7) environment. To plug one into the other in that form... well, I don't think I'm a good enough programmer to really know where to start. >> >> >> >> >> >> >> >> The move to Conda and a more modular CCTBX architecture should make a lot more possible in that direction. Pip would be even better for me personally (ChimeraX can install directly from the PyPI, but doesn't interact with Conda) - but I understand pretty well the substantial challenge that would amount to (not least being that the PyPI imposes a limit - around 100MB from memory? - on the size of an individual package). >> >> >> >> >> >> >> >> Best regards, >> >> >> >> >> >> >> >> Tristan >> >> >> >> >> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote: >> >> >> >>> Hi Graeme, >> >> >> >>> Yes, I know. But “black" is a program doing a very particular task >> >> >> >>> (code formatting from the top of my head). Requiring to use a wrapper >> >> >> >>> for python itself is another level. But ok, I think I am mellowing to >> >> >> >>> the idea after all! Talking with people around me, and extrapolating, >> >> >> >>> I would bet that, right now, a great majority of people interested by >> >> >> >>> cctbx in pip have already used the cctbx, so they know about the >> >> >> >>> Python wrapper, and they would not be too sanguine about that. My >> >> >> >>> concern is for the future, when pip will be the first time some people >> >> >> >>> use cctbx. Big fat warning notices on PyPI page and a better error >> >> >> >>> message when cctbx fails because LIBTBX_BUILD is not set would be >> >> >> >>> needed but that could be all right. >> >> >> >>> If we do a pip installer, we should aim at a minimal install: cctbx, >> >> >> >>> iotbx and their dependencies, and that’s it. >> >> >> >>> Best wishes, >> >> >> >>> Luc >> >> >> >>>> On 23 Aug 2019, at 07:17, Graeme.Winter(a)Diamond.ac.uk <Graeme.Winter(a)diamond.ac.uk> wrote: >> >> >> >>>> Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g. >> >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat $(which black) >> >> >> >>>> /Library/Frameworks/Python.framework/Versions/3.6/bin/black >> >> >> >>>> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 >> >> >> >>>> # -*- coding: utf-8 -*- >> >> >> >>>> import re >> >> >> >>>> import sys >> >> >> >>>> from black import main >> >> >> >>>> if __name__ == '__main__': >> >> >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) >> >> >> >>>> sys.exit(main()) >> >> >> >>>> So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this. >> >> >> >>>> Best wishes Graeme >> >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote: >> >> >> >>>> Hi, >> >> >> >>>> Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs? >> >> >> >>>> Best wishes, >> >> >> >>>> Luc >> >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote: >> >> >> >>>> Hi, >> >> >> >>>> I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.org<http://anaconda.org/>, and then to advertise it hard to every potential user out there. >> >> >> >>>> Best wishes, >> >> >> >>>> Luc >> >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster <asbrewster(a)lbl.gov<mailto:[email protected]>> wrote: >> >> >> >>>> Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython…). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not. >> >> >> >>>> All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support. >> >> >> >>>> There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6). >> >> >> >>>> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4. >> >> >> >>>> Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :) >> >> >> >>>> Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users. >> >> >> >>>> Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort. >> >> >> >>>> Thanks, >> >> >> >>>> -Aaron >> >> >> >>>> _______________________________________________ >> >> >> >>>> cctbxbb mailing list >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >>>> _______________________________________________ >> >> >> >>>> cctbxbb mailing list >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >>>> _______________________________________________ >> >> >> >>>> cctbxbb mailing list >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >>>> -- >> >> >> >>>> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. >> >> >> >>>> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. >> >> >> >>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. >> >> >> >>>> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom >> >> >> >>>> _______________________________________________ >> >> >> >>>> cctbxbb mailing list >> >> >> >>>> cctbxbb(a)phenix-online.org >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >>> _______________________________________________ >> >> >> >>> cctbxbb mailing list >> >> >> >>> cctbxbb(a)phenix-online.org >> >> >> >>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> >> cctbxbb mailing list >> >> >> >> cctbxbb(a)phenix-online.org >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> > >> >> >> > >> >> >> > _______________________________________________ >> >> >> > cctbxbb mailing list >> >> >> > cctbxbb(a)phenix-online.org >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> cctbxbb mailing list >> >> >> cctbxbb(a)phenix-online.org >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> > >> >> > _______________________________________________ >> >> > cctbxbb mailing list >> >> > cctbxbb(a)phenix-online.org >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> -- >> >> Gergely Katona, PhD >> >> Associate Professor >> >> Department of Chemistry and Molecular Biology, University of Gothenburg >> >> Box 462, 40530 Göteborg, Sweden >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 >> >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se >> >> >> >> _______________________________________________ >> >> cctbxbb mailing list >> >> cctbxbb(a)phenix-online.org >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> > >> > _______________________________________________ >> > cctbxbb mailing list >> > cctbxbb(a)phenix-online.org >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> -- >> Gergely Katona, PhD >> Associate Professor >> Department of Chemistry and Molecular Biology, University of Gothenburg >> Box 462, 40530 Göteborg, Sweden >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se >> >> _______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb -- Gergely Katona, PhD Associate Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: gergely.katona(a)gu.se

6 years, 1 month

Re: [cctbxbb] Boost Python 1.56

by markus.gerstel＠diamond.ac.uk

Hi Billy, thank you very much. This should help the dials installer greatly, which blew up from 158 MB to 226 MB :) There are still some nonfunctional files in the boost distribution, which can be removed with find libs -type d | grep "/$test\|doc\|dot_net_example\|example\|examples\|tutorial\|xmldoc$$" | xargs rm -r find libs -type f | grep "$.html\|Jamfile.v2$$" | xargs rm find . -type d -empty -delete and by packing it using gzip -9 instead of the default level gzip compression you can get the download size to below 10MB. (simply "export GZIP=-9" before tar cfz ...) (Incidentally - is there a valid reason why we use gzip instead of at least bzip2 or even xz?) -Markus ________________________________ From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of Billy Poon [bkpoon(a)lbl.gov] Sent: Friday, April 14, 2017 01:02 To: cctbx mailing list Subject: Re: [cctbxbb] Boost Python 1.56 Hi Graeme, I hope Markus's commands were helpful. The only tricky thing was the deletion of the build directory. If you had set up the cctbx environment, then python would have pointed to something in the build directory. You would have need to explicitly use ./base/bin/python or the system python with bootstrap.py instead. Also, I stripped extraneous stuff from the Boost tarball according to how earlier boost tarballs were built. This will improve download times and fix the Windows installation. Basically, only the following things are kept, LICENSE_1_0.txt boost/ libs/date_time libs/detail libs/filesystem libs/program_options libs/python libs/system libs/thread It's available now if you use authenticated access (--cciuser flag for bootstrap.py) and will propagate for unauthenticated access (http://cci.lbl.gov/repositories) by 9:30 pm Pacific time (or 5:30 am UK time). Let me know of any issues. Thanks! -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org On Thu, Apr 13, 2017 at 8:02 AM, R.D. Oeffner <rdo20(a)cam.ac.uk<mailto:[email protected]>> wrote: Sorry this was to Billy only. Please ignore. Rob On 13/04/2017 16:00, R.D. Oeffner wrote: Hi Billy, As you'll know the Windows builds failed with the new boost libraries. this is due to some filenames exceeding the 255 char limit on Windows. It worked fine on my nightly build because the build folder isn't as deeply nested as on pulse and flux. On pulse and flux it dies in tarfile.TarFile._extract_member() which I have made a monkey patch for which appears to work by ignoring the offending files. However, as all the long filenames seem to come from html folders within boost presumably these are just documentation and it would be better to exclude these altogether in the cci subset of boost. People can always go online if they want to read up on the documentation for boost. Besides bootstrap.py is already quite bloated. Let me know what you think, Rob On 12/04/2017 21:47, Billy Poon wrote: Hi everyone, Boost 1.56 is now available and the tests have been updated. To update an existing installtion, you have to delete the "build" directory first. If you don't, I guess some files are not recompiled with the new Boost headers, which will cause cctbx_project/scitbx/random/tests/tst_random.py to expect the old values. Then run bootstrap.py with hot, update, and build to rebuild. cd <installation directory> rm -fr build python bootstrap.py hot update build --builder=cctbx Let me know if there are any issues. Thanks! -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709<tel:(510)%20486-5709> Fax: (510) 486-5909<tel:(510)%20486-5909> Web: https://phenix-online.org On Fri, Apr 7, 2017 at 1:08 PM, Billy Poon <bkpoon(a)lbl.gov<mailto:[email protected]>> wrote: I can do the switch to Boost 1.56 next Wednesday now that Dials 1.5 has been released. I'm just double-checking some other Phenix tests. Once the update is live, bootstrap.py should delete the existing modules/boost directory and replace it with the new one. And then scons should recompile anything that depends on boost. If in doubt, you can manually delete the module/boost and build directories. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709<tel:(510)%20486-5709> Fax: (510) 486-5909<tel:(510)%20486-5909> Web: https://phenix-online.org On Thu, Apr 6, 2017 at 9:32 AM, Billy Poon <bkpoon(a)lbl.gov<mailto:[email protected]>> wrote: Hi James, I agree with Graeme's tests and I would add cctbx_regression.test_nightly That command is a shortcut for running the test modules for libtbx, boost_adaptbx, scitbx, cctbx, iotbx, dxtbx, and smtbx. The mmtbx module is also tested if chem_data is available. Should we add rstbx to the shortcut (cctbx_project/cctbx/command_line/cctbx_test_nightly.py)? -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709<tel:(510)%20486-5709> Fax: (510) 486-5909<tel:(510)%20486-5909> Web: https://phenix-online.org On Thu, Apr 6, 2017 at 1:37 AM, <richard.gildea(a)diamond.ac.uk<mailto:[email protected]>> wrote: I've made a start on transcribing this document here: https://github.com/cctbx/cctbx_project/wiki/cctbx-Developer-Guidance It probably still needs cleaning up a bit (e.g. I couldn't figure out quickly how to do 3-level list nesting, 1.a.i etc.) and updating to reflect current practice (e.g. git not svn). Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078<tel:%2B441235%2077%208078> Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE ________________________________________ From: cctbxbb-bounces(a)phenix-online.org<mailto:[email protected]> [cctbxbb-bounces(a)phenix-online.org<mailto:[email protected]>] on behalf of Pavel Afonine [pafonine(a)lbl.gov<mailto:[email protected]>] Sent: 06 April 2017 09:07 To: cctbx mailing list Subject: Re: [cctbxbb] Boost Python 1.56 Hi Graeme, hm.. this is a good question. We've been through back-and-forth iterations of editing this file and I think the latest I have is from Paul. But I can't find a non-PDF version of it. Paul: do you have an editable version of this file? Thanks, Pavel On 4/6/17 13:45, Graeme.Winter(a)diamond.ac.uk<mailto:[email protected]> wrote: > Hi Pavel > > These all seem sensible > > If you have the original non pdf document it may be easier to transcribe this over.. > > I also note that it lacks the actual detail on how to run tests! However would be happy to add this once on wiki > > Best wishes Graeme > > On 6 Apr 2017, at 04:00, Pavel Afonine <pafonine(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: > > Not sure if that answers your questions but once upon a time we here at Berkeley tried to write a some sort of document that was supposed to answer questions like this. Attached. By no means it is complete, up-to-date, etc, but it might be worth reading for anyone who contributes to cctbx. (Even not sure if I'm sending the latest version). > Unfortunately, nobody bothered to put it in some central place. > > Pavel > > On 4/6/17 10:51, James Holton wrote: > Hey Billy, > > On a related note. How do I run these regression tests before committing something into Git? Is there a document on dials regression testing I can't find? > > -James > > On Apr 5, 2017, at 3:38 PM, Billy Poon <bkpoon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: > > Hi all, > > I tested Boost 1.56 on our buildbot servers and got some new test failures with > > cctbx_project/scitbx/array_family/boost_python/tst_flex.py > cctbx_project/scitbx/random/tests/tst_random.py > > The full log for CentOS 6 can be found at > > http://cci-vm-6.lbl.gov:8010/builders/phenix-nightly-intel-linux-2.6-x86_64… > > It looks like the errors are related to random number generation. For a given seed, would the sequence of numbers change when Boost is changed? I did a diff between Boost 1.56 and the current Boost and could not see any changes that immediately stood out as being related to random numbers. > > Are these tests failing for others as well? > > -- > Billy K. Poon > Research Scientist, Molecular Biophysics and Integrated Bioimaging > Lawrence Berkeley National Laboratory > 1 Cyclotron Road, M/S 33R0345 > Berkeley, CA 94720 > Tel: (510) 486-5709<tel:%28510%29%20486-5709> > Fax: (510) 486-5909<tel:%28510%29%20486-5909> > Web: https://phenix-online.org<https://phenix-online.org/> > > On Wed, Apr 5, 2017 at 8:12 AM, Charles Ballard <charles.ballard(a)stfc.ac.uk<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: > FYI, we (CCP4) have been using 1.56 for building cctbx/phaser/dials for the last while with no issues. Don't know about 1.60, but 1.59 causes issues with the boost python make_getter and make_setter (initialisation of none const reference if the passed type is a temporary). > > Charles > > On 3 Apr 2017, at 14:31, Luc Bourhis wrote: > > Hi all, > > everybody seemed to agree but then it was proposed to move straight to Boost 1.60, and this caused troubles. Could we consider again to move to at least 1.56? As far as I can tell, this does not cause any issue and as stated one year ago, it would help me and Olex 2. > > Thanks, > > Luc > > On 10 Feb 2016, at 15:17, Nicholas Sauter <nksauter(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: > > Nigel, Billy & Aaron, > > I completely endorse this move to Boost 1.56. Can we update our build? > > Nick > > Nicholas K. Sauter, Ph. D. > Computer Staff Scientist, Molecular Biophysics and Integrated Bioimaging Division > Lawrence Berkeley National Laboratory > 1 Cyclotron Rd., Bldg. 33R0345 > Berkeley, CA 94720 > (510) 486-5713<tel:%28510%29%20486-5713><tel:%28510%29%20486-5713> > > On Wed, Feb 10, 2016 at 2:41 PM, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: > Hi, > > I have improvements to the smtbx on their way to be committed which require Boost version 1.56. This is related to Boost.Threads, whose support I re-activated a few months ago on Nick's request. I need the function boost::thread::physical_concurrency which returns the number of physical cores on the machine, as opposed to virtual cores when hyperthreading is enabled (which it is by default on any Intel machine). That function is not available in Boost 1.55 which is the version currently used in the nightly tests: it appeared in 1.56. > > So, would it be possible to move to Boost 1.56? Otherwise, I will need to backport that function. Not too difficult but not thrilling. > > Best wishes, > > Luc > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> > http://phenix-online.org/mailman/listinfo/cctbxbb > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> > http://phenix-online.org/mailman/listinfo/cctbxbb > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> > http://phenix-online.org/mailman/listinfo/cctbxbb > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> > http://phenix-online.org/mailman/listinfo/cctbxbb > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> > http://phenix-online.org/mailman/listinfo/cctbxbb > > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> > http://phenix-online.org/mailman/listinfo/cctbxbb > > > <cctbx-developer-guidance-08-2015.pdf>_______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> > http://phenix-online.org/mailman/listinfo/cctbxbb > > _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org<mailto:[email protected]> http://phenix-online.org/mailman/listinfo/cctbxbb

8 years, 9 months

Re: [phenixbb] AUTOSOL for MAD data with error message-'Need a symfile for solve'

by Thomas C. Terwilliger

Hi Frank and Graeme and Mohan, Yes, P 2 21 21 should be allowed in autosol. I see that it doesn't work and I'll fix that. All the best, Tom T >> Um, this one has been hacked to death on ccp4bb: there is in fact >> nothing wrong or "non-standard" with that setting. If the program won't >> handle it, email the authors and ask them to fix it. >> >> Reindexing is an almighty pain in the butt. >> >> That said: it sounds like you may be pointing at an out-of-date >> symmetry operator file; I forget which environment variable it is, >> check the manual, but point it to an up-to-date ccp4 installation, and I >> suspect it will be fine. >> >> phx >> >> >> >> >> On 02/02/2010 13:49, Graeme Winter wrote: >>> Hi Mohan, >>> >>> P22121 is not the standard setting for this spacegroup - in >>> International Tables the convention is to have the spacegroup's unique >>> axis as C, e.g. P21212 in this case. If you run this reflection file >>> through reindex as follows: >>> >>> reindex hklin in.mtz hklout out.mtz<< eof >>> reindex k,l,h >>> eof >>> >>> Then it should work just fine. An alternative would be to compose a >>> solve "symm" file and put this in the right place. This would look >>> something like: >>> >>> 4 Equiv positions, P22121 SG # 3018 >>> X,Y,Z >>> X,-Y,-Z >>> -X,1/2+Y,1/2-Z >>> -X,1/2-Y,1/2+Z >>> >>> (copied from spacegroup #3018 in the CCP4 symop.lib) which would need to >>> go in >>> >>> phenix-1.6-289/solve_resolve/ext_ref_files >>> >>> On balance, reindexing is probably easiest! >>> >>> Best wishes, >>> >>> Graeme >>> >>> On 2 February 2010 13:38,<m.b.rajasekaran(a)reading.ac.uk> wrote: >>> >>>> >>>> Dear all, >>>> I have a query regarding the Autosol option in the >>>> PHENIX-1.6-289 version. We are trying to process a MAD data for a zinc >>>> bound >>>> protein crystal with AuTOSOL. We uploaded the scaled files, sequence >>>> and all >>>> other details and started running AUTOSOL. But the program stopped with >>>> the >>>> following error message pasted below mentioning some missing of >>>> symfile. We >>>> were not able to locate any parameters file on the installation >>>> directory. >>>> It would be helpful if we get some useful suggestions on this. >>>> >>>> Thanks in advance, >>>> Mohan >>>> *********************************************************** >>>> >>>> PHENIX AutoSol Tue Feb 2 12:37:41 2010 >>>> >>>> ************************************************************ >>>> >>>> Working directory: /home/sar06mbr/30JanDiamondI02/107/107_php22121 >>>> PHENIX VERSION: 1.6 of 29-01-2010 PHENIX RELEASE_TAG : 289 PHENIX MTYPE >>>> : >>>> intel-linux-2.6 PHENIX MVERSION : suse AutoSol_start AutoSol Run 2 Tue >>>> Feb 2 >>>> 12:37:41 2010 INPUT FILE LIST: >>>> ['/home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_2_P22121_scala2.mtz', >>>> '/home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_3_P22121_scala2.mtz', >>>> '/home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_4_P22121_scala2.mtz'] >>>> >>>> Copied /home/sar06mbr/30JanDiamondI02/107/107_php22121/result.seq to >>>> /home/sar06mbr/30JanDiamondI02/107/107_php22121/AutoSol_run_2_/sequence_autosol.dat >>>> >>>> Setting chain_type to PROTEIN Setting defaults for data_quality >>>> moderate >>>> Setting thorough_denmod to Yes Settin fix_xyz_after_denmod to False >>>> Setting >>>> max_ha_iterations to 2 Setting fixscattfactors to No Settin defaults >>>> for >>>> thoroughness quick Setting best_of_n_hyss_always to 1 Setting >>>> max_extra_unique_solutions to 0 Settin ha_iteration to No Setting >>>> max_choices to 1 Setting number_of_models to 1 Setting number_of_builds >>>> to 1 >>>> Setting test_mask_type to No Setting n_cycle_build to 0 Setting >>>> thorough_loop_fit to Setting create_scoring_table to No Not truncating >>>> ha >>>> sites at start of resolve as this is not PHASER SAD phasing >>>> >>>> Trying to guess refinement file for later use from >>>> ['/home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_2_P22121_scala2.mtz', >>>> '/home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_3_P22121_scala2.mtz', >>>> '/home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_4_P22121_scala2.mtz'] >>>> >>>> Choosing guess of refinement file for later use from all files: >>>> ['/home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_2_P22121_scala2.mtz', >>>> '/home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_3_P22121_scala2.mtz', >>>> '/home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_4_P22121_scala2.mtz'] >>>> >>>> FILE TYPE ccp4_mtz >>>> >>>> COLUMN LABELS: ['H', 'K', 'L', 'FreeR_flag', 'F_107_2re', >>>> 'SIGF_107_2re', >>>> 'F_107_2re(+)', 'SIGF_107_2re(+)', 'F_107_2re(-)', 'SIGF_107_2re(-)', >>>> 'DANO_107_2re', 'SIGDANO_107_2re', 'IMEAN_107_2re', 'SIGIMEAN_107_2re', >>>> 'I_107_2re(+)', 'SIGI_107_2re(+)', 'I_107_2re(-)', 'SIGI_107_2re(-)'] >>>> >>>> GUESS FILE TYPE MERGE TYPE mtz premerged TARGET LABELS ['F1', 'SIGF1', >>>> 'DANO1', 'SIGDANO1'] Unit cell: (34.79, 108.92, 134.88, 90, 90, 90) >>>> Space >>>> group: P 2 21 21 (No. 18) CONTENTS: >>>> ['/home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_2_P22121_scala2.mtz', >>>> 'mtz', 'premerged', 'P 2 21 21', [34.790000915527344, >>>> 108.92009735107422, >>>> 134.8800048828125, 90.0, 90.0, 90.0], 2.4890566908055836, ['F1', >>>> 'SIGF1', >>>> 'DANO1', 'SIGDANO1']] >>>> >>>> File >>>> /home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_2_P22121_scala2.mtz >>>> RES: 2.5 >>>> >>>> FILE TYPE ccp4_mtz COLUMN LABELS: ['H', 'K', 'L', 'FreeR_flag', >>>> 'F_107_3re', >>>> 'SIGF_107_3re', 'F_107_3re(+)', 'SIGF_107_3re(+)', 'F_107_3re(-)', >>>> 'SIGF_107_3re(-)', 'DANO_107_3re', 'SIGDANO_107_3re', 'IMEAN_107_3re', >>>> 'SIGIMEAN_107_3re', 'I_107_3re(+)', 'SIGI_107_3re(+)', 'I_107_3re(-)', >>>> 'SIGI_107_3re(-)'] >>>> >>>> GUESS FILE TYPE MERGE TYPE mtz premerged TARGET LABELS ['F1', 'SIGF1', >>>> 'DANO1', 'SIGDANO1'] Unit cell: (34.85, 109.18, 135.27, 90, 90, 90) >>>> Space >>>> group: P 2 21 21 (No. 18) CONTENTS: >>>> ['/home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_3_P22121_scala2.mtz', >>>> 'mtz', 'premerged', 'P 2 21 21', [34.849998474121094, >>>> 109.18000030517578, >>>> 135.26980590820312, 90.0, 90.0, 90.0], 2.4889952912293629, ['F1', >>>> 'SIGF1', >>>> 'DANO1', 'SIGDANO1']] >>>> >>>> File >>>> /home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_3_P22121_scala2.mtz >>>> RES: 2.5 >>>> >>>> FILE TYPE ccp4_mtz COLUMN LABELS: ['H', 'K', 'L', 'FreeR_flag', >>>> 'F_107_4re', >>>> 'SIGF_107_4re', 'F_107_4re(+)', 'SIGF_107_4re(+)', 'F_107_4re(-)', >>>> 'SIGF_107_4re(-)', 'DANO_107_4re', 'SIGDANO_107_4re', 'IMEAN_107_4re', >>>> 'SIGIMEAN_107_4re', 'I_107_4re(+)', 'SIGI_107_4re(+)', 'I_107_4re(-)', >>>> 'SIGI_107_4re(-)'] >>>> >>>> GUESS FIL TARGET LABELS ['F1', 'SIGF1', 'DANO1', 'SIGDANO1'] Unit cell: >>>> (34.87, 109.2, 135.33, 90, 90, 90) Space group: P 2 21 21 (No. 18) >>>> CONTENTS: >>>> ['/home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_4_P22121_scala2.mtz', >>>> 'mtz', 'premerged', 'P 2 21 21', [34.869998931884766, >>>> 109.19999694824219, >>>> 135.32989501953125, 90.0, 90.0, 90.0], 2.4890134311310943, ['F1', >>>> 'SIGF1', >>>> 'DANO1', 'SIGDANO1']] >>>> >>>> File >>>> /home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_4_P22121_scala2.mtz >>>> RES: 2.5 >>>> >>>> Guess of datafile for refinement: >>>> /home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_3_P22121_scala2.mtz >>>> >>>> Using >>>> /home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_3_P22121_scala2.mtz >>>> for refinement >>>> >>>> Specify >>>> input_refinement_file=/home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_3_P22121_scala2.mtz >>>> to change this >>>> >>>> HKLIN ENTRY: >>>> /home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_2_P22121_scala2.mtz >>>> >>>> FILE TYPE ccp4_mtz >>>> >>>> COLUMN LABELS: ['H', 'K', 'L', 'FreeR_flag', 'F_107_2re', >>>> 'SIGF_107_2re', >>>> 'F_107_2re(+)', 'SIGF_107_2re(+)', 'F_107_2re(-)', 'SIGF_107_2re(-)', >>>> 'DANO_107_2re', 'SIGDANO_107_2re', 'IMEAN_107_2re', 'SIGIMEAN_107_2re', >>>> 'I_107_2re(+)', 'SIGI_107_2re(+)', 'I_107_2re(-)', 'SIGI_107_2re(-)'] >>>> >>>> GUESS FILE TYPE MERGE TYPE mtz premerged TARGET LABELS ['F1', 'SIGF1', >>>> 'DANO1', 'SIGDANO1'] Unit cell: (34.79, 108.92, 134.88, 90, 90, 90) >>>> Space >>>> group: P 2 21 21 (No. 18) CONTENTS: >>>> ['/home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_2_P22121_scala2.mtz', >>>> 'mtz', 'premerged', 'P 2 21 21', [34.790000915527344, >>>> 108.92009735107422, >>>> 134.8800048828125, 90.0, 90.0, 90.0], 2.4890566908055836, ['F1', >>>> 'SIGF1', >>>> 'DANO1', 'SIGDANO1']] >>>> >>>> Inverse hand of space group: P 2 21 21 Creating sg entry from >>>> /home/sar06mbr/30JanDiamondI02/107/107_php22121/M75_Zn_107_2_P22121_scala2.mtz >>>> Unit cell: (34.79, 108.92, 134.88, 90, 90, 90) Space group: P 2 21 21 >>>> (No. >>>> 18) Space group name is: P 2 21 21 symbol is: p22121 >>>> >>>> **************************************** >>>> >>>> AutoSol Input failed >>>> >>>> Need a symfile for solve >>>> >>>> **************************************** >>>> >>>> _______________________________________________ >>>> phenixbb mailing list >>>> phenixbb(a)phenix-online.org >>>> http://phenix-online.org/mailman/listinfo/phenixbb >>>> >>>> >>> _______________________________________________ >>> phenixbb mailing list >>> phenixbb(a)phenix-online.org >>> http://phenix-online.org/mailman/listinfo/phenixbb >>> >> _______________________________________________ >> phenixbb mailing list >> phenixbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/phenixbb >>

16 years

Re: [cctbxbb] some thoughts on cctbx and pip

by Gergely Katona

Hi Billy, Conda install went fine with your instructions after rearranging the channels by putting cctbx last. I removed all environmental variables and previous build of cctbx. import sys print (sys.path, sys.prefix) ['/home/gergely/anaconda3/lib/python36.zip', '/home/gergely/anaconda3/lib/python3.6', '/home/gergely/anaconda3/lib/python3.6/lib-dynload', '', '/home/gergely/anaconda3/lib/python3.6/site-packages', '/home/gergely/anaconda3/lib/python3.6/site-packages/IPython/extensions', '/home/gergely/.ipython'] /home/gergely/anaconda3 Many imports went fine including pymc3, but I encountered problems with these three: from cctbx import miller import iotbx.pdb from iotbx import reflection_file_reader, mtz I also got type error when handling space groups. --------------------------------------------------------------------------- KeyError Traceback (most recent call last) <ipython-input-5-cedbd4f84f3d> in <module> 25 from cctbx import uctbx 26 from cctbx import sgtbx ---> 27 from cctbx import miller 28 #from iotbx import reflection_file_reader, mtz 29 sns.set_context("poster") ~/anaconda3/lib/python3.6/site-packages/cctbx/miller/__init__.py in <module> 11 12 from cctbx import crystal ---> 13 from cctbx import maptbx 14 from cctbx import sgtbx 15 from cctbx.sgtbx import lattice_symmetry ~/anaconda3/lib/python3.6/site-packages/cctbx/maptbx/__init__.py in <module> 15 from libtbx import adopt_init_args 16 from libtbx.utils import Sorry ---> 17 import libtbx.load_env 18 import math 19 import sys, os ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module> 3 import libtbx.env_config 4 import os ----> 5 libtbx.env = libtbx.env_config.unpickle() 6 libtbx.env.set_os_environ_all_dist() 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME") ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle() 2603 2604 def unpickle(): -> 2605 build_path = os.environ["LIBTBX_BUILD"] 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path) 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb") ~/anaconda3/lib/python3.6/os.py in __getitem__(self, key) 667 except KeyError: 668 # raise KeyError with the original key value --> 669 raise KeyError(key) from None 670 return self.decodevalue(value) 671 KeyError: 'LIBTBX_BUILD' --------------------------------------------------------------------------- ArgumentError Traceback (most recent call last) <ipython-input-7-2acd3a9ce26a> in <module> 26 return ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar 27 ---> 28 ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar=initializecrystal() <ipython-input-7-2acd3a9ce26a> in initializecrystal() 11 uc = uctbx.unit_cell(unit_cell) 12 wavelength = 1.54980 ---> 13 xtal_symm = crystal.symmetry(unit_cell=unit_cell, space_group_symbol="P 43 21 2") 14 15 ms = miller.build_set(crystal_symmetry=xtal_symm,anomalous_flag=True, d_min=1.61) ~/anaconda3/lib/python3.6/site-packages/cctbx/crystal/__init__.py in __init__(self, unit_cell, space_group_symbol, space_group_info, space_group, correct_rhombohedral_setting_if_necessary, assert_is_compatible_unit_cell, raise_sorry_if_incompatible_unit_cell, force_compatible_unit_cell) 74 if (space_group_symbol is not None): 75 self._space_group_info = sgtbx.space_group_info( ---> 76 symbol=space_group_symbol) 77 elif (space_group is not None): 78 if (isinstance(space_group, sgtbx.space_group)): ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in __init__(self, symbol, table_id, group, number, space_group_t_den) 100 assert group is None 101 if (table_id is None): --> 102 symbols = space_group_symbols(symbol) 103 else: 104 if (isinstance(symbol, int)): symbol = str(symbol) ArgumentError: Python argument types in space_group_symbols.__init__(space_group_symbols, str) did not match C++ signature: __init__(_object*, int space_group_number) __init__(_object*, int space_group_number, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > extension='') __init__(_object*, int space_group_number, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > extension='', std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > table_id='') __init__(_object*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > symbol) __init__(_object*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > symbol, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > table_id='') When I tried to set LIBTBX_BUILD to /home/gergely/anaconda3 (this is sys.prefix on my system), I got other problems when importing. Can LIBTBX_BUILD be set in the conda package? --------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) <ipython-input-1-7e2b144826cc> in <module> 26 from cctbx import sgtbx 27 #from cctbx import miller ---> 28 from iotbx import reflection_file_reader, mtz 29 sns.set_context("poster") 30 plt.rcParams.update({'figure.autolayout': True}) ~/anaconda3/lib/python3.6/site-packages/iotbx/reflection_file_reader.py in <module> 53 54 from __future__ import absolute_import, division, print_function ---> 55 from iotbx import mtz 56 from iotbx.scalepack import merge as scalepack_merge 57 from iotbx.scalepack import no_merge_original_index as scalepack_no_merge ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/__init__.py in <module> 9 import iotbx_mtz_ext as ext 10 ---> 11 from iotbx.mtz import extract_from_symmetry_lib 12 from cctbx import xray 13 import cctbx.xray.observation_types ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/extract_from_symmetry_lib.py in <module> 1 from __future__ import absolute_import, division, print_function 2 from cctbx import sgtbx ----> 3 import libtbx.load_env 4 import os.path as op 5 from six.moves import range ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module> 3 import libtbx.env_config 4 import os ----> 5 libtbx.env = libtbx.env_config.unpickle() 6 libtbx.env.set_os_environ_all_dist() 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME") ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle() 2605 build_path = os.environ["LIBTBX_BUILD"] 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path) -> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb") 2608 env = pickle.load(libtbx_env) 2609 if (env.python_version_major_minor != sys.version_info[:2]): FileNotFoundError: [Errno 2] No such file or directory: '/home/gergely/anaconda3/libtbx_env' On Wed, Dec 18, 2019 at 9:10 AM Billy Poon <BKPoon(a)lbl.gov> wrote: > > Hi Gergely, > > I've uploaded linux packages to a new channel, cctbx-dev, and you can install it with > > conda install -c cctbx-dev cctbx > > in your current environment. You should set your ~/.condarc file to pull the other dependencies from the conda-forge channel first, so put conda-forge above cctbx. Mine looks like > > channels: > - conda-forge > - defaults > - cctbx > > Do you need dxtbx for your scripts? This package does not build that part. I think the plan is to build a separate conda package for dxtbx so that it can be updated more frequently. I can rebuild the packages to include it for testing, but the one being submitted to conda-forge will not have it. > > Also, your error message is probably due to version of HDF5 that the development build installs. The bootstrap.py script will install 1.10.4, but your other dependency is looking for 1.10.5. Installing this cctbx conda package should install 1.10.5, which should fix the issue. Also, I'm updating those environments and 1.10.5 will be the new default version. > > Lastly, the dispatchers will not work in these packages because the old paths during the build process are still in them and many of them expect some additional information that has not been updated in the packages yet. I'm in the process of doing that and will update the cctbx-dev channel when that's done. However, by starting python, you can import cctbx modules. So you can run commands like > > from scitbx.array_family import flex > a = flex.random_double(1000000) > b = flex.min_max_mean_double(a) > b.min > b.max > b.mean > > Thanks! > > -- > Billy K. Poon > Research Scientist, Molecular Biophysics and Integrated Bioimaging > Lawrence Berkeley National Laboratory > 1 Cyclotron Road, M/S 33R0345 > Berkeley, CA 94720 > Tel: (510) 486-5709 > Fax: (510) 486-5909 > Web: https://phenix-online.org > > > On Tue, Dec 17, 2019 at 5:39 AM Gergely Katona <gkatona(a)gmail.com> wrote: >> >> Dear Billy, >> >> Thank you for the detailed explanation, I look forward to do further >> testing! I aim to pool all modules under the same environment, if >> there are conflicts then I just try to reshuffle the order of imports. >> So far this did not cause problems for me even when I was using system >> python, but of course it is not the most prudent thing to do. With >> anaconda everything is much more standardized and isolated already and >> without being superuser I can have my familiar environment at any >> synchrotron based cluster. It is great that cctbx will be an integral >> part of this ecosystem and this was also the last thing holding me >> back from adopting python3. About hdf5, pymc3 requires it and >> importing causes a kernel restart with the following error messages, >> curiously if I import h5py first this can be avoided. >> >> Best wishes, >> >> Gergely >> >> import pymc3 as pm >> >> Warning! ***HDF5 library version mismatched error*** >> The HDF5 header files used to compile this application do not match >> the version used by the HDF5 library to which this application is linked. >> Data corruption or segmentation faults may occur if the application continues. >> This can happen when an application was compiled by one version of HDF5 but >> linked with a different version of static or shared HDF5 library. >> You should recompile the application or check your shared library related >> settings such as 'LD_LIBRARY_PATH'. >> You can, at your own risk, disable this warning by setting the environment >> variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'. >> Setting it to 2 or higher will suppress the warning messages totally. >> Headers are 1.10.4, library is 1.10.5 >> SUMMARY OF THE HDF5 CONFIGURATION >> ================================= >> >> General Information: >> ------------------- >> HDF5 Version: 1.10.5 >> Configured on: Tue Oct 22 12:02:13 UTC 2019 >> Configured by: conda@16247e67ecd5 >> Host system: x86_64-conda_cos6-linux-gnu >> Uname information: Linux 16247e67ecd5 4.15.0-1059-azure >> #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64 >> GNU/Linux >> Byte sex: little-endian >> Installation point: /home/gergely/anaconda3 >> >> Compiling Options: >> ------------------ >> Build Mode: production >> Debugging Symbols: no >> Asserts: no >> Profiling: no >> Optimization Level: high >> >> Linking Options: >> ---------------- >> Libraries: static, shared >> Statically Linked Executables: >> LDFLAGS: -Wl,-O2 -Wl,--sort-common >> -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags >> -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib >> -Wl,-rpath-link,/home/gergely/anaconda3/lib >> -L/home/gergely/anaconda3/lib >> H5_LDFLAGS: >> AM_LDFLAGS: -L/home/gergely/anaconda3/lib >> Extra libraries: -lrt -lpthread -lz -ldl -lm >> Archiver: >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar >> AR_FLAGS: cr >> Ranlib: >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib >> >> Languages: >> ---------- >> C: yes >> C Compiler: >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc >> CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 >> -I/home/gergely/anaconda3/include >> H5_CPPFLAGS: -D_GNU_SOURCE >> -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API >> AM_CPPFLAGS: -I/home/gergely/anaconda3/include >> C Flags: -march=nocona -mtune=haswell >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include >> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5 >> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix >> H5 C Flags: -std=c99 -pedantic -Wall -Wextra >> -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion >> -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal >> -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations >> -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs >> -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls >> -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef >> -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings >> -finline-functions -s -Wno-inline -Wno-aggregate-return >> -Wno-missing-format-attribute -Wno-missing-noreturn -O >> AM C Flags: >> Shared C Library: yes >> Static C Library: yes >> >> >> Fortran: yes >> Fortran Compiler: >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran >> Fortran Flags: >> H5 Fortran Flags: -pedantic -Wall -Wextra -Wunderflow >> -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2 >> AM Fortran Flags: >> Shared Fortran Library: yes >> Static Fortran Library: yes >> >> C++: yes >> C++ Compiler: >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++ >> C++ Flags: -fvisibility-inlines-hidden >> -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include >> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5 >> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix >> H5 C++ Flags: -pedantic -Wall -W -Wundef -Wshadow >> -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion >> -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual >> -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor >> -Wctor-dtor-privacy -Wabi -finline-functions -s -O >> AM C++ Flags: >> Shared C++ Library: yes >> Static C++ Library: yes >> >> Java: no >> >> >> Features: >> --------- >> Parallel HDF5: no >> Parallel Filtered Dataset Writes: no >> Large Parallel I/O: no >> High-level library: yes >> Threadsafety: yes >> Default API mapping: v110 >> With deprecated public symbols: yes >> I/O filters (external): deflate(zlib) >> MPE: no >> Direct VFD: no >> dmalloc: no >> Packages w/ extra debug output: none >> API tracing: no >> Using memory checker: yes >> Memory allocation sanity checks: no >> Function stack tracing: no >> Strict file format checks: no >> Optimization instrumentation: no >> >> On Tue, Dec 17, 2019 at 8:35 AM Billy Poon <BKPoon(a)lbl.gov> wrote: >> > >> > Hi Gergely, >> > >> > Let me build the test package tomorrow. All the gritty details for building with conda is being finalized and the official documentation will be updated to describe the steps. It would be too confusing to keep changing the documentation as the process evolves. You do have the general process, though, which is summarized as follows. >> > >> > 1) Installing dependencies. The cctbx_dependencies metapackage was an initial approach for managing the CCTBX dependencies, but after contacting the conda-forge folks, they recommended using the --only-deps flag. So when the CCTBX conda package is available, you'll be able to get a set of dependencies with, >> > >> > conda install -c conda-forge --only-deps cctbx >> > >> > By default, the bootstrap.py file will automatically install a set of dependencies in the "conda_base" directory (and a conda installation if one is not found). It just uses standard conda environment files located in libtbx/auto_build/conda_envs, so you do not need to install cctbx_dependencies as a separate step. The environment files avoid channel issues by explicitly defining the channel to pull the packages from and the cctbx channel just stores copies of packages from conda-forge. There were issues earlier where the conda-forge packages would sometimes be moved to a different label. The --use-conda flag also accepts a path to $CONDA_PREFIX if you want to use a specific environment for testing. >> > >> > 2) Building. The bootstrap.py file handles that with SCons. >> > >> > 3) Running. After building, there should be a setpaths.sh (and .csh) file that adds build/bin to your path. The build/bin directory has our dispatchers, which are just scripts that set up the environment variables for you. This prevents other programs from loading our libraries, whose versions may conflict. You should see that there is a "python" dispatcher, which is a convenience for developers. Otherwise, you can use libtbx.python, which will be able to import CCTBX modules. >> > >> > The running part is where the conda package for CCTBX will be different than this build. Since our Python files and extensions modules will be in the "site-packages" directory for Python, the PYTHONPATH variable will not be needed (and conda suggests that that variable not be set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so LD_LIBRARY_PATH is not needed. In an active environment, PATH will already be modified. And then our LIBTBX_BUILD directory can be set to sys.prefix. >> > >> > So with the conda package, you would only need to activate your conda environment and CCTBX should integrate with other conda packages. What is the conflict with hdf5? That's something that should be fixed. Thanks! >> > >> > -- >> > Billy K. Poon >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging >> > Lawrence Berkeley National Laboratory >> > 1 Cyclotron Road, M/S 33R0345 >> > Berkeley, CA 94720 >> > Tel: (510) 486-5709 >> > Fax: (510) 486-5909 >> > Web: https://phenix-online.org >> > >> > >> > On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona <gkatona(a)gmail.com> wrote: >> >> >> >> Dear Billy, >> >> >> >> Thank you for this update and for your efforts! I found a solution and >> >> indeed most things already work in anaconda3. The steps I took (even >> >> if these instructions will have short expiry date): >> >> >> >> Modified .condarc with: >> >> channels: >> >> - cctbx >> >> - conda-forge >> >> - defaults >> >> >> >> run >> >> conda conda install cctbx_dependencies python=3.6 >> >> >> >> I expect this will work without python=3.6 in the near future. >> >> >> >> Then compiling cctbx with anaconda3 python went without any problem >> >> when using these flags: >> >> python bootstrap.py --use-conda --python3 --nproc=12 >> >> >> >> Then I probably did the unorthodox thing and sourced these directories: >> >> >> >> setenv LIBTBX_BUILD /home/gergely/cctbx/build >> >> setenv PATH ${PATH}:/home/gergely/cctbx/build/bin >> >> setenv PYTHONPATH >> >> /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH >> >> setenv LD_LIBRARY_PATH >> >> /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH >> >> >> >> There is probably a better way to put this into conda environment. >> >> With these steps I could run one of my scripts depending on cctbx. The >> >> only problem I found is that hdf5 library had conflict with another >> >> package in conda, but I does not find this as a showstopper. >> >> >> >> Best wishes, >> >> >> >> Gergely >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Fri, Dec 13, 2019 at 7:23 PM Billy Poon <BKPoon(a)lbl.gov> wrote: >> >> > >> >> > Hi Gergely, >> >> > >> >> > It's still a work in progress. I'm sorting out some Windows issues right now. I can probably build a test package on a separate channel for people that want to test it (let's say next week?). I'll provide instructions, but basically, the test conda package will be in its own separate channel and the dependencies will be pulled from the conda-forge channel. I want most things to be working correctly on Python 2.7, 3.6, 3.7, and 3.8 on all 3 platforms. >> >> > >> >> > Thanks! >> >> > >> >> > -- >> >> > Billy K. Poon >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging >> >> > Lawrence Berkeley National Laboratory >> >> > 1 Cyclotron Road, M/S 33R0345 >> >> > Berkeley, CA 94720 >> >> > Tel: (510) 486-5709 >> >> > Fax: (510) 486-5909 >> >> > Web: https://phenix-online.org >> >> > >> >> > >> >> > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona <gkatona(a)gmail.com> wrote: >> >> >> >> >> >> Dear Billy, >> >> >> >> >> >> This sounds very promising and exciting. I am not sure if cctbx is >> >> >> already functional as a conda package in anaconda3 (Linux) or this is >> >> >> still work in progress. My technical expertise does not allow me to >> >> >> tell the difference. What I tried: >> >> >> >> >> >> Fresh install of anaconda3. Adding - cctbx and - conda-forge to >> >> >> .condarc . Installing pyside2 with conda. Running conda install >> >> >> conda_dependencies . I get a lot package version conflict, and I >> >> >> cannot import cctbx or iotbx to anaconda python. Am I following the >> >> >> right instructions? Or it is too early to expect that cctbx works when >> >> >> installed through conda? >> >> >> >> >> >> Best wishes, >> >> >> >> >> >> Gergely >> >> >> >> >> >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon <BKPoon(a)lbl.gov> wrote: >> >> >> > >> >> >> > Hi all, >> >> >> > >> >> >> > For a brief update, I have submitted a recipe for cctbxlite to conda-forge (https://github.com/conda-forge/staged-recipes/pull/10021) and support for Python 3.7 and 3.8 is being added (https://github.com/cctbx/cctbx_project/pull/409). With some fixes for Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8. Some additional changes will be needed to get Windows to work with Python 3 and for tests to pass with Boost 1.70.0. That will enable the conda-forge recipe to build for all platforms and for all supported versions of Python. >> >> >> > >> >> >> > Currently, the conda-forge recipe will install into the conda python and cctbx imports can be done without sourcing the environment scripts. It looks like a lot of the environment variables being set in the dispatchers can be removed since the Python files and C++ extensions are in the right places. I'll update the libtbx_env file so that commands that load the environment can work correctly. >> >> >> > >> >> >> > -- >> >> >> > Billy K. Poon >> >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging >> >> >> > Lawrence Berkeley National Laboratory >> >> >> > 1 Cyclotron Road, M/S 33R0345 >> >> >> > Berkeley, CA 94720 >> >> >> > Tel: (510) 486-5709 >> >> >> > Fax: (510) 486-5909 >> >> >> > Web: https://phenix-online.org >> >> >> > >> >> >> > >> >> >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll <tic20(a)cam.ac.uk> wrote: >> >> >> >> >> >> >> >> Hi Luc, >> >> >> >> >> >> >> >> That sounds promising. From there, I’d need to work out how to make a fully-packaged installer (basically a modified wheel file) for the ChimeraX ToolShed - the aim is for the end user to not have to worry about any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would need to be set dynamically before first import - but doesn’t seem insurmountable. >> >> >> >> >> >> >> >> Thanks, >> >> >> >> >> >> >> >> Tristan >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Tristan Croll >> >> >> >> Research Fellow >> >> >> >> Cambridge Institute for Medical Research >> >> >> >> University of Cambridge CB2 0XY >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis <luc_j_bourhis(a)mac.com> wrote: >> >> >> >> > >> >> >> >> > Hi Tristan, >> >> >> >> > >> >> >> >> > cctbx could be built to use your ChimeraX python, now that cctbx is moving to Python 3. The option —with-python is there for that with the bootstrap script. The specific environment setup boil down to setting two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a framework such as ChimeraX, that should not be difficult to ensure those two variables are set. >> >> >> >> > >> >> >> >> > Best wishes, >> >> >> >> > >> >> >> >> > Luc >> >> >> >> > >> >> >> >> > >> >> >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll <tic20(a)cam.ac.uk> wrote: >> >> >> >> >> >> >> >> >> >> To add my two cents on this: probably the second-most common question I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". The honest answer to that is, "I didn't know how." >> >> >> >> >> >> >> >> >> >> Still don't, really - although the current developments are rather promising. The problem I've faced is that CCTBX was designed as its own self-contained Python (2.7, until very recently) environment, with its own interpreter and a lot of very specific environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is *also* its own self-contained Python (3.7) environment. To plug one into the other in that form... well, I don't think I'm a good enough programmer to really know where to start. >> >> >> >> >> >> >> >> >> >> The move to Conda and a more modular CCTBX architecture should make a lot more possible in that direction. Pip would be even better for me personally (ChimeraX can install directly from the PyPI, but doesn't interact with Conda) - but I understand pretty well the substantial challenge that would amount to (not least being that the PyPI imposes a limit - around 100MB from memory? - on the size of an individual package). >> >> >> >> >> >> >> >> >> >> Best regards, >> >> >> >> >> >> >> >> >> >> Tristan >> >> >> >> >> >> >> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote: >> >> >> >> >>> Hi Graeme, >> >> >> >> >>> Yes, I know. But “black" is a program doing a very particular task >> >> >> >> >>> (code formatting from the top of my head). Requiring to use a wrapper >> >> >> >> >>> for python itself is another level. But ok, I think I am mellowing to >> >> >> >> >>> the idea after all! Talking with people around me, and extrapolating, >> >> >> >> >>> I would bet that, right now, a great majority of people interested by >> >> >> >> >>> cctbx in pip have already used the cctbx, so they know about the >> >> >> >> >>> Python wrapper, and they would not be too sanguine about that. My >> >> >> >> >>> concern is for the future, when pip will be the first time some people >> >> >> >> >>> use cctbx. Big fat warning notices on PyPI page and a better error >> >> >> >> >>> message when cctbx fails because LIBTBX_BUILD is not set would be >> >> >> >> >>> needed but that could be all right. >> >> >> >> >>> If we do a pip installer, we should aim at a minimal install: cctbx, >> >> >> >> >>> iotbx and their dependencies, and that’s it. >> >> >> >> >>> Best wishes, >> >> >> >> >>> Luc >> >> >> >> >>>> On 23 Aug 2019, at 07:17, Graeme.Winter(a)Diamond.ac.uk <Graeme.Winter(a)diamond.ac.uk> wrote: >> >> >> >> >>>> Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g. >> >> >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat $(which black) >> >> >> >> >>>> /Library/Frameworks/Python.framework/Versions/3.6/bin/black >> >> >> >> >>>> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 >> >> >> >> >>>> # -*- coding: utf-8 -*- >> >> >> >> >>>> import re >> >> >> >> >>>> import sys >> >> >> >> >>>> from black import main >> >> >> >> >>>> if __name__ == '__main__': >> >> >> >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) >> >> >> >> >>>> sys.exit(main()) >> >> >> >> >>>> So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this. >> >> >> >> >>>> Best wishes Graeme >> >> >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote: >> >> >> >> >>>> Hi, >> >> >> >> >>>> Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs? >> >> >> >> >>>> Best wishes, >> >> >> >> >>>> Luc >> >> >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote: >> >> >> >> >>>> Hi, >> >> >> >> >>>> I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.org<http://anaconda.org/>, and then to advertise it hard to every potential user out there. >> >> >> >> >>>> Best wishes, >> >> >> >> >>>> Luc >> >> >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster <asbrewster(a)lbl.gov<mailto:[email protected]>> wrote: >> >> >> >> >>>> Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython…). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not. >> >> >> >> >>>> All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support. >> >> >> >> >>>> There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6). >> >> >> >> >>>> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4. >> >> >> >> >>>> Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :) >> >> >> >> >>>> Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users. >> >> >> >> >>>> Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort. >> >> >> >> >>>> Thanks, >> >> >> >> >>>> -Aaron >> >> >> >> >>>> _______________________________________________ >> >> >> >> >>>> cctbxbb mailing list >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >>>> _______________________________________________ >> >> >> >> >>>> cctbxbb mailing list >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >>>> _______________________________________________ >> >> >> >> >>>> cctbxbb mailing list >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >>>> -- >> >> >> >> >>>> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. >> >> >> >> >>>> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. >> >> >> >> >>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. >> >> >> >> >>>> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom >> >> >> >> >>>> _______________________________________________ >> >> >> >> >>>> cctbxbb mailing list >> >> >> >> >>>> cctbxbb(a)phenix-online.org >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >>> _______________________________________________ >> >> >> >> >>> cctbxbb mailing list >> >> >> >> >>> cctbxbb(a)phenix-online.org >> >> >> >> >>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> >> >> cctbxbb mailing list >> >> >> >> >> cctbxbb(a)phenix-online.org >> >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> > >> >> >> >> > >> >> >> >> > _______________________________________________ >> >> >> >> > cctbxbb mailing list >> >> >> >> > cctbxbb(a)phenix-online.org >> >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> >> cctbxbb mailing list >> >> >> >> cctbxbb(a)phenix-online.org >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> > >> >> >> > _______________________________________________ >> >> >> > cctbxbb mailing list >> >> >> > cctbxbb(a)phenix-online.org >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Gergely Katona, PhD >> >> >> Associate Professor >> >> >> Department of Chemistry and Molecular Biology, University of Gothenburg >> >> >> Box 462, 40530 Göteborg, Sweden >> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 >> >> >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se >> >> >> >> >> >> _______________________________________________ >> >> >> cctbxbb mailing list >> >> >> cctbxbb(a)phenix-online.org >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> > >> >> > _______________________________________________ >> >> > cctbxbb mailing list >> >> > cctbxbb(a)phenix-online.org >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> -- >> >> Gergely Katona, PhD >> >> Associate Professor >> >> Department of Chemistry and Molecular Biology, University of Gothenburg >> >> Box 462, 40530 Göteborg, Sweden >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 >> >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se >> >> >> >> _______________________________________________ >> >> cctbxbb mailing list >> >> cctbxbb(a)phenix-online.org >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> > >> > _______________________________________________ >> > cctbxbb mailing list >> > cctbxbb(a)phenix-online.org >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> -- >> Gergely Katona, PhD >> Associate Professor >> Department of Chemistry and Molecular Biology, University of Gothenburg >> Box 462, 40530 Göteborg, Sweden >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se >> >> _______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb -- Gergely Katona, PhD Associate Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: gergely.katona(a)gu.se

6 years, 1 month

Re: [cctbxbb] Boost Python 1.56

by Marcin Wojdyr

Hi Aaron, yes, I didn't run test yesterday. I did run it today and at first 20 failed. It's mostly this problem: https://github.com/boostorg/python/issues/56 Other failures I got were probably not related to the boost update. I've committed 3 fixes. I started looking at 4th, which was: File "/home/marcin/dials/modules/cctbx_project/iotbx/mtz/tst_ext.py", line 1093, in exercise_modifiers assert approx_equal(v, [-1]*4) but I must leave now. Does this test also fail for you? Marcin On Wed, Feb 10, 2016 at 11:20 PM, Aaron Brewster <asbrewster(a)lbl.gov> wrote: > Hi Marcin, thanks for the detective work. I've reproduced your results on > Centos 7 using the cctbx bootstrap script (not sure from your email if > that's what you used): > > svn export > svn://svn.code.sf.net/p/cctbx/code/trunk/libtbx/auto_build/bootstrap.py > python bootstrap.py --builder=dials hot update > cd modules > mv boost oldboost > wget > https://sourceforge.net/projects/boost/files/latest/download?source=files -O > ./boost_1_60_0.tar.gz > mv boost_1_60_0 boost > vi boost/boost/rational.hpp (revert > https://github.com/boostorg/rational/commit/5fddb3f889cd2a2fe59cdcae182f8b3…, > this is necessary to compile) > cd .. > python bootstrap.py --builder=dials base build > > I then exercised the cctbx regression tests: > > source build/setpaths.sh > mkdir tests > cd tests > cctbx_regression.test_nightly > > 17 tests failed, but I think they are fixable, most of them being interface > changes in boost. I ran the tests again after restoring the original boost > and recompiling. There were still 3 errors, but I need to verify those > errors from a completely clean system that hadn't seen boost 1.60 before I'd > think they were real. > > Anyhow the next step is to track down those 17 errors, fix them, then try > the whole process again with the phenix builder and the full set of dials > and phenix tests. > > -Aaron > > > > > > On Wed, Feb 10, 2016 at 10:45 AM, Marcin Wojdyr <wojdyr(a)gmail.com> wrote: >> >> To help a bit I went through compilation of cctbx/Dials with Boost 1.60. >> >> There is a lot of warnings from Boost.Python, >> about deprecated header: >> >> https://github.com/boostorg/python/commit/0a4c76b9ac16974d7d4f164cf61790952… >> and unused typedefs. After adding -Wno-unused-local-typedefs and >> removing the deprecated #include almost all warnings are gone. >> >> Then there is one error, copy&pasted below. >> I tracked it down to this change in boost.rational: >> >> https://github.com/boostorg/rational/commit/5fddb3f889cd2a2fe59cdcae182f8b3… >> I just reverted this change to finish compilation, but I suppose a >> proper fix is also not difficult. >> >> I don't know if this is relevant, but I'm using Boost compiled outside of >> cctbx. >> I'm attaching a patch that we use in CCP4 for this. It's based on >> Debian/Gentoo patches. >> >> cheers, >> Marcin >> >> >> g++ -o boost_adaptbx/rational_ext.o -c >> -I/home/marcin/dials/boost_160/include -Wno-unused-local-typedefs >> -DBOOST_PYTHON_MAX_BASES=2 -fPIC -fno-strict-aliasing -w -DNDEBUG -O3 >> -ffast-math -DBOOST_ALL_NO_LIB -I/home/marcin/dials/build/include >> -I/home/marcin/miniconda2/include/python2.7 >> /home/marcin/dials/modules/cctbx_project/boost_adaptbx/rational_ext.cpp >> In file included from >> >> /home/marcin/dials/boost_160/include/boost/preprocessor/iteration/detail/iter/forward1.hpp:47:0, >> from >> /home/marcin/dials/boost_160/include/boost/python/detail/invoke.hpp:63, >> from >> /home/marcin/dials/boost_160/include/boost/python/detail/caller.hpp:16, >> from >> >> /home/marcin/dials/boost_160/include/boost/python/object/function_handle.hpp:8, >> from >> >> /home/marcin/dials/boost_160/include/boost/python/converter/arg_to_python.hpp:19, >> from >> /home/marcin/dials/boost_160/include/boost/python/call.hpp:15, >> from >> /home/marcin/dials/boost_160/include/boost/python/object_core.hpp:14, >> from >> /home/marcin/dials/boost_160/include/boost/python/args.hpp:25, >> from >> /home/marcin/dials/boost_160/include/boost/python/make_function.hpp:11, >> from >> /home/marcin/dials/boost_160/include/boost/python/def.hpp:11, >> from >> /home/marcin/dials/modules/cctbx_project/boost_adaptbx/rational_ext.cpp:2: >> /home/marcin/dials/boost_160/include/boost/python/detail/invoke.hpp: >> In instantiation of 'PyObject* >> boost::python::detail::invoke(boost::python::detail::invoke_tag_<false, >> true>, const RC&, F&, TC&) [with RC = >> >> boost::python::detail::specify_a_return_value_policy_to_wrap_functions_returning<const >> int&>; F = const int& (boost::rational<int>::*)()const; TC = >> boost::python::arg_from_python<boost::rational<int>&>; PyObject = >> _object]': >> >> /home/marcin/dials/boost_160/include/boost/python/detail/caller.hpp:223:13: >> required from 'PyObject* >> boost::python::detail::caller_arity<1u>::impl<F, Policies, >> Sig>::operator()(PyObject*, PyObject*) [with F = const int& >> (boost::rational<int>::*)()const; Policies = >> boost::python::default_call_policies; Sig = boost::mpl::vector2<const >> int&, boost::rational<int>&>; PyObject = _object]' >> >> /home/marcin/dials/boost_160/include/boost/python/object/py_function.hpp:38:33: >> required from 'PyObject* >> >> boost::python::objects::caller_py_function_impl<Caller>::operator()(PyObject*, >> PyObject*) [with Caller = boost::python::detail::caller<const int& >> (boost::rational<int>::*)()const, >> boost::python::default_call_policies, boost::mpl::vector2<const int&, >> boost::rational<int>&> >; PyObject = _object]' >> >> /home/marcin/dials/modules/cctbx_project/boost_adaptbx/rational_ext.cpp:231:1: >> required from here >> /home/marcin/dials/boost_160/include/boost/python/detail/invoke.hpp:88:90: >> error: no match for call to '(const >> >> boost::python::detail::specify_a_return_value_policy_to_wrap_functions_returning<const >> int&>) (const int&)' >> return rc( (tc().*f)(BOOST_PP_ENUM_BINARY_PARAMS_Z(1, N, ac, () >> BOOST_PP_INTERCEPT)) ); >> >> ^ >> In file included from >> >> /home/marcin/dials/boost_160/include/boost/python/object/function_handle.hpp:8:0, >> from >> >> /home/marcin/dials/boost_160/include/boost/python/converter/arg_to_python.hpp:19, >> from >> /home/marcin/dials/boost_160/include/boost/python/call.hpp:15, >> from >> /home/marcin/dials/boost_160/include/boost/python/object_core.hpp:14, >> from >> /home/marcin/dials/boost_160/include/boost/python/args.hpp:25, >> from >> /home/marcin/dials/boost_160/include/boost/python/make_function.hpp:11, >> from >> /home/marcin/dials/boost_160/include/boost/python/def.hpp:11, >> from >> /home/marcin/dials/modules/cctbx_project/boost_adaptbx/rational_ext.cpp:2: >> /home/marcin/dials/boost_160/include/boost/python/detail/caller.hpp: >> In instantiation of 'static const PyTypeObject* >> >> boost::python::detail::converter_target_type<ResultConverter>::get_pytype() >> [with ResultConverter = >> >> boost::python::detail::specify_a_return_value_policy_to_wrap_functions_returning<const >> int&>; PyTypeObject = _typeobject]': >> >> /home/marcin/dials/boost_160/include/boost/python/detail/caller.hpp:240:19: >> required from 'static boost::python::detail::py_func_sig_info >> boost::python::detail::caller_arity<1u>::impl<F, Policies, >> Sig>::signature() [with F = const int& >> (boost::rational<int>::*)()const; Policies = >> boost::python::default_call_policies; Sig = boost::mpl::vector2<const >> int&, boost::rational<int>&>]' >> >> /home/marcin/dials/boost_160/include/boost/python/object/py_function.hpp:48:35: >> required from 'boost::python::detail::py_func_sig_info >> boost::python::objects::caller_py_function_impl<Caller>::signature() >> const [with Caller = boost::python::detail::caller<const int& >> (boost::rational<int>::*)()const, >> boost::python::default_call_policies, boost::mpl::vector2<const int&, >> boost::rational<int>&> >]' >> >> /home/marcin/dials/modules/cctbx_project/boost_adaptbx/rational_ext.cpp:231:1: >> required from here >> >> /home/marcin/dials/boost_160/include/boost/python/detail/caller.hpp:102:109: >> error: 'struct >> boost::python::detail::specify_a_return_value_policy_to_wrap_functions_returning<const >> int&>' has no member named 'get_pytype' >> return create_result_converter((PyObject*)0, (ResultConverter >> *)0, (ResultConverter *)0).get_pytype(); >> >> ^ >> scons: *** [boost_adaptbx/rational_ext.o] Error 1 >> scons: building terminated because of errors. >> >> >> >> >> >> On Wed, Feb 10, 2016 at 4:44 PM, Aaron Brewster <asbrewster(a)lbl.gov> >> wrote: >> > Hi Nick, Billy and I will work on this today, >> > >> > -Aaron >> > >> > On Wed, Feb 10, 2016 at 6:40 AM, Marcin Wojdyr <wojdyr(a)gmail.com> wrote: >> >> >> >> FWIW the latest version of Boost is 1.60. >> >> CCP4 6.5 was using cctbx compiled with Boost 1.56 >> >> and CCP4 7.0 uses Boost 1.58. >> >> >> >> Marcin >> >> >> >> On Wed, Feb 10, 2016 at 2:17 PM, Nicholas Sauter <nksauter(a)lbl.gov> >> >> wrote: >> >> > Nigel, Billy & Aaron, >> >> > >> >> > I completely endorse this move to Boost 1.56. Can we update our >> >> > build? >> >> > >> >> > Nick >> >> > >> >> > Nicholas K. Sauter, Ph. D. >> >> > Computer Staff Scientist, Molecular Biophysics and Integrated >> >> > Bioimaging >> >> > Division >> >> > Lawrence Berkeley National Laboratory >> >> > 1 Cyclotron Rd., Bldg. 33R0345 >> >> > Berkeley, CA 94720 >> >> > (510) 486-5713 >> >> > >> >> > On Wed, Feb 10, 2016 at 2:41 PM, Luc Bourhis <luc_j_bourhis(a)mac.com> >> >> > wrote: >> >> >> >> >> >> Hi, >> >> >> >> >> >> I have improvements to the smtbx on their way to be committed which >> >> >> require Boost version 1.56. This is related to Boost.Threads, whose >> >> >> support >> >> >> I re-activated a few months ago on Nick’s request. I need the >> >> >> function >> >> >> boost::thread::physical_concurrency which returns the number of >> >> >> physical >> >> >> cores on the machine, as opposed to virtual cores when >> >> >> hyperthreading >> >> >> is >> >> >> enabled (which it is by default on any Intel machine). That function >> >> >> is >> >> >> not >> >> >> available in Boost 1.55 which is the version currently used in the >> >> >> nightly >> >> >> tests: it appeared in 1.56. >> >> >> >> >> >> So, would it be possible to move to Boost 1.56? Otherwise, I will >> >> >> need >> >> >> to >> >> >> backport that function. Not too difficult but not thrilling. >> >> >> >> >> >> Best wishes, >> >> >> >> >> >> Luc >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> cctbxbb mailing list >> >> >> cctbxbb(a)phenix-online.org >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> > >> >> > >> >> > >> >> > _______________________________________________ >> >> > cctbxbb mailing list >> >> > cctbxbb(a)phenix-online.org >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> > >> >> >> >> _______________________________________________ >> >> cctbxbb mailing list >> >> cctbxbb(a)phenix-online.org >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> > >> > >> > >> > _______________________________________________ >> > cctbxbb mailing list >> > cctbxbb(a)phenix-online.org >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> > >> >> _______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb >> > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb >

10 years

Re: [cctbxbb] Boost Python 1.56

by Billy Poon

Hi everyone, Boost 1.56 is now available and the tests have been updated. To update an existing installtion, you have to delete the "build" directory first. If you don't, I guess some files are not recompiled with the new Boost headers, which will cause cctbx_project/scitbx/random/tests/tst_random.py to expect the old values. Then run bootstrap.py with hot, update, and build to rebuild. cd <installation directory> rm -fr build python bootstrap.py hot update build --builder=cctbx Let me know if there are any issues. Thanks! -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org On Fri, Apr 7, 2017 at 1:08 PM, Billy Poon <bkpoon(a)lbl.gov> wrote: > I can do the switch to Boost 1.56 next Wednesday now that Dials 1.5 has > been released. I'm just double-checking some other Phenix tests. > > Once the update is live, bootstrap.py should delete the existing > modules/boost directory and replace it with the new one. And then scons > should recompile anything that depends on boost. If in doubt, you can > manually delete the module/boost and build directories. > > -- > Billy K. Poon > Research Scientist, Molecular Biophysics and Integrated Bioimaging > Lawrence Berkeley National Laboratory > 1 Cyclotron Road, M/S 33R0345 > Berkeley, CA 94720 > Tel: (510) 486-5709 > Fax: (510) 486-5909 > Web: https://phenix-online.org > > On Thu, Apr 6, 2017 at 9:32 AM, Billy Poon <bkpoon(a)lbl.gov> wrote: > >> Hi James, >> >> I agree with Graeme's tests and I would add >> >> cctbx_regression.test_nightly >> >> That command is a shortcut for running the test modules for libtbx, >> boost_adaptbx, scitbx, cctbx, iotbx, dxtbx, and smtbx. The mmtbx module is >> also tested if chem_data is available. Should we add rstbx to the shortcut >> (cctbx_project/cctbx/command_line/cctbx_test_nightly.py)? >> >> -- >> Billy K. Poon >> Research Scientist, Molecular Biophysics and Integrated Bioimaging >> Lawrence Berkeley National Laboratory >> 1 Cyclotron Road, M/S 33R0345 >> Berkeley, CA 94720 >> Tel: (510) 486-5709 >> Fax: (510) 486-5909 >> Web: https://phenix-online.org >> >> On Thu, Apr 6, 2017 at 1:37 AM, <richard.gildea(a)diamond.ac.uk> wrote: >> >>> I've made a start on transcribing this document here: >>> >>> https://github.com/cctbx/cctbx_project/wiki/cctbx-Developer-Guidance >>> >>> It probably still needs cleaning up a bit (e.g. I couldn't figure out >>> quickly how to do 3-level list nesting, 1.a.i etc.) and updating to reflect >>> current practice (e.g. git not svn). >>> >>> Cheers, >>> >>> Richard >>> >>> Dr Richard Gildea >>> Data Analysis Scientist >>> Tel: +441235 77 8078 >>> >>> Diamond Light Source Ltd. >>> Diamond House >>> Harwell Science & Innovation Campus >>> Didcot >>> Oxfordshire >>> OX11 0DE >>> >>> ________________________________________ >>> From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces@phenix-online >>> .org] on behalf of Pavel Afonine [pafonine(a)lbl.gov] >>> Sent: 06 April 2017 09:07 >>> To: cctbx mailing list >>> Subject: Re: [cctbxbb] Boost Python 1.56 >>> >>> Hi Graeme, >>> >>> hm.. this is a good question. We've been through back-and-forth >>> iterations of editing this file and I think the latest I have is from >>> Paul. But I can't find a non-PDF version of it. Paul: do you have an >>> editable version of this file? >>> >>> Thanks, >>> Pavel >>> >>> On 4/6/17 13:45, Graeme.Winter(a)diamond.ac.uk wrote: >>> > Hi Pavel >>> > >>> > These all seem sensible >>> > >>> > If you have the original non pdf document it may be easier to >>> transcribe this over.. >>> > >>> > I also note that it lacks the actual detail on how to run tests! >>> However would be happy to add this once on wiki >>> > >>> > Best wishes Graeme >>> > >>> > On 6 Apr 2017, at 04:00, Pavel Afonine <pafonine(a)lbl.gov<mailto:pafon >>> ine(a)lbl.gov>> wrote: >>> > >>> > Not sure if that answers your questions but once upon a time we here >>> at Berkeley tried to write a some sort of document that was supposed to >>> answer questions like this. Attached. By no means it is complete, >>> up-to-date, etc, but it might be worth reading for anyone who contributes >>> to cctbx. (Even not sure if I'm sending the latest version). >>> > Unfortunately, nobody bothered to put it in some central place. >>> > >>> > Pavel >>> > >>> > On 4/6/17 10:51, James Holton wrote: >>> > Hey Billy, >>> > >>> > On a related note. How do I run these regression tests before >>> committing something into Git? Is there a document on dials regression >>> testing I can't find? >>> > >>> > -James >>> > >>> > On Apr 5, 2017, at 3:38 PM, Billy Poon <bkpoon(a)lbl.gov<mailto:bkpoon@ >>> lbl.gov>> wrote: >>> > >>> > Hi all, >>> > >>> > I tested Boost 1.56 on our buildbot servers and got some new test >>> failures with >>> > >>> > cctbx_project/scitbx/array_family/boost_python/tst_flex.py >>> > cctbx_project/scitbx/random/tests/tst_random.py >>> > >>> > The full log for CentOS 6 can be found at >>> > >>> > http://cci-vm-6.lbl.gov:8010/builders/phenix-nightly-intel-l >>> inux-2.6-x86_64-centos6/builds/601/steps/test%20cctbx_regres >>> sion.test_nightly/logs/stdio >>> > >>> > It looks like the errors are related to random number generation. For >>> a given seed, would the sequence of numbers change when Boost is changed? I >>> did a diff between Boost 1.56 and the current Boost and could not see any >>> changes that immediately stood out as being related to random numbers. >>> > >>> > Are these tests failing for others as well? >>> > >>> > -- >>> > Billy K. Poon >>> > Research Scientist, Molecular Biophysics and Integrated Bioimaging >>> > Lawrence Berkeley National Laboratory >>> > 1 Cyclotron Road, M/S 33R0345 >>> > Berkeley, CA 94720 >>> > Tel: (510) 486-5709 >>> > Fax: (510) 486-5909 >>> > Web: https://phenix-online.org<https://phenix-online.org/> >>> > >>> > On Wed, Apr 5, 2017 at 8:12 AM, Charles Ballard < >>> charles.ballard(a)stfc.ac.uk<mailto:[email protected]>> wrote: >>> > FYI, we (CCP4) have been using 1.56 for building cctbx/phaser/dials >>> for the last while with no issues. Don't know about 1.60, but 1.59 causes >>> issues with the boost python make_getter and make_setter (initialisation of >>> none const reference if the passed type is a temporary). >>> > >>> > Charles >>> > >>> > On 3 Apr 2017, at 14:31, Luc Bourhis wrote: >>> > >>> > Hi all, >>> > >>> > everybody seemed to agree but then it was proposed to move straight to >>> Boost 1.60, and this caused troubles. Could we consider again to move to at >>> least 1.56? As far as I can tell, this does not cause any issue and as >>> stated one year ago, it would help me and Olex 2. >>> > >>> > Thanks, >>> > >>> > Luc >>> > >>> > On 10 Feb 2016, at 15:17, Nicholas Sauter <nksauter(a)lbl.gov<mailto: >>> nksauter(a)lbl.gov>> wrote: >>> > >>> > Nigel, Billy & Aaron, >>> > >>> > I completely endorse this move to Boost 1.56. Can we update our build? >>> > >>> > Nick >>> > >>> > Nicholas K. Sauter, Ph. D. >>> > Computer Staff Scientist, Molecular Biophysics and Integrated >>> Bioimaging Division >>> > Lawrence Berkeley National Laboratory >>> > 1 Cyclotron Rd., Bldg. 33R0345 >>> > Berkeley, CA 94720 >>> > (510) 486-5713<tel:%28510%29%20486-5713> >>> > >>> > On Wed, Feb 10, 2016 at 2:41 PM, Luc Bourhis <luc_j_bourhis(a)mac.com >>> <mailto:[email protected]>> wrote: >>> > Hi, >>> > >>> > I have improvements to the smtbx on their way to be committed which >>> require Boost version 1.56. This is related to Boost.Threads, whose support >>> I re-activated a few months ago on Nick’s request. I need the function >>> boost::thread::physical_concurrency which returns the number of >>> physical cores on the machine, as opposed to virtual cores when >>> hyperthreading is enabled (which it is by default on any Intel machine). >>> That function is not available in Boost 1.55 which is the version currently >>> used in the nightly tests: it appeared in 1.56. >>> > >>> > So, would it be possible to move to Boost 1.56? Otherwise, I will need >>> to backport that function. Not too difficult but not thrilling. >>> > >>> > Best wishes, >>> > >>> > Luc >>> > >>> > >>> > _______________________________________________ >>> > cctbxbb mailing list >>> > cctbxbb(a)phenix-online.org<mailto:[email protected]> >>> > http://phenix-online.org/mailman/listinfo/cctbxbb >>> > >>> > _______________________________________________ >>> > cctbxbb mailing list >>> > cctbxbb(a)phenix-online.org<mailto:[email protected]> >>> > http://phenix-online.org/mailman/listinfo/cctbxbb >>> > >>> > _______________________________________________ >>> > cctbxbb mailing list >>> > cctbxbb(a)phenix-online.org<mailto:[email protected]> >>> > http://phenix-online.org/mailman/listinfo/cctbxbb >>> > >>> > >>> > _______________________________________________ >>> > cctbxbb mailing list >>> > cctbxbb(a)phenix-online.org<mailto:[email protected]> >>> > http://phenix-online.org/mailman/listinfo/cctbxbb >>> > >>> > >>> > _______________________________________________ >>> > cctbxbb mailing list >>> > cctbxbb(a)phenix-online.org<mailto:[email protected]> >>> > http://phenix-online.org/mailman/listinfo/cctbxbb >>> > >>> > >>> > >>> > _______________________________________________ >>> > cctbxbb mailing list >>> > cctbxbb(a)phenix-online.org<mailto:[email protected]> >>> > http://phenix-online.org/mailman/listinfo/cctbxbb >>> > >>> > >>> > <cctbx-developer-guidance-08-2015.pdf>______________________ >>> _________________________ >>> > cctbxbb mailing list >>> > cctbxbb(a)phenix-online.org<mailto:[email protected]> >>> > http://phenix-online.org/mailman/listinfo/cctbxbb >>> > >>> > >>> >>> _______________________________________________ >>> cctbxbb mailing list >>> cctbxbb(a)phenix-online.org >>> http://phenix-online.org/mailman/listinfo/cctbxbb >>> >>> -- >>> This e-mail and any attachments may contain confidential, copyright and >>> or privileged material, and are for the use of the intended addressee only. >>> If you are not the intended addressee or an authorised recipient of the >>> addressee please notify us of receipt by returning the e-mail and do not >>> use, copy, retain, distribute or disclose the information in or attached to >>> the e-mail. >>> Any opinions expressed within this e-mail are those of the individual >>> and not necessarily of Diamond Light Source Ltd. >>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any >>> attachments are free from viruses and we cannot accept liability for any >>> damage which you may sustain as a result of software viruses which may be >>> transmitted in or with the message. >>> Diamond Light Source Limited (company no. 4375679). Registered in >>> England and Wales with its registered office at Diamond House, Harwell >>> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom >>> >>> >>> _______________________________________________ >>> cctbxbb mailing list >>> cctbxbb(a)phenix-online.org >>> http://phenix-online.org/mailman/listinfo/cctbxbb >>> >> >> >

8 years, 9 months

Re: [cctbxbb] some thoughts on cctbx and pip

by Billy Poon

Hi Gergely, That's the other information that hasn't been updated yet in the conda package. :) For LIBTBX_BUILD, I can set up environment variables sort of how the conda gcc package sets up environment variables. I have to double check to see if they're set after installation of the package and after activation of an environment. The variable will just be a copy of $CONDA_PREFIX. The libtbx_env file is a file that exists in $LIBTBX_BUILD that basically stores configuration information. I'll have to modify the contents to update the locations of the different modules since the conda package will not have a "modules" directory. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org On Wed, Dec 18, 2019 at 2:08 AM Gergely Katona <gkatona(a)gmail.com> wrote: > Hi Billy, > > Conda install went fine with your instructions after rearranging the > channels by putting cctbx last. I removed all environmental variables > and previous build of cctbx. > > import sys > print (sys.path, sys.prefix) > > ['/home/gergely/anaconda3/lib/python36.zip', > '/home/gergely/anaconda3/lib/python3.6', > '/home/gergely/anaconda3/lib/python3.6/lib-dynload', '', > '/home/gergely/anaconda3/lib/python3.6/site-packages', > '/home/gergely/anaconda3/lib/python3.6/site-packages/IPython/extensions', > '/home/gergely/.ipython'] /home/gergely/anaconda3 > > Many imports went fine including pymc3, but I encountered problems > with these three: > > from cctbx import miller > import iotbx.pdb > from iotbx import reflection_file_reader, mtz > > > I also got type error when handling space groups. > > > > --------------------------------------------------------------------------- > KeyError Traceback (most recent call last) > <ipython-input-5-cedbd4f84f3d> in <module> > 25 from cctbx import uctbx > 26 from cctbx import sgtbx > ---> 27 from cctbx import miller > 28 #from iotbx import reflection_file_reader, mtz > 29 sns.set_context("poster") > > ~/anaconda3/lib/python3.6/site-packages/cctbx/miller/__init__.py in > <module> > 11 > 12 from cctbx import crystal > ---> 13 from cctbx import maptbx > 14 from cctbx import sgtbx > 15 from cctbx.sgtbx import lattice_symmetry > > ~/anaconda3/lib/python3.6/site-packages/cctbx/maptbx/__init__.py in > <module> > 15 from libtbx import adopt_init_args > 16 from libtbx.utils import Sorry > ---> 17 import libtbx.load_env > 18 import math > 19 import sys, os > > ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module> > 3 import libtbx.env_config > 4 import os > ----> 5 libtbx.env = libtbx.env_config.unpickle() > 6 libtbx.env.set_os_environ_all_dist() > 7 libtbx.env.dispatcher_name = > os.environ.get("LIBTBX_DISPATCHER_NAME") > > ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle() > 2603 > 2604 def unpickle(): > -> 2605 build_path = os.environ["LIBTBX_BUILD"] > 2606 > set_preferred_sys_prefix_and_sys_executable(build_path=build_path) > 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb") > > ~/anaconda3/lib/python3.6/os.py in __getitem__(self, key) > 667 except KeyError: > 668 # raise KeyError with the original key value > --> 669 raise KeyError(key) from None > 670 return self.decodevalue(value) > 671 > > KeyError: 'LIBTBX_BUILD' > > > > --------------------------------------------------------------------------- > ArgumentError Traceback (most recent call last) > <ipython-input-7-2acd3a9ce26a> in <module> > 26 return > ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar > 27 > ---> 28 > ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar=initializecrystal() > > <ipython-input-7-2acd3a9ce26a> in initializecrystal() > 11 uc = uctbx.unit_cell(unit_cell) > 12 wavelength = 1.54980 > ---> 13 xtal_symm = crystal.symmetry(unit_cell=unit_cell, > space_group_symbol="P 43 21 2") > 14 > 15 ms = > miller.build_set(crystal_symmetry=xtal_symm,anomalous_flag=True, > d_min=1.61) > > ~/anaconda3/lib/python3.6/site-packages/cctbx/crystal/__init__.py in > __init__(self, unit_cell, space_group_symbol, space_group_info, > space_group, correct_rhombohedral_setting_if_necessary, > assert_is_compatible_unit_cell, raise_sorry_if_incompatible_unit_cell, > force_compatible_unit_cell) > 74 if (space_group_symbol is not None): > 75 self._space_group_info = sgtbx.space_group_info( > ---> 76 symbol=space_group_symbol) > 77 elif (space_group is not None): > 78 if (isinstance(space_group, sgtbx.space_group)): > > ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in > __init__(self, symbol, table_id, group, number, space_group_t_den) > 100 assert group is None > 101 if (table_id is None): > --> 102 symbols = space_group_symbols(symbol) > 103 else: > 104 if (isinstance(symbol, int)): symbol = str(symbol) > > ArgumentError: Python argument types in > space_group_symbols.__init__(space_group_symbols, str) > did not match C++ signature: > __init__(_object*, int space_group_number) > __init__(_object*, int space_group_number, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > extension='') > __init__(_object*, int space_group_number, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > extension='', std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > table_id='') > __init__(_object*, std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > symbol) > __init__(_object*, std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > symbol, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > table_id='') > > > When I tried to set LIBTBX_BUILD to /home/gergely/anaconda3 (this is > sys.prefix on my system), I got other problems when importing. Can > LIBTBX_BUILD be set in the conda package? > > > > --------------------------------------------------------------------------- > FileNotFoundError Traceback (most recent call last) > <ipython-input-1-7e2b144826cc> in <module> > 26 from cctbx import sgtbx > 27 #from cctbx import miller > ---> 28 from iotbx import reflection_file_reader, mtz > 29 sns.set_context("poster") > 30 plt.rcParams.update({'figure.autolayout': True}) > > ~/anaconda3/lib/python3.6/site-packages/iotbx/reflection_file_reader.py > in <module> > 53 > 54 from __future__ import absolute_import, division, print_function > ---> 55 from iotbx import mtz > 56 from iotbx.scalepack import merge as scalepack_merge > 57 from iotbx.scalepack import no_merge_original_index as > scalepack_no_merge > > ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/__init__.py in <module> > 9 import iotbx_mtz_ext as ext > 10 > ---> 11 from iotbx.mtz import extract_from_symmetry_lib > 12 from cctbx import xray > 13 import cctbx.xray.observation_types > > > ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/extract_from_symmetry_lib.py > in <module> > 1 from __future__ import absolute_import, division, print_function > 2 from cctbx import sgtbx > ----> 3 import libtbx.load_env > 4 import os.path as op > 5 from six.moves import range > > ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module> > 3 import libtbx.env_config > 4 import os > ----> 5 libtbx.env = libtbx.env_config.unpickle() > 6 libtbx.env.set_os_environ_all_dist() > 7 libtbx.env.dispatcher_name = > os.environ.get("LIBTBX_DISPATCHER_NAME") > > ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle() > 2605 build_path = os.environ["LIBTBX_BUILD"] > 2606 > set_preferred_sys_prefix_and_sys_executable(build_path=build_path) > -> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb") > 2608 env = pickle.load(libtbx_env) > 2609 if (env.python_version_major_minor != sys.version_info[:2]): > > FileNotFoundError: [Errno 2] No such file or directory: > '/home/gergely/anaconda3/libtbx_env' > > On Wed, Dec 18, 2019 at 9:10 AM Billy Poon <BKPoon(a)lbl.gov> wrote: > > > > Hi Gergely, > > > > I've uploaded linux packages to a new channel, cctbx-dev, and you can > install it with > > > > conda install -c cctbx-dev cctbx > > > > in your current environment. You should set your ~/.condarc file to pull > the other dependencies from the conda-forge channel first, so put > conda-forge above cctbx. Mine looks like > > > > channels: > > - conda-forge > > - defaults > > - cctbx > > > > Do you need dxtbx for your scripts? This package does not build that > part. I think the plan is to build a separate conda package for dxtbx so > that it can be updated more frequently. I can rebuild the packages to > include it for testing, but the one being submitted to conda-forge will not > have it. > > > > Also, your error message is probably due to version of HDF5 that the > development build installs. The bootstrap.py script will install 1.10.4, > but your other dependency is looking for 1.10.5. Installing this cctbx > conda package should install 1.10.5, which should fix the issue. Also, I'm > updating those environments and 1.10.5 will be the new default version. > > > > Lastly, the dispatchers will not work in these packages because the old > paths during the build process are still in them and many of them expect > some additional information that has not been updated in the packages yet. > I'm in the process of doing that and will update the cctbx-dev channel when > that's done. However, by starting python, you can import cctbx modules. So > you can run commands like > > > > from scitbx.array_family import flex > > a = flex.random_double(1000000) > > b = flex.min_max_mean_double(a) > > b.min > > b.max > > b.mean > > > > Thanks! > > > > -- > > Billy K. Poon > > Research Scientist, Molecular Biophysics and Integrated Bioimaging > > Lawrence Berkeley National Laboratory > > 1 Cyclotron Road, M/S 33R0345 > > Berkeley, CA 94720 > > Tel: (510) 486-5709 > > Fax: (510) 486-5909 > > Web: https://phenix-online.org > > > > > > On Tue, Dec 17, 2019 at 5:39 AM Gergely Katona <gkatona(a)gmail.com> > wrote: > >> > >> Dear Billy, > >> > >> Thank you for the detailed explanation, I look forward to do further > >> testing! I aim to pool all modules under the same environment, if > >> there are conflicts then I just try to reshuffle the order of imports. > >> So far this did not cause problems for me even when I was using system > >> python, but of course it is not the most prudent thing to do. With > >> anaconda everything is much more standardized and isolated already and > >> without being superuser I can have my familiar environment at any > >> synchrotron based cluster. It is great that cctbx will be an integral > >> part of this ecosystem and this was also the last thing holding me > >> back from adopting python3. About hdf5, pymc3 requires it and > >> importing causes a kernel restart with the following error messages, > >> curiously if I import h5py first this can be avoided. > >> > >> Best wishes, > >> > >> Gergely > >> > >> import pymc3 as pm > >> > >> Warning! ***HDF5 library version mismatched error*** > >> The HDF5 header files used to compile this application do not match > >> the version used by the HDF5 library to which this application is > linked. > >> Data corruption or segmentation faults may occur if the application > continues. > >> This can happen when an application was compiled by one version of HDF5 > but > >> linked with a different version of static or shared HDF5 library. > >> You should recompile the application or check your shared library > related > >> settings such as 'LD_LIBRARY_PATH'. > >> You can, at your own risk, disable this warning by setting the > environment > >> variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'. > >> Setting it to 2 or higher will suppress the warning messages totally. > >> Headers are 1.10.4, library is 1.10.5 > >> SUMMARY OF THE HDF5 CONFIGURATION > >> ================================= > >> > >> General Information: > >> ------------------- > >> HDF5 Version: 1.10.5 > >> Configured on: Tue Oct 22 12:02:13 UTC 2019 > >> Configured by: conda@16247e67ecd5 > >> Host system: x86_64-conda_cos6-linux-gnu > >> Uname information: Linux 16247e67ecd5 4.15.0-1059-azure > >> #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64 > >> GNU/Linux > >> Byte sex: little-endian > >> Installation point: /home/gergely/anaconda3 > >> > >> Compiling Options: > >> ------------------ > >> Build Mode: production > >> Debugging Symbols: no > >> Asserts: no > >> Profiling: no > >> Optimization Level: high > >> > >> Linking Options: > >> ---------------- > >> Libraries: static, shared > >> Statically Linked Executables: > >> LDFLAGS: -Wl,-O2 -Wl,--sort-common > >> -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags > >> -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib > >> -Wl,-rpath-link,/home/gergely/anaconda3/lib > >> -L/home/gergely/anaconda3/lib > >> H5_LDFLAGS: > >> AM_LDFLAGS: -L/home/gergely/anaconda3/lib > >> Extra libraries: -lrt -lpthread -lz -ldl -lm > >> Archiver: > >> > /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar > >> AR_FLAGS: cr > >> Ranlib: > >> > /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib > >> > >> Languages: > >> ---------- > >> C: yes > >> C Compiler: > >> > /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc > >> CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 > >> -I/home/gergely/anaconda3/include > >> H5_CPPFLAGS: -D_GNU_SOURCE > >> -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API > >> AM_CPPFLAGS: -I/home/gergely/anaconda3/include > >> C Flags: -march=nocona -mtune=haswell > >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 > >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include > >> > -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5 > >> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix > >> H5 C Flags: -std=c99 -pedantic -Wall -Wextra > >> -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion > >> -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal > >> -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations > >> -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs > >> -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls > >> -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef > >> -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings > >> -finline-functions -s -Wno-inline -Wno-aggregate-return > >> -Wno-missing-format-attribute -Wno-missing-noreturn -O > >> AM C Flags: > >> Shared C Library: yes > >> Static C Library: yes > >> > >> > >> Fortran: yes > >> Fortran Compiler: > >> > /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran > >> Fortran Flags: > >> H5 Fortran Flags: -pedantic -Wall -Wextra -Wunderflow > >> -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2 > >> AM Fortran Flags: > >> Shared Fortran Library: yes > >> Static Fortran Library: yes > >> > >> C++: yes > >> C++ Compiler: > >> > /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++ > >> C++ Flags: -fvisibility-inlines-hidden > >> -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell > >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 > >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include > >> > -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5 > >> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix > >> H5 C++ Flags: -pedantic -Wall -W -Wundef -Wshadow > >> -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion > >> -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual > >> -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor > >> -Wctor-dtor-privacy -Wabi -finline-functions -s -O > >> AM C++ Flags: > >> Shared C++ Library: yes > >> Static C++ Library: yes > >> > >> Java: no > >> > >> > >> Features: > >> --------- > >> Parallel HDF5: no > >> Parallel Filtered Dataset Writes: no > >> Large Parallel I/O: no > >> High-level library: yes > >> Threadsafety: yes > >> Default API mapping: v110 > >> With deprecated public symbols: yes > >> I/O filters (external): deflate(zlib) > >> MPE: no > >> Direct VFD: no > >> dmalloc: no > >> Packages w/ extra debug output: none > >> API tracing: no > >> Using memory checker: yes > >> Memory allocation sanity checks: no > >> Function stack tracing: no > >> Strict file format checks: no > >> Optimization instrumentation: no > >> > >> On Tue, Dec 17, 2019 at 8:35 AM Billy Poon <BKPoon(a)lbl.gov> wrote: > >> > > >> > Hi Gergely, > >> > > >> > Let me build the test package tomorrow. All the gritty details for > building with conda is being finalized and the official documentation will > be updated to describe the steps. It would be too confusing to keep > changing the documentation as the process evolves. You do have the general > process, though, which is summarized as follows. > >> > > >> > 1) Installing dependencies. The cctbx_dependencies metapackage was an > initial approach for managing the CCTBX dependencies, but after contacting > the conda-forge folks, they recommended using the --only-deps flag. So when > the CCTBX conda package is available, you'll be able to get a set of > dependencies with, > >> > > >> > conda install -c conda-forge --only-deps cctbx > >> > > >> > By default, the bootstrap.py file will automatically install a set of > dependencies in the "conda_base" directory (and a conda installation if one > is not found). It just uses standard conda environment files located in > libtbx/auto_build/conda_envs, so you do not need to install > cctbx_dependencies as a separate step. The environment files avoid channel > issues by explicitly defining the channel to pull the packages from and the > cctbx channel just stores copies of packages from conda-forge. There were > issues earlier where the conda-forge packages would sometimes be moved to a > different label. The --use-conda flag also accepts a path to $CONDA_PREFIX > if you want to use a specific environment for testing. > >> > > >> > 2) Building. The bootstrap.py file handles that with SCons. > >> > > >> > 3) Running. After building, there should be a setpaths.sh (and .csh) > file that adds build/bin to your path. The build/bin directory has our > dispatchers, which are just scripts that set up the environment variables > for you. This prevents other programs from loading our libraries, whose > versions may conflict. You should see that there is a "python" dispatcher, > which is a convenience for developers. Otherwise, you can use > libtbx.python, which will be able to import CCTBX modules. > >> > > >> > The running part is where the conda package for CCTBX will be > different than this build. Since our Python files and extensions modules > will be in the "site-packages" directory for Python, the PYTHONPATH > variable will not be needed (and conda suggests that that variable not be > set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so > LD_LIBRARY_PATH is not needed. In an active environment, PATH will already > be modified. And then our LIBTBX_BUILD directory can be set to sys.prefix. > >> > > >> > So with the conda package, you would only need to activate your conda > environment and CCTBX should integrate with other conda packages. What is > the conflict with hdf5? That's something that should be fixed. Thanks! > >> > > >> > -- > >> > Billy K. Poon > >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging > >> > Lawrence Berkeley National Laboratory > >> > 1 Cyclotron Road, M/S 33R0345 > >> > Berkeley, CA 94720 > >> > Tel: (510) 486-5709 > >> > Fax: (510) 486-5909 > >> > Web: https://phenix-online.org > >> > > >> > > >> > On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona <gkatona(a)gmail.com> > wrote: > >> >> > >> >> Dear Billy, > >> >> > >> >> Thank you for this update and for your efforts! I found a solution > and > >> >> indeed most things already work in anaconda3. The steps I took (even > >> >> if these instructions will have short expiry date): > >> >> > >> >> Modified .condarc with: > >> >> channels: > >> >> - cctbx > >> >> - conda-forge > >> >> - defaults > >> >> > >> >> run > >> >> conda conda install cctbx_dependencies python=3.6 > >> >> > >> >> I expect this will work without python=3.6 in the near future. > >> >> > >> >> Then compiling cctbx with anaconda3 python went without any problem > >> >> when using these flags: > >> >> python bootstrap.py --use-conda --python3 --nproc=12 > >> >> > >> >> Then I probably did the unorthodox thing and sourced these > directories: > >> >> > >> >> setenv LIBTBX_BUILD /home/gergely/cctbx/build > >> >> setenv PATH ${PATH}:/home/gergely/cctbx/build/bin > >> >> setenv PYTHONPATH > >> >> > /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH > >> >> setenv LD_LIBRARY_PATH > >> >> > /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH > >> >> > >> >> There is probably a better way to put this into conda environment. > >> >> With these steps I could run one of my scripts depending on cctbx. > The > >> >> only problem I found is that hdf5 library had conflict with another > >> >> package in conda, but I does not find this as a showstopper. > >> >> > >> >> Best wishes, > >> >> > >> >> Gergely > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> On Fri, Dec 13, 2019 at 7:23 PM Billy Poon <BKPoon(a)lbl.gov> wrote: > >> >> > > >> >> > Hi Gergely, > >> >> > > >> >> > It's still a work in progress. I'm sorting out some Windows issues > right now. I can probably build a test package on a separate channel for > people that want to test it (let's say next week?). I'll provide > instructions, but basically, the test conda package will be in its own > separate channel and the dependencies will be pulled from the conda-forge > channel. I want most things to be working correctly on Python 2.7, 3.6, > 3.7, and 3.8 on all 3 platforms. > >> >> > > >> >> > Thanks! > >> >> > > >> >> > -- > >> >> > Billy K. Poon > >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging > >> >> > Lawrence Berkeley National Laboratory > >> >> > 1 Cyclotron Road, M/S 33R0345 > >> >> > Berkeley, CA 94720 > >> >> > Tel: (510) 486-5709 > >> >> > Fax: (510) 486-5909 > >> >> > Web: https://phenix-online.org > >> >> > > >> >> > > >> >> > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona <gkatona(a)gmail.com> > wrote: > >> >> >> > >> >> >> Dear Billy, > >> >> >> > >> >> >> This sounds very promising and exciting. I am not sure if cctbx is > >> >> >> already functional as a conda package in anaconda3 (Linux) or > this is > >> >> >> still work in progress. My technical expertise does not allow me > to > >> >> >> tell the difference. What I tried: > >> >> >> > >> >> >> Fresh install of anaconda3. Adding - cctbx and - conda-forge to > >> >> >> .condarc . Installing pyside2 with conda. Running conda install > >> >> >> conda_dependencies . I get a lot package version conflict, and I > >> >> >> cannot import cctbx or iotbx to anaconda python. Am I following > the > >> >> >> right instructions? Or it is too early to expect that cctbx works > when > >> >> >> installed through conda? > >> >> >> > >> >> >> Best wishes, > >> >> >> > >> >> >> Gergely > >> >> >> > >> >> >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon <BKPoon(a)lbl.gov> > wrote: > >> >> >> > > >> >> >> > Hi all, > >> >> >> > > >> >> >> > For a brief update, I have submitted a recipe for cctbxlite to > conda-forge (https://github.com/conda-forge/staged-recipes/pull/10021) > and support for Python 3.7 and 3.8 is being added ( > https://github.com/cctbx/cctbx_project/pull/409). With some fixes for > Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms > (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8. > Some additional changes will be needed to get Windows to work with Python 3 > and for tests to pass with Boost 1.70.0. That will enable the conda-forge > recipe to build for all platforms and for all supported versions of Python. > >> >> >> > > >> >> >> > Currently, the conda-forge recipe will install into the conda > python and cctbx imports can be done without sourcing the environment > scripts. It looks like a lot of the environment variables being set in the > dispatchers can be removed since the Python files and C++ extensions are in > the right places. I'll update the libtbx_env file so that commands that > load the environment can work correctly. > >> >> >> > > >> >> >> > -- > >> >> >> > Billy K. Poon > >> >> >> > Research Scientist, Molecular Biophysics and Integrated > Bioimaging > >> >> >> > Lawrence Berkeley National Laboratory > >> >> >> > 1 Cyclotron Road, M/S 33R0345 > >> >> >> > Berkeley, CA 94720 > >> >> >> > Tel: (510) 486-5709 > >> >> >> > Fax: (510) 486-5909 > >> >> >> > Web: https://phenix-online.org > >> >> >> > > >> >> >> > > >> >> >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll <tic20(a)cam.ac.uk> > wrote: > >> >> >> >> > >> >> >> >> Hi Luc, > >> >> >> >> > >> >> >> >> That sounds promising. From there, I’d need to work out how to > make a fully-packaged installer (basically a modified wheel file) for the > ChimeraX ToolShed - the aim is for the end user to not have to worry about > any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would > need to be set dynamically before first import - but doesn’t seem > insurmountable. > >> >> >> >> > >> >> >> >> Thanks, > >> >> >> >> > >> >> >> >> Tristan > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> Tristan Croll > >> >> >> >> Research Fellow > >> >> >> >> Cambridge Institute for Medical Research > >> >> >> >> University of Cambridge CB2 0XY > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis <luc_j_bourhis(a)mac.com> > wrote: > >> >> >> >> > > >> >> >> >> > Hi Tristan, > >> >> >> >> > > >> >> >> >> > cctbx could be built to use your ChimeraX python, now that > cctbx is moving to Python 3. The option —with-python is there for that with > the bootstrap script. The specific environment setup boil down to setting > two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux, > PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a > framework such as ChimeraX, that should not be difficult to ensure those > two variables are set. > >> >> >> >> > > >> >> >> >> > Best wishes, > >> >> >> >> > > >> >> >> >> > Luc > >> >> >> >> > > >> >> >> >> > > >> >> >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll <tic20(a)cam.ac.uk> > wrote: > >> >> >> >> >> > >> >> >> >> >> To add my two cents on this: probably the second-most > common question I've had about ISOLDE's implementation is, "why didn't you > use CCTBX?". The honest answer to that is, "I didn't know how." > >> >> >> >> >> > >> >> >> >> >> Still don't, really - although the current developments are > rather promising. The problem I've faced is that CCTBX was designed as its > own self-contained Python (2.7, until very recently) environment, with its > own interpreter and a lot of very specific environment setup. Meanwhile I'm > developing ISOLDE in ChimeraX, which is *also* its own self-contained > Python (3.7) environment. To plug one into the other in that form... well, > I don't think I'm a good enough programmer to really know where to start. > >> >> >> >> >> > >> >> >> >> >> The move to Conda and a more modular CCTBX architecture > should make a lot more possible in that direction. Pip would be even better > for me personally (ChimeraX can install directly from the PyPI, but doesn't > interact with Conda) - but I understand pretty well the substantial > challenge that would amount to (not least being that the PyPI imposes a > limit - around 100MB from memory? - on the size of an individual package). > >> >> >> >> >> > >> >> >> >> >> Best regards, > >> >> >> >> >> > >> >> >> >> >> Tristan > >> >> >> >> >> > >> >> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote: > >> >> >> >> >>> Hi Graeme, > >> >> >> >> >>> Yes, I know. But “black" is a program doing a very > particular task > >> >> >> >> >>> (code formatting from the top of my head). Requiring to > use a wrapper > >> >> >> >> >>> for python itself is another level. But ok, I think I am > mellowing to > >> >> >> >> >>> the idea after all! Talking with people around me, and > extrapolating, > >> >> >> >> >>> I would bet that, right now, a great majority of people > interested by > >> >> >> >> >>> cctbx in pip have already used the cctbx, so they know > about the > >> >> >> >> >>> Python wrapper, and they would not be too sanguine about > that. My > >> >> >> >> >>> concern is for the future, when pip will be the first time > some people > >> >> >> >> >>> use cctbx. Big fat warning notices on PyPI page and a > better error > >> >> >> >> >>> message when cctbx fails because LIBTBX_BUILD is not set > would be > >> >> >> >> >>> needed but that could be all right. > >> >> >> >> >>> If we do a pip installer, we should aim at a minimal > install: cctbx, > >> >> >> >> >>> iotbx and their dependencies, and that’s it. > >> >> >> >> >>> Best wishes, > >> >> >> >> >>> Luc > >> >> >> >> >>>> On 23 Aug 2019, at 07:17, Graeme.Winter(a)Diamond.ac.uk < > Graeme.Winter(a)diamond.ac.uk> wrote: > >> >> >> >> >>>> Without discussing the merits of this or whether we > _choose_ to make the move to supporting PIP, I am certain it would be > _possible_ - many other packages make dispatcher scripts when you pip > install them e.g. > >> >> >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat $(which > black) > >> >> >> >> >>>> > /Library/Frameworks/Python.framework/Versions/3.6/bin/black > >> >> >> >> >>>> > #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 > >> >> >> >> >>>> # -*- coding: utf-8 -*- > >> >> >> >> >>>> import re > >> >> >> >> >>>> import sys > >> >> >> >> >>>> from black import main > >> >> >> >> >>>> if __name__ == '__main__': > >> >> >> >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', > sys.argv[0]) > >> >> >> >> >>>> sys.exit(main()) > >> >> >> >> >>>> So we _could_ work around the absence of LIBTBX_BUILD > etc. in the system. Whether or not we elect to do the work is a different > question, and it seems clear that here are very mixed opinions on this. > >> >> >> >> >>>> Best wishes Graeme > >> >> >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis < > luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote: > >> >> >> >> >>>> Hi, > >> >> >> >> >>>> Even if we managed to ship our the boost dynamic > libraries with pip, it would still not be pip-like, as we would still need > our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip > packages work with the standard python exe. LD_LIBRARY_PATH, we could get > around that by changing the way we compile, using -Wl,-R, which is the > runtime equivalent of build time -L. That’s a significant change that would > need to be tested. But there is no way around setting LIBTBX_BUILD right > now. Leaving that to the user is horrible. Perhaps there is a way to hack > libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip > installs? > >> >> >> >> >>>> Best wishes, > >> >> >> >> >>>> Luc > >> >> >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis < > luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote: > >> >> >> >> >>>> Hi, > >> >> >> >> >>>> I did look into that many years ago, and even toyed with > building a pip installer. What stopped me is the exact conclusion you > reached too: the user would not have the pip experience he expects. You are > right that it is a lot of effort but is it worth it? Considering that > remark, I don’t think so. Now, Conda was created specifically to go beyond > pip pure-python-only support. Since cctbx has garnered support for Conda, > the best avenue imho is to go the extra length to have a package on > Anaconda.org<http://anaconda.org/>, and then to advertise it hard to > every potential user out there. > >> >> >> >> >>>> Best wishes, > >> >> >> >> >>>> Luc > >> >> >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster < > asbrewster(a)lbl.gov<mailto:[email protected]>> wrote: > >> >> >> >> >>>> Hi, to avoid clouding Dorothee's documentation email > thread, which I think is a highly useful enterprise, here's some thoughts > about putting cctbx into pip. Pip doesn't install non-python dependencies > well. I don't think boost is available as a package on pip (at least the > package version we use). wxPython4 isn't portable through pip ( > https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython…). > MPI libraries are system dependent. If cctbx were a pure python package, > pip would be fine, but cctbx is not. > >> >> >> >> >>>> All that said, we could build a manylinux1 version of > cctbx and upload it to PyPi (I'm just learning about this). For a pip > package to be portable (which is a requirement for cctbx), it needs to > conform to PEP513, the manylinux1 standard ( > https://www.python.org/dev/peps/pep-0513/). For example, numpy is built > according to this standard (see https://pypi.org/project/numpy/#files, > where you'll see the manylinux1 wheel). Note, the manylinux1 standard is > built with Centos 5.11 which we no longer support. > >> >> >> >> >>>> There is also a manylinux2010 standard, which is based on > Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a > more attainable target (note though by default C++11 is not supported on > Centos 6). > >> >> >> >> >>>> If we built a manylinuxX version of cctbx and uploaded it > to PyPi, the user would need all the non-python dependencies. There's no > way to specify these in pip. For example, cctbx requires boost 1.63 or > better. The user will need to have it in a place their python can find it, > or we could package it ourselves and supply it, similar to how the pip h5py > package now comes with an hd5f library, or how the pip numpy package > includes an openblas library. We'd have to do the same for any packages we > depend on that aren't on pip using the manylinux standards, such as > wxPython4. > >> >> >> >> >>>> Further, we need to think about how dials and other > cctbx-based packages interact. If pip install cctbx is set up, how does > pip install dials work, such that any dials shared libraries can find the > cctbx libraries? Can shared libraries from one pip package link against > libraries in another pip package? Would each package need to supply its > own boost? Possibly this is well understood in the pip field, but not by > me :) > >> >> >> >> >>>> Finally, there's the option of providing a source pip > package. This would require the full compiler toolchain for any given > platform (macOS, linux, windows). These are likely available for > developers, but not for general users. > >> >> >> >> >>>> Anyway, these are some of the obstacles. Not saying it > isn't possible, it's just a lot of effort. > >> >> >> >> >>>> Thanks, > >> >> >> >> >>>> -Aaron > >> >> >> >> >>>> _______________________________________________ > >> >> >> >> >>>> cctbxbb mailing list > >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto: > cctbxbb(a)phenix-online.org> > >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> >> >> >>>> _______________________________________________ > >> >> >> >> >>>> cctbxbb mailing list > >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto: > cctbxbb(a)phenix-online.org> > >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> >> >> >>>> _______________________________________________ > >> >> >> >> >>>> cctbxbb mailing list > >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto: > cctbxbb(a)phenix-online.org> > >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> >> >> >>>> -- > >> >> >> >> >>>> This e-mail and any attachments may contain confidential, > copyright and or privileged material, and are for the use of the intended > addressee only. If you are not the intended addressee or an authorised > recipient of the addressee please notify us of receipt by returning the > e-mail and do not use, copy, retain, distribute or disclose the information > in or attached to the e-mail. > >> >> >> >> >>>> Any opinions expressed within this e-mail are those of > the individual and not necessarily of Diamond Light Source Ltd. > >> >> >> >> >>>> Diamond Light Source Ltd. cannot guarantee that this > e-mail or any attachments are free from viruses and we cannot accept > liability for any damage which you may sustain as a result of software > viruses which may be transmitted in or with the message. > >> >> >> >> >>>> Diamond Light Source Limited (company no. 4375679). > Registered in England and Wales with its registered office at Diamond > House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 > 0DE, United Kingdom > >> >> >> >> >>>> _______________________________________________ > >> >> >> >> >>>> cctbxbb mailing list > >> >> >> >> >>>> cctbxbb(a)phenix-online.org > >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> >> >> >>> _______________________________________________ > >> >> >> >> >>> cctbxbb mailing list > >> >> >> >> >>> cctbxbb(a)phenix-online.org > >> >> >> >> >>> http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> _______________________________________________ > >> >> >> >> >> cctbxbb mailing list > >> >> >> >> >> cctbxbb(a)phenix-online.org > >> >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > _______________________________________________ > >> >> >> >> > cctbxbb mailing list > >> >> >> >> > cctbxbb(a)phenix-online.org > >> >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> >> >> > >> >> >> >> > >> >> >> >> _______________________________________________ > >> >> >> >> cctbxbb mailing list > >> >> >> >> cctbxbb(a)phenix-online.org > >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> >> > > >> >> >> > _______________________________________________ > >> >> >> > cctbxbb mailing list > >> >> >> > cctbxbb(a)phenix-online.org > >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> >> > >> >> >> > >> >> >> > >> >> >> -- > >> >> >> Gergely Katona, PhD > >> >> >> Associate Professor > >> >> >> Department of Chemistry and Molecular Biology, University of > Gothenburg > >> >> >> Box 462, 40530 Göteborg, Sweden > >> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 > >> >> >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se > >> >> >> > >> >> >> _______________________________________________ > >> >> >> cctbxbb mailing list > >> >> >> cctbxbb(a)phenix-online.org > >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> > > >> >> > _______________________________________________ > >> >> > cctbxbb mailing list > >> >> > cctbxbb(a)phenix-online.org > >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> > >> >> > >> >> > >> >> -- > >> >> Gergely Katona, PhD > >> >> Associate Professor > >> >> Department of Chemistry and Molecular Biology, University of > Gothenburg > >> >> Box 462, 40530 Göteborg, Sweden > >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 > >> >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se > >> >> > >> >> _______________________________________________ > >> >> cctbxbb mailing list > >> >> cctbxbb(a)phenix-online.org > >> >> http://phenix-online.org/mailman/listinfo/cctbxbb > >> > > >> > _______________________________________________ > >> > cctbxbb mailing list > >> > cctbxbb(a)phenix-online.org > >> > http://phenix-online.org/mailman/listinfo/cctbxbb > >> > >> > >> > >> -- > >> Gergely Katona, PhD > >> Associate Professor > >> Department of Chemistry and Molecular Biology, University of Gothenburg > >> Box 462, 40530 Göteborg, Sweden > >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 > >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se > >> > >> _______________________________________________ > >> cctbxbb mailing list > >> cctbxbb(a)phenix-online.org > >> http://phenix-online.org/mailman/listinfo/cctbxbb > > > > _______________________________________________ > > cctbxbb mailing list > > cctbxbb(a)phenix-online.org > > http://phenix-online.org/mailman/listinfo/cctbxbb > > > > -- > Gergely Katona, PhD > Associate Professor > Department of Chemistry and Molecular Biology, University of Gothenburg > Box 462, 40530 Göteborg, Sweden > Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 > Web: http://katonalab.eu, Email: gergely.katona(a)gu.se > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb >

6 years, 1 month

Re: [cctbxbb] Boost Python 1.56

by Aaron Brewster

Hi Marcin, thanks for the fixes. I'll re-run the tests today and see if tst_ext.py is failing for me too. -Aaron On Thu, Feb 11, 2016 at 10:50 AM, Marcin Wojdyr <wojdyr(a)gmail.com> wrote: > Hi Aaron, > yes, I didn't run test yesterday. I did run it today and at first 20 > failed. > It's mostly this problem: > https://github.com/boostorg/python/issues/56 > > Other failures I got were probably not related to the boost update. > I've committed 3 fixes. > > I started looking at 4th, which was: > File "/home/marcin/dials/modules/cctbx_project/iotbx/mtz/tst_ext.py", > line 1093, in exercise_modifiers > assert approx_equal(v, [-1]*4) > but I must leave now. Does this test also fail for you? > > Marcin > > On Wed, Feb 10, 2016 at 11:20 PM, Aaron Brewster <asbrewster(a)lbl.gov> > wrote: > > Hi Marcin, thanks for the detective work. I've reproduced your results > on > > Centos 7 using the cctbx bootstrap script (not sure from your email if > > that's what you used): > > > > svn export > > svn://svn.code.sf.net/p/cctbx/code/trunk/libtbx/auto_build/bootstrap.py > > python bootstrap.py --builder=dials hot update > > cd modules > > mv boost oldboost > > wget > > > https://sourceforge.net/projects/boost/files/latest/download?source=files > -O > > ./boost_1_60_0.tar.gz > > mv boost_1_60_0 boost > > vi boost/boost/rational.hpp (revert > > > https://github.com/boostorg/rational/commit/5fddb3f889cd2a2fe59cdcae182f8b3… > , > > this is necessary to compile) > > cd .. > > python bootstrap.py --builder=dials base build > > > > I then exercised the cctbx regression tests: > > > > source build/setpaths.sh > > mkdir tests > > cd tests > > cctbx_regression.test_nightly > > > > 17 tests failed, but I think they are fixable, most of them being > interface > > changes in boost. I ran the tests again after restoring the original > boost > > and recompiling. There were still 3 errors, but I need to verify those > > errors from a completely clean system that hadn't seen boost 1.60 before > I'd > > think they were real. > > > > Anyhow the next step is to track down those 17 errors, fix them, then try > > the whole process again with the phenix builder and the full set of dials > > and phenix tests. > > > > -Aaron > > > > > > > > > > > > On Wed, Feb 10, 2016 at 10:45 AM, Marcin Wojdyr <wojdyr(a)gmail.com> > wrote: > >> > >> To help a bit I went through compilation of cctbx/Dials with Boost 1.60. > >> > >> There is a lot of warnings from Boost.Python, > >> about deprecated header: > >> > >> > https://github.com/boostorg/python/commit/0a4c76b9ac16974d7d4f164cf61790952… > >> and unused typedefs. After adding -Wno-unused-local-typedefs and > >> removing the deprecated #include almost all warnings are gone. > >> > >> Then there is one error, copy&pasted below. > >> I tracked it down to this change in boost.rational: > >> > >> > https://github.com/boostorg/rational/commit/5fddb3f889cd2a2fe59cdcae182f8b3… > >> I just reverted this change to finish compilation, but I suppose a > >> proper fix is also not difficult. > >> > >> I don't know if this is relevant, but I'm using Boost compiled outside > of > >> cctbx. > >> I'm attaching a patch that we use in CCP4 for this. It's based on > >> Debian/Gentoo patches. > >> > >> cheers, > >> Marcin > >> > >> > >> g++ -o boost_adaptbx/rational_ext.o -c > >> -I/home/marcin/dials/boost_160/include -Wno-unused-local-typedefs > >> -DBOOST_PYTHON_MAX_BASES=2 -fPIC -fno-strict-aliasing -w -DNDEBUG -O3 > >> -ffast-math -DBOOST_ALL_NO_LIB -I/home/marcin/dials/build/include > >> -I/home/marcin/miniconda2/include/python2.7 > >> /home/marcin/dials/modules/cctbx_project/boost_adaptbx/rational_ext.cpp > >> In file included from > >> > >> > /home/marcin/dials/boost_160/include/boost/preprocessor/iteration/detail/iter/forward1.hpp:47:0, > >> from > >> /home/marcin/dials/boost_160/include/boost/python/detail/invoke.hpp:63, > >> from > >> /home/marcin/dials/boost_160/include/boost/python/detail/caller.hpp:16, > >> from > >> > >> > /home/marcin/dials/boost_160/include/boost/python/object/function_handle.hpp:8, > >> from > >> > >> > /home/marcin/dials/boost_160/include/boost/python/converter/arg_to_python.hpp:19, > >> from > >> /home/marcin/dials/boost_160/include/boost/python/call.hpp:15, > >> from > >> /home/marcin/dials/boost_160/include/boost/python/object_core.hpp:14, > >> from > >> /home/marcin/dials/boost_160/include/boost/python/args.hpp:25, > >> from > >> /home/marcin/dials/boost_160/include/boost/python/make_function.hpp:11, > >> from > >> /home/marcin/dials/boost_160/include/boost/python/def.hpp:11, > >> from > >> > /home/marcin/dials/modules/cctbx_project/boost_adaptbx/rational_ext.cpp:2: > >> /home/marcin/dials/boost_160/include/boost/python/detail/invoke.hpp: > >> In instantiation of 'PyObject* > >> boost::python::detail::invoke(boost::python::detail::invoke_tag_<false, > >> true>, const RC&, F&, TC&) [with RC = > >> > >> > boost::python::detail::specify_a_return_value_policy_to_wrap_functions_returning<const > >> int&>; F = const int& (boost::rational<int>::*)()const; TC = > >> boost::python::arg_from_python<boost::rational<int>&>; PyObject = > >> _object]': > >> > >> > /home/marcin/dials/boost_160/include/boost/python/detail/caller.hpp:223:13: > >> required from 'PyObject* > >> boost::python::detail::caller_arity<1u>::impl<F, Policies, > >> Sig>::operator()(PyObject*, PyObject*) [with F = const int& > >> (boost::rational<int>::*)()const; Policies = > >> boost::python::default_call_policies; Sig = boost::mpl::vector2<const > >> int&, boost::rational<int>&>; PyObject = _object]' > >> > >> > /home/marcin/dials/boost_160/include/boost/python/object/py_function.hpp:38:33: > >> required from 'PyObject* > >> > >> > boost::python::objects::caller_py_function_impl<Caller>::operator()(PyObject*, > >> PyObject*) [with Caller = boost::python::detail::caller<const int& > >> (boost::rational<int>::*)()const, > >> boost::python::default_call_policies, boost::mpl::vector2<const int&, > >> boost::rational<int>&> >; PyObject = _object]' > >> > >> > /home/marcin/dials/modules/cctbx_project/boost_adaptbx/rational_ext.cpp:231:1: > >> required from here > >> > /home/marcin/dials/boost_160/include/boost/python/detail/invoke.hpp:88:90: > >> error: no match for call to '(const > >> > >> > boost::python::detail::specify_a_return_value_policy_to_wrap_functions_returning<const > >> int&>) (const int&)' > >> return rc( (tc().*f)(BOOST_PP_ENUM_BINARY_PARAMS_Z(1, N, ac, () > >> BOOST_PP_INTERCEPT)) ); > >> > >> ^ > >> In file included from > >> > >> > /home/marcin/dials/boost_160/include/boost/python/object/function_handle.hpp:8:0, > >> from > >> > >> > /home/marcin/dials/boost_160/include/boost/python/converter/arg_to_python.hpp:19, > >> from > >> /home/marcin/dials/boost_160/include/boost/python/call.hpp:15, > >> from > >> /home/marcin/dials/boost_160/include/boost/python/object_core.hpp:14, > >> from > >> /home/marcin/dials/boost_160/include/boost/python/args.hpp:25, > >> from > >> /home/marcin/dials/boost_160/include/boost/python/make_function.hpp:11, > >> from > >> /home/marcin/dials/boost_160/include/boost/python/def.hpp:11, > >> from > >> > /home/marcin/dials/modules/cctbx_project/boost_adaptbx/rational_ext.cpp:2: > >> /home/marcin/dials/boost_160/include/boost/python/detail/caller.hpp: > >> In instantiation of 'static const PyTypeObject* > >> > >> > boost::python::detail::converter_target_type<ResultConverter>::get_pytype() > >> [with ResultConverter = > >> > >> > boost::python::detail::specify_a_return_value_policy_to_wrap_functions_returning<const > >> int&>; PyTypeObject = _typeobject]': > >> > >> > /home/marcin/dials/boost_160/include/boost/python/detail/caller.hpp:240:19: > >> required from 'static boost::python::detail::py_func_sig_info > >> boost::python::detail::caller_arity<1u>::impl<F, Policies, > >> Sig>::signature() [with F = const int& > >> (boost::rational<int>::*)()const; Policies = > >> boost::python::default_call_policies; Sig = boost::mpl::vector2<const > >> int&, boost::rational<int>&>]' > >> > >> > /home/marcin/dials/boost_160/include/boost/python/object/py_function.hpp:48:35: > >> required from 'boost::python::detail::py_func_sig_info > >> boost::python::objects::caller_py_function_impl<Caller>::signature() > >> const [with Caller = boost::python::detail::caller<const int& > >> (boost::rational<int>::*)()const, > >> boost::python::default_call_policies, boost::mpl::vector2<const int&, > >> boost::rational<int>&> >]' > >> > >> > /home/marcin/dials/modules/cctbx_project/boost_adaptbx/rational_ext.cpp:231:1: > >> required from here > >> > >> > /home/marcin/dials/boost_160/include/boost/python/detail/caller.hpp:102:109: > >> error: 'struct > >> > boost::python::detail::specify_a_return_value_policy_to_wrap_functions_returning<const > >> int&>' has no member named 'get_pytype' > >> return create_result_converter((PyObject*)0, (ResultConverter > >> *)0, (ResultConverter *)0).get_pytype(); > >> > >> ^ > >> scons: *** [boost_adaptbx/rational_ext.o] Error 1 > >> scons: building terminated because of errors. > >> > >> > >> > >> > >> > >> On Wed, Feb 10, 2016 at 4:44 PM, Aaron Brewster <asbrewster(a)lbl.gov> > >> wrote: > >> > Hi Nick, Billy and I will work on this today, > >> > > >> > -Aaron > >> > > >> > On Wed, Feb 10, 2016 at 6:40 AM, Marcin Wojdyr <wojdyr(a)gmail.com> > wrote: > >> >> > >> >> FWIW the latest version of Boost is 1.60. > >> >> CCP4 6.5 was using cctbx compiled with Boost 1.56 > >> >> and CCP4 7.0 uses Boost 1.58. > >> >> > >> >> Marcin > >> >> > >> >> On Wed, Feb 10, 2016 at 2:17 PM, Nicholas Sauter <nksauter(a)lbl.gov> > >> >> wrote: > >> >> > Nigel, Billy & Aaron, > >> >> > > >> >> > I completely endorse this move to Boost 1.56. Can we update our > >> >> > build? > >> >> > > >> >> > Nick > >> >> > > >> >> > Nicholas K. Sauter, Ph. D. > >> >> > Computer Staff Scientist, Molecular Biophysics and Integrated > >> >> > Bioimaging > >> >> > Division > >> >> > Lawrence Berkeley National Laboratory > >> >> > 1 Cyclotron Rd., Bldg. 33R0345 > >> >> > Berkeley, CA 94720 > >> >> > (510) 486-5713 > >> >> > > >> >> > On Wed, Feb 10, 2016 at 2:41 PM, Luc Bourhis < > luc_j_bourhis(a)mac.com> > >> >> > wrote: > >> >> >> > >> >> >> Hi, > >> >> >> > >> >> >> I have improvements to the smtbx on their way to be committed > which > >> >> >> require Boost version 1.56. This is related to Boost.Threads, > whose > >> >> >> support > >> >> >> I re-activated a few months ago on Nick’s request. I need the > >> >> >> function > >> >> >> boost::thread::physical_concurrency which returns the number of > >> >> >> physical > >> >> >> cores on the machine, as opposed to virtual cores when > >> >> >> hyperthreading > >> >> >> is > >> >> >> enabled (which it is by default on any Intel machine). That > function > >> >> >> is > >> >> >> not > >> >> >> available in Boost 1.55 which is the version currently used in the > >> >> >> nightly > >> >> >> tests: it appeared in 1.56. > >> >> >> > >> >> >> So, would it be possible to move to Boost 1.56? Otherwise, I will > >> >> >> need > >> >> >> to > >> >> >> backport that function. Not too difficult but not thrilling. > >> >> >> > >> >> >> Best wishes, > >> >> >> > >> >> >> Luc > >> >> >> > >> >> >> > >> >> >> _______________________________________________ > >> >> >> cctbxbb mailing list > >> >> >> cctbxbb(a)phenix-online.org > >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> > > >> >> > > >> >> > > >> >> > _______________________________________________ > >> >> > cctbxbb mailing list > >> >> > cctbxbb(a)phenix-online.org > >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb > >> >> > > >> >> > >> >> _______________________________________________ > >> >> cctbxbb mailing list > >> >> cctbxbb(a)phenix-online.org > >> >> http://phenix-online.org/mailman/listinfo/cctbxbb > >> > > >> > > >> > > >> > _______________________________________________ > >> > cctbxbb mailing list > >> > cctbxbb(a)phenix-online.org > >> > http://phenix-online.org/mailman/listinfo/cctbxbb > >> > > >> > >> _______________________________________________ > >> cctbxbb mailing list > >> cctbxbb(a)phenix-online.org > >> http://phenix-online.org/mailman/listinfo/cctbxbb > >> > > > > > > _______________________________________________ > > cctbxbb mailing list > > cctbxbb(a)phenix-online.org > > http://phenix-online.org/mailman/listinfo/cctbxbb > > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb >

10 years

Jump to page: