[cctbxbb] some thoughts on cctbx and pip
Billy Poon
BKPoon at lbl.gov
Wed Dec 18 11:25:07 PST 2019
Hi Gergely,
That's the other information that hasn't been updated yet in the conda
package. :)
For LIBTBX_BUILD, I can set up environment variables sort of how the conda
gcc package sets up environment variables. I have to double check to see if
they're set after installation of the package and after activation of an
environment. The variable will just be a copy of $CONDA_PREFIX.
The libtbx_env file is a file that exists in $LIBTBX_BUILD that basically
stores configuration information. I'll have to modify the contents to
update the locations of the different modules since the conda package will
not have a "modules" directory.
--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709
Fax: (510) 486-5909
Web: https://phenix-online.org
On Wed, Dec 18, 2019 at 2:08 AM Gergely Katona <gkatona at gmail.com> wrote:
> Hi Billy,
>
> Conda install went fine with your instructions after rearranging the
> channels by putting cctbx last. I removed all environmental variables
> and previous build of cctbx.
>
> import sys
> print (sys.path, sys.prefix)
>
> ['/home/gergely/anaconda3/lib/python36.zip',
> '/home/gergely/anaconda3/lib/python3.6',
> '/home/gergely/anaconda3/lib/python3.6/lib-dynload', '',
> '/home/gergely/anaconda3/lib/python3.6/site-packages',
> '/home/gergely/anaconda3/lib/python3.6/site-packages/IPython/extensions',
> '/home/gergely/.ipython'] /home/gergely/anaconda3
>
> Many imports went fine including pymc3, but I encountered problems
> with these three:
>
> from cctbx import miller
> import iotbx.pdb
> from iotbx import reflection_file_reader, mtz
>
>
> I also got type error when handling space groups.
>
>
>
> ---------------------------------------------------------------------------
> KeyError Traceback (most recent call last)
> <ipython-input-5-cedbd4f84f3d> in <module>
> 25 from cctbx import uctbx
> 26 from cctbx import sgtbx
> ---> 27 from cctbx import miller
> 28 #from iotbx import reflection_file_reader, mtz
> 29 sns.set_context("poster")
>
> ~/anaconda3/lib/python3.6/site-packages/cctbx/miller/__init__.py in
> <module>
> 11
> 12 from cctbx import crystal
> ---> 13 from cctbx import maptbx
> 14 from cctbx import sgtbx
> 15 from cctbx.sgtbx import lattice_symmetry
>
> ~/anaconda3/lib/python3.6/site-packages/cctbx/maptbx/__init__.py in
> <module>
> 15 from libtbx import adopt_init_args
> 16 from libtbx.utils import Sorry
> ---> 17 import libtbx.load_env
> 18 import math
> 19 import sys, os
>
> ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module>
> 3 import libtbx.env_config
> 4 import os
> ----> 5 libtbx.env = libtbx.env_config.unpickle()
> 6 libtbx.env.set_os_environ_all_dist()
> 7 libtbx.env.dispatcher_name =
> os.environ.get("LIBTBX_DISPATCHER_NAME")
>
> ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle()
> 2603
> 2604 def unpickle():
> -> 2605 build_path = os.environ["LIBTBX_BUILD"]
> 2606
> set_preferred_sys_prefix_and_sys_executable(build_path=build_path)
> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")
>
> ~/anaconda3/lib/python3.6/os.py in __getitem__(self, key)
> 667 except KeyError:
> 668 # raise KeyError with the original key value
> --> 669 raise KeyError(key) from None
> 670 return self.decodevalue(value)
> 671
>
> KeyError: 'LIBTBX_BUILD'
>
>
>
> ---------------------------------------------------------------------------
> ArgumentError Traceback (most recent call last)
> <ipython-input-7-2acd3a9ce26a> in <module>
> 26 return
> ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar
> 27
> ---> 28
> ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar=initializecrystal()
>
> <ipython-input-7-2acd3a9ce26a> in initializecrystal()
> 11 uc = uctbx.unit_cell(unit_cell)
> 12 wavelength = 1.54980
> ---> 13 xtal_symm = crystal.symmetry(unit_cell=unit_cell,
> space_group_symbol="P 43 21 2")
> 14
> 15 ms =
> miller.build_set(crystal_symmetry=xtal_symm,anomalous_flag=True,
> d_min=1.61)
>
> ~/anaconda3/lib/python3.6/site-packages/cctbx/crystal/__init__.py in
> __init__(self, unit_cell, space_group_symbol, space_group_info,
> space_group, correct_rhombohedral_setting_if_necessary,
> assert_is_compatible_unit_cell, raise_sorry_if_incompatible_unit_cell,
> force_compatible_unit_cell)
> 74 if (space_group_symbol is not None):
> 75 self._space_group_info = sgtbx.space_group_info(
> ---> 76 symbol=space_group_symbol)
> 77 elif (space_group is not None):
> 78 if (isinstance(space_group, sgtbx.space_group)):
>
> ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in
> __init__(self, symbol, table_id, group, number, space_group_t_den)
> 100 assert group is None
> 101 if (table_id is None):
> --> 102 symbols = space_group_symbols(symbol)
> 103 else:
> 104 if (isinstance(symbol, int)): symbol = str(symbol)
>
> ArgumentError: Python argument types in
> space_group_symbols.__init__(space_group_symbols, str)
> did not match C++ signature:
> __init__(_object*, int space_group_number)
> __init__(_object*, int space_group_number,
> std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > extension='')
> __init__(_object*, int space_group_number,
> std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > extension='', std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > table_id='')
> __init__(_object*, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > symbol)
> __init__(_object*, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > symbol,
> std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > table_id='')
>
>
> When I tried to set LIBTBX_BUILD to /home/gergely/anaconda3 (this is
> sys.prefix on my system), I got other problems when importing. Can
> LIBTBX_BUILD be set in the conda package?
>
>
>
> ---------------------------------------------------------------------------
> FileNotFoundError Traceback (most recent call last)
> <ipython-input-1-7e2b144826cc> in <module>
> 26 from cctbx import sgtbx
> 27 #from cctbx import miller
> ---> 28 from iotbx import reflection_file_reader, mtz
> 29 sns.set_context("poster")
> 30 plt.rcParams.update({'figure.autolayout': True})
>
> ~/anaconda3/lib/python3.6/site-packages/iotbx/reflection_file_reader.py
> in <module>
> 53
> 54 from __future__ import absolute_import, division, print_function
> ---> 55 from iotbx import mtz
> 56 from iotbx.scalepack import merge as scalepack_merge
> 57 from iotbx.scalepack import no_merge_original_index as
> scalepack_no_merge
>
> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/__init__.py in <module>
> 9 import iotbx_mtz_ext as ext
> 10
> ---> 11 from iotbx.mtz import extract_from_symmetry_lib
> 12 from cctbx import xray
> 13 import cctbx.xray.observation_types
>
>
> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/extract_from_symmetry_lib.py
> in <module>
> 1 from __future__ import absolute_import, division, print_function
> 2 from cctbx import sgtbx
> ----> 3 import libtbx.load_env
> 4 import os.path as op
> 5 from six.moves import range
>
> ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module>
> 3 import libtbx.env_config
> 4 import os
> ----> 5 libtbx.env = libtbx.env_config.unpickle()
> 6 libtbx.env.set_os_environ_all_dist()
> 7 libtbx.env.dispatcher_name =
> os.environ.get("LIBTBX_DISPATCHER_NAME")
>
> ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle()
> 2605 build_path = os.environ["LIBTBX_BUILD"]
> 2606
> set_preferred_sys_prefix_and_sys_executable(build_path=build_path)
> -> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")
> 2608 env = pickle.load(libtbx_env)
> 2609 if (env.python_version_major_minor != sys.version_info[:2]):
>
> FileNotFoundError: [Errno 2] No such file or directory:
> '/home/gergely/anaconda3/libtbx_env'
>
> On Wed, Dec 18, 2019 at 9:10 AM Billy Poon <BKPoon at lbl.gov> wrote:
> >
> > Hi Gergely,
> >
> > I've uploaded linux packages to a new channel, cctbx-dev, and you can
> install it with
> >
> > conda install -c cctbx-dev cctbx
> >
> > in your current environment. You should set your ~/.condarc file to pull
> the other dependencies from the conda-forge channel first, so put
> conda-forge above cctbx. Mine looks like
> >
> > channels:
> > - conda-forge
> > - defaults
> > - cctbx
> >
> > Do you need dxtbx for your scripts? This package does not build that
> part. I think the plan is to build a separate conda package for dxtbx so
> that it can be updated more frequently. I can rebuild the packages to
> include it for testing, but the one being submitted to conda-forge will not
> have it.
> >
> > Also, your error message is probably due to version of HDF5 that the
> development build installs. The bootstrap.py script will install 1.10.4,
> but your other dependency is looking for 1.10.5. Installing this cctbx
> conda package should install 1.10.5, which should fix the issue. Also, I'm
> updating those environments and 1.10.5 will be the new default version.
> >
> > Lastly, the dispatchers will not work in these packages because the old
> paths during the build process are still in them and many of them expect
> some additional information that has not been updated in the packages yet.
> I'm in the process of doing that and will update the cctbx-dev channel when
> that's done. However, by starting python, you can import cctbx modules. So
> you can run commands like
> >
> > from scitbx.array_family import flex
> > a = flex.random_double(1000000)
> > b = flex.min_max_mean_double(a)
> > b.min
> > b.max
> > b.mean
> >
> > Thanks!
> >
> > --
> > Billy K. Poon
> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> > Lawrence Berkeley National Laboratory
> > 1 Cyclotron Road, M/S 33R0345
> > Berkeley, CA 94720
> > Tel: (510) 486-5709
> > Fax: (510) 486-5909
> > Web: https://phenix-online.org
> >
> >
> > On Tue, Dec 17, 2019 at 5:39 AM Gergely Katona <gkatona at gmail.com>
> wrote:
> >>
> >> Dear Billy,
> >>
> >> Thank you for the detailed explanation, I look forward to do further
> >> testing! I aim to pool all modules under the same environment, if
> >> there are conflicts then I just try to reshuffle the order of imports.
> >> So far this did not cause problems for me even when I was using system
> >> python, but of course it is not the most prudent thing to do. With
> >> anaconda everything is much more standardized and isolated already and
> >> without being superuser I can have my familiar environment at any
> >> synchrotron based cluster. It is great that cctbx will be an integral
> >> part of this ecosystem and this was also the last thing holding me
> >> back from adopting python3. About hdf5, pymc3 requires it and
> >> importing causes a kernel restart with the following error messages,
> >> curiously if I import h5py first this can be avoided.
> >>
> >> Best wishes,
> >>
> >> Gergely
> >>
> >> import pymc3 as pm
> >>
> >> Warning! ***HDF5 library version mismatched error***
> >> The HDF5 header files used to compile this application do not match
> >> the version used by the HDF5 library to which this application is
> linked.
> >> Data corruption or segmentation faults may occur if the application
> continues.
> >> This can happen when an application was compiled by one version of HDF5
> but
> >> linked with a different version of static or shared HDF5 library.
> >> You should recompile the application or check your shared library
> related
> >> settings such as 'LD_LIBRARY_PATH'.
> >> You can, at your own risk, disable this warning by setting the
> environment
> >> variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.
> >> Setting it to 2 or higher will suppress the warning messages totally.
> >> Headers are 1.10.4, library is 1.10.5
> >> SUMMARY OF THE HDF5 CONFIGURATION
> >> =================================
> >>
> >> General Information:
> >> -------------------
> >> HDF5 Version: 1.10.5
> >> Configured on: Tue Oct 22 12:02:13 UTC 2019
> >> Configured by: conda at 16247e67ecd5
> >> Host system: x86_64-conda_cos6-linux-gnu
> >> Uname information: Linux 16247e67ecd5 4.15.0-1059-azure
> >> #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64
> >> GNU/Linux
> >> Byte sex: little-endian
> >> Installation point: /home/gergely/anaconda3
> >>
> >> Compiling Options:
> >> ------------------
> >> Build Mode: production
> >> Debugging Symbols: no
> >> Asserts: no
> >> Profiling: no
> >> Optimization Level: high
> >>
> >> Linking Options:
> >> ----------------
> >> Libraries: static, shared
> >> Statically Linked Executables:
> >> LDFLAGS: -Wl,-O2 -Wl,--sort-common
> >> -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags
> >> -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib
> >> -Wl,-rpath-link,/home/gergely/anaconda3/lib
> >> -L/home/gergely/anaconda3/lib
> >> H5_LDFLAGS:
> >> AM_LDFLAGS: -L/home/gergely/anaconda3/lib
> >> Extra libraries: -lrt -lpthread -lz -ldl -lm
> >> Archiver:
> >>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar
> >> AR_FLAGS: cr
> >> Ranlib:
> >>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib
> >>
> >> Languages:
> >> ----------
> >> C: yes
> >> C Compiler:
> >>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc
> >> CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2
> >> -I/home/gergely/anaconda3/include
> >> H5_CPPFLAGS: -D_GNU_SOURCE
> >> -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API
> >> AM_CPPFLAGS: -I/home/gergely/anaconda3/include
> >> C Flags: -march=nocona -mtune=haswell
> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include
> >>
> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5
> >> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix
> >> H5 C Flags: -std=c99 -pedantic -Wall -Wextra
> >> -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion
> >> -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal
> >> -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations
> >> -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs
> >> -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls
> >> -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef
> >> -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings
> >> -finline-functions -s -Wno-inline -Wno-aggregate-return
> >> -Wno-missing-format-attribute -Wno-missing-noreturn -O
> >> AM C Flags:
> >> Shared C Library: yes
> >> Static C Library: yes
> >>
> >>
> >> Fortran: yes
> >> Fortran Compiler:
> >>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran
> >> Fortran Flags:
> >> H5 Fortran Flags: -pedantic -Wall -Wextra -Wunderflow
> >> -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2
> >> AM Fortran Flags:
> >> Shared Fortran Library: yes
> >> Static Fortran Library: yes
> >>
> >> C++: yes
> >> C++ Compiler:
> >>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++
> >> C++ Flags: -fvisibility-inlines-hidden
> >> -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell
> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include
> >>
> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5
> >> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix
> >> H5 C++ Flags: -pedantic -Wall -W -Wundef -Wshadow
> >> -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion
> >> -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual
> >> -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor
> >> -Wctor-dtor-privacy -Wabi -finline-functions -s -O
> >> AM C++ Flags:
> >> Shared C++ Library: yes
> >> Static C++ Library: yes
> >>
> >> Java: no
> >>
> >>
> >> Features:
> >> ---------
> >> Parallel HDF5: no
> >> Parallel Filtered Dataset Writes: no
> >> Large Parallel I/O: no
> >> High-level library: yes
> >> Threadsafety: yes
> >> Default API mapping: v110
> >> With deprecated public symbols: yes
> >> I/O filters (external): deflate(zlib)
> >> MPE: no
> >> Direct VFD: no
> >> dmalloc: no
> >> Packages w/ extra debug output: none
> >> API tracing: no
> >> Using memory checker: yes
> >> Memory allocation sanity checks: no
> >> Function stack tracing: no
> >> Strict file format checks: no
> >> Optimization instrumentation: no
> >>
> >> On Tue, Dec 17, 2019 at 8:35 AM Billy Poon <BKPoon at lbl.gov> wrote:
> >> >
> >> > Hi Gergely,
> >> >
> >> > Let me build the test package tomorrow. All the gritty details for
> building with conda is being finalized and the official documentation will
> be updated to describe the steps. It would be too confusing to keep
> changing the documentation as the process evolves. You do have the general
> process, though, which is summarized as follows.
> >> >
> >> > 1) Installing dependencies. The cctbx_dependencies metapackage was an
> initial approach for managing the CCTBX dependencies, but after contacting
> the conda-forge folks, they recommended using the --only-deps flag. So when
> the CCTBX conda package is available, you'll be able to get a set of
> dependencies with,
> >> >
> >> > conda install -c conda-forge --only-deps cctbx
> >> >
> >> > By default, the bootstrap.py file will automatically install a set of
> dependencies in the "conda_base" directory (and a conda installation if one
> is not found). It just uses standard conda environment files located in
> libtbx/auto_build/conda_envs, so you do not need to install
> cctbx_dependencies as a separate step. The environment files avoid channel
> issues by explicitly defining the channel to pull the packages from and the
> cctbx channel just stores copies of packages from conda-forge. There were
> issues earlier where the conda-forge packages would sometimes be moved to a
> different label. The --use-conda flag also accepts a path to $CONDA_PREFIX
> if you want to use a specific environment for testing.
> >> >
> >> > 2) Building. The bootstrap.py file handles that with SCons.
> >> >
> >> > 3) Running. After building, there should be a setpaths.sh (and .csh)
> file that adds build/bin to your path. The build/bin directory has our
> dispatchers, which are just scripts that set up the environment variables
> for you. This prevents other programs from loading our libraries, whose
> versions may conflict. You should see that there is a "python" dispatcher,
> which is a convenience for developers. Otherwise, you can use
> libtbx.python, which will be able to import CCTBX modules.
> >> >
> >> > The running part is where the conda package for CCTBX will be
> different than this build. Since our Python files and extensions modules
> will be in the "site-packages" directory for Python, the PYTHONPATH
> variable will not be needed (and conda suggests that that variable not be
> set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so
> LD_LIBRARY_PATH is not needed. In an active environment, PATH will already
> be modified. And then our LIBTBX_BUILD directory can be set to sys.prefix.
> >> >
> >> > So with the conda package, you would only need to activate your conda
> environment and CCTBX should integrate with other conda packages. What is
> the conflict with hdf5? That's something that should be fixed. Thanks!
> >> >
> >> > --
> >> > Billy K. Poon
> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> >> > Lawrence Berkeley National Laboratory
> >> > 1 Cyclotron Road, M/S 33R0345
> >> > Berkeley, CA 94720
> >> > Tel: (510) 486-5709
> >> > Fax: (510) 486-5909
> >> > Web: https://phenix-online.org
> >> >
> >> >
> >> > On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona <gkatona at gmail.com>
> wrote:
> >> >>
> >> >> Dear Billy,
> >> >>
> >> >> Thank you for this update and for your efforts! I found a solution
> and
> >> >> indeed most things already work in anaconda3. The steps I took (even
> >> >> if these instructions will have short expiry date):
> >> >>
> >> >> Modified .condarc with:
> >> >> channels:
> >> >> - cctbx
> >> >> - conda-forge
> >> >> - defaults
> >> >>
> >> >> run
> >> >> conda conda install cctbx_dependencies python=3.6
> >> >>
> >> >> I expect this will work without python=3.6 in the near future.
> >> >>
> >> >> Then compiling cctbx with anaconda3 python went without any problem
> >> >> when using these flags:
> >> >> python bootstrap.py --use-conda --python3 --nproc=12
> >> >>
> >> >> Then I probably did the unorthodox thing and sourced these
> directories:
> >> >>
> >> >> setenv LIBTBX_BUILD /home/gergely/cctbx/build
> >> >> setenv PATH ${PATH}:/home/gergely/cctbx/build/bin
> >> >> setenv PYTHONPATH
> >> >>
> /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH
> >> >> setenv LD_LIBRARY_PATH
> >> >>
> /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH
> >> >>
> >> >> There is probably a better way to put this into conda environment.
> >> >> With these steps I could run one of my scripts depending on cctbx.
> The
> >> >> only problem I found is that hdf5 library had conflict with another
> >> >> package in conda, but I does not find this as a showstopper.
> >> >>
> >> >> Best wishes,
> >> >>
> >> >> Gergely
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On Fri, Dec 13, 2019 at 7:23 PM Billy Poon <BKPoon at lbl.gov> wrote:
> >> >> >
> >> >> > Hi Gergely,
> >> >> >
> >> >> > It's still a work in progress. I'm sorting out some Windows issues
> right now. I can probably build a test package on a separate channel for
> people that want to test it (let's say next week?). I'll provide
> instructions, but basically, the test conda package will be in its own
> separate channel and the dependencies will be pulled from the conda-forge
> channel. I want most things to be working correctly on Python 2.7, 3.6,
> 3.7, and 3.8 on all 3 platforms.
> >> >> >
> >> >> > Thanks!
> >> >> >
> >> >> > --
> >> >> > Billy K. Poon
> >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> >> >> > Lawrence Berkeley National Laboratory
> >> >> > 1 Cyclotron Road, M/S 33R0345
> >> >> > Berkeley, CA 94720
> >> >> > Tel: (510) 486-5709
> >> >> > Fax: (510) 486-5909
> >> >> > Web: https://phenix-online.org
> >> >> >
> >> >> >
> >> >> > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona <gkatona at gmail.com>
> wrote:
> >> >> >>
> >> >> >> Dear Billy,
> >> >> >>
> >> >> >> This sounds very promising and exciting. I am not sure if cctbx is
> >> >> >> already functional as a conda package in anaconda3 (Linux) or
> this is
> >> >> >> still work in progress. My technical expertise does not allow me
> to
> >> >> >> tell the difference. What I tried:
> >> >> >>
> >> >> >> Fresh install of anaconda3. Adding - cctbx and - conda-forge to
> >> >> >> .condarc . Installing pyside2 with conda. Running conda install
> >> >> >> conda_dependencies . I get a lot package version conflict, and I
> >> >> >> cannot import cctbx or iotbx to anaconda python. Am I following
> the
> >> >> >> right instructions? Or it is too early to expect that cctbx works
> when
> >> >> >> installed through conda?
> >> >> >>
> >> >> >> Best wishes,
> >> >> >>
> >> >> >> Gergely
> >> >> >>
> >> >> >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon <BKPoon at lbl.gov>
> wrote:
> >> >> >> >
> >> >> >> > Hi all,
> >> >> >> >
> >> >> >> > For a brief update, I have submitted a recipe for cctbxlite to
> conda-forge (https://github.com/conda-forge/staged-recipes/pull/10021)
> and support for Python 3.7 and 3.8 is being added (
> https://github.com/cctbx/cctbx_project/pull/409). With some fixes for
> Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms
> (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8.
> Some additional changes will be needed to get Windows to work with Python 3
> and for tests to pass with Boost 1.70.0. That will enable the conda-forge
> recipe to build for all platforms and for all supported versions of Python.
> >> >> >> >
> >> >> >> > Currently, the conda-forge recipe will install into the conda
> python and cctbx imports can be done without sourcing the environment
> scripts. It looks like a lot of the environment variables being set in the
> dispatchers can be removed since the Python files and C++ extensions are in
> the right places. I'll update the libtbx_env file so that commands that
> load the environment can work correctly.
> >> >> >> >
> >> >> >> > --
> >> >> >> > Billy K. Poon
> >> >> >> > Research Scientist, Molecular Biophysics and Integrated
> Bioimaging
> >> >> >> > Lawrence Berkeley National Laboratory
> >> >> >> > 1 Cyclotron Road, M/S 33R0345
> >> >> >> > Berkeley, CA 94720
> >> >> >> > Tel: (510) 486-5709
> >> >> >> > Fax: (510) 486-5909
> >> >> >> > Web: https://phenix-online.org
> >> >> >> >
> >> >> >> >
> >> >> >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll <tic20 at cam.ac.uk>
> wrote:
> >> >> >> >>
> >> >> >> >> Hi Luc,
> >> >> >> >>
> >> >> >> >> That sounds promising. From there, I’d need to work out how to
> make a fully-packaged installer (basically a modified wheel file) for the
> ChimeraX ToolShed - the aim is for the end user to not have to worry about
> any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would
> need to be set dynamically before first import - but doesn’t seem
> insurmountable.
> >> >> >> >>
> >> >> >> >> Thanks,
> >> >> >> >>
> >> >> >> >> Tristan
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Tristan Croll
> >> >> >> >> Research Fellow
> >> >> >> >> Cambridge Institute for Medical Research
> >> >> >> >> University of Cambridge CB2 0XY
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis <luc_j_bourhis at mac.com>
> wrote:
> >> >> >> >> >
> >> >> >> >> > Hi Tristan,
> >> >> >> >> >
> >> >> >> >> > cctbx could be built to use your ChimeraX python, now that
> cctbx is moving to Python 3. The option —with-python is there for that with
> the bootstrap script. The specific environment setup boil down to setting
> two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux,
> PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a
> framework such as ChimeraX, that should not be difficult to ensure those
> two variables are set.
> >> >> >> >> >
> >> >> >> >> > Best wishes,
> >> >> >> >> >
> >> >> >> >> > Luc
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll <tic20 at cam.ac.uk>
> wrote:
> >> >> >> >> >>
> >> >> >> >> >> To add my two cents on this: probably the second-most
> common question I've had about ISOLDE's implementation is, "why didn't you
> use CCTBX?". The honest answer to that is, "I didn't know how."
> >> >> >> >> >>
> >> >> >> >> >> Still don't, really - although the current developments are
> rather promising. The problem I've faced is that CCTBX was designed as its
> own self-contained Python (2.7, until very recently) environment, with its
> own interpreter and a lot of very specific environment setup. Meanwhile I'm
> developing ISOLDE in ChimeraX, which is *also* its own self-contained
> Python (3.7) environment. To plug one into the other in that form... well,
> I don't think I'm a good enough programmer to really know where to start.
> >> >> >> >> >>
> >> >> >> >> >> The move to Conda and a more modular CCTBX architecture
> should make a lot more possible in that direction. Pip would be even better
> for me personally (ChimeraX can install directly from the PyPI, but doesn't
> interact with Conda) - but I understand pretty well the substantial
> challenge that would amount to (not least being that the PyPI imposes a
> limit - around 100MB from memory? - on the size of an individual package).
> >> >> >> >> >>
> >> >> >> >> >> Best regards,
> >> >> >> >> >>
> >> >> >> >> >> Tristan
> >> >> >> >> >>
> >> >> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote:
> >> >> >> >> >>> Hi Graeme,
> >> >> >> >> >>> Yes, I know. But “black" is a program doing a very
> particular task
> >> >> >> >> >>> (code formatting from the top of my head). Requiring to
> use a wrapper
> >> >> >> >> >>> for python itself is another level. But ok, I think I am
> mellowing to
> >> >> >> >> >>> the idea after all! Talking with people around me, and
> extrapolating,
> >> >> >> >> >>> I would bet that, right now, a great majority of people
> interested by
> >> >> >> >> >>> cctbx in pip have already used the cctbx, so they know
> about the
> >> >> >> >> >>> Python wrapper, and they would not be too sanguine about
> that. My
> >> >> >> >> >>> concern is for the future, when pip will be the first time
> some people
> >> >> >> >> >>> use cctbx. Big fat warning notices on PyPI page and a
> better error
> >> >> >> >> >>> message when cctbx fails because LIBTBX_BUILD is not set
> would be
> >> >> >> >> >>> needed but that could be all right.
> >> >> >> >> >>> If we do a pip installer, we should aim at a minimal
> install: cctbx,
> >> >> >> >> >>> iotbx and their dependencies, and that’s it.
> >> >> >> >> >>> Best wishes,
> >> >> >> >> >>> Luc
> >> >> >> >> >>>> On 23 Aug 2019, at 07:17, Graeme.Winter at Diamond.ac.uk <
> Graeme.Winter at diamond.ac.uk> wrote:
> >> >> >> >> >>>> Without discussing the merits of this or whether we
> _choose_ to make the move to supporting PIP, I am certain it would be
> _possible_ - many other packages make dispatcher scripts when you pip
> install them e.g.
> >> >> >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat $(which
> black)
> >> >> >> >> >>>>
> /Library/Frameworks/Python.framework/Versions/3.6/bin/black
> >> >> >> >> >>>>
> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
> >> >> >> >> >>>> # -*- coding: utf-8 -*-
> >> >> >> >> >>>> import re
> >> >> >> >> >>>> import sys
> >> >> >> >> >>>> from black import main
> >> >> >> >> >>>> if __name__ == '__main__':
> >> >> >> >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '',
> sys.argv[0])
> >> >> >> >> >>>> sys.exit(main())
> >> >> >> >> >>>> So we _could_ work around the absence of LIBTBX_BUILD
> etc. in the system. Whether or not we elect to do the work is a different
> question, and it seems clear that here are very mixed opinions on this.
> >> >> >> >> >>>> Best wishes Graeme
> >> >> >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis <
> luc_j_bourhis at mac.com<mailto:luc_j_bourhis at mac.com>> wrote:
> >> >> >> >> >>>> Hi,
> >> >> >> >> >>>> Even if we managed to ship our the boost dynamic
> libraries with pip, it would still not be pip-like, as we would still need
> our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip
> packages work with the standard python exe. LD_LIBRARY_PATH, we could get
> around that by changing the way we compile, using -Wl,-R, which is the
> runtime equivalent of build time -L. That’s a significant change that would
> need to be tested. But there is no way around setting LIBTBX_BUILD right
> now. Leaving that to the user is horrible. Perhaps there is a way to hack
> libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip
> installs?
> >> >> >> >> >>>> Best wishes,
> >> >> >> >> >>>> Luc
> >> >> >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis <
> luc_j_bourhis at mac.com<mailto:luc_j_bourhis at mac.com>> wrote:
> >> >> >> >> >>>> Hi,
> >> >> >> >> >>>> I did look into that many years ago, and even toyed with
> building a pip installer. What stopped me is the exact conclusion you
> reached too: the user would not have the pip experience he expects. You are
> right that it is a lot of effort but is it worth it? Considering that
> remark, I don’t think so. Now, Conda was created specifically to go beyond
> pip pure-python-only support. Since cctbx has garnered support for Conda,
> the best avenue imho is to go the extra length to have a package on
> Anaconda.org<http://anaconda.org/>, and then to advertise it hard to
> every potential user out there.
> >> >> >> >> >>>> Best wishes,
> >> >> >> >> >>>> Luc
> >> >> >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster <
> asbrewster at lbl.gov<mailto:asbrewster at lbl.gov>> wrote:
> >> >> >> >> >>>> Hi, to avoid clouding Dorothee's documentation email
> thread, which I think is a highly useful enterprise, here's some thoughts
> about putting cctbx into pip. Pip doesn't install non-python dependencies
> well. I don't think boost is available as a package on pip (at least the
> package version we use). wxPython4 isn't portable through pip (
> https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-Phoenix_using_pip).
> MPI libraries are system dependent. If cctbx were a pure python package,
> pip would be fine, but cctbx is not.
> >> >> >> >> >>>> All that said, we could build a manylinux1 version of
> cctbx and upload it to PyPi (I'm just learning about this). For a pip
> package to be portable (which is a requirement for cctbx), it needs to
> conform to PEP513, the manylinux1 standard (
> https://www.python.org/dev/peps/pep-0513/). For example, numpy is built
> according to this standard (see https://pypi.org/project/numpy/#files,
> where you'll see the manylinux1 wheel). Note, the manylinux1 standard is
> built with Centos 5.11 which we no longer support.
> >> >> >> >> >>>> There is also a manylinux2010 standard, which is based on
> Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a
> more attainable target (note though by default C++11 is not supported on
> Centos 6).
> >> >> >> >> >>>> If we built a manylinuxX version of cctbx and uploaded it
> to PyPi, the user would need all the non-python dependencies. There's no
> way to specify these in pip. For example, cctbx requires boost 1.63 or
> better. The user will need to have it in a place their python can find it,
> or we could package it ourselves and supply it, similar to how the pip h5py
> package now comes with an hd5f library, or how the pip numpy package
> includes an openblas library. We'd have to do the same for any packages we
> depend on that aren't on pip using the manylinux standards, such as
> wxPython4.
> >> >> >> >> >>>> Further, we need to think about how dials and other
> cctbx-based packages interact. If pip install cctbx is set up, how does
> pip install dials work, such that any dials shared libraries can find the
> cctbx libraries? Can shared libraries from one pip package link against
> libraries in another pip package? Would each package need to supply its
> own boost? Possibly this is well understood in the pip field, but not by
> me :)
> >> >> >> >> >>>> Finally, there's the option of providing a source pip
> package. This would require the full compiler toolchain for any given
> platform (macOS, linux, windows). These are likely available for
> developers, but not for general users.
> >> >> >> >> >>>> Anyway, these are some of the obstacles. Not saying it
> isn't possible, it's just a lot of effort.
> >> >> >> >> >>>> Thanks,
> >> >> >> >> >>>> -Aaron
> >> >> >> >> >>>> _______________________________________________
> >> >> >> >> >>>> cctbxbb mailing list
> >> >> >> >> >>>> cctbxbb at phenix-online.org<mailto:
> cctbxbb at phenix-online.org>
> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >> >> >>>> _______________________________________________
> >> >> >> >> >>>> cctbxbb mailing list
> >> >> >> >> >>>> cctbxbb at phenix-online.org<mailto:
> cctbxbb at phenix-online.org>
> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >> >> >>>> _______________________________________________
> >> >> >> >> >>>> cctbxbb mailing list
> >> >> >> >> >>>> cctbxbb at phenix-online.org<mailto:
> cctbxbb at phenix-online.org>
> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >> >> >>>> --
> >> >> >> >> >>>> This e-mail and any attachments may contain confidential,
> copyright and or privileged material, and are for the use of the intended
> addressee only. If you are not the intended addressee or an authorised
> recipient of the addressee please notify us of receipt by returning the
> e-mail and do not use, copy, retain, distribute or disclose the information
> in or attached to the e-mail.
> >> >> >> >> >>>> Any opinions expressed within this e-mail are those of
> the individual and not necessarily of Diamond Light Source Ltd.
> >> >> >> >> >>>> Diamond Light Source Ltd. cannot guarantee that this
> e-mail or any attachments are free from viruses and we cannot accept
> liability for any damage which you may sustain as a result of software
> viruses which may be transmitted in or with the message.
> >> >> >> >> >>>> Diamond Light Source Limited (company no. 4375679).
> Registered in England and Wales with its registered office at Diamond
> House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11
> 0DE, United Kingdom
> >> >> >> >> >>>> _______________________________________________
> >> >> >> >> >>>> cctbxbb mailing list
> >> >> >> >> >>>> cctbxbb at phenix-online.org
> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >> >> >>> _______________________________________________
> >> >> >> >> >>> cctbxbb mailing list
> >> >> >> >> >>> cctbxbb at phenix-online.org
> >> >> >> >> >>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> _______________________________________________
> >> >> >> >> >> cctbxbb mailing list
> >> >> >> >> >> cctbxbb at phenix-online.org
> >> >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > _______________________________________________
> >> >> >> >> > cctbxbb mailing list
> >> >> >> >> > cctbxbb at phenix-online.org
> >> >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> _______________________________________________
> >> >> >> >> cctbxbb mailing list
> >> >> >> >> cctbxbb at phenix-online.org
> >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >> >
> >> >> >> > _______________________________________________
> >> >> >> > cctbxbb mailing list
> >> >> >> > cctbxbb at phenix-online.org
> >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Gergely Katona, PhD
> >> >> >> Associate Professor
> >> >> >> Department of Chemistry and Molecular Biology, University of
> Gothenburg
> >> >> >> Box 462, 40530 Göteborg, Sweden
> >> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> >> >> >> Web: http://katonalab.eu, Email: gergely.katona at gu.se
> >> >> >>
> >> >> >> _______________________________________________
> >> >> >> cctbxbb mailing list
> >> >> >> cctbxbb at phenix-online.org
> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >
> >> >> > _______________________________________________
> >> >> > cctbxbb mailing list
> >> >> > cctbxbb at phenix-online.org
> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Gergely Katona, PhD
> >> >> Associate Professor
> >> >> Department of Chemistry and Molecular Biology, University of
> Gothenburg
> >> >> Box 462, 40530 Göteborg, Sweden
> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> >> >> Web: http://katonalab.eu, Email: gergely.katona at gu.se
> >> >>
> >> >> _______________________________________________
> >> >> cctbxbb mailing list
> >> >> cctbxbb at phenix-online.org
> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >
> >> > _______________________________________________
> >> > cctbxbb mailing list
> >> > cctbxbb at phenix-online.org
> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
> >>
> >>
> >>
> >> --
> >> Gergely Katona, PhD
> >> Associate Professor
> >> Department of Chemistry and Molecular Biology, University of Gothenburg
> >> Box 462, 40530 Göteborg, Sweden
> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> >> Web: http://katonalab.eu, Email: gergely.katona at gu.se
> >>
> >> _______________________________________________
> >> cctbxbb mailing list
> >> cctbxbb at phenix-online.org
> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >
> > _______________________________________________
> > cctbxbb mailing list
> > cctbxbb at phenix-online.org
> > http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
>
> --
> Gergely Katona, PhD
> Associate Professor
> Department of Chemistry and Molecular Biology, University of Gothenburg
> Box 462, 40530 Göteborg, Sweden
> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> Web: http://katonalab.eu, Email: gergely.katona at gu.se
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20191218/89aa8391/attachment.htm>
More information about the cctbxbb
mailing list