[cctbxbb] some thoughts on cctbx and pip
Billy Poon
BKPoon at lbl.gov
Fri Jan 3 12:04:25 PST 2020
Hi Gergely,
Happy New Year!
Can you provide the output to "conda list" for each environment?
I suspect that the old version of cctbx might still be around. Is there a
file named PYTHON_VERSION_MAJOR_MINOR in "/home/gergely/anaconda3/lib"? If
so, delete that file. To make sure you are downloading a new copy of the
package, clear the cache where conda keeps copies of all the packages. To
do that, you can run "conda clean -y --all". You can remove the "-y" flag
if you want to see what will be deleted. The hash for the new linux package
should be h8b68381_0.
The last error looks like an incompatible version of boost or gcc library
was installed. The output from "conda list" should show the boost and
boost-cpp packages coming from conda-forge. I would avoid installing into
the root or base environment because that usually pulls from the default
channels. All the dependencies should be pulled from the conda-forge
channel.
--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709
Fax: (510) 486-5909
Web: https://phenix-online.org
On Fri, Jan 3, 2020 at 2:20 PM Gergely Katona <gkatona at gmail.com> wrote:
> I found some other errors when using the test environment.
>
>
> ---------------------------------------------------------------------------
> RuntimeError Traceback (most recent call last)
> <ipython-input-1-fe308b345b3f> in <module>
> 10 import scipy as sp
> 11 from scipy import linalg
> ---> 12 import iotbx.pdb
> 13 import cctbx
> 14 import pandas as pd
>
> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/__init__.py
> in <module>
> 8
> 9 import iotbx.pdb.records
> ---> 10 import iotbx.pdb.hierarchy
> 11 from scitbx import matrix
> 12
>
> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/hierarchy.py
> in <module>
> 14 from six.moves import cStringIO as StringIO
> 15 from iotbx.pdb import hy36encode, hy36decode
> ---> 16 import iotbx.cif.model
> 17 from cctbx import crystal
> 18 from libtbx import group_args
>
> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/cif/__init__.py
> in <module>
> 19
> 20 from cctbx.array_family import flex
> ---> 21 from cctbx import miller
> 22 from iotbx.cif import model, builders, geometry
> 23 from libtbx.containers import OrderedDict
>
> ~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/miller/__init__.py
> in <module>
> 11
> 12 from cctbx import crystal
> ---> 13 from cctbx import maptbx
> 14 from cctbx import sgtbx
> 15 from cctbx.sgtbx import lattice_symmetry
>
> ~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/maptbx/__init__.py
> in <module>
> 15 from libtbx import adopt_init_args
> 16 from libtbx.utils import Sorry
> ---> 17 import libtbx.load_env
> 18 import math
> 19 import sys, os
>
> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/load_env.py in
> <module>
> 3 import libtbx.env_config
> 4 import os
> ----> 5 libtbx.env = libtbx.env_config.unpickle()
> 6 libtbx.env.set_os_environ_all_dist()
> 7 libtbx.env.dispatcher_name =
> os.environ.get("LIBTBX_DISPATCHER_NAME")
>
> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/env_config.py
> in unpickle()
> 2736 env = pickle.load(libtbx_env)
> 2737 if (env.python_version_major_minor != sys.version_info[:2]):
> -> 2738 env.raise_python_version_incompatible()
> 2739 if (op.realpath(build_path) != op.realpath(abs(env.build_path))):
> 2740 env.build_path.reset(build_path)
>
> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/env_config.py
> in raise_python_version_incompatible(self, prev_pvmm)
> 469 if (prev_pvmm is None):
> 470 prev_pvmm = "%d.%d" % self.python_version_major_minor
> --> 471 raise RuntimeError("Python version incompatible with this
> build:\n"
> 472 + " Build directory: %s\n" %
> show_string(abs(self.build_path))
> 473 + " Python version used initially: %s\n" % prev_pvmm
>
> RuntimeError: Python version incompatible with this build:
> Build directory: "/home/gergely/anaconda3"
> Python version used initially: 3.6
> Python version in use now: 3.8
>
>
> Interestingly in the base environment all import worked, but I got an
> error at:
>
>
> ArgumentError Traceback (most recent call last)
> <ipython-input-4-5bc71c36e121> in <module>
> 33 atm_odd.set_b(0)
> 34
> ---> 35 odd.write_pdb_file("bdiff_in_even_000.pdb") #skriver ny pdb
> med dessa inkluderade
> 36 dfWBvals=pd.DataFrame(rows_list)
>
> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in
> write_pdb_file(self, file_name, open_append, crystal_symmetry,
> cryst1_z, write_scale_records, append_end, atom_hetatm, sigatm,
> anisou, siguij)
> 930 siguij=True):
> 931 if (crystal_symmetry is Auto):
> --> 932 crystal_symmetry = self.crystal_symmetry()
> 933 if (cryst1_z is Auto):
> 934 cryst1_z = self.extract_cryst1_z_columns()
>
> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in
> crystal_symmetry(self, crystal_symmetry, weak_symmetry)
> 1218 crystal_symmetry=None,
> 1219 weak_symmetry=False):
> -> 1220 self_symmetry = self.crystal_symmetry_from_cryst1()
> 1221 if (self_symmetry is None):
> 1222 self_symmetry = self._crystal_symmetry_from_cns_remark_sg()
>
> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in
> crystal_symmetry_from_cryst1(self)
> 1195 for line in self.crystallographic_section():
> 1196 if (line.startswith("CRYST1")):
> -> 1197 return
> cryst1_interpretation.crystal_symmetry(cryst1_record=line)
> 1198 return None
> 1199
>
> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/cryst1_interpretation.py
> in crystal_symmetry(cryst1_record)
> 136 space_group_info=None)
> 137 space_group_info =
> categorize(cryst1_record.sgroup).space_group_info(
> --> 138 unit_cell=u)
> 139 return crystal.symmetry(unit_cell=u,
> space_group_info=space_group_info)
>
> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/cryst1_interpretation.py
> in space_group_info(self, unit_cell)
> 85 if (self.symbol is None): return None
> 86 if (self.category is None):
> ---> 87 try: return sgtbx.space_group_info(self.symbol)
> 88 except RuntimeError: return None
> 89 if (isinstance(unit_cell, uctbx.ext.unit_cell)):
>
> ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in
> __init__(self, symbol, table_id, group, number, space_group_t_den)
> 100 assert group is None
> 101 if (table_id is None):
> --> 102 symbols = space_group_symbols(symbol)
> 103 else:
> 104 if (isinstance(symbol, int)): symbol = str(symbol)
>
> ArgumentError: Python argument types in
> space_group_symbols.__init__(space_group_symbols, str)
> did not match C++ signature:
> __init__(_object*, int space_group_number)
> __init__(_object*, int space_group_number,
> std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > extension='')
> __init__(_object*, int space_group_number,
> std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > extension='', std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > table_id='')
> __init__(_object*, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > symbol)
> __init__(_object*, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > symbol,
> std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > table_id='')
>
>
> Gergely
>
>
>
> On Fri, Jan 3, 2020 at 1:44 PM Gergely Katona <gkatona at gmail.com> wrote:
> >
> > Dear Billy,
> >
> > I did some tests installing in my base conda was not troublefree in
> > python 3.8. Running iotbx.fetch_pdb --all --mtz 1kp8 resulted in boost
> > related error when converting to mtz. Installing into a test
> > environment went fine with iotbx.fetch_pdb --all --mtz 1kp8 running
> > successfully. I could add pymc3, jupyter and seaborn and iotbx still
> > did not break. My LIBTBX_BUILD is not set however in the test
> > environment, which breaks miller when I was trying to import it.
> > Manually setting the LIBTBX_BUILD to CONDA_PREFIX (setenv LIBTBX_BUILD
> > $CONDA_PREFIX) fixes this and one of my script works fine! I will come
> > back if I find some problems with my other scripts, but so far this
> > looks promising!
> >
> >
> > Happy new year!
> >
> > Gergely
> >
> >
> >
> >
> >
> > On Fri, Dec 20, 2019 at 8:16 PM Billy Poon <BKPoon at lbl.gov> wrote:
> > >
> > > Hi everyone,
> > >
> > > I updated the conda packages on the cctbx-dev channel and there are
> also now packages for macOS. The old packages have been removed so you can
> create a new environment with
> > >
> > > conda create -n test -c cctbx-dev cctbx python=2.7
> > > conda activate test
> > >
> > > where "test" is the environment name and 2.7 is the python version.
> For the python version, you can also pick 3.6, 3.7, or 3.8. I think the
> latest conda will install 3.8 as the default python if the python argument
> is not provided. Or install into an existing environment (with python
> already installed) with the same command as before,
> > >
> > > conda install -c cctbx-dev cctbx
> > >
> > > Again, your .condarc file should have conda-forge as the first channel
> so that dependencies can be pulled correctly.
> > >
> > > The dispatchers should work now and $LIBTBX_BUILD will be set on
> activation of an environment (and unset on environment deactivation) or
> after installing into an existing environment, so you can do more
> interesting things like
> > >
> > > iotbx.fetch_pdb --all --mtz 1kp8
> > >
> > > However, this package does not install databases, so
> > >
> > > molprobity.molprobity 1kp8.pdb 1kp8.mtz
> > >
> > > will fail because the database is not available (also probe and reduce
> are not built).
> > >
> > > Some known issues in no particular order,
> > >
> > > 1) You can run "libtbx.run_tests_parallel module=cctbx nproc=Auto" to
> run tests, but not all the test files were copied, so I got 30 failures
> here. The final package will probably not have the test files.
> > > 2) The other python dispatchers, like "cctbx.python", were not
> updated, so those will fail. But you can just run "python" in the active
> environment.
> > > 3) Parts of mmtbx still need to be updated for Python 3.
> > > 4) The LIBTBX_BUILD variable should be set for sh and csh shells. Let
> me know if it's not working for your shell (i.e. "echo $LIBTBX_BUILD"
> should show the same thing as $CONDA_PREFIX).
> > > 5) One future step is to enable building other CCTBX-like modules with
> this conda package. This will require some bookkeeping updates so that
> modules in the conda environment and the modules being built are tracked
> properly. Also, making sure that all the necessary headers exist in
> $PREFIX/include.
> > >
> > > Gergely, let me know if your scripts are working.
> > >
> > > Happy holidays!
> > >
> > > --
> > > Billy K. Poon
> > > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> > > Lawrence Berkeley National Laboratory
> > > 1 Cyclotron Road, M/S 33R0345
> > > Berkeley, CA 94720
> > > Tel: (510) 486-5709
> > > Fax: (510) 486-5909
> > > Web: https://phenix-online.org
> > >
> > >
> > > On Wed, Dec 18, 2019 at 11:25 AM Billy Poon <BKPoon at lbl.gov> wrote:
> > >>
> > >> Hi Gergely,
> > >>
> > >> That's the other information that hasn't been updated yet in the
> conda package. :)
> > >>
> > >> For LIBTBX_BUILD, I can set up environment variables sort of how the
> conda gcc package sets up environment variables. I have to double check to
> see if they're set after installation of the package and after activation
> of an environment. The variable will just be a copy of $CONDA_PREFIX.
> > >>
> > >> The libtbx_env file is a file that exists in $LIBTBX_BUILD that
> basically stores configuration information. I'll have to modify the
> contents to update the locations of the different modules since the conda
> package will not have a "modules" directory.
> > >>
> > >> --
> > >> Billy K. Poon
> > >> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> > >> Lawrence Berkeley National Laboratory
> > >> 1 Cyclotron Road, M/S 33R0345
> > >> Berkeley, CA 94720
> > >> Tel: (510) 486-5709
> > >> Fax: (510) 486-5909
> > >> Web: https://phenix-online.org
> > >>
> > >>
> > >> On Wed, Dec 18, 2019 at 2:08 AM Gergely Katona <gkatona at gmail.com>
> wrote:
> > >>>
> > >>> Hi Billy,
> > >>>
> > >>> Conda install went fine with your instructions after rearranging the
> > >>> channels by putting cctbx last. I removed all environmental variables
> > >>> and previous build of cctbx.
> > >>>
> > >>> import sys
> > >>> print (sys.path, sys.prefix)
> > >>>
> > >>> ['/home/gergely/anaconda3/lib/python36.zip',
> > >>> '/home/gergely/anaconda3/lib/python3.6',
> > >>> '/home/gergely/anaconda3/lib/python3.6/lib-dynload', '',
> > >>> '/home/gergely/anaconda3/lib/python3.6/site-packages',
> > >>>
> '/home/gergely/anaconda3/lib/python3.6/site-packages/IPython/extensions',
> > >>> '/home/gergely/.ipython'] /home/gergely/anaconda3
> > >>>
> > >>> Many imports went fine including pymc3, but I encountered problems
> > >>> with these three:
> > >>>
> > >>> from cctbx import miller
> > >>> import iotbx.pdb
> > >>> from iotbx import reflection_file_reader, mtz
> > >>>
> > >>>
> > >>> I also got type error when handling space groups.
> > >>>
> > >>>
> > >>>
> > >>>
> ---------------------------------------------------------------------------
> > >>> KeyError Traceback (most recent
> call last)
> > >>> <ipython-input-5-cedbd4f84f3d> in <module>
> > >>> 25 from cctbx import uctbx
> > >>> 26 from cctbx import sgtbx
> > >>> ---> 27 from cctbx import miller
> > >>> 28 #from iotbx import reflection_file_reader, mtz
> > >>> 29 sns.set_context("poster")
> > >>>
> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/miller/__init__.py in
> <module>
> > >>> 11
> > >>> 12 from cctbx import crystal
> > >>> ---> 13 from cctbx import maptbx
> > >>> 14 from cctbx import sgtbx
> > >>> 15 from cctbx.sgtbx import lattice_symmetry
> > >>>
> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/maptbx/__init__.py in
> <module>
> > >>> 15 from libtbx import adopt_init_args
> > >>> 16 from libtbx.utils import Sorry
> > >>> ---> 17 import libtbx.load_env
> > >>> 18 import math
> > >>> 19 import sys, os
> > >>>
> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in
> <module>
> > >>> 3 import libtbx.env_config
> > >>> 4 import os
> > >>> ----> 5 libtbx.env = libtbx.env_config.unpickle()
> > >>> 6 libtbx.env.set_os_environ_all_dist()
> > >>> 7 libtbx.env.dispatcher_name =
> os.environ.get("LIBTBX_DISPATCHER_NAME")
> > >>>
> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in
> unpickle()
> > >>> 2603
> > >>> 2604 def unpickle():
> > >>> -> 2605 build_path = os.environ["LIBTBX_BUILD"]
> > >>> 2606
> set_preferred_sys_prefix_and_sys_executable(build_path=build_path)
> > >>> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")
> > >>>
> > >>> ~/anaconda3/lib/python3.6/os.py in __getitem__(self, key)
> > >>> 667 except KeyError:
> > >>> 668 # raise KeyError with the original key value
> > >>> --> 669 raise KeyError(key) from None
> > >>> 670 return self.decodevalue(value)
> > >>> 671
> > >>>
> > >>> KeyError: 'LIBTBX_BUILD'
> > >>>
> > >>>
> > >>>
> > >>>
> ---------------------------------------------------------------------------
> > >>> ArgumentError Traceback (most recent
> call last)
> > >>> <ipython-input-7-2acd3a9ce26a> in <module>
> > >>> 26 return
> ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar
> > >>> 27
> > >>> ---> 28
> ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar=initializecrystal()
> > >>>
> > >>> <ipython-input-7-2acd3a9ce26a> in initializecrystal()
> > >>> 11 uc = uctbx.unit_cell(unit_cell)
> > >>> 12 wavelength = 1.54980
> > >>> ---> 13 xtal_symm = crystal.symmetry(unit_cell=unit_cell,
> > >>> space_group_symbol="P 43 21 2")
> > >>> 14
> > >>> 15 ms =
> > >>> miller.build_set(crystal_symmetry=xtal_symm,anomalous_flag=True,
> > >>> d_min=1.61)
> > >>>
> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/crystal/__init__.py in
> > >>> __init__(self, unit_cell, space_group_symbol, space_group_info,
> > >>> space_group, correct_rhombohedral_setting_if_necessary,
> > >>> assert_is_compatible_unit_cell,
> raise_sorry_if_incompatible_unit_cell,
> > >>> force_compatible_unit_cell)
> > >>> 74 if (space_group_symbol is not None):
> > >>> 75 self._space_group_info = sgtbx.space_group_info(
> > >>> ---> 76 symbol=space_group_symbol)
> > >>> 77 elif (space_group is not None):
> > >>> 78 if (isinstance(space_group, sgtbx.space_group)):
> > >>>
> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in
> > >>> __init__(self, symbol, table_id, group, number, space_group_t_den)
> > >>> 100 assert group is None
> > >>> 101 if (table_id is None):
> > >>> --> 102 symbols = space_group_symbols(symbol)
> > >>> 103 else:
> > >>> 104 if (isinstance(symbol, int)): symbol = str(symbol)
> > >>>
> > >>> ArgumentError: Python argument types in
> > >>> space_group_symbols.__init__(space_group_symbols, str)
> > >>> did not match C++ signature:
> > >>> __init__(_object*, int space_group_number)
> > >>> __init__(_object*, int space_group_number,
> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
> > >>> std::allocator<char> > extension='')
> > >>> __init__(_object*, int space_group_number,
> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
> > >>> std::allocator<char> > extension='', std::__cxx11::basic_string<char,
> > >>> std::char_traits<char>, std::allocator<char> > table_id='')
> > >>> __init__(_object*, std::__cxx11::basic_string<char,
> > >>> std::char_traits<char>, std::allocator<char> > symbol)
> > >>> __init__(_object*, std::__cxx11::basic_string<char,
> > >>> std::char_traits<char>, std::allocator<char> > symbol,
> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
> > >>> std::allocator<char> > table_id='')
> > >>>
> > >>>
> > >>> When I tried to set LIBTBX_BUILD to /home/gergely/anaconda3 (this is
> > >>> sys.prefix on my system), I got other problems when importing. Can
> > >>> LIBTBX_BUILD be set in the conda package?
> > >>>
> > >>>
> > >>>
> > >>>
> ---------------------------------------------------------------------------
> > >>> FileNotFoundError Traceback (most recent
> call last)
> > >>> <ipython-input-1-7e2b144826cc> in <module>
> > >>> 26 from cctbx import sgtbx
> > >>> 27 #from cctbx import miller
> > >>> ---> 28 from iotbx import reflection_file_reader, mtz
> > >>> 29 sns.set_context("poster")
> > >>> 30 plt.rcParams.update({'figure.autolayout': True})
> > >>>
> > >>>
> ~/anaconda3/lib/python3.6/site-packages/iotbx/reflection_file_reader.py
> > >>> in <module>
> > >>> 53
> > >>> 54 from __future__ import absolute_import, division,
> print_function
> > >>> ---> 55 from iotbx import mtz
> > >>> 56 from iotbx.scalepack import merge as scalepack_merge
> > >>> 57 from iotbx.scalepack import no_merge_original_index as
> > >>> scalepack_no_merge
> > >>>
> > >>> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/__init__.py in
> <module>
> > >>> 9 import iotbx_mtz_ext as ext
> > >>> 10
> > >>> ---> 11 from iotbx.mtz import extract_from_symmetry_lib
> > >>> 12 from cctbx import xray
> > >>> 13 import cctbx.xray.observation_types
> > >>>
> > >>>
> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/extract_from_symmetry_lib.py
> > >>> in <module>
> > >>> 1 from __future__ import absolute_import, division,
> print_function
> > >>> 2 from cctbx import sgtbx
> > >>> ----> 3 import libtbx.load_env
> > >>> 4 import os.path as op
> > >>> 5 from six.moves import range
> > >>>
> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in
> <module>
> > >>> 3 import libtbx.env_config
> > >>> 4 import os
> > >>> ----> 5 libtbx.env = libtbx.env_config.unpickle()
> > >>> 6 libtbx.env.set_os_environ_all_dist()
> > >>> 7 libtbx.env.dispatcher_name =
> os.environ.get("LIBTBX_DISPATCHER_NAME")
> > >>>
> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in
> unpickle()
> > >>> 2605 build_path = os.environ["LIBTBX_BUILD"]
> > >>> 2606
> set_preferred_sys_prefix_and_sys_executable(build_path=build_path)
> > >>> -> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")
> > >>> 2608 env = pickle.load(libtbx_env)
> > >>> 2609 if (env.python_version_major_minor !=
> sys.version_info[:2]):
> > >>>
> > >>> FileNotFoundError: [Errno 2] No such file or directory:
> > >>> '/home/gergely/anaconda3/libtbx_env'
> > >>>
> > >>> On Wed, Dec 18, 2019 at 9:10 AM Billy Poon <BKPoon at lbl.gov> wrote:
> > >>> >
> > >>> > Hi Gergely,
> > >>> >
> > >>> > I've uploaded linux packages to a new channel, cctbx-dev, and you
> can install it with
> > >>> >
> > >>> > conda install -c cctbx-dev cctbx
> > >>> >
> > >>> > in your current environment. You should set your ~/.condarc file
> to pull the other dependencies from the conda-forge channel first, so put
> conda-forge above cctbx. Mine looks like
> > >>> >
> > >>> > channels:
> > >>> > - conda-forge
> > >>> > - defaults
> > >>> > - cctbx
> > >>> >
> > >>> > Do you need dxtbx for your scripts? This package does not build
> that part. I think the plan is to build a separate conda package for dxtbx
> so that it can be updated more frequently. I can rebuild the packages to
> include it for testing, but the one being submitted to conda-forge will not
> have it.
> > >>> >
> > >>> > Also, your error message is probably due to version of HDF5 that
> the development build installs. The bootstrap.py script will install
> 1.10.4, but your other dependency is looking for 1.10.5. Installing this
> cctbx conda package should install 1.10.5, which should fix the issue.
> Also, I'm updating those environments and 1.10.5 will be the new default
> version.
> > >>> >
> > >>> > Lastly, the dispatchers will not work in these packages because
> the old paths during the build process are still in them and many of them
> expect some additional information that has not been updated in the
> packages yet. I'm in the process of doing that and will update the
> cctbx-dev channel when that's done. However, by starting python, you can
> import cctbx modules. So you can run commands like
> > >>> >
> > >>> > from scitbx.array_family import flex
> > >>> > a = flex.random_double(1000000)
> > >>> > b = flex.min_max_mean_double(a)
> > >>> > b.min
> > >>> > b.max
> > >>> > b.mean
> > >>> >
> > >>> > Thanks!
> > >>> >
> > >>> > --
> > >>> > Billy K. Poon
> > >>> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> > >>> > Lawrence Berkeley National Laboratory
> > >>> > 1 Cyclotron Road, M/S 33R0345
> > >>> > Berkeley, CA 94720
> > >>> > Tel: (510) 486-5709
> > >>> > Fax: (510) 486-5909
> > >>> > Web: https://phenix-online.org
> > >>> >
> > >>> >
> > >>> > On Tue, Dec 17, 2019 at 5:39 AM Gergely Katona <gkatona at gmail.com>
> wrote:
> > >>> >>
> > >>> >> Dear Billy,
> > >>> >>
> > >>> >> Thank you for the detailed explanation, I look forward to do
> further
> > >>> >> testing! I aim to pool all modules under the same environment, if
> > >>> >> there are conflicts then I just try to reshuffle the order of
> imports.
> > >>> >> So far this did not cause problems for me even when I was using
> system
> > >>> >> python, but of course it is not the most prudent thing to do. With
> > >>> >> anaconda everything is much more standardized and isolated
> already and
> > >>> >> without being superuser I can have my familiar environment at any
> > >>> >> synchrotron based cluster. It is great that cctbx will be an
> integral
> > >>> >> part of this ecosystem and this was also the last thing holding me
> > >>> >> back from adopting python3. About hdf5, pymc3 requires it and
> > >>> >> importing causes a kernel restart with the following error
> messages,
> > >>> >> curiously if I import h5py first this can be avoided.
> > >>> >>
> > >>> >> Best wishes,
> > >>> >>
> > >>> >> Gergely
> > >>> >>
> > >>> >> import pymc3 as pm
> > >>> >>
> > >>> >> Warning! ***HDF5 library version mismatched error***
> > >>> >> The HDF5 header files used to compile this application do not
> match
> > >>> >> the version used by the HDF5 library to which this application is
> linked.
> > >>> >> Data corruption or segmentation faults may occur if the
> application continues.
> > >>> >> This can happen when an application was compiled by one version
> of HDF5 but
> > >>> >> linked with a different version of static or shared HDF5 library.
> > >>> >> You should recompile the application or check your shared library
> related
> > >>> >> settings such as 'LD_LIBRARY_PATH'.
> > >>> >> You can, at your own risk, disable this warning by setting the
> environment
> > >>> >> variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.
> > >>> >> Setting it to 2 or higher will suppress the warning messages
> totally.
> > >>> >> Headers are 1.10.4, library is 1.10.5
> > >>> >> SUMMARY OF THE HDF5 CONFIGURATION
> > >>> >> =================================
> > >>> >>
> > >>> >> General Information:
> > >>> >> -------------------
> > >>> >> HDF5 Version: 1.10.5
> > >>> >> Configured on: Tue Oct 22 12:02:13 UTC 2019
> > >>> >> Configured by: conda at 16247e67ecd5
> > >>> >> Host system: x86_64-conda_cos6-linux-gnu
> > >>> >> Uname information: Linux 16247e67ecd5
> 4.15.0-1059-azure
> > >>> >> #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64
> > >>> >> GNU/Linux
> > >>> >> Byte sex: little-endian
> > >>> >> Installation point: /home/gergely/anaconda3
> > >>> >>
> > >>> >> Compiling Options:
> > >>> >> ------------------
> > >>> >> Build Mode: production
> > >>> >> Debugging Symbols: no
> > >>> >> Asserts: no
> > >>> >> Profiling: no
> > >>> >> Optimization Level: high
> > >>> >>
> > >>> >> Linking Options:
> > >>> >> ----------------
> > >>> >> Libraries: static, shared
> > >>> >> Statically Linked Executables:
> > >>> >> LDFLAGS: -Wl,-O2 -Wl,--sort-common
> > >>> >> -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags
> > >>> >> -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib
> > >>> >> -Wl,-rpath-link,/home/gergely/anaconda3/lib
> > >>> >> -L/home/gergely/anaconda3/lib
> > >>> >> H5_LDFLAGS:
> > >>> >> AM_LDFLAGS: -L/home/gergely/anaconda3/lib
> > >>> >> Extra libraries: -lrt -lpthread -lz -ldl -lm
> > >>> >> Archiver:
> > >>> >>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar
> > >>> >> AR_FLAGS: cr
> > >>> >> Ranlib:
> > >>> >>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib
> > >>> >>
> > >>> >> Languages:
> > >>> >> ----------
> > >>> >> C: yes
> > >>> >> C Compiler:
> > >>> >>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc
> > >>> >> CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2
> > >>> >> -I/home/gergely/anaconda3/include
> > >>> >> H5_CPPFLAGS: -D_GNU_SOURCE
> > >>> >> -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API
> > >>> >> AM_CPPFLAGS:
> -I/home/gergely/anaconda3/include
> > >>> >> C Flags: -march=nocona -mtune=haswell
> > >>> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
> > >>> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include
> > >>> >>
> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5
> > >>> >>
> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix
> > >>> >> H5 C Flags: -std=c99 -pedantic -Wall
> -Wextra
> > >>> >> -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual
> -Wconversion
> > >>> >> -Wdeclaration-after-statement -Wdisabled-optimization
> -Wfloat-equal
> > >>> >> -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations
> > >>> >> -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs
> > >>> >> -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls
> > >>> >> -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum
> -Wundef
> > >>> >> -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings
> > >>> >> -finline-functions -s -Wno-inline -Wno-aggregate-return
> > >>> >> -Wno-missing-format-attribute -Wno-missing-noreturn -O
> > >>> >> AM C Flags:
> > >>> >> Shared C Library: yes
> > >>> >> Static C Library: yes
> > >>> >>
> > >>> >>
> > >>> >> Fortran: yes
> > >>> >> Fortran Compiler:
> > >>> >>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran
> > >>> >> Fortran Flags:
> > >>> >> H5 Fortran Flags: -pedantic -Wall -Wextra
> -Wunderflow
> > >>> >> -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2
> > >>> >> AM Fortran Flags:
> > >>> >> Shared Fortran Library: yes
> > >>> >> Static Fortran Library: yes
> > >>> >>
> > >>> >> C++: yes
> > >>> >> C++ Compiler:
> > >>> >>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++
> > >>> >> C++ Flags: -fvisibility-inlines-hidden
> > >>> >> -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell
> > >>> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
> > >>> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include
> > >>> >>
> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5
> > >>> >>
> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix
> > >>> >> H5 C++ Flags: -pedantic -Wall -W -Wundef
> -Wshadow
> > >>> >> -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings
> -Wconversion
> > >>> >> -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual
> > >>> >> -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor
> > >>> >> -Wctor-dtor-privacy -Wabi -finline-functions -s -O
> > >>> >> AM C++ Flags:
> > >>> >> Shared C++ Library: yes
> > >>> >> Static C++ Library: yes
> > >>> >>
> > >>> >> Java: no
> > >>> >>
> > >>> >>
> > >>> >> Features:
> > >>> >> ---------
> > >>> >> Parallel HDF5: no
> > >>> >> Parallel Filtered Dataset Writes: no
> > >>> >> Large Parallel I/O: no
> > >>> >> High-level library: yes
> > >>> >> Threadsafety: yes
> > >>> >> Default API mapping: v110
> > >>> >> With deprecated public symbols: yes
> > >>> >> I/O filters (external): deflate(zlib)
> > >>> >> MPE: no
> > >>> >> Direct VFD: no
> > >>> >> dmalloc: no
> > >>> >> Packages w/ extra debug output: none
> > >>> >> API tracing: no
> > >>> >> Using memory checker: yes
> > >>> >> Memory allocation sanity checks: no
> > >>> >> Function stack tracing: no
> > >>> >> Strict file format checks: no
> > >>> >> Optimization instrumentation: no
> > >>> >>
> > >>> >> On Tue, Dec 17, 2019 at 8:35 AM Billy Poon <BKPoon at lbl.gov>
> wrote:
> > >>> >> >
> > >>> >> > Hi Gergely,
> > >>> >> >
> > >>> >> > Let me build the test package tomorrow. All the gritty details
> for building with conda is being finalized and the official documentation
> will be updated to describe the steps. It would be too confusing to keep
> changing the documentation as the process evolves. You do have the general
> process, though, which is summarized as follows.
> > >>> >> >
> > >>> >> > 1) Installing dependencies. The cctbx_dependencies metapackage
> was an initial approach for managing the CCTBX dependencies, but after
> contacting the conda-forge folks, they recommended using the --only-deps
> flag. So when the CCTBX conda package is available, you'll be able to get a
> set of dependencies with,
> > >>> >> >
> > >>> >> > conda install -c conda-forge --only-deps cctbx
> > >>> >> >
> > >>> >> > By default, the bootstrap.py file will automatically install a
> set of dependencies in the "conda_base" directory (and a conda installation
> if one is not found). It just uses standard conda environment files located
> in libtbx/auto_build/conda_envs, so you do not need to install
> cctbx_dependencies as a separate step. The environment files avoid channel
> issues by explicitly defining the channel to pull the packages from and the
> cctbx channel just stores copies of packages from conda-forge. There were
> issues earlier where the conda-forge packages would sometimes be moved to a
> different label. The --use-conda flag also accepts a path to $CONDA_PREFIX
> if you want to use a specific environment for testing.
> > >>> >> >
> > >>> >> > 2) Building. The bootstrap.py file handles that with SCons.
> > >>> >> >
> > >>> >> > 3) Running. After building, there should be a setpaths.sh (and
> .csh) file that adds build/bin to your path. The build/bin directory has
> our dispatchers, which are just scripts that set up the environment
> variables for you. This prevents other programs from loading our libraries,
> whose versions may conflict. You should see that there is a "python"
> dispatcher, which is a convenience for developers. Otherwise, you can use
> libtbx.python, which will be able to import CCTBX modules.
> > >>> >> >
> > >>> >> > The running part is where the conda package for CCTBX will be
> different than this build. Since our Python files and extensions modules
> will be in the "site-packages" directory for Python, the PYTHONPATH
> variable will not be needed (and conda suggests that that variable not be
> set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so
> LD_LIBRARY_PATH is not needed. In an active environment, PATH will already
> be modified. And then our LIBTBX_BUILD directory can be set to sys.prefix.
> > >>> >> >
> > >>> >> > So with the conda package, you would only need to activate your
> conda environment and CCTBX should integrate with other conda packages.
> What is the conflict with hdf5? That's something that should be fixed.
> Thanks!
> > >>> >> >
> > >>> >> > --
> > >>> >> > Billy K. Poon
> > >>> >> > Research Scientist, Molecular Biophysics and Integrated
> Bioimaging
> > >>> >> > Lawrence Berkeley National Laboratory
> > >>> >> > 1 Cyclotron Road, M/S 33R0345
> > >>> >> > Berkeley, CA 94720
> > >>> >> > Tel: (510) 486-5709
> > >>> >> > Fax: (510) 486-5909
> > >>> >> > Web: https://phenix-online.org
> > >>> >> >
> > >>> >> >
> > >>> >> > On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona <
> gkatona at gmail.com> wrote:
> > >>> >> >>
> > >>> >> >> Dear Billy,
> > >>> >> >>
> > >>> >> >> Thank you for this update and for your efforts! I found a
> solution and
> > >>> >> >> indeed most things already work in anaconda3. The steps I took
> (even
> > >>> >> >> if these instructions will have short expiry date):
> > >>> >> >>
> > >>> >> >> Modified .condarc with:
> > >>> >> >> channels:
> > >>> >> >> - cctbx
> > >>> >> >> - conda-forge
> > >>> >> >> - defaults
> > >>> >> >>
> > >>> >> >> run
> > >>> >> >> conda conda install cctbx_dependencies python=3.6
> > >>> >> >>
> > >>> >> >> I expect this will work without python=3.6 in the near future.
> > >>> >> >>
> > >>> >> >> Then compiling cctbx with anaconda3 python went without any
> problem
> > >>> >> >> when using these flags:
> > >>> >> >> python bootstrap.py --use-conda --python3 --nproc=12
> > >>> >> >>
> > >>> >> >> Then I probably did the unorthodox thing and sourced these
> directories:
> > >>> >> >>
> > >>> >> >> setenv LIBTBX_BUILD /home/gergely/cctbx/build
> > >>> >> >> setenv PATH ${PATH}:/home/gergely/cctbx/build/bin
> > >>> >> >> setenv PYTHONPATH
> > >>> >> >>
> /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH
> > >>> >> >> setenv LD_LIBRARY_PATH
> > >>> >> >>
> /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH
> > >>> >> >>
> > >>> >> >> There is probably a better way to put this into conda
> environment.
> > >>> >> >> With these steps I could run one of my scripts depending on
> cctbx. The
> > >>> >> >> only problem I found is that hdf5 library had conflict with
> another
> > >>> >> >> package in conda, but I does not find this as a showstopper.
> > >>> >> >>
> > >>> >> >> Best wishes,
> > >>> >> >>
> > >>> >> >> Gergely
> > >>> >> >>
> > >>> >> >>
> > >>> >> >>
> > >>> >> >>
> > >>> >> >>
> > >>> >> >>
> > >>> >> >> On Fri, Dec 13, 2019 at 7:23 PM Billy Poon <BKPoon at lbl.gov>
> wrote:
> > >>> >> >> >
> > >>> >> >> > Hi Gergely,
> > >>> >> >> >
> > >>> >> >> > It's still a work in progress. I'm sorting out some Windows
> issues right now. I can probably build a test package on a separate channel
> for people that want to test it (let's say next week?). I'll provide
> instructions, but basically, the test conda package will be in its own
> separate channel and the dependencies will be pulled from the conda-forge
> channel. I want most things to be working correctly on Python 2.7, 3.6,
> 3.7, and 3.8 on all 3 platforms.
> > >>> >> >> >
> > >>> >> >> > Thanks!
> > >>> >> >> >
> > >>> >> >> > --
> > >>> >> >> > Billy K. Poon
> > >>> >> >> > Research Scientist, Molecular Biophysics and Integrated
> Bioimaging
> > >>> >> >> > Lawrence Berkeley National Laboratory
> > >>> >> >> > 1 Cyclotron Road, M/S 33R0345
> > >>> >> >> > Berkeley, CA 94720
> > >>> >> >> > Tel: (510) 486-5709
> > >>> >> >> > Fax: (510) 486-5909
> > >>> >> >> > Web: https://phenix-online.org
> > >>> >> >> >
> > >>> >> >> >
> > >>> >> >> > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona <
> gkatona at gmail.com> wrote:
> > >>> >> >> >>
> > >>> >> >> >> Dear Billy,
> > >>> >> >> >>
> > >>> >> >> >> This sounds very promising and exciting. I am not sure if
> cctbx is
> > >>> >> >> >> already functional as a conda package in anaconda3 (Linux)
> or this is
> > >>> >> >> >> still work in progress. My technical expertise does not
> allow me to
> > >>> >> >> >> tell the difference. What I tried:
> > >>> >> >> >>
> > >>> >> >> >> Fresh install of anaconda3. Adding - cctbx and -
> conda-forge to
> > >>> >> >> >> .condarc . Installing pyside2 with conda. Running conda
> install
> > >>> >> >> >> conda_dependencies . I get a lot package version conflict,
> and I
> > >>> >> >> >> cannot import cctbx or iotbx to anaconda python. Am I
> following the
> > >>> >> >> >> right instructions? Or it is too early to expect that cctbx
> works when
> > >>> >> >> >> installed through conda?
> > >>> >> >> >>
> > >>> >> >> >> Best wishes,
> > >>> >> >> >>
> > >>> >> >> >> Gergely
> > >>> >> >> >>
> > >>> >> >> >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon <BKPoon at lbl.gov>
> wrote:
> > >>> >> >> >> >
> > >>> >> >> >> > Hi all,
> > >>> >> >> >> >
> > >>> >> >> >> > For a brief update, I have submitted a recipe for
> cctbxlite to conda-forge (
> https://github.com/conda-forge/staged-recipes/pull/10021) and support for
> Python 3.7 and 3.8 is being added (
> https://github.com/cctbx/cctbx_project/pull/409). With some fixes for
> Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms
> (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8.
> Some additional changes will be needed to get Windows to work with Python 3
> and for tests to pass with Boost 1.70.0. That will enable the conda-forge
> recipe to build for all platforms and for all supported versions of Python.
> > >>> >> >> >> >
> > >>> >> >> >> > Currently, the conda-forge recipe will install into the
> conda python and cctbx imports can be done without sourcing the environment
> scripts. It looks like a lot of the environment variables being set in the
> dispatchers can be removed since the Python files and C++ extensions are in
> the right places. I'll update the libtbx_env file so that commands that
> load the environment can work correctly.
> > >>> >> >> >> >
> > >>> >> >> >> > --
> > >>> >> >> >> > Billy K. Poon
> > >>> >> >> >> > Research Scientist, Molecular Biophysics and Integrated
> Bioimaging
> > >>> >> >> >> > Lawrence Berkeley National Laboratory
> > >>> >> >> >> > 1 Cyclotron Road, M/S 33R0345
> > >>> >> >> >> > Berkeley, CA 94720
> > >>> >> >> >> > Tel: (510) 486-5709
> > >>> >> >> >> > Fax: (510) 486-5909
> > >>> >> >> >> > Web: https://phenix-online.org
> > >>> >> >> >> >
> > >>> >> >> >> >
> > >>> >> >> >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll <
> tic20 at cam.ac.uk> wrote:
> > >>> >> >> >> >>
> > >>> >> >> >> >> Hi Luc,
> > >>> >> >> >> >>
> > >>> >> >> >> >> That sounds promising. From there, I’d need to work out
> how to make a fully-packaged installer (basically a modified wheel file)
> for the ChimeraX ToolShed - the aim is for the end user to not have to
> worry about any of this. That adds a couple of complications - e.g.
> $LIBTBX_BUILD would need to be set dynamically before first import - but
> doesn’t seem insurmountable.
> > >>> >> >> >> >>
> > >>> >> >> >> >> Thanks,
> > >>> >> >> >> >>
> > >>> >> >> >> >> Tristan
> > >>> >> >> >> >>
> > >>> >> >> >> >>
> > >>> >> >> >> >>
> > >>> >> >> >> >> Tristan Croll
> > >>> >> >> >> >> Research Fellow
> > >>> >> >> >> >> Cambridge Institute for Medical Research
> > >>> >> >> >> >> University of Cambridge CB2 0XY
> > >>> >> >> >> >>
> > >>> >> >> >> >>
> > >>> >> >> >> >>
> > >>> >> >> >> >>
> > >>> >> >> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis <
> luc_j_bourhis at mac.com> wrote:
> > >>> >> >> >> >> >
> > >>> >> >> >> >> > Hi Tristan,
> > >>> >> >> >> >> >
> > >>> >> >> >> >> > cctbx could be built to use your ChimeraX python, now
> that cctbx is moving to Python 3. The option —with-python is there for that
> with the bootstrap script. The specific environment setup boil down to
> setting two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on
> Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a
> framework such as ChimeraX, that should not be difficult to ensure those
> two variables are set.
> > >>> >> >> >> >> >
> > >>> >> >> >> >> > Best wishes,
> > >>> >> >> >> >> >
> > >>> >> >> >> >> > Luc
> > >>> >> >> >> >> >
> > >>> >> >> >> >> >
> > >>> >> >> >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll <
> tic20 at cam.ac.uk> wrote:
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> To add my two cents on this: probably the second-most
> common question I've had about ISOLDE's implementation is, "why didn't you
> use CCTBX?". The honest answer to that is, "I didn't know how."
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> Still don't, really - although the current
> developments are rather promising. The problem I've faced is that CCTBX was
> designed as its own self-contained Python (2.7, until very recently)
> environment, with its own interpreter and a lot of very specific
> environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is
> *also* its own self-contained Python (3.7) environment. To plug one into
> the other in that form... well, I don't think I'm a good enough programmer
> to really know where to start.
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> The move to Conda and a more modular CCTBX
> architecture should make a lot more possible in that direction. Pip would
> be even better for me personally (ChimeraX can install directly from the
> PyPI, but doesn't interact with Conda) - but I understand pretty well the
> substantial challenge that would amount to (not least being that the PyPI
> imposes a limit - around 100MB from memory? - on the size of an individual
> package).
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> Best regards,
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> Tristan
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote:
> > >>> >> >> >> >> >>> Hi Graeme,
> > >>> >> >> >> >> >>> Yes, I know. But “black" is a program doing a very
> particular task
> > >>> >> >> >> >> >>> (code formatting from the top of my head). Requiring
> to use a wrapper
> > >>> >> >> >> >> >>> for python itself is another level. But ok, I think
> I am mellowing to
> > >>> >> >> >> >> >>> the idea after all! Talking with people around me,
> and extrapolating,
> > >>> >> >> >> >> >>> I would bet that, right now, a great majority of
> people interested by
> > >>> >> >> >> >> >>> cctbx in pip have already used the cctbx, so they
> know about the
> > >>> >> >> >> >> >>> Python wrapper, and they would not be too sanguine
> about that. My
> > >>> >> >> >> >> >>> concern is for the future, when pip will be the
> first time some people
> > >>> >> >> >> >> >>> use cctbx. Big fat warning notices on PyPI page and
> a better error
> > >>> >> >> >> >> >>> message when cctbx fails because LIBTBX_BUILD is not
> set would be
> > >>> >> >> >> >> >>> needed but that could be all right.
> > >>> >> >> >> >> >>> If we do a pip installer, we should aim at a minimal
> install: cctbx,
> > >>> >> >> >> >> >>> iotbx and their dependencies, and that’s it.
> > >>> >> >> >> >> >>> Best wishes,
> > >>> >> >> >> >> >>> Luc
> > >>> >> >> >> >> >>>> On 23 Aug 2019, at 07:17,
> Graeme.Winter at Diamond.ac.uk <Graeme.Winter at diamond.ac.uk> wrote:
> > >>> >> >> >> >> >>>> Without discussing the merits of this or whether we
> _choose_ to make the move to supporting PIP, I am certain it would be
> _possible_ - many other packages make dispatcher scripts when you pip
> install them e.g.
> > >>> >> >> >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat
> $(which black)
> > >>> >> >> >> >> >>>>
> /Library/Frameworks/Python.framework/Versions/3.6/bin/black
> > >>> >> >> >> >> >>>>
> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
> > >>> >> >> >> >> >>>> # -*- coding: utf-8 -*-
> > >>> >> >> >> >> >>>> import re
> > >>> >> >> >> >> >>>> import sys
> > >>> >> >> >> >> >>>> from black import main
> > >>> >> >> >> >> >>>> if __name__ == '__main__':
> > >>> >> >> >> >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$',
> '', sys.argv[0])
> > >>> >> >> >> >> >>>> sys.exit(main())
> > >>> >> >> >> >> >>>> So we _could_ work around the absence of
> LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is
> a different question, and it seems clear that here are very mixed opinions
> on this.
> > >>> >> >> >> >> >>>> Best wishes Graeme
> > >>> >> >> >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis <
> luc_j_bourhis at mac.com<mailto:luc_j_bourhis at mac.com>> wrote:
> > >>> >> >> >> >> >>>> Hi,
> > >>> >> >> >> >> >>>> Even if we managed to ship our the boost dynamic
> libraries with pip, it would still not be pip-like, as we would still need
> our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip
> packages work with the standard python exe. LD_LIBRARY_PATH, we could get
> around that by changing the way we compile, using -Wl,-R, which is the
> runtime equivalent of build time -L. That’s a significant change that would
> need to be tested. But there is no way around setting LIBTBX_BUILD right
> now. Leaving that to the user is horrible. Perhaps there is a way to hack
> libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip
> installs?
> > >>> >> >> >> >> >>>> Best wishes,
> > >>> >> >> >> >> >>>> Luc
> > >>> >> >> >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis <
> luc_j_bourhis at mac.com<mailto:luc_j_bourhis at mac.com>> wrote:
> > >>> >> >> >> >> >>>> Hi,
> > >>> >> >> >> >> >>>> I did look into that many years ago, and even toyed
> with building a pip installer. What stopped me is the exact conclusion you
> reached too: the user would not have the pip experience he expects. You are
> right that it is a lot of effort but is it worth it? Considering that
> remark, I don’t think so. Now, Conda was created specifically to go beyond
> pip pure-python-only support. Since cctbx has garnered support for Conda,
> the best avenue imho is to go the extra length to have a package on
> Anaconda.org<http://anaconda.org/>, and then to advertise it hard to
> every potential user out there.
> > >>> >> >> >> >> >>>> Best wishes,
> > >>> >> >> >> >> >>>> Luc
> > >>> >> >> >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster <
> asbrewster at lbl.gov<mailto:asbrewster at lbl.gov>> wrote:
> > >>> >> >> >> >> >>>> Hi, to avoid clouding Dorothee's documentation
> email thread, which I think is a highly useful enterprise, here's some
> thoughts about putting cctbx into pip. Pip doesn't install non-python
> dependencies well. I don't think boost is available as a package on pip
> (at least the package version we use). wxPython4 isn't portable through
> pip (
> https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-Phoenix_using_pip).
> MPI libraries are system dependent. If cctbx were a pure python package,
> pip would be fine, but cctbx is not.
> > >>> >> >> >> >> >>>> All that said, we could build a manylinux1 version
> of cctbx and upload it to PyPi (I'm just learning about this). For a pip
> package to be portable (which is a requirement for cctbx), it needs to
> conform to PEP513, the manylinux1 standard (
> https://www.python.org/dev/peps/pep-0513/). For example, numpy is built
> according to this standard (see https://pypi.org/project/numpy/#files,
> where you'll see the manylinux1 wheel). Note, the manylinux1 standard is
> built with Centos 5.11 which we no longer support.
> > >>> >> >> >> >> >>>> There is also a manylinux2010 standard, which is
> based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is
> likely a more attainable target (note though by default C++11 is not
> supported on Centos 6).
> > >>> >> >> >> >> >>>> If we built a manylinuxX version of cctbx and
> uploaded it to PyPi, the user would need all the non-python dependencies.
> There's no way to specify these in pip. For example, cctbx requires boost
> 1.63 or better. The user will need to have it in a place their python can
> find it, or we could package it ourselves and supply it, similar to how the
> pip h5py package now comes with an hd5f library, or how the pip numpy
> package includes an openblas library. We'd have to do the same for any
> packages we depend on that aren't on pip using the manylinux standards,
> such as wxPython4.
> > >>> >> >> >> >> >>>> Further, we need to think about how dials and other
> cctbx-based packages interact. If pip install cctbx is set up, how does
> pip install dials work, such that any dials shared libraries can find the
> cctbx libraries? Can shared libraries from one pip package link against
> libraries in another pip package? Would each package need to supply its
> own boost? Possibly this is well understood in the pip field, but not by
> me :)
> > >>> >> >> >> >> >>>> Finally, there's the option of providing a source
> pip package. This would require the full compiler toolchain for any given
> platform (macOS, linux, windows). These are likely available for
> developers, but not for general users.
> > >>> >> >> >> >> >>>> Anyway, these are some of the obstacles. Not
> saying it isn't possible, it's just a lot of effort.
> > >>> >> >> >> >> >>>> Thanks,
> > >>> >> >> >> >> >>>> -Aaron
> > >>> >> >> >> >> >>>> _______________________________________________
> > >>> >> >> >> >> >>>> cctbxbb mailing list
> > >>> >> >> >> >> >>>> cctbxbb at phenix-online.org<mailto:
> cctbxbb at phenix-online.org>
> > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> > >>> >> >> >> >> >>>> _______________________________________________
> > >>> >> >> >> >> >>>> cctbxbb mailing list
> > >>> >> >> >> >> >>>> cctbxbb at phenix-online.org<mailto:
> cctbxbb at phenix-online.org>
> > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> > >>> >> >> >> >> >>>> _______________________________________________
> > >>> >> >> >> >> >>>> cctbxbb mailing list
> > >>> >> >> >> >> >>>> cctbxbb at phenix-online.org<mailto:
> cctbxbb at phenix-online.org>
> > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> > >>> >> >> >> >> >>>> --
> > >>> >> >> >> >> >>>> This e-mail and any attachments may contain
> confidential, copyright and or privileged material, and are for the use of
> the intended addressee only. If you are not the intended addressee or an
> authorised recipient of the addressee please notify us of receipt by
> returning the e-mail and do not use, copy, retain, distribute or disclose
> the information in or attached to the e-mail.
> > >>> >> >> >> >> >>>> Any opinions expressed within this e-mail are those
> of the individual and not necessarily of Diamond Light Source Ltd.
> > >>> >> >> >> >> >>>> Diamond Light Source Ltd. cannot guarantee that
> this e-mail or any attachments are free from viruses and we cannot accept
> liability for any damage which you may sustain as a result of software
> viruses which may be transmitted in or with the message.
> > >>> >> >> >> >> >>>> Diamond Light Source Limited (company no. 4375679).
> Registered in England and Wales with its registered office at Diamond
> House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11
> 0DE, United Kingdom
> > >>> >> >> >> >> >>>> _______________________________________________
> > >>> >> >> >> >> >>>> cctbxbb mailing list
> > >>> >> >> >> >> >>>> cctbxbb at phenix-online.org
> > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> > >>> >> >> >> >> >>> _______________________________________________
> > >>> >> >> >> >> >>> cctbxbb mailing list
> > >>> >> >> >> >> >>> cctbxbb at phenix-online.org
> > >>> >> >> >> >> >>> http://phenix-online.org/mailman/listinfo/cctbxbb
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> _______________________________________________
> > >>> >> >> >> >> >> cctbxbb mailing list
> > >>> >> >> >> >> >> cctbxbb at phenix-online.org
> > >>> >> >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> > >>> >> >> >> >> >
> > >>> >> >> >> >> >
> > >>> >> >> >> >> > _______________________________________________
> > >>> >> >> >> >> > cctbxbb mailing list
> > >>> >> >> >> >> > cctbxbb at phenix-online.org
> > >>> >> >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
> > >>> >> >> >> >>
> > >>> >> >> >> >>
> > >>> >> >> >> >> _______________________________________________
> > >>> >> >> >> >> cctbxbb mailing list
> > >>> >> >> >> >> cctbxbb at phenix-online.org
> > >>> >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> > >>> >> >> >> >
> > >>> >> >> >> > _______________________________________________
> > >>> >> >> >> > cctbxbb mailing list
> > >>> >> >> >> > cctbxbb at phenix-online.org
> > >>> >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
> > >>> >> >> >>
> > >>> >> >> >>
> > >>> >> >> >>
> > >>> >> >> >> --
> > >>> >> >> >> Gergely Katona, PhD
> > >>> >> >> >> Associate Professor
> > >>> >> >> >> Department of Chemistry and Molecular Biology, University
> of Gothenburg
> > >>> >> >> >> Box 462, 40530 Göteborg, Sweden
> > >>> >> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax:
> +46-31-786-3910
> > >>> >> >> >> Web: http://katonalab.eu, Email: gergely.katona at gu.se
> > >>> >> >> >>
> > >>> >> >> >> _______________________________________________
> > >>> >> >> >> cctbxbb mailing list
> > >>> >> >> >> cctbxbb at phenix-online.org
> > >>> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> > >>> >> >> >
> > >>> >> >> > _______________________________________________
> > >>> >> >> > cctbxbb mailing list
> > >>> >> >> > cctbxbb at phenix-online.org
> > >>> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
> > >>> >> >>
> > >>> >> >>
> > >>> >> >>
> > >>> >> >> --
> > >>> >> >> Gergely Katona, PhD
> > >>> >> >> Associate Professor
> > >>> >> >> Department of Chemistry and Molecular Biology, University of
> Gothenburg
> > >>> >> >> Box 462, 40530 Göteborg, Sweden
> > >>> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax:
> +46-31-786-3910
> > >>> >> >> Web: http://katonalab.eu, Email: gergely.katona at gu.se
> > >>> >> >>
> > >>> >> >> _______________________________________________
> > >>> >> >> cctbxbb mailing list
> > >>> >> >> cctbxbb at phenix-online.org
> > >>> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> > >>> >> >
> > >>> >> > _______________________________________________
> > >>> >> > cctbxbb mailing list
> > >>> >> > cctbxbb at phenix-online.org
> > >>> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
> > >>> >>
> > >>> >>
> > >>> >>
> > >>> >> --
> > >>> >> Gergely Katona, PhD
> > >>> >> Associate Professor
> > >>> >> Department of Chemistry and Molecular Biology, University of
> Gothenburg
> > >>> >> Box 462, 40530 Göteborg, Sweden
> > >>> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> > >>> >> Web: http://katonalab.eu, Email: gergely.katona at gu.se
> > >>> >>
> > >>> >> _______________________________________________
> > >>> >> cctbxbb mailing list
> > >>> >> cctbxbb at phenix-online.org
> > >>> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> > >>> >
> > >>> > _______________________________________________
> > >>> > cctbxbb mailing list
> > >>> > cctbxbb at phenix-online.org
> > >>> > http://phenix-online.org/mailman/listinfo/cctbxbb
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Gergely Katona, PhD
> > >>> Associate Professor
> > >>> Department of Chemistry and Molecular Biology, University of
> Gothenburg
> > >>> Box 462, 40530 Göteborg, Sweden
> > >>> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> > >>> Web: http://katonalab.eu, Email: gergely.katona at gu.se
> > >>>
> > >>> _______________________________________________
> > >>> cctbxbb mailing list
> > >>> cctbxbb at phenix-online.org
> > >>> http://phenix-online.org/mailman/listinfo/cctbxbb
> > >
> > > _______________________________________________
> > > cctbxbb mailing list
> > > cctbxbb at phenix-online.org
> > > http://phenix-online.org/mailman/listinfo/cctbxbb
> >
> >
> >
> > --
> > Gergely Katona, Professor
> > Department of Chemistry and Molecular Biology, University of Gothenburg
> > Box 462, 40530 Göteborg, Sweden
> > Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> > Web: http://katonalab.eu, Email: gergely.katona at gu.se
>
>
>
> --
> Gergely Katona, Professor
> Department of Chemistry and Molecular Biology, University of Gothenburg
> Box 462, 40530 Göteborg, Sweden
> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> Web: http://katonalab.eu, Email: gergely.katona at gu.se
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20200103/1eafb4cd/attachment.htm>
More information about the cctbxbb
mailing list