Search results for query "look through"
- 527 messages
[cctbxbb] conda cctbx on Mac 10.15.2 working with python3.8
by Mooers, Blaine H.M. (HSC)
Hi Billy,
Gergely's post yesterday stimulated me to check my conda cctbx setup.
I had earlier reported success with conda cctbx for Python3.6, 3.7, and 3.8.
This morning I found in my path an earlier python3.6 build of cctbx that may
have given me false positives with my conda installs of cctbx.
After removing the old build of cctbx from my path, I modified my zshrc file
as Gergely suggested. I can confirm that results that Gergely reported yesterday.
I used a Mac OS X 10.15.2 with zsh as my shell and anaconda3 installed
as /opt/anaconda. I did my testing in a conda env named cctbx38.
I added 'export LIBTBX_BUILD=$CONDA_PREFIX' in my .zshrc file.
After sourcing my .zshrc file, 'echo $LIBTBX_BUILD' returns '/opt/anaconda/envs/cctbx38' as expected.
The toy script below prints the calculated Miller indices as a tuples in a terminal window.
It was run using 'python millerSet.py'.
Note that the printing of 1 million indices to a jupyter notebook cell is disappointingly slow!!
Reduce each unit cell dimension by two orders of magnitude before trying this at home!
The following code also worked with Python3.6 and Python3.7 in their corresponding cctbx36 and cctbx37 envs.
from cctbx import crystal
from cctbx import miller
ms = miller.build_set(
crystal_symmetry=crystal.symmetry(
space_group_symbol='P4',
unit_cell=(150.8,150.8,250.4,90.0,90.0,90.0)),
anomalous_flag=False,
d_min=1.4)
[print(hkl) for hkl in ms.indices()]
I used conda python3.8.0 built on November 22. Python3.8.1 does not seem to be
available yet from Anaconda.
Best regards,
Blaine
Blaine Mooers, Ph.D.
Associate Professor
Department of Biochemistry and Molecular Biology
College of Medicine
University of Oklahoma Health Sciences Center
S.L. Young Biomedical Research Center (BRC) Rm. 466
975 NE 10th Street, BRC 466
Oklahoma City, OK 73104-5419
________________________________________
From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of cctbxbb-request(a)phenix-online.org [cctbxbb-request(a)phenix-online.org]
Sent: Saturday, January 04, 2020 2:00 PM
To: cctbxbb(a)phenix-online.org
Subject: [EXTERNAL] cctbxbb Digest, Vol 130, Issue 2
Send cctbxbb mailing list submissions to
cctbxbb(a)phenix-online.org
To subscribe or unsubscribe via the World Wide Web, visit
https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
or, via email, send a message with subject or body 'help' to
cctbxbb-request(a)phenix-online.org
You can reach the person managing the list at
cctbxbb-owner(a)phenix-online.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of cctbxbb digest..."
Today's Topics:
1. Re: some thoughts on cctbx and pip (Billy Poon)
----------------------------------------------------------------------
Message: 1
Date: Fri, 3 Jan 2020 21:04:25 +0100
From: Billy Poon <BKPoon(a)lbl.gov>
To: cctbx mailing list <cctbxbb(a)phenix-online.org>
Subject: Re: [cctbxbb] some thoughts on cctbx and pip
Message-ID:
<CAHmYUcbWS7FXA6EPUx=AC2X6m6gPY=GqH-MwGD+uzpTaOKSyjQ(a)mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi Gergely,
Happy New Year!
Can you provide the output to "conda list" for each environment?
I suspect that the old version of cctbx might still be around. Is there a
file named PYTHON_VERSION_MAJOR_MINOR in "/home/gergely/anaconda3/lib"? If
so, delete that file. To make sure you are downloading a new copy of the
package, clear the cache where conda keeps copies of all the packages. To
do that, you can run "conda clean -y --all". You can remove the "-y" flag
if you want to see what will be deleted. The hash for the new linux package
should be h8b68381_0.
The last error looks like an incompatible version of boost or gcc library
was installed. The output from "conda list" should show the boost and
boost-cpp packages coming from conda-forge. I would avoid installing into
the root or base environment because that usually pulls from the default
channels. All the dependencies should be pulled from the conda-forge
channel.
--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709
Fax: (510) 486-5909
Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
On Fri, Jan 3, 2020 at 2:20 PM Gergely Katona <gkatona(a)gmail.com> wrote:
> I found some other errors when using the test environment.
>
>
> ---------------------------------------------------------------------------
> RuntimeError Traceback (most recent call last)
> <ipython-input-1-fe308b345b3f> in <module>
> 10 import scipy as sp
> 11 from scipy import linalg
> ---> 12 import iotbx.pdb
> 13 import cctbx
> 14 import pandas as pd
>
> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
> in <module>
> 8
> 9 import iotbx.pdb.records
> ---> 10 import iotbx.pdb.hierarchy
> 11 from scitbx import matrix
> 12
>
> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__hierarchy.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=QlonHINbJqyu1vTwlBJWKyChCMCCJxpIWIaT5_rrkLc&e=
> in <module>
> 14 from six.moves import cStringIO as StringIO
> 15 from iotbx.pdb import hy36encode, hy36decode
> ---> 16 import iotbx.cif.model
> 17 from cctbx import crystal
> 18 from libtbx import group_args
>
> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/cif/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
> in <module>
> 19
> 20 from cctbx.array_family import flex
> ---> 21 from cctbx import miller
> 22 from iotbx.cif import model, builders, geometry
> 23 from libtbx.containers import OrderedDict
>
> ~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/miller/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
> in <module>
> 11
> 12 from cctbx import crystal
> ---> 13 from cctbx import maptbx
> 14 from cctbx import sgtbx
> 15 from cctbx.sgtbx import lattice_symmetry
>
> ~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/maptbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
> in <module>
> 15 from libtbx import adopt_init_args
> 16 from libtbx.utils import Sorry
> ---> 17 import libtbx.load_env
> 18 import math
> 19 import sys, os
>
> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__load-5Fenv.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=rJbfdPr3ymiz7Ir508_VGIhspAKCOwozKhY5I6OXux0&e= in
> <module>
> 3 import libtbx.env_config
> 4 import os
> ----> 5 libtbx.env = libtbx.env_config.unpickle()
> 6 libtbx.env.set_os_environ_all_dist()
> 7 libtbx.env.dispatcher_name =
> os.environ.get("LIBTBX_DISPATCHER_NAME")
>
> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__env-5Fconfig.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=UrsFhQ0dxNNuzXmOT3D6pnGJ5lXko4mfrKceNgBaQV4&e=
> in unpickle()
> 2736 env = pickle.load(libtbx_env)
> 2737 if (env.python_version_major_minor != sys.version_info[:2]):
> -> 2738 env.raise_python_version_incompatible()
> 2739 if (op.realpath(build_path) != op.realpath(abs(env.build_path))):
> 2740 env.build_path.reset(build_path)
>
> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__env-5Fconfig.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=UrsFhQ0dxNNuzXmOT3D6pnGJ5lXko4mfrKceNgBaQV4&e=
> in raise_python_version_incompatible(self, prev_pvmm)
> 469 if (prev_pvmm is None):
> 470 prev_pvmm = "%d.%d" % self.python_version_major_minor
> --> 471 raise RuntimeError("Python version incompatible with this
> build:\n"
> 472 + " Build directory: %s\n" %
> show_string(abs(self.build_path))
> 473 + " Python version used initially: %s\n" % prev_pvmm
>
> RuntimeError: Python version incompatible with this build:
> Build directory: "/home/gergely/anaconda3"
> Python version used initially: 3.6
> Python version in use now: 3.8
>
>
> Interestingly in the base environment all import worked, but I got an
> error at:
>
>
> ArgumentError Traceback (most recent call last)
> <ipython-input-4-5bc71c36e121> in <module>
> 33 atm_odd.set_b(0)
> 34
> ---> 35 odd.write_pdb_file("bdiff_in_even_000.pdb") #skriver ny pdb
> med dessa inkluderade
> 36 dfWBvals=pd.DataFrame(rows_list)
>
> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
> write_pdb_file(self, file_name, open_append, crystal_symmetry,
> cryst1_z, write_scale_records, append_end, atom_hetatm, sigatm,
> anisou, siguij)
> 930 siguij=True):
> 931 if (crystal_symmetry is Auto):
> --> 932 crystal_symmetry = self.crystal_symmetry()
> 933 if (cryst1_z is Auto):
> 934 cryst1_z = self.extract_cryst1_z_columns()
>
> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
> crystal_symmetry(self, crystal_symmetry, weak_symmetry)
> 1218 crystal_symmetry=None,
> 1219 weak_symmetry=False):
> -> 1220 self_symmetry = self.crystal_symmetry_from_cryst1()
> 1221 if (self_symmetry is None):
> 1222 self_symmetry = self._crystal_symmetry_from_cns_remark_sg()
>
> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
> crystal_symmetry_from_cryst1(self)
> 1195 for line in self.crystallographic_section():
> 1196 if (line.startswith("CRYST1")):
> -> 1197 return
> cryst1_interpretation.crystal_symmetry(cryst1_record=line)
> 1198 return None
> 1199
>
> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__cryst1-5Finterpretation.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=3OchLb18AMp3YpXOuEb8BDOwhl2XLrV1QAfZByH-l2E&e=
> in crystal_symmetry(cryst1_record)
> 136 space_group_info=None)
> 137 space_group_info =
> categorize(cryst1_record.sgroup).space_group_info(
> --> 138 unit_cell=u)
> 139 return crystal.symmetry(unit_cell=u,
> space_group_info=space_group_info)
>
> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__cryst1-5Finterpretation.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=3OchLb18AMp3YpXOuEb8BDOwhl2XLrV1QAfZByH-l2E&e=
> in space_group_info(self, unit_cell)
> 85 if (self.symbol is None): return None
> 86 if (self.category is None):
> ---> 87 try: return sgtbx.space_group_info(self.symbol)
> 88 except RuntimeError: return None
> 89 if (isinstance(unit_cell, uctbx.ext.unit_cell)):
>
> ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
> __init__(self, symbol, table_id, group, number, space_group_t_den)
> 100 assert group is None
> 101 if (table_id is None):
> --> 102 symbols = space_group_symbols(symbol)
> 103 else:
> 104 if (isinstance(symbol, int)): symbol = str(symbol)
>
> ArgumentError: Python argument types in
> space_group_symbols.__init__(space_group_symbols, str)
> did not match C++ signature:
> __init__(_object*, int space_group_number)
> __init__(_object*, int space_group_number,
> std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > extension='')
> __init__(_object*, int space_group_number,
> std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > extension='', std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > table_id='')
> __init__(_object*, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > symbol)
> __init__(_object*, std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> > symbol,
> std::__cxx11::basic_string<char, std::char_traits<char>,
> std::allocator<char> > table_id='')
>
>
> Gergely
>
>
>
> On Fri, Jan 3, 2020 at 1:44 PM Gergely Katona <gkatona(a)gmail.com> wrote:
> >
> > Dear Billy,
> >
> > I did some tests installing in my base conda was not troublefree in
> > python 3.8. Running iotbx.fetch_pdb --all --mtz 1kp8 resulted in boost
> > related error when converting to mtz. Installing into a test
> > environment went fine with iotbx.fetch_pdb --all --mtz 1kp8 running
> > successfully. I could add pymc3, jupyter and seaborn and iotbx still
> > did not break. My LIBTBX_BUILD is not set however in the test
> > environment, which breaks miller when I was trying to import it.
> > Manually setting the LIBTBX_BUILD to CONDA_PREFIX (setenv LIBTBX_BUILD
> > $CONDA_PREFIX) fixes this and one of my script works fine! I will come
> > back if I find some problems with my other scripts, but so far this
> > looks promising!
> >
> >
> > Happy new year!
> >
> > Gergely
> >
> >
> >
> >
> >
> > On Fri, Dec 20, 2019 at 8:16 PM Billy Poon <BKPoon(a)lbl.gov> wrote:
> > >
> > > Hi everyone,
> > >
> > > I updated the conda packages on the cctbx-dev channel and there are
> also now packages for macOS. The old packages have been removed so you can
> create a new environment with
> > >
> > > conda create -n test -c cctbx-dev cctbx python=2.7
> > > conda activate test
> > >
> > > where "test" is the environment name and 2.7 is the python version.
> For the python version, you can also pick 3.6, 3.7, or 3.8. I think the
> latest conda will install 3.8 as the default python if the python argument
> is not provided. Or install into an existing environment (with python
> already installed) with the same command as before,
> > >
> > > conda install -c cctbx-dev cctbx
> > >
> > > Again, your .condarc file should have conda-forge as the first channel
> so that dependencies can be pulled correctly.
> > >
> > > The dispatchers should work now and $LIBTBX_BUILD will be set on
> activation of an environment (and unset on environment deactivation) or
> after installing into an existing environment, so you can do more
> interesting things like
> > >
> > > iotbx.fetch_pdb --all --mtz 1kp8
> > >
> > > However, this package does not install databases, so
> > >
> > > molprobity.molprobity 1kp8.pdb 1kp8.mtz
> > >
> > > will fail because the database is not available (also probe and reduce
> are not built).
> > >
> > > Some known issues in no particular order,
> > >
> > > 1) You can run "libtbx.run_tests_parallel module=cctbx nproc=Auto" to
> run tests, but not all the test files were copied, so I got 30 failures
> here. The final package will probably not have the test files.
> > > 2) The other python dispatchers, like "cctbx.python", were not
> updated, so those will fail. But you can just run "python" in the active
> environment.
> > > 3) Parts of mmtbx still need to be updated for Python 3.
> > > 4) The LIBTBX_BUILD variable should be set for sh and csh shells. Let
> me know if it's not working for your shell (i.e. "echo $LIBTBX_BUILD"
> should show the same thing as $CONDA_PREFIX).
> > > 5) One future step is to enable building other CCTBX-like modules with
> this conda package. This will require some bookkeeping updates so that
> modules in the conda environment and the modules being built are tracked
> properly. Also, making sure that all the necessary headers exist in
> $PREFIX/include.
> > >
> > > Gergely, let me know if your scripts are working.
> > >
> > > Happy holidays!
> > >
> > > --
> > > Billy K. Poon
> > > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> > > Lawrence Berkeley National Laboratory
> > > 1 Cyclotron Road, M/S 33R0345
> > > Berkeley, CA 94720
> > > Tel: (510) 486-5709
> > > Fax: (510) 486-5909
> > > Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
> > >
> > >
> > > On Wed, Dec 18, 2019 at 11:25 AM Billy Poon <BKPoon(a)lbl.gov> wrote:
> > >>
> > >> Hi Gergely,
> > >>
> > >> That's the other information that hasn't been updated yet in the
> conda package. :)
> > >>
> > >> For LIBTBX_BUILD, I can set up environment variables sort of how the
> conda gcc package sets up environment variables. I have to double check to
> see if they're set after installation of the package and after activation
> of an environment. The variable will just be a copy of $CONDA_PREFIX.
> > >>
> > >> The libtbx_env file is a file that exists in $LIBTBX_BUILD that
> basically stores configuration information. I'll have to modify the
> contents to update the locations of the different modules since the conda
> package will not have a "modules" directory.
> > >>
> > >> --
> > >> Billy K. Poon
> > >> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> > >> Lawrence Berkeley National Laboratory
> > >> 1 Cyclotron Road, M/S 33R0345
> > >> Berkeley, CA 94720
> > >> Tel: (510) 486-5709
> > >> Fax: (510) 486-5909
> > >> Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
> > >>
> > >>
> > >> On Wed, Dec 18, 2019 at 2:08 AM Gergely Katona <gkatona(a)gmail.com>
> wrote:
> > >>>
> > >>> Hi Billy,
> > >>>
> > >>> Conda install went fine with your instructions after rearranging the
> > >>> channels by putting cctbx last. I removed all environmental variables
> > >>> and previous build of cctbx.
> > >>>
> > >>> import sys
> > >>> print (sys.path, sys.prefix)
> > >>>
> > >>> ['/home/gergely/anaconda3/lib/https://urldefense.proofpoint.com/v2/url?u=http-3A__python36.zip&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=cUHL8qgeloeUO3oYTQcr9d56RegJk01wJ2i-c3Zo6oE&e= ',
> > >>> '/home/gergely/anaconda3/lib/python3.6',
> > >>> '/home/gergely/anaconda3/lib/python3.6/lib-dynload', '',
> > >>> '/home/gergely/anaconda3/lib/python3.6/site-packages',
> > >>>
> '/home/gergely/anaconda3/lib/python3.6/site-packages/IPython/extensions',
> > >>> '/home/gergely/.ipython'] /home/gergely/anaconda3
> > >>>
> > >>> Many imports went fine including pymc3, but I encountered problems
> > >>> with these three:
> > >>>
> > >>> from cctbx import miller
> > >>> import iotbx.pdb
> > >>> from iotbx import reflection_file_reader, mtz
> > >>>
> > >>>
> > >>> I also got type error when handling space groups.
> > >>>
> > >>>
> > >>>
> > >>>
> ---------------------------------------------------------------------------
> > >>> KeyError Traceback (most recent
> call last)
> > >>> <ipython-input-5-cedbd4f84f3d> in <module>
> > >>> 25 from cctbx import uctbx
> > >>> 26 from cctbx import sgtbx
> > >>> ---> 27 from cctbx import miller
> > >>> 28 #from iotbx import reflection_file_reader, mtz
> > >>> 29 sns.set_context("poster")
> > >>>
> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/miller/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
> <module>
> > >>> 11
> > >>> 12 from cctbx import crystal
> > >>> ---> 13 from cctbx import maptbx
> > >>> 14 from cctbx import sgtbx
> > >>> 15 from cctbx.sgtbx import lattice_symmetry
> > >>>
> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/maptbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
> <module>
> > >>> 15 from libtbx import adopt_init_args
> > >>> 16 from libtbx.utils import Sorry
> > >>> ---> 17 import libtbx.load_env
> > >>> 18 import math
> > >>> 19 import sys, os
> > >>>
> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__load-5Fenv.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=rJbfdPr3ymiz7Ir508_VGIhspAKCOwozKhY5I6OXux0&e= in
> <module>
> > >>> 3 import libtbx.env_config
> > >>> 4 import os
> > >>> ----> 5 libtbx.env = libtbx.env_config.unpickle()
> > >>> 6 libtbx.env.set_os_environ_all_dist()
> > >>> 7 libtbx.env.dispatcher_name =
> os.environ.get("LIBTBX_DISPATCHER_NAME")
> > >>>
> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__env-5Fconfig.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=UrsFhQ0dxNNuzXmOT3D6pnGJ5lXko4mfrKceNgBaQV4&e= in
> unpickle()
> > >>> 2603
> > >>> 2604 def unpickle():
> > >>> -> 2605 build_path = os.environ["LIBTBX_BUILD"]
> > >>> 2606
> set_preferred_sys_prefix_and_sys_executable(build_path=build_path)
> > >>> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")
> > >>>
> > >>> ~/anaconda3/lib/python3.6/https://urldefense.proofpoint.com/v2/url?u=http-3A__os.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=hA1zo5wap0c9wK3DrlWIKqYBIqqbfKXca5jQ9aX5pYU&e= in __getitem__(self, key)
> > >>> 667 except KeyError:
> > >>> 668 # raise KeyError with the original key value
> > >>> --> 669 raise KeyError(key) from None
> > >>> 670 return self.decodevalue(value)
> > >>> 671
> > >>>
> > >>> KeyError: 'LIBTBX_BUILD'
> > >>>
> > >>>
> > >>>
> > >>>
> ---------------------------------------------------------------------------
> > >>> ArgumentError Traceback (most recent
> call last)
> > >>> <ipython-input-7-2acd3a9ce26a> in <module>
> > >>> 26 return
> ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar
> > >>> 27
> > >>> ---> 28
> ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar=initializecrystal()
> > >>>
> > >>> <ipython-input-7-2acd3a9ce26a> in initializecrystal()
> > >>> 11 uc = uctbx.unit_cell(unit_cell)
> > >>> 12 wavelength = 1.54980
> > >>> ---> 13 xtal_symm = crystal.symmetry(unit_cell=unit_cell,
> > >>> space_group_symbol="P 43 21 2")
> > >>> 14
> > >>> 15 ms =
> > >>> miller.build_set(crystal_symmetry=xtal_symm,anomalous_flag=True,
> > >>> d_min=1.61)
> > >>>
> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/crystal/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
> > >>> __init__(self, unit_cell, space_group_symbol, space_group_info,
> > >>> space_group, correct_rhombohedral_setting_if_necessary,
> > >>> assert_is_compatible_unit_cell,
> raise_sorry_if_incompatible_unit_cell,
> > >>> force_compatible_unit_cell)
> > >>> 74 if (space_group_symbol is not None):
> > >>> 75 self._space_group_info = sgtbx.space_group_info(
> > >>> ---> 76 symbol=space_group_symbol)
> > >>> 77 elif (space_group is not None):
> > >>> 78 if (isinstance(space_group, sgtbx.space_group)):
> > >>>
> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
> > >>> __init__(self, symbol, table_id, group, number, space_group_t_den)
> > >>> 100 assert group is None
> > >>> 101 if (table_id is None):
> > >>> --> 102 symbols = space_group_symbols(symbol)
> > >>> 103 else:
> > >>> 104 if (isinstance(symbol, int)): symbol = str(symbol)
> > >>>
> > >>> ArgumentError: Python argument types in
> > >>> space_group_symbols.__init__(space_group_symbols, str)
> > >>> did not match C++ signature:
> > >>> __init__(_object*, int space_group_number)
> > >>> __init__(_object*, int space_group_number,
> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
> > >>> std::allocator<char> > extension='')
> > >>> __init__(_object*, int space_group_number,
> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
> > >>> std::allocator<char> > extension='', std::__cxx11::basic_string<char,
> > >>> std::char_traits<char>, std::allocator<char> > table_id='')
> > >>> __init__(_object*, std::__cxx11::basic_string<char,
> > >>> std::char_traits<char>, std::allocator<char> > symbol)
> > >>> __init__(_object*, std::__cxx11::basic_string<char,
> > >>> std::char_traits<char>, std::allocator<char> > symbol,
> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
> > >>> std::allocator<char> > table_id='')
> > >>>
> > >>>
> > >>> When I tried to set LIBTBX_BUILD to /home/gergely/anaconda3 (this is
> > >>> sys.prefix on my system), I got other problems when importing. Can
> > >>> LIBTBX_BUILD be set in the conda package?
> > >>>
> > >>>
> > >>>
> > >>>
> ---------------------------------------------------------------------------
> > >>> FileNotFoundError Traceback (most recent
> call last)
> > >>> <ipython-input-1-7e2b144826cc> in <module>
> > >>> 26 from cctbx import sgtbx
> > >>> 27 #from cctbx import miller
> > >>> ---> 28 from iotbx import reflection_file_reader, mtz
> > >>> 29 sns.set_context("poster")
> > >>> 30 plt.rcParams.update({'figure.autolayout': True})
> > >>>
> > >>>
> ~/anaconda3/lib/python3.6/site-packages/iotbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__reflection-5Ffile-5Freader.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=U5BvLmuCgufvN_LRxQw9n78p33jNaRrXVbH1BRSXgxg&e=
> > >>> in <module>
> > >>> 53
> > >>> 54 from __future__ import absolute_import, division,
> print_function
> > >>> ---> 55 from iotbx import mtz
> > >>> 56 from iotbx.scalepack import merge as scalepack_merge
> > >>> 57 from iotbx.scalepack import no_merge_original_index as
> > >>> scalepack_no_merge
> > >>>
> > >>> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
> <module>
> > >>> 9 import iotbx_mtz_ext as ext
> > >>> 10
> > >>> ---> 11 from iotbx.mtz import extract_from_symmetry_lib
> > >>> 12 from cctbx import xray
> > >>> 13 import cctbx.xray.observation_types
> > >>>
> > >>>
> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/https://urldefense.proofpoint.com/v2/url?u=http-3A__extract-5Ffrom-5Fsymmetry-5Flib.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=k3WR6VZd4-PDULPXl4O0Vwz9q3LCaocUUhK_ehTn9UE&e=
> > >>> in <module>
> > >>> 1 from __future__ import absolute_import, division,
> print_function
> > >>> 2 from cctbx import sgtbx
> > >>> ----> 3 import libtbx.load_env
> > >>> 4 import os.path as op
> > >>> 5 from six.moves import range
> > >>>
> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__load-5Fenv.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=rJbfdPr3ymiz7Ir508_VGIhspAKCOwozKhY5I6OXux0&e= in
> <module>
> > >>> 3 import libtbx.env_config
> > >>> 4 import os
> > >>> ----> 5 libtbx.env = libtbx.env_config.unpickle()
> > >>> 6 libtbx.env.set_os_environ_all_dist()
> > >>> 7 libtbx.env.dispatcher_name =
> os.environ.get("LIBTBX_DISPATCHER_NAME")
> > >>>
> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__env-5Fconfig.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=UrsFhQ0dxNNuzXmOT3D6pnGJ5lXko4mfrKceNgBaQV4&e= in
> unpickle()
> > >>> 2605 build_path = os.environ["LIBTBX_BUILD"]
> > >>> 2606
> set_preferred_sys_prefix_and_sys_executable(build_path=build_path)
> > >>> -> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")
> > >>> 2608 env = pickle.load(libtbx_env)
> > >>> 2609 if (env.python_version_major_minor !=
> sys.version_info[:2]):
> > >>>
> > >>> FileNotFoundError: [Errno 2] No such file or directory:
> > >>> '/home/gergely/anaconda3/libtbx_env'
> > >>>
> > >>> On Wed, Dec 18, 2019 at 9:10 AM Billy Poon <BKPoon(a)lbl.gov> wrote:
> > >>> >
> > >>> > Hi Gergely,
> > >>> >
> > >>> > I've uploaded linux packages to a new channel, cctbx-dev, and you
> can install it with
> > >>> >
> > >>> > conda install -c cctbx-dev cctbx
> > >>> >
> > >>> > in your current environment. You should set your ~/.condarc file
> to pull the other dependencies from the conda-forge channel first, so put
> conda-forge above cctbx. Mine looks like
> > >>> >
> > >>> > channels:
> > >>> > - conda-forge
> > >>> > - defaults
> > >>> > - cctbx
> > >>> >
> > >>> > Do you need dxtbx for your scripts? This package does not build
> that part. I think the plan is to build a separate conda package for dxtbx
> so that it can be updated more frequently. I can rebuild the packages to
> include it for testing, but the one being submitted to conda-forge will not
> have it.
> > >>> >
> > >>> > Also, your error message is probably due to version of HDF5 that
> the development build installs. The https://urldefense.proofpoint.com/v2/url?u=http-3A__bootstrap.py&d=DwICAg&c… script will install
> 1.10.4, but your other dependency is looking for 1.10.5. Installing this
> cctbx conda package should install 1.10.5, which should fix the issue.
> Also, I'm updating those environments and 1.10.5 will be the new default
> version.
> > >>> >
> > >>> > Lastly, the dispatchers will not work in these packages because
> the old paths during the build process are still in them and many of them
> expect some additional information that has not been updated in the
> packages yet. I'm in the process of doing that and will update the
> cctbx-dev channel when that's done. However, by starting python, you can
> import cctbx modules. So you can run commands like
> > >>> >
> > >>> > from scitbx.array_family import flex
> > >>> > a = flex.random_double(1000000)
> > >>> > b = flex.min_max_mean_double(a)
> > >>> > b.min
> > >>> > b.max
> > >>> > b.mean
> > >>> >
> > >>> > Thanks!
> > >>> >
> > >>> > --
> > >>> > Billy K. Poon
> > >>> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> > >>> > Lawrence Berkeley National Laboratory
> > >>> > 1 Cyclotron Road, M/S 33R0345
> > >>> > Berkeley, CA 94720
> > >>> > Tel: (510) 486-5709
> > >>> > Fax: (510) 486-5909
> > >>> > Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
> > >>> >
> > >>> >
> > >>> > On Tue, Dec 17, 2019 at 5:39 AM Gergely Katona <gkatona(a)gmail.com>
> wrote:
> > >>> >>
> > >>> >> Dear Billy,
> > >>> >>
> > >>> >> Thank you for the detailed explanation, I look forward to do
> further
> > >>> >> testing! I aim to pool all modules under the same environment, if
> > >>> >> there are conflicts then I just try to reshuffle the order of
> imports.
> > >>> >> So far this did not cause problems for me even when I was using
> system
> > >>> >> python, but of course it is not the most prudent thing to do. With
> > >>> >> anaconda everything is much more standardized and isolated
> already and
> > >>> >> without being superuser I can have my familiar environment at any
> > >>> >> synchrotron based cluster. It is great that cctbx will be an
> integral
> > >>> >> part of this ecosystem and this was also the last thing holding me
> > >>> >> back from adopting python3. About hdf5, pymc3 requires it and
> > >>> >> importing causes a kernel restart with the following error
> messages,
> > >>> >> curiously if I import h5py first this can be avoided.
> > >>> >>
> > >>> >> Best wishes,
> > >>> >>
> > >>> >> Gergely
> > >>> >>
> > >>> >> import pymc3 as pm
> > >>> >>
> > >>> >> Warning! ***HDF5 library version mismatched error***
> > >>> >> The HDF5 header files used to compile this application do not
> match
> > >>> >> the version used by the HDF5 library to which this application is
> linked.
> > >>> >> Data corruption or segmentation faults may occur if the
> application continues.
> > >>> >> This can happen when an application was compiled by one version
> of HDF5 but
> > >>> >> linked with a different version of static or shared HDF5 library.
> > >>> >> You should recompile the application or check your shared library
> related
> > >>> >> settings such as 'LD_LIBRARY_PATH'.
> > >>> >> You can, at your own risk, disable this warning by setting the
> environment
> > >>> >> variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.
> > >>> >> Setting it to 2 or higher will suppress the warning messages
> totally.
> > >>> >> Headers are 1.10.4, library is 1.10.5
> > >>> >> SUMMARY OF THE HDF5 CONFIGURATION
> > >>> >> =================================
> > >>> >>
> > >>> >> General Information:
> > >>> >> -------------------
> > >>> >> HDF5 Version: 1.10.5
> > >>> >> Configured on: Tue Oct 22 12:02:13 UTC 2019
> > >>> >> Configured by: conda@16247e67ecd5
> > >>> >> Host system: x86_64-conda_cos6-linux-gnu
> > >>> >> Uname information: Linux 16247e67ecd5
> 4.15.0-1059-azure
> > >>> >> #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64
> > >>> >> GNU/Linux
> > >>> >> Byte sex: little-endian
> > >>> >> Installation point: /home/gergely/anaconda3
> > >>> >>
> > >>> >> Compiling Options:
> > >>> >> ------------------
> > >>> >> Build Mode: production
> > >>> >> Debugging Symbols: no
> > >>> >> Asserts: no
> > >>> >> Profiling: no
> > >>> >> Optimization Level: high
> > >>> >>
> > >>> >> Linking Options:
> > >>> >> ----------------
> > >>> >> Libraries: static, shared
> > >>> >> Statically Linked Executables:
> > >>> >> LDFLAGS: -Wl,-O2 -Wl,--sort-common
> > >>> >> -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags
> > >>> >> -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib
> > >>> >> -Wl,-rpath-link,/home/gergely/anaconda3/lib
> > >>> >> -L/home/gergely/anaconda3/lib
> > >>> >> H5_LDFLAGS:
> > >>> >> AM_LDFLAGS: -L/home/gergely/anaconda3/lib
> > >>> >> Extra libraries: -lrt -lpthread -lz -ldl -lm
> > >>> >> Archiver:
> > >>> >>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar
> > >>> >> AR_FLAGS: cr
> > >>> >> Ranlib:
> > >>> >>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib
> > >>> >>
> > >>> >> Languages:
> > >>> >> ----------
> > >>> >> C: yes
> > >>> >> C Compiler:
> > >>> >>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc
> > >>> >> CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2
> > >>> >> -I/home/gergely/anaconda3/include
> > >>> >> H5_CPPFLAGS: -D_GNU_SOURCE
> > >>> >> -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API
> > >>> >> AM_CPPFLAGS:
> -I/home/gergely/anaconda3/include
> > >>> >> C Flags: -march=nocona -mtune=haswell
> > >>> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
> > >>> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include
> > >>> >>
> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5
> > >>> >>
> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix
> > >>> >> H5 C Flags: -std=c99 -pedantic -Wall
> -Wextra
> > >>> >> -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual
> -Wconversion
> > >>> >> -Wdeclaration-after-statement -Wdisabled-optimization
> -Wfloat-equal
> > >>> >> -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations
> > >>> >> -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs
> > >>> >> -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls
> > >>> >> -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum
> -Wundef
> > >>> >> -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings
> > >>> >> -finline-functions -s -Wno-inline -Wno-aggregate-return
> > >>> >> -Wno-missing-format-attribute -Wno-missing-noreturn -O
> > >>> >> AM C Flags:
> > >>> >> Shared C Library: yes
> > >>> >> Static C Library: yes
> > >>> >>
> > >>> >>
> > >>> >> Fortran: yes
> > >>> >> Fortran Compiler:
> > >>> >>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran
> > >>> >> Fortran Flags:
> > >>> >> H5 Fortran Flags: -pedantic -Wall -Wextra
> -Wunderflow
> > >>> >> -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2
> > >>> >> AM Fortran Flags:
> > >>> >> Shared Fortran Library: yes
> > >>> >> Static Fortran Library: yes
> > >>> >>
> > >>> >> C++: yes
> > >>> >> C++ Compiler:
> > >>> >>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++
> > >>> >> C++ Flags: -fvisibility-inlines-hidden
> > >>> >> -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell
> > >>> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
> > >>> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include
> > >>> >>
> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5
> > >>> >>
> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix
> > >>> >> H5 C++ Flags: -pedantic -Wall -W -Wundef
> -Wshadow
> > >>> >> -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings
> -Wconversion
> > >>> >> -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual
> > >>> >> -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor
> > >>> >> -Wctor-dtor-privacy -Wabi -finline-functions -s -O
> > >>> >> AM C++ Flags:
> > >>> >> Shared C++ Library: yes
> > >>> >> Static C++ Library: yes
> > >>> >>
> > >>> >> Java: no
> > >>> >>
> > >>> >>
> > >>> >> Features:
> > >>> >> ---------
> > >>> >> Parallel HDF5: no
> > >>> >> Parallel Filtered Dataset Writes: no
> > >>> >> Large Parallel I/O: no
> > >>> >> High-level library: yes
> > >>> >> Threadsafety: yes
> > >>> >> Default API mapping: v110
> > >>> >> With deprecated public symbols: yes
> > >>> >> I/O filters (external): deflate(zlib)
> > >>> >> MPE: no
> > >>> >> Direct VFD: no
> > >>> >> dmalloc: no
> > >>> >> Packages w/ extra debug output: none
> > >>> >> API tracing: no
> > >>> >> Using memory checker: yes
> > >>> >> Memory allocation sanity checks: no
> > >>> >> Function stack tracing: no
> > >>> >> Strict file format checks: no
> > >>> >> Optimization instrumentation: no
> > >>> >>
> > >>> >> On Tue, Dec 17, 2019 at 8:35 AM Billy Poon <BKPoon(a)lbl.gov>
> wrote:
> > >>> >> >
> > >>> >> > Hi Gergely,
> > >>> >> >
> > >>> >> > Let me build the test package tomorrow. All the gritty details
> for building with conda is being finalized and the official documentation
> will be updated to describe the steps. It would be too confusing to keep
> changing the documentation as the process evolves. You do have the general
> process, though, which is summarized as follows.
> > >>> >> >
> > >>> >> > 1) Installing dependencies. The cctbx_dependencies metapackage
> was an initial approach for managing the CCTBX dependencies, but after
> contacting the conda-forge folks, they recommended using the --only-deps
> flag. So when the CCTBX conda package is available, you'll be able to get a
> set of dependencies with,
> > >>> >> >
> > >>> >> > conda install -c conda-forge --only-deps cctbx
> > >>> >> >
> > >>> >> > By default, the https://urldefense.proofpoint.com/v2/url?u=http-3A__bootstrap.py&d=DwICAg&c… file will automatically install a
> set of dependencies in the "conda_base" directory (and a conda installation
> if one is not found). It just uses standard conda environment files located
> in libtbx/auto_build/conda_envs, so you do not need to install
> cctbx_dependencies as a separate step. The environment files avoid channel
> issues by explicitly defining the channel to pull the packages from and the
> cctbx channel just stores copies of packages from conda-forge. There were
> issues earlier where the conda-forge packages would sometimes be moved to a
> different label. The --use-conda flag also accepts a path to $CONDA_PREFIX
> if you want to use a specific environment for testing.
> > >>> >> >
> > >>> >> > 2) Building. The https://urldefense.proofpoint.com/v2/url?u=http-3A__bootstrap.py&d=DwICAg&c… file handles that with SCons.
> > >>> >> >
> > >>> >> > 3) Running. After building, there should be a https://urldefense.proofpoint.com/v2/url?u=http-3A__setpaths.sh&d=DwICAg&c=… (and
> .csh) file that adds build/bin to your path. The build/bin directory has
> our dispatchers, which are just scripts that set up the environment
> variables for you. This prevents other programs from loading our libraries,
> whose versions may conflict. You should see that there is a "python"
> dispatcher, which is a convenience for developers. Otherwise, you can use
> libtbx.python, which will be able to import CCTBX modules.
> > >>> >> >
> > >>> >> > The running part is where the conda package for CCTBX will be
> different than this build. Since our Python files and extensions modules
> will be in the "site-packages" directory for Python, the PYTHONPATH
> variable will not be needed (and conda suggests that that variable not be
> set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so
> LD_LIBRARY_PATH is not needed. In an active environment, PATH will already
> be modified. And then our LIBTBX_BUILD directory can be set to sys.prefix.
> > >>> >> >
> > >>> >> > So with the conda package, you would only need to activate your
> conda environment and CCTBX should integrate with other conda packages.
> What is the conflict with hdf5? That's something that should be fixed.
> Thanks!
> > >>> >> >
> > >>> >> > --
> > >>> >> > Billy K. Poon
> > >>> >> > Research Scientist, Molecular Biophysics and Integrated
> Bioimaging
> > >>> >> > Lawrence Berkeley National Laboratory
> > >>> >> > 1 Cyclotron Road, M/S 33R0345
> > >>> >> > Berkeley, CA 94720
> > >>> >> > Tel: (510) 486-5709
> > >>> >> > Fax: (510) 486-5909
> > >>> >> > Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
> > >>> >> >
> > >>> >> >
> > >>> >> > On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona <
> gkatona(a)gmail.com> wrote:
> > >>> >> >>
> > >>> >> >> Dear Billy,
> > >>> >> >>
> > >>> >> >> Thank you for this update and for your efforts! I found a
> solution and
> > >>> >> >> indeed most things already work in anaconda3. The steps I took
> (even
> > >>> >> >> if these instructions will have short expiry date):
> > >>> >> >>
> > >>> >> >> Modified .condarc with:
> > >>> >> >> channels:
> > >>> >> >> - cctbx
> > >>> >> >> - conda-forge
> > >>> >> >> - defaults
> > >>> >> >>
> > >>> >> >> run
> > >>> >> >> conda conda install cctbx_dependencies python=3.6
> > >>> >> >>
> > >>> >> >> I expect this will work without python=3.6 in the near future.
> > >>> >> >>
> > >>> >> >> Then compiling cctbx with anaconda3 python went without any
> problem
> > >>> >> >> when using these flags:
> > >>> >> >> python https://urldefense.proofpoint.com/v2/url?u=http-3A__bootstrap.py&d=DwICAg&c… --use-conda --python3 --nproc=12
> > >>> >> >>
> > >>> >> >> Then I probably did the unorthodox thing and sourced these
> directories:
> > >>> >> >>
> > >>> >> >> setenv LIBTBX_BUILD /home/gergely/cctbx/build
> > >>> >> >> setenv PATH ${PATH}:/home/gergely/cctbx/build/bin
> > >>> >> >> setenv PYTHONPATH
> > >>> >> >>
> /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH
> > >>> >> >> setenv LD_LIBRARY_PATH
> > >>> >> >>
> /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH
> > >>> >> >>
> > >>> >> >> There is probably a better way to put this into conda
> environment.
> > >>> >> >> With these steps I could run one of my scripts depending on
> cctbx. The
> > >>> >> >> only problem I found is that hdf5 library had conflict with
> another
> > >>> >> >> package in conda, but I does not find this as a showstopper.
> > >>> >> >>
> > >>> >> >> Best wishes,
> > >>> >> >>
> > >>> >> >> Gergely
> > >>> >> >>
> > >>> >> >>
> > >>> >> >>
> > >>> >> >>
> > >>> >> >>
> > >>> >> >>
> > >>> >> >> On Fri, Dec 13, 2019 at 7:23 PM Billy Poon <BKPoon(a)lbl.gov>
> wrote:
> > >>> >> >> >
> > >>> >> >> > Hi Gergely,
> > >>> >> >> >
> > >>> >> >> > It's still a work in progress. I'm sorting out some Windows
> issues right now. I can probably build a test package on a separate channel
> for people that want to test it (let's say next week?). I'll provide
> instructions, but basically, the test conda package will be in its own
> separate channel and the dependencies will be pulled from the conda-forge
> channel. I want most things to be working correctly on Python 2.7, 3.6,
> 3.7, and 3.8 on all 3 platforms.
> > >>> >> >> >
> > >>> >> >> > Thanks!
> > >>> >> >> >
> > >>> >> >> > --
> > >>> >> >> > Billy K. Poon
> > >>> >> >> > Research Scientist, Molecular Biophysics and Integrated
> Bioimaging
> > >>> >> >> > Lawrence Berkeley National Laboratory
> > >>> >> >> > 1 Cyclotron Road, M/S 33R0345
> > >>> >> >> > Berkeley, CA 94720
> > >>> >> >> > Tel: (510) 486-5709
> > >>> >> >> > Fax: (510) 486-5909
> > >>> >> >> > Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
> > >>> >> >> >
> > >>> >> >> >
> > >>> >> >> > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona <
> gkatona(a)gmail.com> wrote:
> > >>> >> >> >>
> > >>> >> >> >> Dear Billy,
> > >>> >> >> >>
> > >>> >> >> >> This sounds very promising and exciting. I am not sure if
> cctbx is
> > >>> >> >> >> already functional as a conda package in anaconda3 (Linux)
> or this is
> > >>> >> >> >> still work in progress. My technical expertise does not
> allow me to
> > >>> >> >> >> tell the difference. What I tried:
> > >>> >> >> >>
> > >>> >> >> >> Fresh install of anaconda3. Adding - cctbx and -
> conda-forge to
> > >>> >> >> >> .condarc . Installing pyside2 with conda. Running conda
> install
> > >>> >> >> >> conda_dependencies . I get a lot package version conflict,
> and I
> > >>> >> >> >> cannot import cctbx or iotbx to anaconda python. Am I
> following the
> > >>> >> >> >> right instructions? Or it is too early to expect that cctbx
> works when
> > >>> >> >> >> installed through conda?
> > >>> >> >> >>
> > >>> >> >> >> Best wishes,
> > >>> >> >> >>
> > >>> >> >> >> Gergely
> > >>> >> >> >>
> > >>> >> >> >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon <BKPoon(a)lbl.gov>
> wrote:
> > >>> >> >> >> >
> > >>> >> >> >> > Hi all,
> > >>> >> >> >> >
> > >>> >> >> >> > For a brief update, I have submitted a recipe for
> cctbxlite to conda-forge (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_conda-2Dfor… ) and support for
> Python 3.7 and 3.8 is being added (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_cctbx_cctbx… ). With some fixes for
> Windows (https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_cctbx_cctbx… ), all platforms
> (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8.
> Some additional changes will be needed to get Windows to work with Python 3
> and for tests to pass with Boost 1.70.0. That will enable the conda-forge
> recipe to build for all platforms and for all supported versions of Python.
> > >>> >> >> >> >
> > >>> >> >> >> > Currently, the conda-forge recipe will install into the
> conda python and cctbx imports can be done without sourcing the environment
> scripts. It looks like a lot of the environment variables being set in the
> dispatchers can be removed since the Python files and C++ extensions are in
> the right places. I'll update the libtbx_env file so that commands that
> load the environment can work correctly.
> > >>> >> >> >> >
> > >>> >> >> >> > --
> > >>> >> >> >> > Billy K. Poon
> > >>> >> >> >> > Research Scientist, Molecular Biophysics and Integrated
> Bioimaging
> > >>> >> >> >> > Lawrence Berkeley National Laboratory
> > >>> >> >> >> > 1 Cyclotron Road, M/S 33R0345
> > >>> >> >> >> > Berkeley, CA 94720
> > >>> >> >> >> > Tel: (510) 486-5709
> > >>> >> >> >> > Fax: (510) 486-5909
> > >>> >> >> >> > Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
> > >>> >> >> >> >
> > >>> >> >> >> >
> > >>> >> >> >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll <
> tic20(a)cam.ac.uk> wrote:
> > >>> >> >> >> >>
> > >>> >> >> >> >> Hi Luc,
> > >>> >> >> >> >>
> > >>> >> >> >> >> That sounds promising. From there, I?d need to work out
> how to make a fully-packaged installer (basically a modified wheel file)
> for the ChimeraX ToolShed - the aim is for the end user to not have to
> worry about any of this. That adds a couple of complications - e.g.
> $LIBTBX_BUILD would need to be set dynamically before first import - but
> doesn?t seem insurmountable.
> > >>> >> >> >> >>
> > >>> >> >> >> >> Thanks,
> > >>> >> >> >> >>
> > >>> >> >> >> >> Tristan
> > >>> >> >> >> >>
> > >>> >> >> >> >>
> > >>> >> >> >> >>
> > >>> >> >> >> >> Tristan Croll
> > >>> >> >> >> >> Research Fellow
> > >>> >> >> >> >> Cambridge Institute for Medical Research
> > >>> >> >> >> >> University of Cambridge CB2 0XY
> > >>> >> >> >> >>
> > >>> >> >> >> >>
> > >>> >> >> >> >>
> > >>> >> >> >> >>
> > >>> >> >> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis <
> luc_j_bourhis(a)mac.com> wrote:
> > >>> >> >> >> >> >
> > >>> >> >> >> >> > Hi Tristan,
> > >>> >> >> >> >> >
> > >>> >> >> >> >> > cctbx could be built to use your ChimeraX python, now
> that cctbx is moving to Python 3. The option ?with-python is there for that
> with the bootstrap script. The specific environment setup boil down to
> setting two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on
> Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a
> framework such as ChimeraX, that should not be difficult to ensure those
> two variables are set.
> > >>> >> >> >> >> >
> > >>> >> >> >> >> > Best wishes,
> > >>> >> >> >> >> >
> > >>> >> >> >> >> > Luc
> > >>> >> >> >> >> >
> > >>> >> >> >> >> >
> > >>> >> >> >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll <
> tic20(a)cam.ac.uk> wrote:
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> To add my two cents on this: probably the second-most
> common question I've had about ISOLDE's implementation is, "why didn't you
> use CCTBX?". The honest answer to that is, "I didn't know how."
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> Still don't, really - although the current
> developments are rather promising. The problem I've faced is that CCTBX was
> designed as its own self-contained Python (2.7, until very recently)
> environment, with its own interpreter and a lot of very specific
> environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is
> *also* its own self-contained Python (3.7) environment. To plug one into
> the other in that form... well, I don't think I'm a good enough programmer
> to really know where to start.
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> The move to Conda and a more modular CCTBX
> architecture should make a lot more possible in that direction. Pip would
> be even better for me personally (ChimeraX can install directly from the
> PyPI, but doesn't interact with Conda) - but I understand pretty well the
> substantial challenge that would amount to (not least being that the PyPI
> imposes a limit - around 100MB from memory? - on the size of an individual
> package).
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> Best regards,
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> Tristan
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote:
> > >>> >> >> >> >> >>> Hi Graeme,
> > >>> >> >> >> >> >>> Yes, I know. But ?black" is a program doing a very
> particular task
> > >>> >> >> >> >> >>> (code formatting from the top of my head). Requiring
> to use a wrapper
> > >>> >> >> >> >> >>> for python itself is another level. But ok, I think
> I am mellowing to
> > >>> >> >> >> >> >>> the idea after all! Talking with people around me,
> and extrapolating,
> > >>> >> >> >> >> >>> I would bet that, right now, a great majority of
> people interested by
> > >>> >> >> >> >> >>> cctbx in pip have already used the cctbx, so they
> know about the
> > >>> >> >> >> >> >>> Python wrapper, and they would not be too sanguine
> about that. My
> > >>> >> >> >> >> >>> concern is for the future, when pip will be the
> first time some people
> > >>> >> >> >> >> >>> use cctbx. Big fat warning notices on PyPI page and
> a better error
> > >>> >> >> >> >> >>> message when cctbx fails because LIBTBX_BUILD is not
> set would be
> > >>> >> >> >> >> >>> needed but that could be all right.
> > >>> >> >> >> >> >>> If we do a pip installer, we should aim at a minimal
> install: cctbx,
> > >>> >> >> >> >> >>> iotbx and their dependencies, and that?s it.
> > >>> >> >> >> >> >>> Best wishes,
> > >>> >> >> >> >> >>> Luc
> > >>> >> >> >> >> >>>> On 23 Aug 2019, at 07:17,
> Graeme.Winter(a)Diamond.ac.uk <Graeme.Winter(a)diamond.ac.uk> wrote:
> > >>> >> >> >> >> >>>> Without discussing the merits of this or whether we
> _choose_ to make the move to supporting PIP, I am certain it would be
> _possible_ - many other packages make dispatcher scripts when you pip
> install them e.g.
> > >>> >> >> >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat
> $(which black)
> > >>> >> >> >> >> >>>>
> /Library/Frameworks/Python.framework/Versions/3.6/bin/black
> > >>> >> >> >> >> >>>>
> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
> > >>> >> >> >> >> >>>> # -*- coding: utf-8 -*-
> > >>> >> >> >> >> >>>> import re
> > >>> >> >> >> >> >>>> import sys
> > >>> >> >> >> >> >>>> from black import main
> > >>> >> >> >> >> >>>> if __name__ == '__main__':
> > >>> >> >> >> >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$',
> '', sys.argv[0])
> > >>> >> >> >> >> >>>> sys.exit(main())
> > >>> >> >> >> >> >>>> So we _could_ work around the absence of
> LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is
> a different question, and it seems clear that here are very mixed opinions
> on this.
> > >>> >> >> >> >> >>>> Best wishes Graeme
> > >>> >> >> >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis <
> luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote:
> > >>> >> >> >> >> >>>> Hi,
> > >>> >> >> >> >> >>>> Even if we managed to ship our the boost dynamic
> libraries with pip, it would still not be pip-like, as we would still need
> our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip
> packages work with the standard python exe. LD_LIBRARY_PATH, we could get
> around that by changing the way we compile, using -Wl,-R, which is the
> runtime equivalent of build time -L. That?s a significant change that would
> need to be tested. But there is no way around setting LIBTBX_BUILD right
> now. Leaving that to the user is horrible. Perhaps there is a way to hack
> libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__env-5Fconfig.py&… so that we can hardwire LIBTBX_BUILD in there when pip
> installs?
> > >>> >> >> >> >> >>>> Best wishes,
> > >>> >> >> >> >> >>>> Luc
> > >>> >> >> >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis <
> luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote:
> > >>> >> >> >> >> >>>> Hi,
> > >>> >> >> >> >> >>>> I did look into that many years ago, and even toyed
> with building a pip installer. What stopped me is the exact conclusion you
> reached too: the user would not have the pip experience he expects. You are
> right that it is a lot of effort but is it worth it? Considering that
> remark, I don?t think so. Now, Conda was created specifically to go beyond
> pip pure-python-only support. Since cctbx has garnered support for Conda,
> the best avenue imho is to go the extra length to have a package on
> https://urldefense.proofpoint.com/v2/url?u=http-3A__Anaconda.org&d=DwICAg&c… <https://urldefense.proofpoint.com/v2/url?u=http-3A__anaconda.org_&d=DwICAg&… >, and then to advertise it hard to
> every potential user out there.
> > >>> >> >> >> >> >>>> Best wishes,
> > >>> >> >> >> >> >>>> Luc
> > >>> >> >> >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster <
> asbrewster(a)lbl.gov<mailto:[email protected]>> wrote:
> > >>> >> >> >> >> >>>> Hi, to avoid clouding Dorothee's documentation
> email thread, which I think is a highly useful enterprise, here's some
> thoughts about putting cctbx into pip. Pip doesn't install non-python
> dependencies well. I don't think boost is available as a package on pip
> (at least the package version we use). wxPython4 isn't portable through
> pip (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.wxpython.org_How-… ).
> MPI libraries are system dependent. If cctbx were a pure python package,
> pip would be fine, but cctbx is not.
> > >>> >> >> >> >> >>>> All that said, we could build a manylinux1 version
> of cctbx and upload it to PyPi (I'm just learning about this). For a pip
> package to be portable (which is a requirement for cctbx), it needs to
> conform to PEP513, the manylinux1 standard (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.python.org_dev_pep… ). For example, numpy is built
> according to this standard (see https://urldefense.proofpoint.com/v2/url?u=https-3A__pypi.org_project_numpy… ,
> where you'll see the manylinux1 wheel). Note, the manylinux1 standard is
> built with Centos 5.11 which we no longer support.
> > >>> >> >> >> >> >>>> There is also a manylinux2010 standard, which is
> based on Centos 6 (https://urldefense.proofpoint.com/v2/url?u=https-3A__www.python.org_dev_pep… ). This is
> likely a more attainable target (note though by default C++11 is not
> supported on Centos 6).
> > >>> >> >> >> >> >>>> If we built a manylinuxX version of cctbx and
> uploaded it to PyPi, the user would need all the non-python dependencies.
> There's no way to specify these in pip. For example, cctbx requires boost
> 1.63 or better. The user will need to have it in a place their python can
> find it, or we could package it ourselves and supply it, similar to how the
> pip h5py package now comes with an hd5f library, or how the pip numpy
> package includes an openblas library. We'd have to do the same for any
> packages we depend on that aren't on pip using the manylinux standards,
> such as wxPython4.
> > >>> >> >> >> >> >>>> Further, we need to think about how dials and other
> cctbx-based packages interact. If pip install cctbx is set up, how does
> pip install dials work, such that any dials shared libraries can find the
> cctbx libraries? Can shared libraries from one pip package link against
> libraries in another pip package? Would each package need to supply its
> own boost? Possibly this is well understood in the pip field, but not by
> me :)
> > >>> >> >> >> >> >>>> Finally, there's the option of providing a source
> pip package. This would require the full compiler toolchain for any given
> platform (macOS, linux, windows). These are likely available for
> developers, but not for general users.
> > >>> >> >> >> >> >>>> Anyway, these are some of the obstacles. Not
> saying it isn't possible, it's just a lot of effort.
> > >>> >> >> >> >> >>>> Thanks,
> > >>> >> >> >> >> >>>> -Aaron
> > >>> >> >> >> >> >>>> _______________________________________________
> > >>> >> >> >> >> >>>> cctbxbb mailing list
> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:
> cctbxbb(a)phenix-online.org>
> > >>> >> >> >> >> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> > >>> >> >> >> >> >>>> _______________________________________________
> > >>> >> >> >> >> >>>> cctbxbb mailing list
> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:
> cctbxbb(a)phenix-online.org>
> > >>> >> >> >> >> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> > >>> >> >> >> >> >>>> _______________________________________________
> > >>> >> >> >> >> >>>> cctbxbb mailing list
> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:
> cctbxbb(a)phenix-online.org>
> > >>> >> >> >> >> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> > >>> >> >> >> >> >>>> --
> > >>> >> >> >> >> >>>> This e-mail and any attachments may contain
> confidential, copyright and or privileged material, and are for the use of
> the intended addressee only. If you are not the intended addressee or an
> authorised recipient of the addressee please notify us of receipt by
> returning the e-mail and do not use, copy, retain, distribute or disclose
> the information in or attached to the e-mail.
> > >>> >> >> >> >> >>>> Any opinions expressed within this e-mail are those
> of the individual and not necessarily of Diamond Light Source Ltd.
> > >>> >> >> >> >> >>>> Diamond Light Source Ltd. cannot guarantee that
> this e-mail or any attachments are free from viruses and we cannot accept
> liability for any damage which you may sustain as a result of software
> viruses which may be transmitted in or with the message.
> > >>> >> >> >> >> >>>> Diamond Light Source Limited (company no. 4375679).
> Registered in England and Wales with its registered office at Diamond
> House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11
> 0DE, United Kingdom
> > >>> >> >> >> >> >>>> _______________________________________________
> > >>> >> >> >> >> >>>> cctbxbb mailing list
> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org
> > >>> >> >> >> >> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> > >>> >> >> >> >> >>> _______________________________________________
> > >>> >> >> >> >> >>> cctbxbb mailing list
> > >>> >> >> >> >> >>> cctbxbb(a)phenix-online.org
> > >>> >> >> >> >> >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> _______________________________________________
> > >>> >> >> >> >> >> cctbxbb mailing list
> > >>> >> >> >> >> >> cctbxbb(a)phenix-online.org
> > >>> >> >> >> >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> > >>> >> >> >> >> >
> > >>> >> >> >> >> >
> > >>> >> >> >> >> > _______________________________________________
> > >>> >> >> >> >> > cctbxbb mailing list
> > >>> >> >> >> >> > cctbxbb(a)phenix-online.org
> > >>> >> >> >> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> > >>> >> >> >> >>
> > >>> >> >> >> >>
> > >>> >> >> >> >> _______________________________________________
> > >>> >> >> >> >> cctbxbb mailing list
> > >>> >> >> >> >> cctbxbb(a)phenix-online.org
> > >>> >> >> >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> > >>> >> >> >> >
> > >>> >> >> >> > _______________________________________________
> > >>> >> >> >> > cctbxbb mailing list
> > >>> >> >> >> > cctbxbb(a)phenix-online.org
> > >>> >> >> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> > >>> >> >> >>
> > >>> >> >> >>
> > >>> >> >> >>
> > >>> >> >> >> --
> > >>> >> >> >> Gergely Katona, PhD
> > >>> >> >> >> Associate Professor
> > >>> >> >> >> Department of Chemistry and Molecular Biology, University
> of Gothenburg
> > >>> >> >> >> Box 462, 40530 G?teborg, Sweden
> > >>> >> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax:
> +46-31-786-3910
> > >>> >> >> >> Web: https://urldefense.proofpoint.com/v2/url?u=http-3A__katonalab.eu&d=DwICAg&c… , Email: gergely.katona(a)gu.se
> > >>> >> >> >>
> > >>> >> >> >> _______________________________________________
> > >>> >> >> >> cctbxbb mailing list
> > >>> >> >> >> cctbxbb(a)phenix-online.org
> > >>> >> >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> > >>> >> >> >
> > >>> >> >> > _______________________________________________
> > >>> >> >> > cctbxbb mailing list
> > >>> >> >> > cctbxbb(a)phenix-online.org
> > >>> >> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> > >>> >> >>
> > >>> >> >>
> > >>> >> >>
> > >>> >> >> --
> > >>> >> >> Gergely Katona, PhD
> > >>> >> >> Associate Professor
> > >>> >> >> Department of Chemistry and Molecular Biology, University of
> Gothenburg
> > >>> >> >> Box 462, 40530 G?teborg, Sweden
> > >>> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax:
> +46-31-786-3910
> > >>> >> >> Web: https://urldefense.proofpoint.com/v2/url?u=http-3A__katonalab.eu&d=DwICAg&c… , Email: gergely.katona(a)gu.se
> > >>> >> >>
> > >>> >> >> _______________________________________________
> > >>> >> >> cctbxbb mailing list
> > >>> >> >> cctbxbb(a)phenix-online.org
> > >>> >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> > >>> >> >
> > >>> >> > _______________________________________________
> > >>> >> > cctbxbb mailing list
> > >>> >> > cctbxbb(a)phenix-online.org
> > >>> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> > >>> >>
> > >>> >>
> > >>> >>
> > >>> >> --
> > >>> >> Gergely Katona, PhD
> > >>> >> Associate Professor
> > >>> >> Department of Chemistry and Molecular Biology, University of
> Gothenburg
> > >>> >> Box 462, 40530 G?teborg, Sweden
> > >>> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> > >>> >> Web: https://urldefense.proofpoint.com/v2/url?u=http-3A__katonalab.eu&d=DwICAg&c… , Email: gergely.katona(a)gu.se
> > >>> >>
> > >>> >> _______________________________________________
> > >>> >> cctbxbb mailing list
> > >>> >> cctbxbb(a)phenix-online.org
> > >>> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> > >>> >
> > >>> > _______________________________________________
> > >>> > cctbxbb mailing list
> > >>> > cctbxbb(a)phenix-online.org
> > >>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Gergely Katona, PhD
> > >>> Associate Professor
> > >>> Department of Chemistry and Molecular Biology, University of
> Gothenburg
> > >>> Box 462, 40530 G?teborg, Sweden
> > >>> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> > >>> Web: https://urldefense.proofpoint.com/v2/url?u=http-3A__katonalab.eu&d=DwICAg&c… , Email: gergely.katona(a)gu.se
> > >>>
> > >>> _______________________________________________
> > >>> cctbxbb mailing list
> > >>> cctbxbb(a)phenix-online.org
> > >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> > >
> > > _______________________________________________
> > > cctbxbb mailing list
> > > cctbxbb(a)phenix-online.org
> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> >
> >
> >
> > --
> > Gergely Katona, Professor
> > Department of Chemistry and Molecular Biology, University of Gothenburg
> > Box 462, 40530 G?teborg, Sweden
> > Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> > Web: https://urldefense.proofpoint.com/v2/url?u=http-3A__katonalab.eu&d=DwICAg&c… , Email: gergely.katona(a)gu.se
>
>
>
> --
> Gergely Katona, Professor
> Department of Chemistry and Molecular Biology, University of Gothenburg
> Box 462, 40530 G?teborg, Sweden
> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> Web: https://urldefense.proofpoint.com/v2/url?u=http-3A__katonalab.eu&d=DwICAg&c… , Email: gergely.katona(a)gu.se
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>
6 years, 1 month
Re: [phenixbb] phenix.map_to_model input mtz file failure --caution on using map_to_model with X-ray data
by Edward A. Berry
On 06/13/2017 07:44 PM, Dale Tronrud wrote:
>
> First we have to agree on exactly what we are talking about. I
> presumed we were talking about real space refinement against an
> experimental map such as one gets in cryo-EM. In that case there are no
> Fobs. Reciprocal space is a fiction and should be avoided.
>
> If you are working, instead, with a 2Fo-Fc map then just do
> reciprocal space refinement. I don't know of any reason to do
> whole-molecule real space refinement when you are working with crystal
> diffraction data. Reciprocal space is where the experiment lives and
> the analysis should be done there.
>
> In cases of model building it is computationally quicker to do a
> local real space refinement to touch up a model just so you can see if
> it looks reasonable before going back into reciprocal space. This real
> space refinement is quick-and-dirty and any flaws will be erased by the
> proper reciprocal space refinement that follows.
I have no argument with that! And I do realize how significant cryo-EM has
become to structural biology with the advent of direct electron detectors.
I was still stuck on the perhaps not so important theoretical question of
"to exclude or not to exclude" the free-R set in real-space refinement
prior to reciprocal space refinement.
> As for "neighborhood correlation" I was thinking of cryo-EM maps.
> Since the individual measurements (pictures) are real space in nature, I
> can't imagine an experimental error in the voltage of one voxel wouldn't
> tend to show up similarly in its neighbors. The whole group of voxels
> will be illuminated by electrons who all had very similar histories
> passing through the microscope.
>
> We have similar situations with diffraction data. A reflection whose
> neighbor is in a shadow has a much higher chance of being shadowed
> itself. Our spots, however, are much further apart on the detector than
> the voxels of an EM image.
>
> There is another type of correlation that is probably more important.
> Our diffraction spots are separated enough that you cannot predict the
> intensity of a reflection based on its neighbors. You can make a very
> good prediction of the darkness of a voxel based on its neighbors. If
> you leave out one voxel, as a test set member, you could easily deduce
> its hidden value without even building a molecular model - just
> interpolate. You can't do that with diffraction data.
>
> This means if you want to leave out a chunk of map data for a test
> set you have to pull out a big enough piece (many contiguous voxels)
> that you can't deduce anything about their opaqueness from the remaining
> image. To do this you have to know something about how the microscope
> works.
>
> Dale Tronrud
>
> On 6/13/2017 4:08 PM, Edward A. Berry wrote:
>>> To unbias you would have to calculate a
>>> new map with current Fcalc's for every iteration of the model, but this
>>> method would not take into account the neighborhood correlation present
>>> in experimental maps.)
>>
>> Thanks, Dale,
>> Could you explain this "neighborhood correlation"?
>>
>> My very simple (maybe too simple) understanding of how real space would
>> bias reflections is as follows:
>>
>> You make a map using phases (and Fc?) from the current model and Fobs.
>> But you omit the free set.
>>
>> Now if you take the fourier transform of that unmodified map, you would
>> get back
>> exactly the coefficients you put in: 2Fo-Fc (?) for the working
>> reflections,
>> and zero for the free set.
>>
>> Then you make modifications to the model to make its density match as
>> nearly
>> as possible the density of the map. If you were able to make the density of
>> the model exactly match that of the map, then the Fc for the model would
>> be that of the map.
>>
>> Of course you can never make the density of the model exactly match that of
>> the map - modelization is the severest form of density modification.
>> But, to the extent that you make the model's density more nearly like that
>> of the map, the Fourier transform of the new model will be more
>> like that of the map.
>>
>> That means for the working reflections, Fc will get closer to 2Fo-Fc
>> which brings them closer to Fo; and the R-work improves.
>> (If there is error in the Fobs, that will be reflected in the map,
>> and the Fcalc of the model will tend toward these eromeous Fobs
>> (fitting the error) and Rwork will get better than it should (bias).)
>> Free reflections will move closer to zero, and most likely Rfree
>> will get worse.
>>
>> I think that's all consistent with what you wrote, but then
>> I had the impression that the bias could be prevented by making the
>> map with Fc for the test set (proposed in an old paper by Ivan
>> Rayment. That way the free reflections get are following the
>> process by their coupling to neighboring reflections in reciprocal
>> space (neighborhood correlation?), the same way they do in reciprocal
>> space refinement, rather than the Fobs being used. The information
>> in these free Fcalc is coming from the neighboring working reflections
>> due to redundancy of information in a finely sampled molecular transform.
>>
>> Ed
>>
>>
>>
>> On 06/13/2017 05:40 PM, Dale Tronrud wrote:
>>>
>>>
>>> On 6/13/2017 12:30 PM, Edward A. Berry wrote:
>>>> Thanks, Pavel,
>>>> I really appreciate your taking the time to generate the example.
>>>>
>>>> While I agree with Tim and Ian that refinement to convergence should
>>>> remove the bias making it perhaps not a serious problem, my question was
>>>> in fact whether there is any bias immediately after the refinement.
>>>>
>>>> I will need to study this example a bit, but one thing I notice is
>>>> that you are doing exactly what I was guessing, comparing Rfree
>>>> after real-space refinement with and without using the free set.
>>>> Then, I still think, we
>>>>>>> have to think about how much of that difference results from
>>>>>>> bias towards the observed values (when the reflections are included)
>>>>>>> and
>>>>>>> how much is from bias towards zero (when the free set is excluded).
>>>>
>>> Of course the model is refined as though the test set Fourier
>>> components were equal to zero. In reciprocal space refinement when you
>>> leave a reflection out of the "sum over all reflections" when
>>> calculating the difference map you are saying that you have no opinion
>>> about the amplitude of that reflection. When you calculate a real space
>>> map from Fourier coefficients you can't not have an opinion, i.e. you
>>> can't leave a term out of the sum you can only set that term to zero.
>>> If your model produces a prediction for that term which is not equal to
>>> zero it will be penalized. (If you set that term to Fcalc you tie your
>>> model to its starting point. To unbias you would have to calculate a
>>> new map with current Fcalc's for every iteration of the model, but this
>>> method would not take into account the neighborhood correlation present
>>> in experimental maps.)
>>>
>>> What this means is that Rfree is not a meaningful stat for assessing
>>> overfitting of real space refinement. This is hardly a surprise. A
>>> test of a refinement protocol has to be based on the mathematics of that
>>> protocol, not the protocol you happened to have used yesterday. If you
>>> want an unbiased estimate of the quality of a real space refinement you
>>> have to leave out a region of the map and then see how well the model
>>> fits that region. This is harder to do in an automated fashion and
>>> there will be a lot of caveats about your results (e.g. you know about
>>> the ability to fit one region but does that generalize to other areas?).
>>> If you recall there are a lot of caveats about Rfree too - we have just
>>> stopped worrying about them. (e.g. low resolution vrs high resolution
>>> reflections, choosing based on shells or randomly, what to do about
>>> ncs...)
>>>
>>> I think you should consider yourself on the wrong track if you come
>>> up with a statistical test, but haven't given any thought to the actual
>>> experiment that produced your map.
>>>
>>> Dale Tronrud
>>>
>>>> Things I need to look at-
>>>> What are R and R-free for the original refined model
>>>> What are R and R-free after shaking (did RSR lower R but not Rfree, or
>>>> did it raise Rfree?
>>>> What if RSR is done using a map made with fill-in strategy?
>>>>
>>>> Ed
>>>>
>>>> On 06/13/2017 02:15 PM, Pavel Afonine wrote:
>>>>> Hi Ed,
>>>>>
>>>>> Including free-r reflections into map calculation and then using such
>>>>> map in real-space refinement of entire model will affect Rfree. Here
>>>>> is a simple example that illustrates my statement, step-by-step:
>>>>>
>>>>> 1) Get data and model from PDB:
>>>>>
>>>>> phenix.fetch_pdb 1f8t --mtz
>>>>>
>>>>> 2) Compute two 2mFo-DFc maps: one includes all reflections the other
>>>>> one has no free-r terms:
>>>>>
>>>>> phenix.python run.py 1f8t.{pdb,mtz}
>>>>>
>>>>> This will create an MTZ file (map_coeffs.mtz) that contains Fourier
>>>>> map coefficients for both maps.
>>>>>
>>>>> 3) Shake model a bit:
>>>>>
>>>>> phenix.dynamics 1f8t.pdb number_of_steps=500
>>>>>
>>>>> 4) Run real-space refinement using two maps:
>>>>>
>>>>> phenix.real_space_refine map_coeffs.mtz 1f8t_shaken.pdb
>>>>> label="work,PHIwork" ncs_constraints=false output.file_name_prefix=work
>>>>>
>>>>> phenix.real_space_refine map_coeffs.mtz 1f8t_shaken.pdb
>>>>> label="all,PHIall" ncs_constraints=false output.file_name_prefix=all
>>>>>
>>>>> 5) Compute R-factors using data and real-space refined models:
>>>>>
>>>>> phenix.model_vs_data 1f8t.mtz all_real_space_refined.pdb
>>>>> r_work(re-computed) : 0.2419
>>>>> r_free(re-computed) : 0.2441
>>>>>
>>>>> phenix.model_vs_data 1f8t.mtz work_real_space_refined.pdb
>>>>> r_work(re-computed) : 0.2444
>>>>> r_free(re-computed) : 0.2756
>>>>>
>>>>> The result is self-explicable and is inline with Tom's reply to Wei.
>>>>>
>>>>> All files necessary to reproduce calculations above are here:
>>>>> http://cci.lbl.gov/~afonine/tmp/
>>>>>
>>>>> All the best,
>>>>> Pavel
>>>>>
>>>>>
>>>>> On 6/8/17 10:05, Tim Gruene wrote:
>>>>>> Hi Ed,
>>>>>>
>>>>>> including the 'free' reflections in the map for modelling does not
>>>>>> taint the
>>>>>> value of Rfree. That is a misconception that i s very persistent (as
>>>>>> prejudice
>>>>>> usually are). I believe it was Ian Tickle who formulated that when
>>>>>> you simply
>>>>>> refine long enough towards convergence, all reflections excluded from
>>>>>> refinement
>>>>>> will become independent, i.e. you can assign a new set for Rfree
>>>>>> every time
>>>>>> you refine, if you wish so.
>>>>>>
>>>>>> This concept is the reason why Rcomplete (the "better" equivalent to
>>>>>> Rfree for
>>>>>> small data sets with < 10,000 unique reflections), introduced by Axel
>>>>>> Brunger,
>>>>>> works, as we could demonstrate in doi: 10.1073/pnas.1502136112
>>>>>>
>>>>>> So nothing to worry about when including all reflections in map
>>>>>> calculations.
>>>>>>
>>>>>> Cheers,
>>>>>> Tim
>>>>>>
>>>>>> On Thursday, June 8, 2017 12:42:53 PM CEST Edward A. Berry wrote:
>>>>>>> Hi, Tom,
>>>>>>> Please forgive what may be a silly question from an outsider who
>>>>>>> hasn't
>>>>>>> really kept up with the crystallography literature or even all the
>>>>>>> Phenix
>>>>>>> newsletters- What is the evidence that including the free set in
>>>>>>> real space
>>>>>>> refinement biases R-free of the resulting model? Is this Rfree also
>>>>>>> biased
>>>>>>> when map coefficients use "fill-in" for the excluded free
>>>>>>> reflections (and
>>>>>>> is that what phenix.remove_free_from_map does?).
>>>>>>>
>>>>>>> My point is that literally excluding the free reflections, as
>>>>>>> opposed to
>>>>>>> substituting their values with Fc, will bias the free set toward
>>>>>>> grossly
>>>>>>> incorrect values (namely zero) and therefore greatly worsen R-free.
>>>>>>> Thus if
>>>>>>> the evidence for bias is that you get worse R-free when you
>>>>>>> exclude the
>>>>>>> free set, you have to think about how much of that difference
>>>>>>> results from
>>>>>>> bias towards the observed values (when the reflections are included)
>>>>>>> and
>>>>>>> how much is from bias towards zero (when the free set is excluded).
>>>>>>> (Again, I realize this may be all very well understood by the
>>>>>>> crystallography community and properly taken care of in phenix; I'm
>>>>>>> just
>>>>>>> asking for my own information) eab
>>>>>>>
>>>>>>> On 06/08/2017 07:28 AM, Terwilliger, Thomas Charles wrote:
>>>>>>>> Hi Wei,
>>>>>>>>
>>>>>>>>
>>>>>>>> I want to give a word of caution about how to use
>>>>>>>> phenix.map_to_model on
>>>>>>>> crystallographic data...The bottom line is you should remove the
>>>>>>>> test set
>>>>>>>> from your map coefficients before running phenix.map_to model on
>>>>>>>> X-ray
>>>>>>>> data. Here is why:
>>>>>>>>
>>>>>>>>
>>>>>>>> phenix.map_to_model uses real-space refinement, which is refinement
>>>>>>>> against the map. If you supply map coefficients that include your
>>>>>>>> test
>>>>>>>> reflections, then you will be refining against data that is in your
>>>>>>>> test
>>>>>>>> set. This will make your Rfree invalid when you go back and
>>>>>>>> refine your
>>>>>>>> model against the original crystallographic data.
>>>>>>>>
>>>>>>>>
>>>>>>>> To remove the test set from your map coefficients you can use:
>>>>>>>>
>>>>>>>>
>>>>>>>> phenix.remove_free_from_map map_coeffs=my_map_coeffs.mtz
>>>>>>>> free_in=my_data_file_with_freeR_flags.mtz
>>>>>>>> mtz_out=my_map_coeffs_no_free.mtz
>>>>>>>>
>>>>>>>>
>>>>>>>> Also note that phenix.map_to_model uses a fixed map (it does not do
>>>>>>>> density modification). Consequently for most crystallographic
>>>>>>>> data at
>>>>>>>> moderate resolution or higher phenix.autobuild is going to do much
>>>>>>>> better
>>>>>>>> than phenix.map_to_model.
>>>>>>>>
>>>>>>>>
>>>>>>>> All the best,
>>>>>>>>
>>>>>>>> Tom T
>>>>>>>>
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------- *From:*[email protected]
>>>>>>>> <dingding830106(a)163.com> on behalf ofdancingdream(a)163.com
>>>>>>>> <dancingdream(a)163.com> *Sent:* Tuesday, June 6, 2017 9:16 PM
>>>>>>>> *To:* Terwilliger, Thomas Charles
>>>>>>>> *Cc:*[email protected]
>>>>>>>> *Subject:* Re:Re: [phenixbb] phenix.map_to_model input mtz file
>>>>>>>> failure
>>>>>>>> Dear Thomas,
>>>>>>>> I use CAD to convert the labels from FDM->FWT, PHIDM->PHFWT, then
>>>>>>>> submit
>>>>>>>> this job again (without map_coeffs_labels=... ), and everything
>>>>>>>> seems ok.
>>>>>>>> Thank you very much for you help.
>>>>>>>> Best!
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Wei Ding
>>>>>>>> P.O.Box 603
>>>>>>>> The Institute of Physics,Chinese Academy of Sciences
>>>>>>>> Beijing,China
>>>>>>>> 100190
>>>>>>>> Tel: +86-10-82649083
>>>>>>>>
>>>>>>>> E-mail:[email protected] <mailto:[email protected]>
>>>>>>>>
>>>>>>>> At 2017-06-07 10:32:14, "Terwilliger, Thomas Charles"
>>>>>> <terwilliger(a)lanl.gov> wrote:
>>>>>>>> Hi Wei,
>>>>>>>>
>>>>>>>>
>>>>>>>> I'm sorry for the trouble!
>>>>>>>>
>>>>>>>>
>>>>>>>> If you supply an MTZ file that has FWT,PHFWT or similar
>>>>>>>> labels, then
>>>>>>>> you can skip the "labels=...." statement and it should run.
>>>>>>>>
>>>>>>>>
>>>>>>>> Let me know if that does not work!
>>>>>>>> All the best,
>>>>>>>>
>>>>>>>> Tom T
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>>
>>>>>>>> ---------- *From:*[email protected]
>>>>>>>> <mailto:[email protected]>
>>>>>>>> <phenixbb-bounces(a)phenix-online.org
>>>>>>>> <mailto:[email protected]>> on behalf of
>>>>>>>> dancingdream(a)163.com <mailto:[email protected]>
>>>>>>>> <dancingdream(a)163.com <mailto:[email protected]>> *Sent:*
>>>>>>>> Tuesday,
>>>>>>>> June 6, 2017 8:19 PM
>>>>>>>> *To:*[email protected]
>>>>>>>> <mailto:[email protected]>
>>>>>>>> *Subject:* [phenixbb] phenix.map_to_model input mtz file
>>>>>>>> failure
>>>>>>>> Dear Phenix bb,
>>>>>>>> I intend to build a initial model by phenix.map_to_model.
>>>>>>>> And the
>>>>>>>> command line is as follows: phenix.map_to_model_1.12rc0-2787
>>>>>>>> map_coeffs_file=../rep_dm.mtz map_coeffs_labels="'FP,SIGFP'
>>>>>>>> 'PHIDM'
>>>>>>>> 'FOMDM'" seq_file=../resolve.seq is_crystal=True
>>>>>>>> use_sg_symmetry=True density_select=False
>>>>>>>> truncate_at_d_min=True
>>>>>>>> and the feedback like this:
>>>>>>>> Sorry: No initial assignment made for map_coeffs. Labels used:
>>>>>>>> FP,SIGFP PHIDM FOMDM. Available labels: ['PHIB', 'FOM',
>>>>>>>> 'HLA,HLB,HLC,HLD', 'FP,SIGFP', 'PHIDM', 'FOMDM', 'FDM',
>>>>>>>> 'HLADM,HLBDM,HLCDM,HLDDM'] NOTE: grouped labels like
>>>>>>>> 'FP,SIGFP' must
>>>>>>>> stay together,
>>>>>>>> have commas, and have no spaces. If they come from an MTZ
>>>>>>>> file,
>>>>>>>> they must be in adjacent columns as well.
>>>>>>>> Suggested labels to use: PHIDM FOMDM
>>>>>>>> I try many other input format of map_coeffs_labels, such as
>>>>>>>> map_coeffs_labels="FP,SIGFP PHIDM FOMDM"
>>>>>>>> map_coeffs_labels=["FP,SIGFP PHIDM FOMDM"]
>>>>>>>> ... ...
>>>>>>>> but the result is the same. Dose anyone can tell me how to fix
>>>>>>>> this
>>>>>>>> problem? Thank a lot.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Wei Ding
>>>>>>>> P.O.Box 603
>>>>>>>> The Institute of Physics,Chinese Academy of Sciences
>>>>>>>> Beijing,China
>>>>>>>> 100190
>>>>>>>> Tel: +86-10-82649083
>>>>>>>> E-mail:[email protected] <mailto:[email protected]>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> phenixbb mailing list
>>>>>>>> phenixbb(a)phenix-online.org
>>>>>>>> http://phenix-online.org/mailman/listinfo/phenixbb
>>>>>>>> Unsubscribe:[email protected]
>>>>>>> _______________________________________________
>>>>>>> phenixbb mailing list
>>>>>>> phenixbb(a)phenix-online.org
>>>>>>> http://phenix-online.org/mailman/listinfo/phenixbb
>>>>>>> Unsubscribe:[email protected]
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> phenixbb mailing list
>>>>>> phenixbb(a)phenix-online.org
>>>>>> http://phenix-online.org/mailman/listinfo/phenixbb
>>>>>> Unsubscribe:[email protected]
>>>>>
>>>> _______________________________________________
>>>> phenixbb mailing list
>>>> phenixbb(a)phenix-online.org
>>>> http://phenix-online.org/mailman/listinfo/phenixbb
>>>> Unsubscribe: phenixbb-leave(a)phenix-online.org
>>> _______________________________________________
>>> phenixbb mailing list
>>> phenixbb(a)phenix-online.org
>>> http://phenix-online.org/mailman/listinfo/phenixbb
>>> Unsubscribe: phenixbb-leave(a)phenix-online.org
>>>
>>
>
8 years, 7 months
Re: [cctbxbb] some thoughts on cctbx and pip
by Gergely Katona
Hi Billy,
Here come the package lists from conda. I will do more testing with
the cleaned condas installation. Indeed, the boost packages came from
the conda-forge channel but for some odd reason in the test
environment not in the base.
Best wishes,
Gergely
On Fri, Jan 3, 2020 at 9:05 PM Billy Poon <BKPoon(a)lbl.gov> wrote:
>
> Hi Gergely,
>
> Happy New Year!
>
> Can you provide the output to "conda list" for each environment?
>
> I suspect that the old version of cctbx might still be around. Is there a file named PYTHON_VERSION_MAJOR_MINOR in "/home/gergely/anaconda3/lib"? If so, delete that file. To make sure you are downloading a new copy of the package, clear the cache where conda keeps copies of all the packages. To do that, you can run "conda clean -y --all". You can remove the "-y" flag if you want to see what will be deleted. The hash for the new linux package should be h8b68381_0.
>
> The last error looks like an incompatible version of boost or gcc library was installed. The output from "conda list" should show the boost and boost-cpp packages coming from conda-forge. I would avoid installing into the root or base environment because that usually pulls from the default channels. All the dependencies should be pulled from the conda-forge channel.
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709
> Fax: (510) 486-5909
> Web: https://phenix-online.org
>
>
> On Fri, Jan 3, 2020 at 2:20 PM Gergely Katona <gkatona(a)gmail.com> wrote:
>>
>> I found some other errors when using the test environment.
>>
>>
>> ---------------------------------------------------------------------------
>> RuntimeError Traceback (most recent call last)
>> <ipython-input-1-fe308b345b3f> in <module>
>> 10 import scipy as sp
>> 11 from scipy import linalg
>> ---> 12 import iotbx.pdb
>> 13 import cctbx
>> 14 import pandas as pd
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/__init__.py
>> in <module>
>> 8
>> 9 import iotbx.pdb.records
>> ---> 10 import iotbx.pdb.hierarchy
>> 11 from scitbx import matrix
>> 12
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/hierarchy.py
>> in <module>
>> 14 from six.moves import cStringIO as StringIO
>> 15 from iotbx.pdb import hy36encode, hy36decode
>> ---> 16 import iotbx.cif.model
>> 17 from cctbx import crystal
>> 18 from libtbx import group_args
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/cif/__init__.py
>> in <module>
>> 19
>> 20 from cctbx.array_family import flex
>> ---> 21 from cctbx import miller
>> 22 from iotbx.cif import model, builders, geometry
>> 23 from libtbx.containers import OrderedDict
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/miller/__init__.py
>> in <module>
>> 11
>> 12 from cctbx import crystal
>> ---> 13 from cctbx import maptbx
>> 14 from cctbx import sgtbx
>> 15 from cctbx.sgtbx import lattice_symmetry
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/maptbx/__init__.py
>> in <module>
>> 15 from libtbx import adopt_init_args
>> 16 from libtbx.utils import Sorry
>> ---> 17 import libtbx.load_env
>> 18 import math
>> 19 import sys, os
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/load_env.py in <module>
>> 3 import libtbx.env_config
>> 4 import os
>> ----> 5 libtbx.env = libtbx.env_config.unpickle()
>> 6 libtbx.env.set_os_environ_all_dist()
>> 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME")
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/env_config.py
>> in unpickle()
>> 2736 env = pickle.load(libtbx_env)
>> 2737 if (env.python_version_major_minor != sys.version_info[:2]):
>> -> 2738 env.raise_python_version_incompatible()
>> 2739 if (op.realpath(build_path) != op.realpath(abs(env.build_path))):
>> 2740 env.build_path.reset(build_path)
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/env_config.py
>> in raise_python_version_incompatible(self, prev_pvmm)
>> 469 if (prev_pvmm is None):
>> 470 prev_pvmm = "%d.%d" % self.python_version_major_minor
>> --> 471 raise RuntimeError("Python version incompatible with this build:\n"
>> 472 + " Build directory: %s\n" % show_string(abs(self.build_path))
>> 473 + " Python version used initially: %s\n" % prev_pvmm
>>
>> RuntimeError: Python version incompatible with this build:
>> Build directory: "/home/gergely/anaconda3"
>> Python version used initially: 3.6
>> Python version in use now: 3.8
>>
>>
>> Interestingly in the base environment all import worked, but I got an error at:
>>
>>
>> ArgumentError Traceback (most recent call last)
>> <ipython-input-4-5bc71c36e121> in <module>
>> 33 atm_odd.set_b(0)
>> 34
>> ---> 35 odd.write_pdb_file("bdiff_in_even_000.pdb") #skriver ny pdb
>> med dessa inkluderade
>> 36 dfWBvals=pd.DataFrame(rows_list)
>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in
>> write_pdb_file(self, file_name, open_append, crystal_symmetry,
>> cryst1_z, write_scale_records, append_end, atom_hetatm, sigatm,
>> anisou, siguij)
>> 930 siguij=True):
>> 931 if (crystal_symmetry is Auto):
>> --> 932 crystal_symmetry = self.crystal_symmetry()
>> 933 if (cryst1_z is Auto):
>> 934 cryst1_z = self.extract_cryst1_z_columns()
>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in
>> crystal_symmetry(self, crystal_symmetry, weak_symmetry)
>> 1218 crystal_symmetry=None,
>> 1219 weak_symmetry=False):
>> -> 1220 self_symmetry = self.crystal_symmetry_from_cryst1()
>> 1221 if (self_symmetry is None):
>> 1222 self_symmetry = self._crystal_symmetry_from_cns_remark_sg()
>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in
>> crystal_symmetry_from_cryst1(self)
>> 1195 for line in self.crystallographic_section():
>> 1196 if (line.startswith("CRYST1")):
>> -> 1197 return
>> cryst1_interpretation.crystal_symmetry(cryst1_record=line)
>> 1198 return None
>> 1199
>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/cryst1_interpretation.py
>> in crystal_symmetry(cryst1_record)
>> 136 space_group_info=None)
>> 137 space_group_info = categorize(cryst1_record.sgroup).space_group_info(
>> --> 138 unit_cell=u)
>> 139 return crystal.symmetry(unit_cell=u,
>> space_group_info=space_group_info)
>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/cryst1_interpretation.py
>> in space_group_info(self, unit_cell)
>> 85 if (self.symbol is None): return None
>> 86 if (self.category is None):
>> ---> 87 try: return sgtbx.space_group_info(self.symbol)
>> 88 except RuntimeError: return None
>> 89 if (isinstance(unit_cell, uctbx.ext.unit_cell)):
>>
>> ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in
>> __init__(self, symbol, table_id, group, number, space_group_t_den)
>> 100 assert group is None
>> 101 if (table_id is None):
>> --> 102 symbols = space_group_symbols(symbol)
>> 103 else:
>> 104 if (isinstance(symbol, int)): symbol = str(symbol)
>>
>> ArgumentError: Python argument types in
>> space_group_symbols.__init__(space_group_symbols, str)
>> did not match C++ signature:
>> __init__(_object*, int space_group_number)
>> __init__(_object*, int space_group_number,
>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> std::allocator<char> > extension='')
>> __init__(_object*, int space_group_number,
>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> std::allocator<char> > extension='', std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> > table_id='')
>> __init__(_object*, std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> > symbol)
>> __init__(_object*, std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> > symbol,
>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> std::allocator<char> > table_id='')
>>
>>
>> Gergely
>>
>>
>>
>> On Fri, Jan 3, 2020 at 1:44 PM Gergely Katona <gkatona(a)gmail.com> wrote:
>> >
>> > Dear Billy,
>> >
>> > I did some tests installing in my base conda was not troublefree in
>> > python 3.8. Running iotbx.fetch_pdb --all --mtz 1kp8 resulted in boost
>> > related error when converting to mtz. Installing into a test
>> > environment went fine with iotbx.fetch_pdb --all --mtz 1kp8 running
>> > successfully. I could add pymc3, jupyter and seaborn and iotbx still
>> > did not break. My LIBTBX_BUILD is not set however in the test
>> > environment, which breaks miller when I was trying to import it.
>> > Manually setting the LIBTBX_BUILD to CONDA_PREFIX (setenv LIBTBX_BUILD
>> > $CONDA_PREFIX) fixes this and one of my script works fine! I will come
>> > back if I find some problems with my other scripts, but so far this
>> > looks promising!
>> >
>> >
>> > Happy new year!
>> >
>> > Gergely
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Dec 20, 2019 at 8:16 PM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >
>> > > Hi everyone,
>> > >
>> > > I updated the conda packages on the cctbx-dev channel and there are also now packages for macOS. The old packages have been removed so you can create a new environment with
>> > >
>> > > conda create -n test -c cctbx-dev cctbx python=2.7
>> > > conda activate test
>> > >
>> > > where "test" is the environment name and 2.7 is the python version. For the python version, you can also pick 3.6, 3.7, or 3.8. I think the latest conda will install 3.8 as the default python if the python argument is not provided. Or install into an existing environment (with python already installed) with the same command as before,
>> > >
>> > > conda install -c cctbx-dev cctbx
>> > >
>> > > Again, your .condarc file should have conda-forge as the first channel so that dependencies can be pulled correctly.
>> > >
>> > > The dispatchers should work now and $LIBTBX_BUILD will be set on activation of an environment (and unset on environment deactivation) or after installing into an existing environment, so you can do more interesting things like
>> > >
>> > > iotbx.fetch_pdb --all --mtz 1kp8
>> > >
>> > > However, this package does not install databases, so
>> > >
>> > > molprobity.molprobity 1kp8.pdb 1kp8.mtz
>> > >
>> > > will fail because the database is not available (also probe and reduce are not built).
>> > >
>> > > Some known issues in no particular order,
>> > >
>> > > 1) You can run "libtbx.run_tests_parallel module=cctbx nproc=Auto" to run tests, but not all the test files were copied, so I got 30 failures here. The final package will probably not have the test files.
>> > > 2) The other python dispatchers, like "cctbx.python", were not updated, so those will fail. But you can just run "python" in the active environment.
>> > > 3) Parts of mmtbx still need to be updated for Python 3.
>> > > 4) The LIBTBX_BUILD variable should be set for sh and csh shells. Let me know if it's not working for your shell (i.e. "echo $LIBTBX_BUILD" should show the same thing as $CONDA_PREFIX).
>> > > 5) One future step is to enable building other CCTBX-like modules with this conda package. This will require some bookkeeping updates so that modules in the conda environment and the modules being built are tracked properly. Also, making sure that all the necessary headers exist in $PREFIX/include.
>> > >
>> > > Gergely, let me know if your scripts are working.
>> > >
>> > > Happy holidays!
>> > >
>> > > --
>> > > Billy K. Poon
>> > > Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > > Lawrence Berkeley National Laboratory
>> > > 1 Cyclotron Road, M/S 33R0345
>> > > Berkeley, CA 94720
>> > > Tel: (510) 486-5709
>> > > Fax: (510) 486-5909
>> > > Web: https://phenix-online.org
>> > >
>> > >
>> > > On Wed, Dec 18, 2019 at 11:25 AM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >>
>> > >> Hi Gergely,
>> > >>
>> > >> That's the other information that hasn't been updated yet in the conda package. :)
>> > >>
>> > >> For LIBTBX_BUILD, I can set up environment variables sort of how the conda gcc package sets up environment variables. I have to double check to see if they're set after installation of the package and after activation of an environment. The variable will just be a copy of $CONDA_PREFIX.
>> > >>
>> > >> The libtbx_env file is a file that exists in $LIBTBX_BUILD that basically stores configuration information. I'll have to modify the contents to update the locations of the different modules since the conda package will not have a "modules" directory.
>> > >>
>> > >> --
>> > >> Billy K. Poon
>> > >> Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > >> Lawrence Berkeley National Laboratory
>> > >> 1 Cyclotron Road, M/S 33R0345
>> > >> Berkeley, CA 94720
>> > >> Tel: (510) 486-5709
>> > >> Fax: (510) 486-5909
>> > >> Web: https://phenix-online.org
>> > >>
>> > >>
>> > >> On Wed, Dec 18, 2019 at 2:08 AM Gergely Katona <gkatona(a)gmail.com> wrote:
>> > >>>
>> > >>> Hi Billy,
>> > >>>
>> > >>> Conda install went fine with your instructions after rearranging the
>> > >>> channels by putting cctbx last. I removed all environmental variables
>> > >>> and previous build of cctbx.
>> > >>>
>> > >>> import sys
>> > >>> print (sys.path, sys.prefix)
>> > >>>
>> > >>> ['/home/gergely/anaconda3/lib/python36.zip',
>> > >>> '/home/gergely/anaconda3/lib/python3.6',
>> > >>> '/home/gergely/anaconda3/lib/python3.6/lib-dynload', '',
>> > >>> '/home/gergely/anaconda3/lib/python3.6/site-packages',
>> > >>> '/home/gergely/anaconda3/lib/python3.6/site-packages/IPython/extensions',
>> > >>> '/home/gergely/.ipython'] /home/gergely/anaconda3
>> > >>>
>> > >>> Many imports went fine including pymc3, but I encountered problems
>> > >>> with these three:
>> > >>>
>> > >>> from cctbx import miller
>> > >>> import iotbx.pdb
>> > >>> from iotbx import reflection_file_reader, mtz
>> > >>>
>> > >>>
>> > >>> I also got type error when handling space groups.
>> > >>>
>> > >>>
>> > >>>
>> > >>> ---------------------------------------------------------------------------
>> > >>> KeyError Traceback (most recent call last)
>> > >>> <ipython-input-5-cedbd4f84f3d> in <module>
>> > >>> 25 from cctbx import uctbx
>> > >>> 26 from cctbx import sgtbx
>> > >>> ---> 27 from cctbx import miller
>> > >>> 28 #from iotbx import reflection_file_reader, mtz
>> > >>> 29 sns.set_context("poster")
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/miller/__init__.py in <module>
>> > >>> 11
>> > >>> 12 from cctbx import crystal
>> > >>> ---> 13 from cctbx import maptbx
>> > >>> 14 from cctbx import sgtbx
>> > >>> 15 from cctbx.sgtbx import lattice_symmetry
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/maptbx/__init__.py in <module>
>> > >>> 15 from libtbx import adopt_init_args
>> > >>> 16 from libtbx.utils import Sorry
>> > >>> ---> 17 import libtbx.load_env
>> > >>> 18 import math
>> > >>> 19 import sys, os
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module>
>> > >>> 3 import libtbx.env_config
>> > >>> 4 import os
>> > >>> ----> 5 libtbx.env = libtbx.env_config.unpickle()
>> > >>> 6 libtbx.env.set_os_environ_all_dist()
>> > >>> 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME")
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle()
>> > >>> 2603
>> > >>> 2604 def unpickle():
>> > >>> -> 2605 build_path = os.environ["LIBTBX_BUILD"]
>> > >>> 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path)
>> > >>> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/os.py in __getitem__(self, key)
>> > >>> 667 except KeyError:
>> > >>> 668 # raise KeyError with the original key value
>> > >>> --> 669 raise KeyError(key) from None
>> > >>> 670 return self.decodevalue(value)
>> > >>> 671
>> > >>>
>> > >>> KeyError: 'LIBTBX_BUILD'
>> > >>>
>> > >>>
>> > >>>
>> > >>> ---------------------------------------------------------------------------
>> > >>> ArgumentError Traceback (most recent call last)
>> > >>> <ipython-input-7-2acd3a9ce26a> in <module>
>> > >>> 26 return ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar
>> > >>> 27
>> > >>> ---> 28 ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar=initializecrystal()
>> > >>>
>> > >>> <ipython-input-7-2acd3a9ce26a> in initializecrystal()
>> > >>> 11 uc = uctbx.unit_cell(unit_cell)
>> > >>> 12 wavelength = 1.54980
>> > >>> ---> 13 xtal_symm = crystal.symmetry(unit_cell=unit_cell,
>> > >>> space_group_symbol="P 43 21 2")
>> > >>> 14
>> > >>> 15 ms =
>> > >>> miller.build_set(crystal_symmetry=xtal_symm,anomalous_flag=True,
>> > >>> d_min=1.61)
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/crystal/__init__.py in
>> > >>> __init__(self, unit_cell, space_group_symbol, space_group_info,
>> > >>> space_group, correct_rhombohedral_setting_if_necessary,
>> > >>> assert_is_compatible_unit_cell, raise_sorry_if_incompatible_unit_cell,
>> > >>> force_compatible_unit_cell)
>> > >>> 74 if (space_group_symbol is not None):
>> > >>> 75 self._space_group_info = sgtbx.space_group_info(
>> > >>> ---> 76 symbol=space_group_symbol)
>> > >>> 77 elif (space_group is not None):
>> > >>> 78 if (isinstance(space_group, sgtbx.space_group)):
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in
>> > >>> __init__(self, symbol, table_id, group, number, space_group_t_den)
>> > >>> 100 assert group is None
>> > >>> 101 if (table_id is None):
>> > >>> --> 102 symbols = space_group_symbols(symbol)
>> > >>> 103 else:
>> > >>> 104 if (isinstance(symbol, int)): symbol = str(symbol)
>> > >>>
>> > >>> ArgumentError: Python argument types in
>> > >>> space_group_symbols.__init__(space_group_symbols, str)
>> > >>> did not match C++ signature:
>> > >>> __init__(_object*, int space_group_number)
>> > >>> __init__(_object*, int space_group_number,
>> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> > >>> std::allocator<char> > extension='')
>> > >>> __init__(_object*, int space_group_number,
>> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> > >>> std::allocator<char> > extension='', std::__cxx11::basic_string<char,
>> > >>> std::char_traits<char>, std::allocator<char> > table_id='')
>> > >>> __init__(_object*, std::__cxx11::basic_string<char,
>> > >>> std::char_traits<char>, std::allocator<char> > symbol)
>> > >>> __init__(_object*, std::__cxx11::basic_string<char,
>> > >>> std::char_traits<char>, std::allocator<char> > symbol,
>> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> > >>> std::allocator<char> > table_id='')
>> > >>>
>> > >>>
>> > >>> When I tried to set LIBTBX_BUILD to /home/gergely/anaconda3 (this is
>> > >>> sys.prefix on my system), I got other problems when importing. Can
>> > >>> LIBTBX_BUILD be set in the conda package?
>> > >>>
>> > >>>
>> > >>>
>> > >>> ---------------------------------------------------------------------------
>> > >>> FileNotFoundError Traceback (most recent call last)
>> > >>> <ipython-input-1-7e2b144826cc> in <module>
>> > >>> 26 from cctbx import sgtbx
>> > >>> 27 #from cctbx import miller
>> > >>> ---> 28 from iotbx import reflection_file_reader, mtz
>> > >>> 29 sns.set_context("poster")
>> > >>> 30 plt.rcParams.update({'figure.autolayout': True})
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/iotbx/reflection_file_reader.py
>> > >>> in <module>
>> > >>> 53
>> > >>> 54 from __future__ import absolute_import, division, print_function
>> > >>> ---> 55 from iotbx import mtz
>> > >>> 56 from iotbx.scalepack import merge as scalepack_merge
>> > >>> 57 from iotbx.scalepack import no_merge_original_index as
>> > >>> scalepack_no_merge
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/__init__.py in <module>
>> > >>> 9 import iotbx_mtz_ext as ext
>> > >>> 10
>> > >>> ---> 11 from iotbx.mtz import extract_from_symmetry_lib
>> > >>> 12 from cctbx import xray
>> > >>> 13 import cctbx.xray.observation_types
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/extract_from_symmetry_lib.py
>> > >>> in <module>
>> > >>> 1 from __future__ import absolute_import, division, print_function
>> > >>> 2 from cctbx import sgtbx
>> > >>> ----> 3 import libtbx.load_env
>> > >>> 4 import os.path as op
>> > >>> 5 from six.moves import range
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module>
>> > >>> 3 import libtbx.env_config
>> > >>> 4 import os
>> > >>> ----> 5 libtbx.env = libtbx.env_config.unpickle()
>> > >>> 6 libtbx.env.set_os_environ_all_dist()
>> > >>> 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME")
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle()
>> > >>> 2605 build_path = os.environ["LIBTBX_BUILD"]
>> > >>> 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path)
>> > >>> -> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")
>> > >>> 2608 env = pickle.load(libtbx_env)
>> > >>> 2609 if (env.python_version_major_minor != sys.version_info[:2]):
>> > >>>
>> > >>> FileNotFoundError: [Errno 2] No such file or directory:
>> > >>> '/home/gergely/anaconda3/libtbx_env'
>> > >>>
>> > >>> On Wed, Dec 18, 2019 at 9:10 AM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >>> >
>> > >>> > Hi Gergely,
>> > >>> >
>> > >>> > I've uploaded linux packages to a new channel, cctbx-dev, and you can install it with
>> > >>> >
>> > >>> > conda install -c cctbx-dev cctbx
>> > >>> >
>> > >>> > in your current environment. You should set your ~/.condarc file to pull the other dependencies from the conda-forge channel first, so put conda-forge above cctbx. Mine looks like
>> > >>> >
>> > >>> > channels:
>> > >>> > - conda-forge
>> > >>> > - defaults
>> > >>> > - cctbx
>> > >>> >
>> > >>> > Do you need dxtbx for your scripts? This package does not build that part. I think the plan is to build a separate conda package for dxtbx so that it can be updated more frequently. I can rebuild the packages to include it for testing, but the one being submitted to conda-forge will not have it.
>> > >>> >
>> > >>> > Also, your error message is probably due to version of HDF5 that the development build installs. The bootstrap.py script will install 1.10.4, but your other dependency is looking for 1.10.5. Installing this cctbx conda package should install 1.10.5, which should fix the issue. Also, I'm updating those environments and 1.10.5 will be the new default version.
>> > >>> >
>> > >>> > Lastly, the dispatchers will not work in these packages because the old paths during the build process are still in them and many of them expect some additional information that has not been updated in the packages yet. I'm in the process of doing that and will update the cctbx-dev channel when that's done. However, by starting python, you can import cctbx modules. So you can run commands like
>> > >>> >
>> > >>> > from scitbx.array_family import flex
>> > >>> > a = flex.random_double(1000000)
>> > >>> > b = flex.min_max_mean_double(a)
>> > >>> > b.min
>> > >>> > b.max
>> > >>> > b.mean
>> > >>> >
>> > >>> > Thanks!
>> > >>> >
>> > >>> > --
>> > >>> > Billy K. Poon
>> > >>> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > >>> > Lawrence Berkeley National Laboratory
>> > >>> > 1 Cyclotron Road, M/S 33R0345
>> > >>> > Berkeley, CA 94720
>> > >>> > Tel: (510) 486-5709
>> > >>> > Fax: (510) 486-5909
>> > >>> > Web: https://phenix-online.org
>> > >>> >
>> > >>> >
>> > >>> > On Tue, Dec 17, 2019 at 5:39 AM Gergely Katona <gkatona(a)gmail.com> wrote:
>> > >>> >>
>> > >>> >> Dear Billy,
>> > >>> >>
>> > >>> >> Thank you for the detailed explanation, I look forward to do further
>> > >>> >> testing! I aim to pool all modules under the same environment, if
>> > >>> >> there are conflicts then I just try to reshuffle the order of imports.
>> > >>> >> So far this did not cause problems for me even when I was using system
>> > >>> >> python, but of course it is not the most prudent thing to do. With
>> > >>> >> anaconda everything is much more standardized and isolated already and
>> > >>> >> without being superuser I can have my familiar environment at any
>> > >>> >> synchrotron based cluster. It is great that cctbx will be an integral
>> > >>> >> part of this ecosystem and this was also the last thing holding me
>> > >>> >> back from adopting python3. About hdf5, pymc3 requires it and
>> > >>> >> importing causes a kernel restart with the following error messages,
>> > >>> >> curiously if I import h5py first this can be avoided.
>> > >>> >>
>> > >>> >> Best wishes,
>> > >>> >>
>> > >>> >> Gergely
>> > >>> >>
>> > >>> >> import pymc3 as pm
>> > >>> >>
>> > >>> >> Warning! ***HDF5 library version mismatched error***
>> > >>> >> The HDF5 header files used to compile this application do not match
>> > >>> >> the version used by the HDF5 library to which this application is linked.
>> > >>> >> Data corruption or segmentation faults may occur if the application continues.
>> > >>> >> This can happen when an application was compiled by one version of HDF5 but
>> > >>> >> linked with a different version of static or shared HDF5 library.
>> > >>> >> You should recompile the application or check your shared library related
>> > >>> >> settings such as 'LD_LIBRARY_PATH'.
>> > >>> >> You can, at your own risk, disable this warning by setting the environment
>> > >>> >> variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.
>> > >>> >> Setting it to 2 or higher will suppress the warning messages totally.
>> > >>> >> Headers are 1.10.4, library is 1.10.5
>> > >>> >> SUMMARY OF THE HDF5 CONFIGURATION
>> > >>> >> =================================
>> > >>> >>
>> > >>> >> General Information:
>> > >>> >> -------------------
>> > >>> >> HDF5 Version: 1.10.5
>> > >>> >> Configured on: Tue Oct 22 12:02:13 UTC 2019
>> > >>> >> Configured by: conda@16247e67ecd5
>> > >>> >> Host system: x86_64-conda_cos6-linux-gnu
>> > >>> >> Uname information: Linux 16247e67ecd5 4.15.0-1059-azure
>> > >>> >> #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64
>> > >>> >> GNU/Linux
>> > >>> >> Byte sex: little-endian
>> > >>> >> Installation point: /home/gergely/anaconda3
>> > >>> >>
>> > >>> >> Compiling Options:
>> > >>> >> ------------------
>> > >>> >> Build Mode: production
>> > >>> >> Debugging Symbols: no
>> > >>> >> Asserts: no
>> > >>> >> Profiling: no
>> > >>> >> Optimization Level: high
>> > >>> >>
>> > >>> >> Linking Options:
>> > >>> >> ----------------
>> > >>> >> Libraries: static, shared
>> > >>> >> Statically Linked Executables:
>> > >>> >> LDFLAGS: -Wl,-O2 -Wl,--sort-common
>> > >>> >> -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags
>> > >>> >> -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib
>> > >>> >> -Wl,-rpath-link,/home/gergely/anaconda3/lib
>> > >>> >> -L/home/gergely/anaconda3/lib
>> > >>> >> H5_LDFLAGS:
>> > >>> >> AM_LDFLAGS: -L/home/gergely/anaconda3/lib
>> > >>> >> Extra libraries: -lrt -lpthread -lz -ldl -lm
>> > >>> >> Archiver:
>> > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar
>> > >>> >> AR_FLAGS: cr
>> > >>> >> Ranlib:
>> > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib
>> > >>> >>
>> > >>> >> Languages:
>> > >>> >> ----------
>> > >>> >> C: yes
>> > >>> >> C Compiler:
>> > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc
>> > >>> >> CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2
>> > >>> >> -I/home/gergely/anaconda3/include
>> > >>> >> H5_CPPFLAGS: -D_GNU_SOURCE
>> > >>> >> -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API
>> > >>> >> AM_CPPFLAGS: -I/home/gergely/anaconda3/include
>> > >>> >> C Flags: -march=nocona -mtune=haswell
>> > >>> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
>> > >>> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include
>> > >>> >> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5
>> > >>> >> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix
>> > >>> >> H5 C Flags: -std=c99 -pedantic -Wall -Wextra
>> > >>> >> -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion
>> > >>> >> -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal
>> > >>> >> -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations
>> > >>> >> -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs
>> > >>> >> -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls
>> > >>> >> -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef
>> > >>> >> -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings
>> > >>> >> -finline-functions -s -Wno-inline -Wno-aggregate-return
>> > >>> >> -Wno-missing-format-attribute -Wno-missing-noreturn -O
>> > >>> >> AM C Flags:
>> > >>> >> Shared C Library: yes
>> > >>> >> Static C Library: yes
>> > >>> >>
>> > >>> >>
>> > >>> >> Fortran: yes
>> > >>> >> Fortran Compiler:
>> > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran
>> > >>> >> Fortran Flags:
>> > >>> >> H5 Fortran Flags: -pedantic -Wall -Wextra -Wunderflow
>> > >>> >> -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2
>> > >>> >> AM Fortran Flags:
>> > >>> >> Shared Fortran Library: yes
>> > >>> >> Static Fortran Library: yes
>> > >>> >>
>> > >>> >> C++: yes
>> > >>> >> C++ Compiler:
>> > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++
>> > >>> >> C++ Flags: -fvisibility-inlines-hidden
>> > >>> >> -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell
>> > >>> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
>> > >>> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include
>> > >>> >> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5
>> > >>> >> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix
>> > >>> >> H5 C++ Flags: -pedantic -Wall -W -Wundef -Wshadow
>> > >>> >> -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion
>> > >>> >> -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual
>> > >>> >> -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor
>> > >>> >> -Wctor-dtor-privacy -Wabi -finline-functions -s -O
>> > >>> >> AM C++ Flags:
>> > >>> >> Shared C++ Library: yes
>> > >>> >> Static C++ Library: yes
>> > >>> >>
>> > >>> >> Java: no
>> > >>> >>
>> > >>> >>
>> > >>> >> Features:
>> > >>> >> ---------
>> > >>> >> Parallel HDF5: no
>> > >>> >> Parallel Filtered Dataset Writes: no
>> > >>> >> Large Parallel I/O: no
>> > >>> >> High-level library: yes
>> > >>> >> Threadsafety: yes
>> > >>> >> Default API mapping: v110
>> > >>> >> With deprecated public symbols: yes
>> > >>> >> I/O filters (external): deflate(zlib)
>> > >>> >> MPE: no
>> > >>> >> Direct VFD: no
>> > >>> >> dmalloc: no
>> > >>> >> Packages w/ extra debug output: none
>> > >>> >> API tracing: no
>> > >>> >> Using memory checker: yes
>> > >>> >> Memory allocation sanity checks: no
>> > >>> >> Function stack tracing: no
>> > >>> >> Strict file format checks: no
>> > >>> >> Optimization instrumentation: no
>> > >>> >>
>> > >>> >> On Tue, Dec 17, 2019 at 8:35 AM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >>> >> >
>> > >>> >> > Hi Gergely,
>> > >>> >> >
>> > >>> >> > Let me build the test package tomorrow. All the gritty details for building with conda is being finalized and the official documentation will be updated to describe the steps. It would be too confusing to keep changing the documentation as the process evolves. You do have the general process, though, which is summarized as follows.
>> > >>> >> >
>> > >>> >> > 1) Installing dependencies. The cctbx_dependencies metapackage was an initial approach for managing the CCTBX dependencies, but after contacting the conda-forge folks, they recommended using the --only-deps flag. So when the CCTBX conda package is available, you'll be able to get a set of dependencies with,
>> > >>> >> >
>> > >>> >> > conda install -c conda-forge --only-deps cctbx
>> > >>> >> >
>> > >>> >> > By default, the bootstrap.py file will automatically install a set of dependencies in the "conda_base" directory (and a conda installation if one is not found). It just uses standard conda environment files located in libtbx/auto_build/conda_envs, so you do not need to install cctbx_dependencies as a separate step. The environment files avoid channel issues by explicitly defining the channel to pull the packages from and the cctbx channel just stores copies of packages from conda-forge. There were issues earlier where the conda-forge packages would sometimes be moved to a different label. The --use-conda flag also accepts a path to $CONDA_PREFIX if you want to use a specific environment for testing.
>> > >>> >> >
>> > >>> >> > 2) Building. The bootstrap.py file handles that with SCons.
>> > >>> >> >
>> > >>> >> > 3) Running. After building, there should be a setpaths.sh (and .csh) file that adds build/bin to your path. The build/bin directory has our dispatchers, which are just scripts that set up the environment variables for you. This prevents other programs from loading our libraries, whose versions may conflict. You should see that there is a "python" dispatcher, which is a convenience for developers. Otherwise, you can use libtbx.python, which will be able to import CCTBX modules.
>> > >>> >> >
>> > >>> >> > The running part is where the conda package for CCTBX will be different than this build. Since our Python files and extensions modules will be in the "site-packages" directory for Python, the PYTHONPATH variable will not be needed (and conda suggests that that variable not be set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so LD_LIBRARY_PATH is not needed. In an active environment, PATH will already be modified. And then our LIBTBX_BUILD directory can be set to sys.prefix.
>> > >>> >> >
>> > >>> >> > So with the conda package, you would only need to activate your conda environment and CCTBX should integrate with other conda packages. What is the conflict with hdf5? That's something that should be fixed. Thanks!
>> > >>> >> >
>> > >>> >> > --
>> > >>> >> > Billy K. Poon
>> > >>> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > >>> >> > Lawrence Berkeley National Laboratory
>> > >>> >> > 1 Cyclotron Road, M/S 33R0345
>> > >>> >> > Berkeley, CA 94720
>> > >>> >> > Tel: (510) 486-5709
>> > >>> >> > Fax: (510) 486-5909
>> > >>> >> > Web: https://phenix-online.org
>> > >>> >> >
>> > >>> >> >
>> > >>> >> > On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona <gkatona(a)gmail.com> wrote:
>> > >>> >> >>
>> > >>> >> >> Dear Billy,
>> > >>> >> >>
>> > >>> >> >> Thank you for this update and for your efforts! I found a solution and
>> > >>> >> >> indeed most things already work in anaconda3. The steps I took (even
>> > >>> >> >> if these instructions will have short expiry date):
>> > >>> >> >>
>> > >>> >> >> Modified .condarc with:
>> > >>> >> >> channels:
>> > >>> >> >> - cctbx
>> > >>> >> >> - conda-forge
>> > >>> >> >> - defaults
>> > >>> >> >>
>> > >>> >> >> run
>> > >>> >> >> conda conda install cctbx_dependencies python=3.6
>> > >>> >> >>
>> > >>> >> >> I expect this will work without python=3.6 in the near future.
>> > >>> >> >>
>> > >>> >> >> Then compiling cctbx with anaconda3 python went without any problem
>> > >>> >> >> when using these flags:
>> > >>> >> >> python bootstrap.py --use-conda --python3 --nproc=12
>> > >>> >> >>
>> > >>> >> >> Then I probably did the unorthodox thing and sourced these directories:
>> > >>> >> >>
>> > >>> >> >> setenv LIBTBX_BUILD /home/gergely/cctbx/build
>> > >>> >> >> setenv PATH ${PATH}:/home/gergely/cctbx/build/bin
>> > >>> >> >> setenv PYTHONPATH
>> > >>> >> >> /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH
>> > >>> >> >> setenv LD_LIBRARY_PATH
>> > >>> >> >> /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH
>> > >>> >> >>
>> > >>> >> >> There is probably a better way to put this into conda environment.
>> > >>> >> >> With these steps I could run one of my scripts depending on cctbx. The
>> > >>> >> >> only problem I found is that hdf5 library had conflict with another
>> > >>> >> >> package in conda, but I does not find this as a showstopper.
>> > >>> >> >>
>> > >>> >> >> Best wishes,
>> > >>> >> >>
>> > >>> >> >> Gergely
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >> On Fri, Dec 13, 2019 at 7:23 PM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >>> >> >> >
>> > >>> >> >> > Hi Gergely,
>> > >>> >> >> >
>> > >>> >> >> > It's still a work in progress. I'm sorting out some Windows issues right now. I can probably build a test package on a separate channel for people that want to test it (let's say next week?). I'll provide instructions, but basically, the test conda package will be in its own separate channel and the dependencies will be pulled from the conda-forge channel. I want most things to be working correctly on Python 2.7, 3.6, 3.7, and 3.8 on all 3 platforms.
>> > >>> >> >> >
>> > >>> >> >> > Thanks!
>> > >>> >> >> >
>> > >>> >> >> > --
>> > >>> >> >> > Billy K. Poon
>> > >>> >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > >>> >> >> > Lawrence Berkeley National Laboratory
>> > >>> >> >> > 1 Cyclotron Road, M/S 33R0345
>> > >>> >> >> > Berkeley, CA 94720
>> > >>> >> >> > Tel: (510) 486-5709
>> > >>> >> >> > Fax: (510) 486-5909
>> > >>> >> >> > Web: https://phenix-online.org
>> > >>> >> >> >
>> > >>> >> >> >
>> > >>> >> >> > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona <gkatona(a)gmail.com> wrote:
>> > >>> >> >> >>
>> > >>> >> >> >> Dear Billy,
>> > >>> >> >> >>
>> > >>> >> >> >> This sounds very promising and exciting. I am not sure if cctbx is
>> > >>> >> >> >> already functional as a conda package in anaconda3 (Linux) or this is
>> > >>> >> >> >> still work in progress. My technical expertise does not allow me to
>> > >>> >> >> >> tell the difference. What I tried:
>> > >>> >> >> >>
>> > >>> >> >> >> Fresh install of anaconda3. Adding - cctbx and - conda-forge to
>> > >>> >> >> >> .condarc . Installing pyside2 with conda. Running conda install
>> > >>> >> >> >> conda_dependencies . I get a lot package version conflict, and I
>> > >>> >> >> >> cannot import cctbx or iotbx to anaconda python. Am I following the
>> > >>> >> >> >> right instructions? Or it is too early to expect that cctbx works when
>> > >>> >> >> >> installed through conda?
>> > >>> >> >> >>
>> > >>> >> >> >> Best wishes,
>> > >>> >> >> >>
>> > >>> >> >> >> Gergely
>> > >>> >> >> >>
>> > >>> >> >> >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >>> >> >> >> >
>> > >>> >> >> >> > Hi all,
>> > >>> >> >> >> >
>> > >>> >> >> >> > For a brief update, I have submitted a recipe for cctbxlite to conda-forge (https://github.com/conda-forge/staged-recipes/pull/10021) and support for Python 3.7 and 3.8 is being added (https://github.com/cctbx/cctbx_project/pull/409). With some fixes for Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8. Some additional changes will be needed to get Windows to work with Python 3 and for tests to pass with Boost 1.70.0. That will enable the conda-forge recipe to build for all platforms and for all supported versions of Python.
>> > >>> >> >> >> >
>> > >>> >> >> >> > Currently, the conda-forge recipe will install into the conda python and cctbx imports can be done without sourcing the environment scripts. It looks like a lot of the environment variables being set in the dispatchers can be removed since the Python files and C++ extensions are in the right places. I'll update the libtbx_env file so that commands that load the environment can work correctly.
>> > >>> >> >> >> >
>> > >>> >> >> >> > --
>> > >>> >> >> >> > Billy K. Poon
>> > >>> >> >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > >>> >> >> >> > Lawrence Berkeley National Laboratory
>> > >>> >> >> >> > 1 Cyclotron Road, M/S 33R0345
>> > >>> >> >> >> > Berkeley, CA 94720
>> > >>> >> >> >> > Tel: (510) 486-5709
>> > >>> >> >> >> > Fax: (510) 486-5909
>> > >>> >> >> >> > Web: https://phenix-online.org
>> > >>> >> >> >> >
>> > >>> >> >> >> >
>> > >>> >> >> >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll <tic20(a)cam.ac.uk> wrote:
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> Hi Luc,
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> That sounds promising. From there, I’d need to work out how to make a fully-packaged installer (basically a modified wheel file) for the ChimeraX ToolShed - the aim is for the end user to not have to worry about any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would need to be set dynamically before first import - but doesn’t seem insurmountable.
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> Thanks,
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> Tristan
>> > >>> >> >> >> >>
>> > >>> >> >> >> >>
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> Tristan Croll
>> > >>> >> >> >> >> Research Fellow
>> > >>> >> >> >> >> Cambridge Institute for Medical Research
>> > >>> >> >> >> >> University of Cambridge CB2 0XY
>> > >>> >> >> >> >>
>> > >>> >> >> >> >>
>> > >>> >> >> >> >>
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis <luc_j_bourhis(a)mac.com> wrote:
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> > Hi Tristan,
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> > cctbx could be built to use your ChimeraX python, now that cctbx is moving to Python 3. The option —with-python is there for that with the bootstrap script. The specific environment setup boil down to setting two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a framework such as ChimeraX, that should not be difficult to ensure those two variables are set.
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> > Best wishes,
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> > Luc
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll <tic20(a)cam.ac.uk> wrote:
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >> To add my two cents on this: probably the second-most common question I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". The honest answer to that is, "I didn't know how."
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >> Still don't, really - although the current developments are rather promising. The problem I've faced is that CCTBX was designed as its own self-contained Python (2.7, until very recently) environment, with its own interpreter and a lot of very specific environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is *also* its own self-contained Python (3.7) environment. To plug one into the other in that form... well, I don't think I'm a good enough programmer to really know where to start.
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >> The move to Conda and a more modular CCTBX architecture should make a lot more possible in that direction. Pip would be even better for me personally (ChimeraX can install directly from the PyPI, but doesn't interact with Conda) - but I understand pretty well the substantial challenge that would amount to (not least being that the PyPI imposes a limit - around 100MB from memory? - on the size of an individual package).
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >> Best regards,
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >> Tristan
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote:
>> > >>> >> >> >> >> >>> Hi Graeme,
>> > >>> >> >> >> >> >>> Yes, I know. But “black" is a program doing a very particular task
>> > >>> >> >> >> >> >>> (code formatting from the top of my head). Requiring to use a wrapper
>> > >>> >> >> >> >> >>> for python itself is another level. But ok, I think I am mellowing to
>> > >>> >> >> >> >> >>> the idea after all! Talking with people around me, and extrapolating,
>> > >>> >> >> >> >> >>> I would bet that, right now, a great majority of people interested by
>> > >>> >> >> >> >> >>> cctbx in pip have already used the cctbx, so they know about the
>> > >>> >> >> >> >> >>> Python wrapper, and they would not be too sanguine about that. My
>> > >>> >> >> >> >> >>> concern is for the future, when pip will be the first time some people
>> > >>> >> >> >> >> >>> use cctbx. Big fat warning notices on PyPI page and a better error
>> > >>> >> >> >> >> >>> message when cctbx fails because LIBTBX_BUILD is not set would be
>> > >>> >> >> >> >> >>> needed but that could be all right.
>> > >>> >> >> >> >> >>> If we do a pip installer, we should aim at a minimal install: cctbx,
>> > >>> >> >> >> >> >>> iotbx and their dependencies, and that’s it.
>> > >>> >> >> >> >> >>> Best wishes,
>> > >>> >> >> >> >> >>> Luc
>> > >>> >> >> >> >> >>>> On 23 Aug 2019, at 07:17, Graeme.Winter(a)Diamond.ac.uk <Graeme.Winter(a)diamond.ac.uk> wrote:
>> > >>> >> >> >> >> >>>> Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g.
>> > >>> >> >> >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat $(which black)
>> > >>> >> >> >> >> >>>> /Library/Frameworks/Python.framework/Versions/3.6/bin/black
>> > >>> >> >> >> >> >>>> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
>> > >>> >> >> >> >> >>>> # -*- coding: utf-8 -*-
>> > >>> >> >> >> >> >>>> import re
>> > >>> >> >> >> >> >>>> import sys
>> > >>> >> >> >> >> >>>> from black import main
>> > >>> >> >> >> >> >>>> if __name__ == '__main__':
>> > >>> >> >> >> >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
>> > >>> >> >> >> >> >>>> sys.exit(main())
>> > >>> >> >> >> >> >>>> So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this.
>> > >>> >> >> >> >> >>>> Best wishes Graeme
>> > >>> >> >> >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote:
>> > >>> >> >> >> >> >>>> Hi,
>> > >>> >> >> >> >> >>>> Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs?
>> > >>> >> >> >> >> >>>> Best wishes,
>> > >>> >> >> >> >> >>>> Luc
>> > >>> >> >> >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote:
>> > >>> >> >> >> >> >>>> Hi,
>> > >>> >> >> >> >> >>>> I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.org<http://anaconda.org/>, and then to advertise it hard to every potential user out there.
>> > >>> >> >> >> >> >>>> Best wishes,
>> > >>> >> >> >> >> >>>> Luc
>> > >>> >> >> >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster <asbrewster(a)lbl.gov<mailto:[email protected]>> wrote:
>> > >>> >> >> >> >> >>>> Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython…). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not.
>> > >>> >> >> >> >> >>>> All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support.
>> > >>> >> >> >> >> >>>> There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6).
>> > >>> >> >> >> >> >>>> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4.
>> > >>> >> >> >> >> >>>> Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :)
>> > >>> >> >> >> >> >>>> Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users.
>> > >>> >> >> >> >> >>>> Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort.
>> > >>> >> >> >> >> >>>> Thanks,
>> > >>> >> >> >> >> >>>> -Aaron
>> > >>> >> >> >> >> >>>> _______________________________________________
>> > >>> >> >> >> >> >>>> cctbxbb mailing list
>> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >>> >> >> >> >> >>>> _______________________________________________
>> > >>> >> >> >> >> >>>> cctbxbb mailing list
>> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >>> >> >> >> >> >>>> _______________________________________________
>> > >>> >> >> >> >> >>>> cctbxbb mailing list
>> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >>> >> >> >> >> >>>> --
>> > >>> >> >> >> >> >>>> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
>> > >>> >> >> >> >> >>>> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
>> > >>> >> >> >> >> >>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
>> > >>> >> >> >> >> >>>> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>> > >>> >> >> >> >> >>>> _______________________________________________
>> > >>> >> >> >> >> >>>> cctbxbb mailing list
>> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org
>> > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >>> >> >> >> >> >>> _______________________________________________
>> > >>> >> >> >> >> >>> cctbxbb mailing list
>> > >>> >> >> >> >> >>> cctbxbb(a)phenix-online.org
>> > >>> >> >> >> >> >>> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >> _______________________________________________
>> > >>> >> >> >> >> >> cctbxbb mailing list
>> > >>> >> >> >> >> >> cctbxbb(a)phenix-online.org
>> > >>> >> >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> > _______________________________________________
>> > >>> >> >> >> >> > cctbxbb mailing list
>> > >>> >> >> >> >> > cctbxbb(a)phenix-online.org
>> > >>> >> >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >>> >> >> >> >>
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> _______________________________________________
>> > >>> >> >> >> >> cctbxbb mailing list
>> > >>> >> >> >> >> cctbxbb(a)phenix-online.org
>> > >>> >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >>> >> >> >> >
>> > >>> >> >> >> > _______________________________________________
>> > >>> >> >> >> > cctbxbb mailing list
>> > >>> >> >> >> > cctbxbb(a)phenix-online.org
>> > >>> >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >>> >> >> >>
>> > >>> >> >> >>
>> > >>> >> >> >>
>> > >>> >> >> >> --
>> > >>> >> >> >> Gergely Katona, PhD
>> > >>> >> >> >> Associate Professor
>> > >>> >> >> >> Department of Chemistry and Molecular Biology, University of Gothenburg
>> > >>> >> >> >> Box 462, 40530 Göteborg, Sweden
>> > >>> >> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> > >>> >> >> >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
>> > >>> >> >> >>
>> > >>> >> >> >> _______________________________________________
>> > >>> >> >> >> cctbxbb mailing list
>> > >>> >> >> >> cctbxbb(a)phenix-online.org
>> > >>> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >>> >> >> >
>> > >>> >> >> > _______________________________________________
>> > >>> >> >> > cctbxbb mailing list
>> > >>> >> >> > cctbxbb(a)phenix-online.org
>> > >>> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >> --
>> > >>> >> >> Gergely Katona, PhD
>> > >>> >> >> Associate Professor
>> > >>> >> >> Department of Chemistry and Molecular Biology, University of Gothenburg
>> > >>> >> >> Box 462, 40530 Göteborg, Sweden
>> > >>> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> > >>> >> >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
>> > >>> >> >>
>> > >>> >> >> _______________________________________________
>> > >>> >> >> cctbxbb mailing list
>> > >>> >> >> cctbxbb(a)phenix-online.org
>> > >>> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >>> >> >
>> > >>> >> > _______________________________________________
>> > >>> >> > cctbxbb mailing list
>> > >>> >> > cctbxbb(a)phenix-online.org
>> > >>> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >>> >>
>> > >>> >>
>> > >>> >>
>> > >>> >> --
>> > >>> >> Gergely Katona, PhD
>> > >>> >> Associate Professor
>> > >>> >> Department of Chemistry and Molecular Biology, University of Gothenburg
>> > >>> >> Box 462, 40530 Göteborg, Sweden
>> > >>> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> > >>> >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
>> > >>> >>
>> > >>> >> _______________________________________________
>> > >>> >> cctbxbb mailing list
>> > >>> >> cctbxbb(a)phenix-online.org
>> > >>> >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >>> >
>> > >>> > _______________________________________________
>> > >>> > cctbxbb mailing list
>> > >>> > cctbxbb(a)phenix-online.org
>> > >>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >>>
>> > >>>
>> > >>>
>> > >>> --
>> > >>> Gergely Katona, PhD
>> > >>> Associate Professor
>> > >>> Department of Chemistry and Molecular Biology, University of Gothenburg
>> > >>> Box 462, 40530 Göteborg, Sweden
>> > >>> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> > >>> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
>> > >>>
>> > >>> _______________________________________________
>> > >>> cctbxbb mailing list
>> > >>> cctbxbb(a)phenix-online.org
>> > >>> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >
>> > > _______________________________________________
>> > > cctbxbb mailing list
>> > > cctbxbb(a)phenix-online.org
>> > > http://phenix-online.org/mailman/listinfo/cctbxbb
>> >
>> >
>> >
>> > --
>> > Gergely Katona, Professor
>> > Department of Chemistry and Molecular Biology, University of Gothenburg
>> > Box 462, 40530 Göteborg, Sweden
>> > Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> > Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
>>
>>
>>
>> --
>> Gergely Katona, Professor
>> Department of Chemistry and Molecular Biology, University of Gothenburg
>> Box 462, 40530 Göteborg, Sweden
>> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org
>> http://phenix-online.org/mailman/listinfo/cctbxbb
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
--
Gergely Katona, Professor
Department of Chemistry and Molecular Biology, University of Gothenburg
Box 462, 40530 Göteborg, Sweden
Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
6 years, 1 month
Re: [cctbxbb] conda cctbx on Mac 10.15.2 working with python3.8
by Tristan Croll
Hi Blaine,
The speed issue is probably because this list comprehension:
[print(hkl) for hkl in ms.indices()]
... is calling print() (and returning None) a million times over. I
think something like the following should be much faster:
print('\n'.join([str(hkl) for hkl in ms.indices()]))
Best regards,
Tristan
On 2020-01-04 22:00, Mooers, Blaine H.M. (HSC) wrote:
> Hi Billy,
>
> Gergely's post yesterday stimulated me to check my conda cctbx setup.
> I had earlier reported success with conda cctbx for Python3.6, 3.7, and
> 3.8.
> This morning I found in my path an earlier python3.6 build of cctbx
> that may
> have given me false positives with my conda installs of cctbx.
> After removing the old build of cctbx from my path, I modified my zshrc
> file
> as Gergely suggested. I can confirm that results that Gergely reported
> yesterday.
>
> I used a Mac OS X 10.15.2 with zsh as my shell and anaconda3 installed
> as /opt/anaconda. I did my testing in a conda env named cctbx38.
> I added 'export LIBTBX_BUILD=$CONDA_PREFIX' in my .zshrc file.
> After sourcing my .zshrc file, 'echo $LIBTBX_BUILD' returns
> '/opt/anaconda/envs/cctbx38' as expected.
>
> The toy script below prints the calculated Miller indices as a tuples
> in a terminal window.
> It was run using 'python millerSet.py'.
>
> Note that the printing of 1 million indices to a jupyter notebook cell
> is disappointingly slow!!
> Reduce each unit cell dimension by two orders of magnitude before
> trying this at home!
> The following code also worked with Python3.6 and Python3.7 in their
> corresponding cctbx36 and cctbx37 envs.
>
> from cctbx import crystal
> from cctbx import miller
>
> ms = miller.build_set(
> crystal_symmetry=crystal.symmetry(
> space_group_symbol='P4',
> unit_cell=(150.8,150.8,250.4,90.0,90.0,90.0)),
> anomalous_flag=False,
> d_min=1.4)
>
> [print(hkl) for hkl in ms.indices()]
>
> I used conda python3.8.0 built on November 22. Python3.8.1 does not
> seem to be
> available yet from Anaconda.
>
> Best regards,
>
> Blaine
>
> Blaine Mooers, Ph.D.
> Associate Professor
> Department of Biochemistry and Molecular Biology
> College of Medicine
> University of Oklahoma Health Sciences Center
> S.L. Young Biomedical Research Center (BRC) Rm. 466
> 975 NE 10th Street, BRC 466
> Oklahoma City, OK 73104-5419
>
> ________________________________________
> From: cctbxbb-bounces(a)phenix-online.org
> [cctbxbb-bounces(a)phenix-online.org] on behalf of
> cctbxbb-request(a)phenix-online.org [cctbxbb-request(a)phenix-online.org]
> Sent: Saturday, January 04, 2020 2:00 PM
> To: cctbxbb(a)phenix-online.org
> Subject: [EXTERNAL] cctbxbb Digest, Vol 130, Issue 2
>
> Send cctbxbb mailing list submissions to
> cctbxbb(a)phenix-online.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> or, via email, send a message with subject or body 'help' to
> cctbxbb-request(a)phenix-online.org
>
> You can reach the person managing the list at
> cctbxbb-owner(a)phenix-online.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of cctbxbb digest..."
>
>
> Today's Topics:
>
> 1. Re: some thoughts on cctbx and pip (Billy Poon)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 3 Jan 2020 21:04:25 +0100
> From: Billy Poon <BKPoon(a)lbl.gov>
> To: cctbx mailing list <cctbxbb(a)phenix-online.org>
> Subject: Re: [cctbxbb] some thoughts on cctbx and pip
> Message-ID:
>
> <CAHmYUcbWS7FXA6EPUx=AC2X6m6gPY=GqH-MwGD+uzpTaOKSyjQ(a)mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Gergely,
>
> Happy New Year!
>
> Can you provide the output to "conda list" for each environment?
>
> I suspect that the old version of cctbx might still be around. Is there
> a
> file named PYTHON_VERSION_MAJOR_MINOR in "/home/gergely/anaconda3/lib"?
> If
> so, delete that file. To make sure you are downloading a new copy of
> the
> package, clear the cache where conda keeps copies of all the packages.
> To
> do that, you can run "conda clean -y --all". You can remove the "-y"
> flag
> if you want to see what will be deleted. The hash for the new linux
> package
> should be h8b68381_0.
>
> The last error looks like an incompatible version of boost or gcc
> library
> was installed. The output from "conda list" should show the boost and
> boost-cpp packages coming from conda-forge. I would avoid installing
> into
> the root or base environment because that usually pulls from the
> default
> channels. All the dependencies should be pulled from the conda-forge
> channel.
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709
> Fax: (510) 486-5909
> Web:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
>
>
> On Fri, Jan 3, 2020 at 2:20 PM Gergely Katona <gkatona(a)gmail.com>
> wrote:
>
>> I found some other errors when using the test environment.
>>
>>
>> ---------------------------------------------------------------------------
>> RuntimeError Traceback (most recent call
>> last)
>> <ipython-input-1-fe308b345b3f> in <module>
>> 10 import scipy as sp
>> 11 from scipy import linalg
>> ---> 12 import iotbx.pdb
>> 13 import cctbx
>> 14 import pandas as pd
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
>> in <module>
>> 8
>> 9 import iotbx.pdb.records
>> ---> 10 import iotbx.pdb.hierarchy
>> 11 from scitbx import matrix
>> 12
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__hierarchy.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=QlonHINbJqyu1vTwlBJWKyChCMCCJxpIWIaT5_rrkLc&e=
>> in <module>
>> 14 from six.moves import cStringIO as StringIO
>> 15 from iotbx.pdb import hy36encode, hy36decode
>> ---> 16 import iotbx.cif.model
>> 17 from cctbx import crystal
>> 18 from libtbx import group_args
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/cif/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
>> in <module>
>> 19
>> 20 from cctbx.array_family import flex
>> ---> 21 from cctbx import miller
>> 22 from iotbx.cif import model, builders, geometry
>> 23 from libtbx.containers import OrderedDict
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/miller/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
>> in <module>
>> 11
>> 12 from cctbx import crystal
>> ---> 13 from cctbx import maptbx
>> 14 from cctbx import sgtbx
>> 15 from cctbx.sgtbx import lattice_symmetry
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/maptbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
>> in <module>
>> 15 from libtbx import adopt_init_args
>> 16 from libtbx.utils import Sorry
>> ---> 17 import libtbx.load_env
>> 18 import math
>> 19 import sys, os
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__load-5Fenv.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=rJbfdPr3ymiz7Ir508_VGIhspAKCOwozKhY5I6OXux0&e=
>> in
>> <module>
>> 3 import libtbx.env_config
>> 4 import os
>> ----> 5 libtbx.env = libtbx.env_config.unpickle()
>> 6 libtbx.env.set_os_environ_all_dist()
>> 7 libtbx.env.dispatcher_name =
>> os.environ.get("LIBTBX_DISPATCHER_NAME")
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__env-5Fconfig.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=UrsFhQ0dxNNuzXmOT3D6pnGJ5lXko4mfrKceNgBaQV4&e=
>> in unpickle()
>> 2736 env = pickle.load(libtbx_env)
>> 2737 if (env.python_version_major_minor != sys.version_info[:2]):
>> -> 2738 env.raise_python_version_incompatible()
>> 2739 if (op.realpath(build_path) !=
>> op.realpath(abs(env.build_path))):
>> 2740 env.build_path.reset(build_path)
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__env-5Fconfig.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=UrsFhQ0dxNNuzXmOT3D6pnGJ5lXko4mfrKceNgBaQV4&e=
>> in raise_python_version_incompatible(self, prev_pvmm)
>> 469 if (prev_pvmm is None):
>> 470 prev_pvmm = "%d.%d" % self.python_version_major_minor
>> --> 471 raise RuntimeError("Python version incompatible with this
>> build:\n"
>> 472 + " Build directory: %s\n" %
>> show_string(abs(self.build_path))
>> 473 + " Python version used initially: %s\n" % prev_pvmm
>>
>> RuntimeError: Python version incompatible with this build:
>> Build directory: "/home/gergely/anaconda3"
>> Python version used initially: 3.6
>> Python version in use now: 3.8
>>
>>
>> Interestingly in the base environment all import worked, but I got an
>> error at:
>>
>>
>> ArgumentError Traceback (most recent call
>> last)
>> <ipython-input-4-5bc71c36e121> in <module>
>> 33 atm_odd.set_b(0)
>> 34
>> ---> 35 odd.write_pdb_file("bdiff_in_even_000.pdb") #skriver ny pdb
>> med dessa inkluderade
>> 36 dfWBvals=pd.DataFrame(rows_list)
>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
>> in
>> write_pdb_file(self, file_name, open_append, crystal_symmetry,
>> cryst1_z, write_scale_records, append_end, atom_hetatm, sigatm,
>> anisou, siguij)
>> 930 siguij=True):
>> 931 if (crystal_symmetry is Auto):
>> --> 932 crystal_symmetry = self.crystal_symmetry()
>> 933 if (cryst1_z is Auto):
>> 934 cryst1_z = self.extract_cryst1_z_columns()
>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
>> in
>> crystal_symmetry(self, crystal_symmetry, weak_symmetry)
>> 1218 crystal_symmetry=None,
>> 1219 weak_symmetry=False):
>> -> 1220 self_symmetry = self.crystal_symmetry_from_cryst1()
>> 1221 if (self_symmetry is None):
>> 1222 self_symmetry =
>> self._crystal_symmetry_from_cns_remark_sg()
>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
>> in
>> crystal_symmetry_from_cryst1(self)
>> 1195 for line in self.crystallographic_section():
>> 1196 if (line.startswith("CRYST1")):
>> -> 1197 return
>> cryst1_interpretation.crystal_symmetry(cryst1_record=line)
>> 1198 return None
>> 1199
>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__cryst1-5Finterpretation.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=3OchLb18AMp3YpXOuEb8BDOwhl2XLrV1QAfZByH-l2E&e=
>> in crystal_symmetry(cryst1_record)
>> 136 space_group_info=None)
>> 137 space_group_info =
>> categorize(cryst1_record.sgroup).space_group_info(
>> --> 138 unit_cell=u)
>> 139 return crystal.symmetry(unit_cell=u,
>> space_group_info=space_group_info)
>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__cryst1-5Finterpretation.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=3OchLb18AMp3YpXOuEb8BDOwhl2XLrV1QAfZByH-l2E&e=
>> in space_group_info(self, unit_cell)
>> 85 if (self.symbol is None): return None
>> 86 if (self.category is None):
>> ---> 87 try: return sgtbx.space_group_info(self.symbol)
>> 88 except RuntimeError: return None
>> 89 if (isinstance(unit_cell, uctbx.ext.unit_cell)):
>>
>> ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
>> in
>> __init__(self, symbol, table_id, group, number, space_group_t_den)
>> 100 assert group is None
>> 101 if (table_id is None):
>> --> 102 symbols = space_group_symbols(symbol)
>> 103 else:
>> 104 if (isinstance(symbol, int)): symbol = str(symbol)
>>
>> ArgumentError: Python argument types in
>> space_group_symbols.__init__(space_group_symbols, str)
>> did not match C++ signature:
>> __init__(_object*, int space_group_number)
>> __init__(_object*, int space_group_number,
>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> std::allocator<char> > extension='')
>> __init__(_object*, int space_group_number,
>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> std::allocator<char> > extension='', std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> > table_id='')
>> __init__(_object*, std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> > symbol)
>> __init__(_object*, std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> > symbol,
>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> std::allocator<char> > table_id='')
>>
>>
>> Gergely
>>
>>
>>
>> On Fri, Jan 3, 2020 at 1:44 PM Gergely Katona <gkatona(a)gmail.com>
>> wrote:
>> >
>> > Dear Billy,
>> >
>> > I did some tests installing in my base conda was not troublefree in
>> > python 3.8. Running iotbx.fetch_pdb --all --mtz 1kp8 resulted in boost
>> > related error when converting to mtz. Installing into a test
>> > environment went fine with iotbx.fetch_pdb --all --mtz 1kp8 running
>> > successfully. I could add pymc3, jupyter and seaborn and iotbx still
>> > did not break. My LIBTBX_BUILD is not set however in the test
>> > environment, which breaks miller when I was trying to import it.
>> > Manually setting the LIBTBX_BUILD to CONDA_PREFIX (setenv LIBTBX_BUILD
>> > $CONDA_PREFIX) fixes this and one of my script works fine! I will come
>> > back if I find some problems with my other scripts, but so far this
>> > looks promising!
>> >
>> >
>> > Happy new year!
>> >
>> > Gergely
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Dec 20, 2019 at 8:16 PM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >
>> > > Hi everyone,
>> > >
>> > > I updated the conda packages on the cctbx-dev channel and there are
>> also now packages for macOS. The old packages have been removed so you
>> can
>> create a new environment with
>> > >
>> > > conda create -n test -c cctbx-dev cctbx python=2.7
>> > > conda activate test
>> > >
>> > > where "test" is the environment name and 2.7 is the python version.
>> For the python version, you can also pick 3.6, 3.7, or 3.8. I think
>> the
>> latest conda will install 3.8 as the default python if the python
>> argument
>> is not provided. Or install into an existing environment (with python
>> already installed) with the same command as before,
>> > >
>> > > conda install -c cctbx-dev cctbx
>> > >
>> > > Again, your .condarc file should have conda-forge as the first channel
>> so that dependencies can be pulled correctly.
>> > >
>> > > The dispatchers should work now and $LIBTBX_BUILD will be set on
>> activation of an environment (and unset on environment deactivation)
>> or
>> after installing into an existing environment, so you can do more
>> interesting things like
>> > >
>> > > iotbx.fetch_pdb --all --mtz 1kp8
>> > >
>> > > However, this package does not install databases, so
>> > >
>> > > molprobity.molprobity 1kp8.pdb 1kp8.mtz
>> > >
>> > > will fail because the database is not available (also probe and reduce
>> are not built).
>> > >
>> > > Some known issues in no particular order,
>> > >
>> > > 1) You can run "libtbx.run_tests_parallel module=cctbx nproc=Auto" to
>> run tests, but not all the test files were copied, so I got 30
>> failures
>> here. The final package will probably not have the test files.
>> > > 2) The other python dispatchers, like "cctbx.python", were not
>> updated, so those will fail. But you can just run "python" in the
>> active
>> environment.
>> > > 3) Parts of mmtbx still need to be updated for Python 3.
>> > > 4) The LIBTBX_BUILD variable should be set for sh and csh shells. Let
>> me know if it's not working for your shell (i.e. "echo $LIBTBX_BUILD"
>> should show the same thing as $CONDA_PREFIX).
>> > > 5) One future step is to enable building other CCTBX-like modules with
>> this conda package. This will require some bookkeeping updates so that
>> modules in the conda environment and the modules being built are
>> tracked
>> properly. Also, making sure that all the necessary headers exist in
>> $PREFIX/include.
>> > >
>> > > Gergely, let me know if your scripts are working.
>> > >
>> > > Happy holidays!
>> > >
>> > > --
>> > > Billy K. Poon
>> > > Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > > Lawrence Berkeley National Laboratory
>> > > 1 Cyclotron Road, M/S 33R0345
>> > > Berkeley, CA 94720
>> > > Tel: (510) 486-5709
>> > > Fax: (510) 486-5909
>> > > Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
>> > >
>> > >
>> > > On Wed, Dec 18, 2019 at 11:25 AM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >>
>> > >> Hi Gergely,
>> > >>
>> > >> That's the other information that hasn't been updated yet in the
>> conda package. :)
>> > >>
>> > >> For LIBTBX_BUILD, I can set up environment variables sort of how the
>> conda gcc package sets up environment variables. I have to double
>> check to
>> see if they're set after installation of the package and after
>> activation
>> of an environment. The variable will just be a copy of $CONDA_PREFIX.
>> > >>
>> > >> The libtbx_env file is a file that exists in $LIBTBX_BUILD that
>> basically stores configuration information. I'll have to modify the
>> contents to update the locations of the different modules since the
>> conda
>> package will not have a "modules" directory.
>> > >>
>> > >> --
>> > >> Billy K. Poon
>> > >> Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > >> Lawrence Berkeley National Laboratory
>> > >> 1 Cyclotron Road, M/S 33R0345
>> > >> Berkeley, CA 94720
>> > >> Tel: (510) 486-5709
>> > >> Fax: (510) 486-5909
>> > >> Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
>> > >>
>> > >>
>> > >> On Wed, Dec 18, 2019 at 2:08 AM Gergely Katona <gkatona(a)gmail.com>
>> wrote:
>> > >>>
>> > >>> Hi Billy,
>> > >>>
>> > >>> Conda install went fine with your instructions after rearranging the
>> > >>> channels by putting cctbx last. I removed all environmental variables
>> > >>> and previous build of cctbx.
>> > >>>
>> > >>> import sys
>> > >>> print (sys.path, sys.prefix)
>> > >>>
>> > >>> ['/home/gergely/anaconda3/lib/https://urldefense.proofpoint.com/v2/url?u=http-3A__python36.zip&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=cUHL8qgeloeUO3oYTQcr9d56RegJk01wJ2i-c3Zo6oE&e= ',
>> > >>> '/home/gergely/anaconda3/lib/python3.6',
>> > >>> '/home/gergely/anaconda3/lib/python3.6/lib-dynload', '',
>> > >>> '/home/gergely/anaconda3/lib/python3.6/site-packages',
>> > >>>
>> '/home/gergely/anaconda3/lib/python3.6/site-packages/IPython/extensions',
>> > >>> '/home/gergely/.ipython'] /home/gergely/anaconda3
>> > >>>
>> > >>> Many imports went fine including pymc3, but I encountered problems
>> > >>> with these three:
>> > >>>
>> > >>> from cctbx import miller
>> > >>> import iotbx.pdb
>> > >>> from iotbx import reflection_file_reader, mtz
>> > >>>
>> > >>>
>> > >>> I also got type error when handling space groups.
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> ---------------------------------------------------------------------------
>> > >>> KeyError Traceback (most recent
>> call last)
>> > >>> <ipython-input-5-cedbd4f84f3d> in <module>
>> > >>> 25 from cctbx import uctbx
>> > >>> 26 from cctbx import sgtbx
>> > >>> ---> 27 from cctbx import miller
>> > >>> 28 #from iotbx import reflection_file_reader, mtz
>> > >>> 29 sns.set_context("poster")
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/miller/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
>> <module>
>> > >>> 11
>> > >>> 12 from cctbx import crystal
>> > >>> ---> 13 from cctbx import maptbx
>> > >>> 14 from cctbx import sgtbx
>> > >>> 15 from cctbx.sgtbx import lattice_symmetry
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/maptbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
>> <module>
>> > >>> 15 from libtbx import adopt_init_args
>> > >>> 16 from libtbx.utils import Sorry
>> > >>> ---> 17 import libtbx.load_env
>> > >>> 18 import math
>> > >>> 19 import sys, os
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__load-5Fenv.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=rJbfdPr3ymiz7Ir508_VGIhspAKCOwozKhY5I6OXux0&e= in
>> <module>
>> > >>> 3 import libtbx.env_config
>> > >>> 4 import os
>> > >>> ----> 5 libtbx.env = libtbx.env_config.unpickle()
>> > >>> 6 libtbx.env.set_os_environ_all_dist()
>> > >>> 7 libtbx.env.dispatcher_name =
>> os.environ.get("LIBTBX_DISPATCHER_NAME")
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__env-5Fconfig.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=UrsFhQ0dxNNuzXmOT3D6pnGJ5lXko4mfrKceNgBaQV4&e= in
>> unpickle()
>> > >>> 2603
>> > >>> 2604 def unpickle():
>> > >>> -> 2605 build_path = os.environ["LIBTBX_BUILD"]
>> > >>> 2606
>> set_preferred_sys_prefix_and_sys_executable(build_path=build_path)
>> > >>> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/https://urldefense.proofpoint.com/v2/url?u=http-3A__os.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=hA1zo5wap0c9wK3DrlWIKqYBIqqbfKXca5jQ9aX5pYU&e= in __getitem__(self, key)
>> > >>> 667 except KeyError:
>> > >>> 668 # raise KeyError with the original key value
>> > >>> --> 669 raise KeyError(key) from None
>> > >>> 670 return self.decodevalue(value)
>> > >>> 671
>> > >>>
>> > >>> KeyError: 'LIBTBX_BUILD'
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> ---------------------------------------------------------------------------
>> > >>> ArgumentError Traceback (most recent
>> call last)
>> > >>> <ipython-input-7-2acd3a9ce26a> in <module>
>> > >>> 26 return
>> ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar
>> > >>> 27
>> > >>> ---> 28
>> ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar=initializecrystal()
>> > >>>
>> > >>> <ipython-input-7-2acd3a9ce26a> in initializecrystal()
>> > >>> 11 uc = uctbx.unit_cell(unit_cell)
>> > >>> 12 wavelength = 1.54980
>> > >>> ---> 13 xtal_symm = crystal.symmetry(unit_cell=unit_cell,
>> > >>> space_group_symbol="P 43 21 2")
>> > >>> 14
>> > >>> 15 ms =
>> > >>> miller.build_set(crystal_symmetry=xtal_symm,anomalous_flag=True,
>> > >>> d_min=1.61)
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/crystal/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
>> > >>> __init__(self, unit_cell, space_group_symbol, space_group_info,
>> > >>> space_group, correct_rhombohedral_setting_if_necessary,
>> > >>> assert_is_compatible_unit_cell,
>> raise_sorry_if_incompatible_unit_cell,
>> > >>> force_compatible_unit_cell)
>> > >>> 74 if (space_group_symbol is not None):
>> > >>> 75 self._space_group_info = sgtbx.space_group_info(
>> > >>> ---> 76 symbol=space_group_symbol)
>> > >>> 77 elif (space_group is not None):
>> > >>> 78 if (isinstance(space_group, sgtbx.space_group)):
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
>> > >>> __init__(self, symbol, table_id, group, number, space_group_t_den)
>> > >>> 100 assert group is None
>> > >>> 101 if (table_id is None):
>> > >>> --> 102 symbols = space_group_symbols(symbol)
>> > >>> 103 else:
>> > >>> 104 if (isinstance(symbol, int)): symbol = str(symbol)
>> > >>>
>> > >>> ArgumentError: Python argument types in
>> > >>> space_group_symbols.__init__(space_group_symbols, str)
>> > >>> did not match C++ signature:
>> > >>> __init__(_object*, int space_group_number)
>> > >>> __init__(_object*, int space_group_number,
>> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> > >>> std::allocator<char> > extension='')
>> > >>> __init__(_object*, int space_group_number,
>> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> > >>> std::allocator<char> > extension='', std::__cxx11::basic_string<char,
>> > >>> std::char_traits<char>, std::allocator<char> > table_id='')
>> > >>> __init__(_object*, std::__cxx11::basic_string<char,
>> > >>> std::char_traits<char>, std::allocator<char> > symbol)
>> > >>> __init__(_object*, std::__cxx11::basic_string<char,
>> > >>> std::char_traits<char>, std::allocator<char> > symbol,
>> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> > >>> std::allocator<char> > table_id='')
>> > >>>
>> > >>>
>> > >>> When I tried to set LIBTBX_BUILD to /home/gergely/anaconda3 (this is
>> > >>> sys.prefix on my system), I got other problems when importing. Can
>> > >>> LIBTBX_BUILD be set in the conda package?
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> ---------------------------------------------------------------------------
>> > >>> FileNotFoundError Traceback (most recent
>> call last)
>> > >>> <ipython-input-1-7e2b144826cc> in <module>
>> > >>> 26 from cctbx import sgtbx
>> > >>> 27 #from cctbx import miller
>> > >>> ---> 28 from iotbx import reflection_file_reader, mtz
>> > >>> 29 sns.set_context("poster")
>> > >>> 30 plt.rcParams.update({'figure.autolayout': True})
>> > >>>
>> > >>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__reflection-5Ffile-5Freader.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=U5BvLmuCgufvN_LRxQw9n78p33jNaRrXVbH1BRSXgxg&e=
>> > >>> in <module>
>> > >>> 53
>> > >>> 54 from __future__ import absolute_import, division,
>> print_function
>> > >>> ---> 55 from iotbx import mtz
>> > >>> 56 from iotbx.scalepack import merge as scalepack_merge
>> > >>> 57 from iotbx.scalepack import no_merge_original_index as
>> > >>> scalepack_no_merge
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
>> <module>
>> > >>> 9 import iotbx_mtz_ext as ext
>> > >>> 10
>> > >>> ---> 11 from iotbx.mtz import extract_from_symmetry_lib
>> > >>> 12 from cctbx import xray
>> > >>> 13 import cctbx.xray.observation_types
>> > >>>
>> > >>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/https://urldefense.proofpoint.com/v2/url?u=http-3A__extract-5Ffrom-5Fsymmetry-5Flib.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=k3WR6VZd4-PDULPXl4O0Vwz9q3LCaocUUhK_ehTn9UE&e=
>> > >>> in <module>
>> > >>> 1 from __future__ import absolute_import, division,
>> print_function
>> > >>> 2 from cctbx import sgtbx
>> > >>> ----> 3 import libtbx.load_env
>> > >>> 4 import os.path as op
>> > >>> 5 from six.moves import range
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__load-5Fenv.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=rJbfdPr3ymiz7Ir508_VGIhspAKCOwozKhY5I6OXux0&e= in
>> <module>
>> > >>> 3 import libtbx.env_config
>> > >>> 4 import os
>> > >>> ----> 5 libtbx.env = libtbx.env_config.unpickle()
>> > >>> 6 libtbx.env.set_os_environ_all_dist()
>> > >>> 7 libtbx.env.dispatcher_name =
>> os.environ.get("LIBTBX_DISPATCHER_NAME")
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__env-5Fconfig.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=UrsFhQ0dxNNuzXmOT3D6pnGJ5lXko4mfrKceNgBaQV4&e= in
>> unpickle()
>> > >>> 2605 build_path = os.environ["LIBTBX_BUILD"]
>> > >>> 2606
>> set_preferred_sys_prefix_and_sys_executable(build_path=build_path)
>> > >>> -> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")
>> > >>> 2608 env = pickle.load(libtbx_env)
>> > >>> 2609 if (env.python_version_major_minor !=
>> sys.version_info[:2]):
>> > >>>
>> > >>> FileNotFoundError: [Errno 2] No such file or directory:
>> > >>> '/home/gergely/anaconda3/libtbx_env'
>> > >>>
>> > >>> On Wed, Dec 18, 2019 at 9:10 AM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >>> >
>> > >>> > Hi Gergely,
>> > >>> >
>> > >>> > I've uploaded linux packages to a new channel, cctbx-dev, and you
>> can install it with
>> > >>> >
>> > >>> > conda install -c cctbx-dev cctbx
>> > >>> >
>> > >>> > in your current environment. You should set your ~/.condarc file
>> to pull the other dependencies from the conda-forge channel first, so
>> put
>> conda-forge above cctbx. Mine looks like
>> > >>> >
>> > >>> > channels:
>> > >>> > - conda-forge
>> > >>> > - defaults
>> > >>> > - cctbx
>> > >>> >
>> > >>> > Do you need dxtbx for your scripts? This package does not build
>> that part. I think the plan is to build a separate conda package for
>> dxtbx
>> so that it can be updated more frequently. I can rebuild the packages
>> to
>> include it for testing, but the one being submitted to conda-forge
>> will not
>> have it.
>> > >>> >
>> > >>> > Also, your error message is probably due to version of HDF5 that
>> the development build installs. The
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__bootstrap.py&d=DwICAg&c…
>> script will install
>> 1.10.4, but your other dependency is looking for 1.10.5. Installing
>> this
>> cctbx conda package should install 1.10.5, which should fix the issue.
>> Also, I'm updating those environments and 1.10.5 will be the new
>> default
>> version.
>> > >>> >
>> > >>> > Lastly, the dispatchers will not work in these packages because
>> the old paths during the build process are still in them and many of
>> them
>> expect some additional information that has not been updated in the
>> packages yet. I'm in the process of doing that and will update the
>> cctbx-dev channel when that's done. However, by starting python, you
>> can
>> import cctbx modules. So you can run commands like
>> > >>> >
>> > >>> > from scitbx.array_family import flex
>> > >>> > a = flex.random_double(1000000)
>> > >>> > b = flex.min_max_mean_double(a)
>> > >>> > b.min
>> > >>> > b.max
>> > >>> > b.mean
>> > >>> >
>> > >>> > Thanks!
>> > >>> >
>> > >>> > --
>> > >>> > Billy K. Poon
>> > >>> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > >>> > Lawrence Berkeley National Laboratory
>> > >>> > 1 Cyclotron Road, M/S 33R0345
>> > >>> > Berkeley, CA 94720
>> > >>> > Tel: (510) 486-5709
>> > >>> > Fax: (510) 486-5909
>> > >>> > Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
>> > >>> >
>> > >>> >
>> > >>> > On Tue, Dec 17, 2019 at 5:39 AM Gergely Katona <gkatona(a)gmail.com>
>> wrote:
>> > >>> >>
>> > >>> >> Dear Billy,
>> > >>> >>
>> > >>> >> Thank you for the detailed explanation, I look forward to do
>> further
>> > >>> >> testing! I aim to pool all modules under the same environment, if
>> > >>> >> there are conflicts then I just try to reshuffle the order of
>> imports.
>> > >>> >> So far this did not cause problems for me even when I was using
>> system
>> > >>> >> python, but of course it is not the most prudent thing to do. With
>> > >>> >> anaconda everything is much more standardized and isolated
>> already and
>> > >>> >> without being superuser I can have my familiar environment at any
>> > >>> >> synchrotron based cluster. It is great that cctbx will be an
>> integral
>> > >>> >> part of this ecosystem and this was also the last thing holding me
>> > >>> >> back from adopting python3. About hdf5, pymc3 requires it and
>> > >>> >> importing causes a kernel restart with the following error
>> messages,
>> > >>> >> curiously if I import h5py first this can be avoided.
>> > >>> >>
>> > >>> >> Best wishes,
>> > >>> >>
>> > >>> >> Gergely
>> > >>> >>
>> > >>> >> import pymc3 as pm
>> > >>> >>
>> > >>> >> Warning! ***HDF5 library version mismatched error***
>> > >>> >> The HDF5 header files used to compile this application do not
>> match
>> > >>> >> the version used by the HDF5 library to which this application is
>> linked.
>> > >>> >> Data corruption or segmentation faults may occur if the
>> application continues.
>> > >>> >> This can happen when an application was compiled by one version
>> of HDF5 but
>> > >>> >> linked with a different version of static or shared HDF5 library.
>> > >>> >> You should recompile the application or check your shared library
>> related
>> > >>> >> settings such as 'LD_LIBRARY_PATH'.
>> > >>> >> You can, at your own risk, disable this warning by setting the
>> environment
>> > >>> >> variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.
>> > >>> >> Setting it to 2 or higher will suppress the warning messages
>> totally.
>> > >>> >> Headers are 1.10.4, library is 1.10.5
>> > >>> >> SUMMARY OF THE HDF5 CONFIGURATION
>> > >>> >> =================================
>> > >>> >>
>> > >>> >> General Information:
>> > >>> >> -------------------
>> > >>> >> HDF5 Version: 1.10.5
>> > >>> >> Configured on: Tue Oct 22 12:02:13 UTC 2019
>> > >>> >> Configured by: conda@16247e67ecd5
>> > >>> >> Host system: x86_64-conda_cos6-linux-gnu
>> > >>> >> Uname information: Linux 16247e67ecd5
>> 4.15.0-1059-azure
>> > >>> >> #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64
>> > >>> >> GNU/Linux
>> > >>> >> Byte sex: little-endian
>> > >>> >> Installation point: /home/gergely/anaconda3
>> > >>> >>
>> > >>> >> Compiling Options:
>> > >>> >> ------------------
>> > >>> >> Build Mode: production
>> > >>> >> Debugging Symbols: no
>> > >>> >> Asserts: no
>> > >>> >> Profiling: no
>> > >>> >> Optimization Level: high
>> > >>> >>
>> > >>> >> Linking Options:
>> > >>> >> ----------------
>> > >>> >> Libraries: static, shared
>> > >>> >> Statically Linked Executables:
>> > >>> >> LDFLAGS: -Wl,-O2 -Wl,--sort-common
>> > >>> >> -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags
>> > >>> >> -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib
>> > >>> >> -Wl,-rpath-link,/home/gergely/anaconda3/lib
>> > >>> >> -L/home/gergely/anaconda3/lib
>> > >>> >> H5_LDFLAGS:
>> > >>> >> AM_LDFLAGS: -L/home/gergely/anaconda3/lib
>> > >>> >> Extra libraries: -lrt -lpthread -lz -ldl -lm
>> > >>> >> Archiver:
>> > >>> >>
>> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar
>> > >>> >> AR_FLAGS: cr
>> > >>> >> Ranlib:
>> > >>> >>
>> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib
>> > >>> >>
>> > >>> >> Languages:
>> > >>> >> ----------
>> > >>> >> C: yes
>> > >>> >> C Compiler:
>> > >>> >>
>> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc
>> > >>> >> CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2
>> > >>> >> -I/home/gergely/anaconda3/include
>> > >>> >> H5_CPPFLAGS: -D_GNU_SOURCE
>> > >>> >> -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API
>> > >>> >> AM_CPPFLAGS:
>> -I/home/gergely/anaconda3/include
>> > >>> >> C Flags: -march=nocona -mtune=haswell
>> > >>> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
>> > >>> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include
>> > >>> >>
>> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5
>> > >>> >>
>> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix
>> > >>> >> H5 C Flags: -std=c99 -pedantic -Wall
>> -Wextra
>> > >>> >> -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual
>> -Wconversion
>> > >>> >> -Wdeclaration-after-statement -Wdisabled-optimization
>> -Wfloat-equal
>> > >>> >> -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations
>> > >>> >> -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs
>> > >>> >> -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls
>> > >>> >> -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum
>> -Wundef
>> > >>> >> -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings
>> > >>> >> -finline-functions -s -Wno-inline -Wno-aggregate-return
>> > >>> >> -Wno-missing-format-attribute -Wno-missing-noreturn -O
>> > >>> >> AM C Flags:
>> > >>> >> Shared C Library: yes
>> > >>> >> Static C Library: yes
>> > >>> >>
>> > >>> >>
>> > >>> >> Fortran: yes
>> > >>> >> Fortran Compiler:
>> > >>> >>
>> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran
>> > >>> >> Fortran Flags:
>> > >>> >> H5 Fortran Flags: -pedantic -Wall -Wextra
>> -Wunderflow
>> > >>> >> -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2
>> > >>> >> AM Fortran Flags:
>> > >>> >> Shared Fortran Library: yes
>> > >>> >> Static Fortran Library: yes
>> > >>> >>
>> > >>> >> C++: yes
>> > >>> >> C++ Compiler:
>> > >>> >>
>> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++
>> > >>> >> C++ Flags: -fvisibility-inlines-hidden
>> > >>> >> -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell
>> > >>> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
>> > >>> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include
>> > >>> >>
>> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5
>> > >>> >>
>> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix
>> > >>> >> H5 C++ Flags: -pedantic -Wall -W -Wundef
>> -Wshadow
>> > >>> >> -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings
>> -Wconversion
>> > >>> >> -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual
>> > >>> >> -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor
>> > >>> >> -Wctor-dtor-privacy -Wabi -finline-functions -s -O
>> > >>> >> AM C++ Flags:
>> > >>> >> Shared C++ Library: yes
>> > >>> >> Static C++ Library: yes
>> > >>> >>
>> > >>> >> Java: no
>> > >>> >>
>> > >>> >>
>> > >>> >> Features:
>> > >>> >> ---------
>> > >>> >> Parallel HDF5: no
>> > >>> >> Parallel Filtered Dataset Writes: no
>> > >>> >> Large Parallel I/O: no
>> > >>> >> High-level library: yes
>> > >>> >> Threadsafety: yes
>> > >>> >> Default API mapping: v110
>> > >>> >> With deprecated public symbols: yes
>> > >>> >> I/O filters (external): deflate(zlib)
>> > >>> >> MPE: no
>> > >>> >> Direct VFD: no
>> > >>> >> dmalloc: no
>> > >>> >> Packages w/ extra debug output: none
>> > >>> >> API tracing: no
>> > >>> >> Using memory checker: yes
>> > >>> >> Memory allocation sanity checks: no
>> > >>> >> Function stack tracing: no
>> > >>> >> Strict file format checks: no
>> > >>> >> Optimization instrumentation: no
>> > >>> >>
>> > >>> >> On Tue, Dec 17, 2019 at 8:35 AM Billy Poon <BKPoon(a)lbl.gov>
>> wrote:
>> > >>> >> >
>> > >>> >> > Hi Gergely,
>> > >>> >> >
>> > >>> >> > Let me build the test package tomorrow. All the gritty details
>> for building with conda is being finalized and the official
>> documentation
>> will be updated to describe the steps. It would be too confusing to
>> keep
>> changing the documentation as the process evolves. You do have the
>> general
>> process, though, which is summarized as follows.
>> > >>> >> >
>> > >>> >> > 1) Installing dependencies. The cctbx_dependencies metapackage
>> was an initial approach for managing the CCTBX dependencies, but after
>> contacting the conda-forge folks, they recommended using the
>> --only-deps
>> flag. So when the CCTBX conda package is available, you'll be able to
>> get a
>> set of dependencies with,
>> > >>> >> >
>> > >>> >> > conda install -c conda-forge --only-deps cctbx
>> > >>> >> >
>> > >>> >> > By default, the https://urldefense.proofpoint.com/v2/url?u=http-3A__bootstrap.py&d=DwICAg&c… file will automatically install a
>> set of dependencies in the "conda_base" directory (and a conda
>> installation
>> if one is not found). It just uses standard conda environment files
>> located
>> in libtbx/auto_build/conda_envs, so you do not need to install
>> cctbx_dependencies as a separate step. The environment files avoid
>> channel
>> issues by explicitly defining the channel to pull the packages from
>> and the
>> cctbx channel just stores copies of packages from conda-forge. There
>> were
>> issues earlier where the conda-forge packages would sometimes be moved
>> to a
>> different label. The --use-conda flag also accepts a path to
>> $CONDA_PREFIX
>> if you want to use a specific environment for testing.
>> > >>> >> >
>> > >>> >> > 2) Building. The https://urldefense.proofpoint.com/v2/url?u=http-3A__bootstrap.py&d=DwICAg&c… file handles that with SCons.
>> > >>> >> >
>> > >>> >> > 3) Running. After building, there should be a https://urldefense.proofpoint.com/v2/url?u=http-3A__setpaths.sh&d=DwICAg&c=… (and
>> .csh) file that adds build/bin to your path. The build/bin directory
>> has
>> our dispatchers, which are just scripts that set up the environment
>> variables for you. This prevents other programs from loading our
>> libraries,
>> whose versions may conflict. You should see that there is a "python"
>> dispatcher, which is a convenience for developers. Otherwise, you can
>> use
>> libtbx.python, which will be able to import CCTBX modules.
>> > >>> >> >
>> > >>> >> > The running part is where the conda package for CCTBX will be
>> different than this build. Since our Python files and extensions
>> modules
>> will be in the "site-packages" directory for Python, the PYTHONPATH
>> variable will not be needed (and conda suggests that that variable not
>> be
>> set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so
>> LD_LIBRARY_PATH is not needed. In an active environment, PATH will
>> already
>> be modified. And then our LIBTBX_BUILD directory can be set to
>> sys.prefix.
>> > >>> >> >
>> > >>> >> > So with the conda package, you would only need to activate your
>> conda environment and CCTBX should integrate with other conda
>> packages.
>> What is the conflict with hdf5? That's something that should be fixed.
>> Thanks!
>> > >>> >> >
>> > >>> >> > --
>> > >>> >> > Billy K. Poon
>> > >>> >> > Research Scientist, Molecular Biophysics and Integrated
>> Bioimaging
>> > >>> >> > Lawrence Berkeley National Laboratory
>> > >>> >> > 1 Cyclotron Road, M/S 33R0345
>> > >>> >> > Berkeley, CA 94720
>> > >>> >> > Tel: (510) 486-5709
>> > >>> >> > Fax: (510) 486-5909
>> > >>> >> > Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
>> > >>> >> >
>> > >>> >> >
>> > >>> >> > On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona <
>> gkatona(a)gmail.com> wrote:
>> > >>> >> >>
>> > >>> >> >> Dear Billy,
>> > >>> >> >>
>> > >>> >> >> Thank you for this update and for your efforts! I found a
>> solution and
>> > >>> >> >> indeed most things already work in anaconda3. The steps I took
>> (even
>> > >>> >> >> if these instructions will have short expiry date):
>> > >>> >> >>
>> > >>> >> >> Modified .condarc with:
>> > >>> >> >> channels:
>> > >>> >> >> - cctbx
>> > >>> >> >> - conda-forge
>> > >>> >> >> - defaults
>> > >>> >> >>
>> > >>> >> >> run
>> > >>> >> >> conda conda install cctbx_dependencies python=3.6
>> > >>> >> >>
>> > >>> >> >> I expect this will work without python=3.6 in the near future.
>> > >>> >> >>
>> > >>> >> >> Then compiling cctbx with anaconda3 python went without any
>> problem
>> > >>> >> >> when using these flags:
>> > >>> >> >> python https://urldefense.proofpoint.com/v2/url?u=http-3A__bootstrap.py&d=DwICAg&c… --use-conda --python3 --nproc=12
>> > >>> >> >>
>> > >>> >> >> Then I probably did the unorthodox thing and sourced these
>> directories:
>> > >>> >> >>
>> > >>> >> >> setenv LIBTBX_BUILD /home/gergely/cctbx/build
>> > >>> >> >> setenv PATH ${PATH}:/home/gergely/cctbx/build/bin
>> > >>> >> >> setenv PYTHONPATH
>> > >>> >> >>
>> /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH
>> > >>> >> >> setenv LD_LIBRARY_PATH
>> > >>> >> >>
>> /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH
>> > >>> >> >>
>> > >>> >> >> There is probably a better way to put this into conda
>> environment.
>> > >>> >> >> With these steps I could run one of my scripts depending on
>> cctbx. The
>> > >>> >> >> only problem I found is that hdf5 library had conflict with
>> another
>> > >>> >> >> package in conda, but I does not find this as a showstopper.
>> > >>> >> >>
>> > >>> >> >> Best wishes,
>> > >>> >> >>
>> > >>> >> >> Gergely
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >> On Fri, Dec 13, 2019 at 7:23 PM Billy Poon <BKPoon(a)lbl.gov>
>> wrote:
>> > >>> >> >> >
>> > >>> >> >> > Hi Gergely,
>> > >>> >> >> >
>> > >>> >> >> > It's still a work in progress. I'm sorting out some Windows
>> issues right now. I can probably build a test package on a separate
>> channel
>> for people that want to test it (let's say next week?). I'll provide
>> instructions, but basically, the test conda package will be in its own
>> separate channel and the dependencies will be pulled from the
>> conda-forge
>> channel. I want most things to be working correctly on Python 2.7,
>> 3.6,
>> 3.7, and 3.8 on all 3 platforms.
>> > >>> >> >> >
>> > >>> >> >> > Thanks!
>> > >>> >> >> >
>> > >>> >> >> > --
>> > >>> >> >> > Billy K. Poon
>> > >>> >> >> > Research Scientist, Molecular Biophysics and Integrated
>> Bioimaging
>> > >>> >> >> > Lawrence Berkeley National Laboratory
>> > >>> >> >> > 1 Cyclotron Road, M/S 33R0345
>> > >>> >> >> > Berkeley, CA 94720
>> > >>> >> >> > Tel: (510) 486-5709
>> > >>> >> >> > Fax: (510) 486-5909
>> > >>> >> >> > Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
>> > >>> >> >> >
>> > >>> >> >> >
>> > >>> >> >> > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona <
>> gkatona(a)gmail.com> wrote:
>> > >>> >> >> >>
>> > >>> >> >> >> Dear Billy,
>> > >>> >> >> >>
>> > >>> >> >> >> This sounds very promising and exciting. I am not sure if
>> cctbx is
>> > >>> >> >> >> already functional as a conda package in anaconda3 (Linux)
>> or this is
>> > >>> >> >> >> still work in progress. My technical expertise does not
>> allow me to
>> > >>> >> >> >> tell the difference. What I tried:
>> > >>> >> >> >>
>> > >>> >> >> >> Fresh install of anaconda3. Adding - cctbx and -
>> conda-forge to
>> > >>> >> >> >> .condarc . Installing pyside2 with conda. Running conda
>> install
>> > >>> >> >> >> conda_dependencies . I get a lot package version conflict,
>> and I
>> > >>> >> >> >> cannot import cctbx or iotbx to anaconda python. Am I
>> following the
>> > >>> >> >> >> right instructions? Or it is too early to expect that cctbx
>> works when
>> > >>> >> >> >> installed through conda?
>> > >>> >> >> >>
>> > >>> >> >> >> Best wishes,
>> > >>> >> >> >>
>> > >>> >> >> >> Gergely
>> > >>> >> >> >>
>> > >>> >> >> >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon <BKPoon(a)lbl.gov>
>> wrote:
>> > >>> >> >> >> >
>> > >>> >> >> >> > Hi all,
>> > >>> >> >> >> >
>> > >>> >> >> >> > For a brief update, I have submitted a recipe for
>> cctbxlite to conda-forge (
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_conda-2Dfor…
>> ) and support for
>> Python 3.7 and 3.8 is being added (
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_cctbx_cctbx…
>> ). With some fixes for
>> Windows
>> (https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_cctbx_cctbx…
>> ), all platforms
>> (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and
>> 3.8.
>> Some additional changes will be needed to get Windows to work with
>> Python 3
>> and for tests to pass with Boost 1.70.0. That will enable the
>> conda-forge
>> recipe to build for all platforms and for all supported versions of
>> Python.
>> > >>> >> >> >> >
>> > >>> >> >> >> > Currently, the conda-forge recipe will install into the
>> conda python and cctbx imports can be done without sourcing the
>> environment
>> scripts. It looks like a lot of the environment variables being set in
>> the
>> dispatchers can be removed since the Python files and C++ extensions
>> are in
>> the right places. I'll update the libtbx_env file so that commands
>> that
>> load the environment can work correctly.
>> > >>> >> >> >> >
>> > >>> >> >> >> > --
>> > >>> >> >> >> > Billy K. Poon
>> > >>> >> >> >> > Research Scientist, Molecular Biophysics and Integrated
>> Bioimaging
>> > >>> >> >> >> > Lawrence Berkeley National Laboratory
>> > >>> >> >> >> > 1 Cyclotron Road, M/S 33R0345
>> > >>> >> >> >> > Berkeley, CA 94720
>> > >>> >> >> >> > Tel: (510) 486-5709
>> > >>> >> >> >> > Fax: (510) 486-5909
>> > >>> >> >> >> > Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
>> > >>> >> >> >> >
>> > >>> >> >> >> >
>> > >>> >> >> >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll <
>> tic20(a)cam.ac.uk> wrote:
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> Hi Luc,
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> That sounds promising. From there, I?d need to work out
>> how to make a fully-packaged installer (basically a modified wheel
>> file)
>> for the ChimeraX ToolShed - the aim is for the end user to not have to
>> worry about any of this. That adds a couple of complications - e.g.
>> $LIBTBX_BUILD would need to be set dynamically before first import -
>> but
>> doesn?t seem insurmountable.
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> Thanks,
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> Tristan
>> > >>> >> >> >> >>
>> > >>> >> >> >> >>
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> Tristan Croll
>> > >>> >> >> >> >> Research Fellow
>> > >>> >> >> >> >> Cambridge Institute for Medical Research
>> > >>> >> >> >> >> University of Cambridge CB2 0XY
>> > >>> >> >> >> >>
>> > >>> >> >> >> >>
>> > >>> >> >> >> >>
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis <
>> luc_j_bourhis(a)mac.com> wrote:
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> > Hi Tristan,
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> > cctbx could be built to use your ChimeraX python, now
>> that cctbx is moving to Python 3. The option ?with-python is there for
>> that
>> with the bootstrap script. The specific environment setup boil down to
>> setting two environment variable LIBTBX_BUILD and either
>> LD_LIBRARY_PATH on
>> Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work
>> within a
>> framework such as ChimeraX, that should not be difficult to ensure
>> those
>> two variables are set.
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> > Best wishes,
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> > Luc
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll <
>> tic20(a)cam.ac.uk> wrote:
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >> To add my two cents on this: probably the second-most
>> common question I've had about ISOLDE's implementation is, "why didn't
>> you
>> use CCTBX?". The honest answer to that is, "I didn't know how."
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >> Still don't, really - although the current
>> developments are rather promising. The problem I've faced is that
>> CCTBX was
>> designed as its own self-contained Python (2.7, until very recently)
>> environment, with its own interpreter and a lot of very specific
>> environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which
>> is
>> *also* its own self-contained Python (3.7) environment. To plug one
>> into
>> the other in that form... well, I don't think I'm a good enough
>> programmer
>> to really know where to start.
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >> The move to Conda and a more modular CCTBX
>> architecture should make a lot more possible in that direction. Pip
>> would
>> be even better for me personally (ChimeraX can install directly from
>> the
>> PyPI, but doesn't interact with Conda) - but I understand pretty well
>> the
>> substantial challenge that would amount to (not least being that the
>> PyPI
>> imposes a limit - around 100MB from memory? - on the size of an
>> individual
>> package).
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >> Best regards,
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >> Tristan
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote:
>> > >>> >> >> >> >> >>> Hi Graeme,
>> > >>> >> >> >> >> >>> Yes, I know. But ?black" is a program doing a very
>> particular task
>> > >>> >> >> >> >> >>> (code formatting from the top of my head). Requiring
>> to use a wrapper
>> > >>> >> >> >> >> >>> for python itself is another level. But ok, I think
>> I am mellowing to
>> > >>> >> >> >> >> >>> the idea after all! Talking with people around me,
>> and extrapolating,
>> > >>> >> >> >> >> >>> I would bet that, right now, a great majority of
>> people interested by
>> > >>> >> >> >> >> >>> cctbx in pip have already used the cctbx, so they
>> know about the
>> > >>> >> >> >> >> >>> Python wrapper, and they would not be too sanguine
>> about that. My
>> > >>> >> >> >> >> >>> concern is for the future, when pip will be the
>> first time some people
>> > >>> >> >> >> >> >>> use cctbx. Big fat warning notices on PyPI page and
>> a better error
>> > >>> >> >> >> >> >>> message when cctbx fails because LIBTBX_BUILD is not
>> set would be
>> > >>> >> >> >> >> >>> needed but that could be all right.
>> > >>> >> >> >> >> >>> If we do a pip installer, we should aim at a minimal
>> install: cctbx,
>> > >>> >> >> >> >> >>> iotbx and their dependencies, and that?s it.
>> > >>> >> >> >> >> >>> Best wishes,
>> > >>> >> >> >> >> >>> Luc
>> > >>> >> >> >> >> >>>> On 23 Aug 2019, at 07:17,
>> Graeme.Winter(a)Diamond.ac.uk <Graeme.Winter(a)diamond.ac.uk> wrote:
>> > >>> >> >> >> >> >>>> Without discussing the merits of this or whether we
>> _choose_ to make the move to supporting PIP, I am certain it would be
>> _possible_ - many other packages make dispatcher scripts when you pip
>> install them e.g.
>> > >>> >> >> >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat
>> $(which black)
>> > >>> >> >> >> >> >>>>
>> /Library/Frameworks/Python.framework/Versions/3.6/bin/black
>> > >>> >> >> >> >> >>>>
>> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
>> > >>> >> >> >> >> >>>> # -*- coding: utf-8 -*-
>> > >>> >> >> >> >> >>>> import re
>> > >>> >> >> >> >> >>>> import sys
>> > >>> >> >> >> >> >>>> from black import main
>> > >>> >> >> >> >> >>>> if __name__ == '__main__':
>> > >>> >> >> >> >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$',
>> '', sys.argv[0])
>> > >>> >> >> >> >> >>>> sys.exit(main())
>> > >>> >> >> >> >> >>>> So we _could_ work around the absence of
>> LIBTBX_BUILD etc. in the system. Whether or not we elect to do the
>> work is
>> a different question, and it seems clear that here are very mixed
>> opinions
>> on this.
>> > >>> >> >> >> >> >>>> Best wishes Graeme
>> > >>> >> >> >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis <
>> luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote:
>> > >>> >> >> >> >> >>>> Hi,
>> > >>> >> >> >> >> >>>> Even if we managed to ship our the boost dynamic
>> libraries with pip, it would still not be pip-like, as we would still
>> need
>> our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal
>> pip
>> packages work with the standard python exe. LD_LIBRARY_PATH, we could
>> get
>> around that by changing the way we compile, using -Wl,-R, which is the
>> runtime equivalent of build time -L. That?s a significant change that
>> would
>> need to be tested. But there is no way around setting LIBTBX_BUILD
>> right
>> now. Leaving that to the user is horrible. Perhaps there is a way to
>> hack
>> libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__env-5Fconfig.py&…
>> so that we can hardwire LIBTBX_BUILD in there when pip
>> installs?
>> > >>> >> >> >> >> >>>> Best wishes,
>> > >>> >> >> >> >> >>>> Luc
>> > >>> >> >> >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis <
>> luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote:
>> > >>> >> >> >> >> >>>> Hi,
>> > >>> >> >> >> >> >>>> I did look into that many years ago, and even toyed
>> with building a pip installer. What stopped me is the exact conclusion
>> you
>> reached too: the user would not have the pip experience he expects.
>> You are
>> right that it is a lot of effort but is it worth it? Considering that
>> remark, I don?t think so. Now, Conda was created specifically to go
>> beyond
>> pip pure-python-only support. Since cctbx has garnered support for
>> Conda,
>> the best avenue imho is to go the extra length to have a package on
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__Anaconda.org&d=DwICAg&c…
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__anaconda.org_&d=DwICAg&…
>> >, and then to advertise it hard to
>> every potential user out there.
>> > >>> >> >> >> >> >>>> Best wishes,
>> > >>> >> >> >> >> >>>> Luc
>> > >>> >> >> >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster <
>> asbrewster(a)lbl.gov<mailto:[email protected]>> wrote:
>> > >>> >> >> >> >> >>>> Hi, to avoid clouding Dorothee's documentation
>> email thread, which I think is a highly useful enterprise, here's some
>> thoughts about putting cctbx into pip. Pip doesn't install non-python
>> dependencies well. I don't think boost is available as a package on
>> pip
>> (at least the package version we use). wxPython4 isn't portable
>> through
>> pip (
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.wxpython.org_How-…
>> ).
>> MPI libraries are system dependent. If cctbx were a pure python
>> package,
>> pip would be fine, but cctbx is not.
>> > >>> >> >> >> >> >>>> All that said, we could build a manylinux1 version
>> of cctbx and upload it to PyPi (I'm just learning about this). For a
>> pip
>> package to be portable (which is a requirement for cctbx), it needs to
>> conform to PEP513, the manylinux1 standard (
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.python.org_dev_pep…
>> ). For example, numpy is built
>> according to this standard (see
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__pypi.org_project_numpy…
>> ,
>> where you'll see the manylinux1 wheel). Note, the manylinux1 standard
>> is
>> built with Centos 5.11 which we no longer support.
>> > >>> >> >> >> >> >>>> There is also a manylinux2010 standard, which is
>> based on Centos 6
>> (https://urldefense.proofpoint.com/v2/url?u=https-3A__www.python.org_dev_pep…
>> ). This is
>> likely a more attainable target (note though by default C++11 is not
>> supported on Centos 6).
>> > >>> >> >> >> >> >>>> If we built a manylinuxX version of cctbx and
>> uploaded it to PyPi, the user would need all the non-python
>> dependencies.
>> There's no way to specify these in pip. For example, cctbx requires
>> boost
>> 1.63 or better. The user will need to have it in a place their python
>> can
>> find it, or we could package it ourselves and supply it, similar to
>> how the
>> pip h5py package now comes with an hd5f library, or how the pip numpy
>> package includes an openblas library. We'd have to do the same for
>> any
>> packages we depend on that aren't on pip using the manylinux
>> standards,
>> such as wxPython4.
>> > >>> >> >> >> >> >>>> Further, we need to think about how dials and other
>> cctbx-based packages interact. If pip install cctbx is set up, how
>> does
>> pip install dials work, such that any dials shared libraries can find
>> the
>> cctbx libraries? Can shared libraries from one pip package link
>> against
>> libraries in another pip package? Would each package need to supply
>> its
>> own boost? Possibly this is well understood in the pip field, but not
>> by
>> me :)
>> > >>> >> >> >> >> >>>> Finally, there's the option of providing a source
>> pip package. This would require the full compiler toolchain for any
>> given
>> platform (macOS, linux, windows). These are likely available for
>> developers, but not for general users.
>> > >>> >> >> >> >> >>>> Anyway, these are some of the obstacles. Not
>> saying it isn't possible, it's just a lot of effort.
>> > >>> >> >> >> >> >>>> Thanks,
>> > >>> >> >> >> >> >>>> -Aaron
>> > >>> >> >> >> >> >>>> _______________________________________________
>> > >>> >> >> >> >> >>>> cctbxbb mailing list
>> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:
>> cctbxbb(a)phenix-online.org>
>> > >>> >> >> >> >> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >> >> >>>> _______________________________________________
>> > >>> >> >> >> >> >>>> cctbxbb mailing list
>> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:
>> cctbxbb(a)phenix-online.org>
>> > >>> >> >> >> >> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >> >> >>>> _______________________________________________
>> > >>> >> >> >> >> >>>> cctbxbb mailing list
>> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:
>> cctbxbb(a)phenix-online.org>
>> > >>> >> >> >> >> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >> >> >>>> --
>> > >>> >> >> >> >> >>>> This e-mail and any attachments may contain
>> confidential, copyright and or privileged material, and are for the
>> use of
>> the intended addressee only. If you are not the intended addressee or
>> an
>> authorised recipient of the addressee please notify us of receipt by
>> returning the e-mail and do not use, copy, retain, distribute or
>> disclose
>> the information in or attached to the e-mail.
>> > >>> >> >> >> >> >>>> Any opinions expressed within this e-mail are those
>> of the individual and not necessarily of Diamond Light Source Ltd.
>> > >>> >> >> >> >> >>>> Diamond Light Source Ltd. cannot guarantee that
>> this e-mail or any attachments are free from viruses and we cannot
>> accept
>> liability for any damage which you may sustain as a result of software
>> viruses which may be transmitted in or with the message.
>> > >>> >> >> >> >> >>>> Diamond Light Source Limited (company no. 4375679).
>> Registered in England and Wales with its registered office at Diamond
>> House, Harwell Science and Innovation Campus, Didcot, Oxfordshire,
>> OX11
>> 0DE, United Kingdom
>> > >>> >> >> >> >> >>>> _______________________________________________
>> > >>> >> >> >> >> >>>> cctbxbb mailing list
>> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org
>> > >>> >> >> >> >> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >> >> >>> _______________________________________________
>> > >>> >> >> >> >> >>> cctbxbb mailing list
>> > >>> >> >> >> >> >>> cctbxbb(a)phenix-online.org
>> > >>> >> >> >> >> >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >> _______________________________________________
>> > >>> >> >> >> >> >> cctbxbb mailing list
>> > >>> >> >> >> >> >> cctbxbb(a)phenix-online.org
>> > >>> >> >> >> >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> > _______________________________________________
>> > >>> >> >> >> >> > cctbxbb mailing list
>> > >>> >> >> >> >> > cctbxbb(a)phenix-online.org
>> > >>> >> >> >> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >> >>
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> _______________________________________________
>> > >>> >> >> >> >> cctbxbb mailing list
>> > >>> >> >> >> >> cctbxbb(a)phenix-online.org
>> > >>> >> >> >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >> >
>> > >>> >> >> >> > _______________________________________________
>> > >>> >> >> >> > cctbxbb mailing list
>> > >>> >> >> >> > cctbxbb(a)phenix-online.org
>> > >>> >> >> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >>
>> > >>> >> >> >>
>> > >>> >> >> >>
>> > >>> >> >> >> --
>> > >>> >> >> >> Gergely Katona, PhD
>> > >>> >> >> >> Associate Professor
>> > >>> >> >> >> Department of Chemistry and Molecular Biology, University
>> of Gothenburg
>> > >>> >> >> >> Box 462, 40530 G?teborg, Sweden
>> > >>> >> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax:
>> +46-31-786-3910
>> > >>> >> >> >> Web: https://urldefense.proofpoint.com/v2/url?u=http-3A__katonalab.eu&d=DwICAg&c… , Email: gergely.katona(a)gu.se
>> > >>> >> >> >>
>> > >>> >> >> >> _______________________________________________
>> > >>> >> >> >> cctbxbb mailing list
>> > >>> >> >> >> cctbxbb(a)phenix-online.org
>> > >>> >> >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >
>> > >>> >> >> > _______________________________________________
>> > >>> >> >> > cctbxbb mailing list
>> > >>> >> >> > cctbxbb(a)phenix-online.org
>> > >>> >> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >> --
>> > >>> >> >> Gergely Katona, PhD
>> > >>> >> >> Associate Professor
>> > >>> >> >> Department of Chemistry and Molecular Biology, University of
>> Gothenburg
>> > >>> >> >> Box 462, 40530 G?teborg, Sweden
>> > >>> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax:
>> +46-31-786-3910
>> > >>> >> >> Web: https://urldefense.proofpoint.com/v2/url?u=http-3A__katonalab.eu&d=DwICAg&c… , Email: gergely.katona(a)gu.se
>> > >>> >> >>
>> > >>> >> >> _______________________________________________
>> > >>> >> >> cctbxbb mailing list
>> > >>> >> >> cctbxbb(a)phenix-online.org
>> > >>> >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >
>> > >>> >> > _______________________________________________
>> > >>> >> > cctbxbb mailing list
>> > >>> >> > cctbxbb(a)phenix-online.org
>> > >>> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >>
>> > >>> >>
>> > >>> >>
>> > >>> >> --
>> > >>> >> Gergely Katona, PhD
>> > >>> >> Associate Professor
>> > >>> >> Department of Chemistry and Molecular Biology, University of
>> Gothenburg
>> > >>> >> Box 462, 40530 G?teborg, Sweden
>> > >>> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> > >>> >> Web: https://urldefense.proofpoint.com/v2/url?u=http-3A__katonalab.eu&d=DwICAg&c… , Email: gergely.katona(a)gu.se
>> > >>> >>
>> > >>> >> _______________________________________________
>> > >>> >> cctbxbb mailing list
>> > >>> >> cctbxbb(a)phenix-online.org
>> > >>> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >
>> > >>> > _______________________________________________
>> > >>> > cctbxbb mailing list
>> > >>> > cctbxbb(a)phenix-online.org
>> > >>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>>
>> > >>>
>> > >>>
>> > >>> --
>> > >>> Gergely Katona, PhD
>> > >>> Associate Professor
>> > >>> Department of Chemistry and Molecular Biology, University of
>> Gothenburg
>> > >>> Box 462, 40530 G?teborg, Sweden
>> > >>> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> > >>> Web: https://urldefense.proofpoint.com/v2/url?u=http-3A__katonalab.eu&d=DwICAg&c… , Email: gergely.katona(a)gu.se
>> > >>>
>> > >>> _______________________________________________
>> > >>> cctbxbb mailing list
>> > >>> cctbxbb(a)phenix-online.org
>> > >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >
>> > > _______________________________________________
>> > > cctbxbb mailing list
>> > > cctbxbb(a)phenix-online.org
>> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> >
>> >
>> >
>> > --
>> > Gergely Katona, Professor
>> > Department of Chemistry and Molecular Biology, University of Gothenburg
>> > Box 462, 40530 G?teborg, Sweden
>> > Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> > Web: https://urldefense.proofpoint.com/v2/url?u=http-3A__katonalab.eu&d=DwICAg&c… , Email: gergely.katona(a)gu.se
>>
>>
>>
>> --
>> Gergely Katona, Professor
>> Department of Chemistry and Molecular Biology, University of
>> Gothenburg
>> Box 462, 40530 G?teborg, Sweden
>> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> Web:
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__katonalab.eu&d=DwICAg&c…
>> , Email: gergely.katona(a)gu.se
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>>
>
6 years, 1 month
[cctbxbb] conda cctbx on Mac 10.15.2 working with python3.8
by Mooers, Blaine H.M. (HSC)
Hi Tristan,
Thank you very much for the explanation and the suggested solution.
I needed both.
When I tried your list comprehension in a Jupyter notebook with the orginal
problem, it is too fast:
```
IOPub data rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_data_rate_limit`.
Current values:
NotebookApp.iopub_data_rate_limit=1000000.0 (bytes/sec)
NotebookApp.rate_limit_window=3.0 (secs)
```
When I reduced the a and b edges by an order of magnitude (now 12,118 reflections),
your list comprehension is two orders of magnitude faster a for loop (no surprise).
It is also slightly faster than this generator expression coupled to a list
comprehension, so there is no advantage to using the generator here.
hklgen = (hkl for hkl in ms.indices())
print('\n'.join([str(hkl) for hkl in hklgen]))
Best regards,
Blaine
________________________________________
From: Tristan Croll [tic20(a)cam.ac.uk]
Sent: Sunday, January 05, 2020 4:13 AM
To: cctbx mailing list
Cc: Mooers, Blaine H.M. (HSC)
Subject: [EXTERNAL] Re: [cctbxbb] conda cctbx on Mac 10.15.2 working with python3.8
Hi Blaine,
The speed issue is probably because this list comprehension:
[print(hkl) for hkl in ms.indices()]
... is calling print() (and returning None) a million times over. I
think something like the following should be much faster:
print('\n'.join([str(hkl) for hkl in ms.indices()]))
Best regards,
Tristan
On 2020-01-04 22:00, Mooers, Blaine H.M. (HSC) wrote:
> Hi Billy,
>
> Gergely's post yesterday stimulated me to check my conda cctbx setup.
> I had earlier reported success with conda cctbx for Python3.6, 3.7, and
> 3.8.
> This morning I found in my path an earlier python3.6 build of cctbx
> that may
> have given me false positives with my conda installs of cctbx.
> After removing the old build of cctbx from my path, I modified my zshrc
> file
> as Gergely suggested. I can confirm that results that Gergely reported
> yesterday.
>
> I used a Mac OS X 10.15.2 with zsh as my shell and anaconda3 installed
> as /opt/anaconda. I did my testing in a conda env named cctbx38.
> I added 'export LIBTBX_BUILD=$CONDA_PREFIX' in my .zshrc file.
> After sourcing my .zshrc file, 'echo $LIBTBX_BUILD' returns
> '/opt/anaconda/envs/cctbx38' as expected.
>
> The toy script below prints the calculated Miller indices as a tuples
> in a terminal window.
> It was run using 'python https://urldefense.proofpoint.com/v2/url?u=http-3A__millerSet.py&d=DwICAg&c… '.
>
> Note that the printing of 1 million indices to a jupyter notebook cell
> is disappointingly slow!!
> Reduce each unit cell dimension by two orders of magnitude before
> trying this at home!
> The following code also worked with Python3.6 and Python3.7 in their
> corresponding cctbx36 and cctbx37 envs.
>
> from cctbx import crystal
> from cctbx import miller
>
> ms = miller.build_set(
> crystal_symmetry=crystal.symmetry(
> space_group_symbol='P4',
> unit_cell=(150.8,150.8,250.4,90.0,90.0,90.0)),
> anomalous_flag=False,
> d_min=1.4)
>
> [print(hkl) for hkl in ms.indices()]
>
> I used conda python3.8.0 built on November 22. Python3.8.1 does not
> seem to be
> available yet from Anaconda.
>
> Best regards,
>
> Blaine
>
> Blaine Mooers, Ph.D.
> Associate Professor
> Department of Biochemistry and Molecular Biology
> College of Medicine
> University of Oklahoma Health Sciences Center
> S.L. Young Biomedical Research Center (BRC) Rm. 466
> 975 NE 10th Street, BRC 466
> Oklahoma City, OK 73104-5419
>
> ________________________________________
> From: cctbxbb-bounces(a)phenix-online.org
> [cctbxbb-bounces(a)phenix-online.org] on behalf of
> cctbxbb-request(a)phenix-online.org [cctbxbb-request(a)phenix-online.org]
> Sent: Saturday, January 04, 2020 2:00 PM
> To: cctbxbb(a)phenix-online.org
> Subject: [EXTERNAL] cctbxbb Digest, Vol 130, Issue 2
>
> Send cctbxbb mailing list submissions to
> cctbxbb(a)phenix-online.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
> or, via email, send a message with subject or body 'help' to
> cctbxbb-request(a)phenix-online.org
>
> You can reach the person managing the list at
> cctbxbb-owner(a)phenix-online.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of cctbxbb digest..."
>
>
> Today's Topics:
>
> 1. Re: some thoughts on cctbx and pip (Billy Poon)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 3 Jan 2020 21:04:25 +0100
> From: Billy Poon <BKPoon(a)lbl.gov>
> To: cctbx mailing list <cctbxbb(a)phenix-online.org>
> Subject: Re: [cctbxbb] some thoughts on cctbx and pip
> Message-ID:
>
> <CAHmYUcbWS7FXA6EPUx=AC2X6m6gPY=GqH-MwGD+uzpTaOKSyjQ(a)mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Gergely,
>
> Happy New Year!
>
> Can you provide the output to "conda list" for each environment?
>
> I suspect that the old version of cctbx might still be around. Is there
> a
> file named PYTHON_VERSION_MAJOR_MINOR in "/home/gergely/anaconda3/lib"?
> If
> so, delete that file. To make sure you are downloading a new copy of
> the
> package, clear the cache where conda keeps copies of all the packages.
> To
> do that, you can run "conda clean -y --all". You can remove the "-y"
> flag
> if you want to see what will be deleted. The hash for the new linux
> package
> should be h8b68381_0.
>
> The last error looks like an incompatible version of boost or gcc
> library
> was installed. The output from "conda list" should show the boost and
> boost-cpp packages coming from conda-forge. I would avoid installing
> into
> the root or base environment because that usually pulls from the
> default
> channels. All the dependencies should be pulled from the conda-forge
> channel.
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709
> Fax: (510) 486-5909
> Web:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
>
>
> On Fri, Jan 3, 2020 at 2:20 PM Gergely Katona <gkatona(a)gmail.com>
> wrote:
>
>> I found some other errors when using the test environment.
>>
>>
>> ---------------------------------------------------------------------------
>> RuntimeError Traceback (most recent call
>> last)
>> <ipython-input-1-fe308b345b3f> in <module>
>> 10 import scipy as sp
>> 11 from scipy import linalg
>> ---> 12 import iotbx.pdb
>> 13 import cctbx
>> 14 import pandas as pd
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
>> in <module>
>> 8
>> 9 import iotbx.pdb.records
>> ---> 10 import iotbx.pdb.hierarchy
>> 11 from scitbx import matrix
>> 12
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__hierarchy.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=QlonHINbJqyu1vTwlBJWKyChCMCCJxpIWIaT5_rrkLc&e=
>> in <module>
>> 14 from six.moves import cStringIO as StringIO
>> 15 from iotbx.pdb import hy36encode, hy36decode
>> ---> 16 import iotbx.cif.model
>> 17 from cctbx import crystal
>> 18 from libtbx import group_args
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/cif/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
>> in <module>
>> 19
>> 20 from cctbx.array_family import flex
>> ---> 21 from cctbx import miller
>> 22 from iotbx.cif import model, builders, geometry
>> 23 from libtbx.containers import OrderedDict
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/miller/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
>> in <module>
>> 11
>> 12 from cctbx import crystal
>> ---> 13 from cctbx import maptbx
>> 14 from cctbx import sgtbx
>> 15 from cctbx.sgtbx import lattice_symmetry
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/maptbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
>> in <module>
>> 15 from libtbx import adopt_init_args
>> 16 from libtbx.utils import Sorry
>> ---> 17 import libtbx.load_env
>> 18 import math
>> 19 import sys, os
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__load-5Fenv.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=rJbfdPr3ymiz7Ir508_VGIhspAKCOwozKhY5I6OXux0&e=
>> in
>> <module>
>> 3 import libtbx.env_config
>> 4 import os
>> ----> 5 libtbx.env = libtbx.env_config.unpickle()
>> 6 libtbx.env.set_os_environ_all_dist()
>> 7 libtbx.env.dispatcher_name =
>> os.environ.get("LIBTBX_DISPATCHER_NAME")
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__env-5Fconfig.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=UrsFhQ0dxNNuzXmOT3D6pnGJ5lXko4mfrKceNgBaQV4&e=
>> in unpickle()
>> 2736 env = pickle.load(libtbx_env)
>> 2737 if (env.python_version_major_minor != sys.version_info[:2]):
>> -> 2738 env.raise_python_version_incompatible()
>> 2739 if (op.realpath(build_path) !=
>> op.realpath(abs(env.build_path))):
>> 2740 env.build_path.reset(build_path)
>>
>> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__env-5Fconfig.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=UrsFhQ0dxNNuzXmOT3D6pnGJ5lXko4mfrKceNgBaQV4&e=
>> in raise_python_version_incompatible(self, prev_pvmm)
>> 469 if (prev_pvmm is None):
>> 470 prev_pvmm = "%d.%d" % self.python_version_major_minor
>> --> 471 raise RuntimeError("Python version incompatible with this
>> build:\n"
>> 472 + " Build directory: %s\n" %
>> show_string(abs(self.build_path))
>> 473 + " Python version used initially: %s\n" % prev_pvmm
>>
>> RuntimeError: Python version incompatible with this build:
>> Build directory: "/home/gergely/anaconda3"
>> Python version used initially: 3.6
>> Python version in use now: 3.8
>>
>>
>> Interestingly in the base environment all import worked, but I got an
>> error at:
>>
>>
>> ArgumentError Traceback (most recent call
>> last)
>> <ipython-input-4-5bc71c36e121> in <module>
>> 33 atm_odd.set_b(0)
>> 34
>> ---> 35 odd.write_pdb_file("bdiff_in_even_000.pdb") #skriver ny pdb
>> med dessa inkluderade
>> 36 dfWBvals=pd.DataFrame(rows_list)
>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
>> in
>> write_pdb_file(self, file_name, open_append, crystal_symmetry,
>> cryst1_z, write_scale_records, append_end, atom_hetatm, sigatm,
>> anisou, siguij)
>> 930 siguij=True):
>> 931 if (crystal_symmetry is Auto):
>> --> 932 crystal_symmetry = self.crystal_symmetry()
>> 933 if (cryst1_z is Auto):
>> 934 cryst1_z = self.extract_cryst1_z_columns()
>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
>> in
>> crystal_symmetry(self, crystal_symmetry, weak_symmetry)
>> 1218 crystal_symmetry=None,
>> 1219 weak_symmetry=False):
>> -> 1220 self_symmetry = self.crystal_symmetry_from_cryst1()
>> 1221 if (self_symmetry is None):
>> 1222 self_symmetry =
>> self._crystal_symmetry_from_cns_remark_sg()
>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
>> in
>> crystal_symmetry_from_cryst1(self)
>> 1195 for line in self.crystallographic_section():
>> 1196 if (line.startswith("CRYST1")):
>> -> 1197 return
>> cryst1_interpretation.crystal_symmetry(cryst1_record=line)
>> 1198 return None
>> 1199
>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__cryst1-5Finterpretation.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=3OchLb18AMp3YpXOuEb8BDOwhl2XLrV1QAfZByH-l2E&e=
>> in crystal_symmetry(cryst1_record)
>> 136 space_group_info=None)
>> 137 space_group_info =
>> categorize(cryst1_record.sgroup).space_group_info(
>> --> 138 unit_cell=u)
>> 139 return crystal.symmetry(unit_cell=u,
>> space_group_info=space_group_info)
>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/https://urldefense.proofpoint.com/v2/url?u=http-3A__cryst1-5Finterpretation.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=3OchLb18AMp3YpXOuEb8BDOwhl2XLrV1QAfZByH-l2E&e=
>> in space_group_info(self, unit_cell)
>> 85 if (self.symbol is None): return None
>> 86 if (self.category is None):
>> ---> 87 try: return sgtbx.space_group_info(self.symbol)
>> 88 except RuntimeError: return None
>> 89 if (isinstance(unit_cell, uctbx.ext.unit_cell)):
>>
>> ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e=
>> in
>> __init__(self, symbol, table_id, group, number, space_group_t_den)
>> 100 assert group is None
>> 101 if (table_id is None):
>> --> 102 symbols = space_group_symbols(symbol)
>> 103 else:
>> 104 if (isinstance(symbol, int)): symbol = str(symbol)
>>
>> ArgumentError: Python argument types in
>> space_group_symbols.__init__(space_group_symbols, str)
>> did not match C++ signature:
>> __init__(_object*, int space_group_number)
>> __init__(_object*, int space_group_number,
>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> std::allocator<char> > extension='')
>> __init__(_object*, int space_group_number,
>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> std::allocator<char> > extension='', std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> > table_id='')
>> __init__(_object*, std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> > symbol)
>> __init__(_object*, std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> > symbol,
>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> std::allocator<char> > table_id='')
>>
>>
>> Gergely
>>
>>
>>
>> On Fri, Jan 3, 2020 at 1:44 PM Gergely Katona <gkatona(a)gmail.com>
>> wrote:
>> >
>> > Dear Billy,
>> >
>> > I did some tests installing in my base conda was not troublefree in
>> > python 3.8. Running iotbx.fetch_pdb --all --mtz 1kp8 resulted in boost
>> > related error when converting to mtz. Installing into a test
>> > environment went fine with iotbx.fetch_pdb --all --mtz 1kp8 running
>> > successfully. I could add pymc3, jupyter and seaborn and iotbx still
>> > did not break. My LIBTBX_BUILD is not set however in the test
>> > environment, which breaks miller when I was trying to import it.
>> > Manually setting the LIBTBX_BUILD to CONDA_PREFIX (setenv LIBTBX_BUILD
>> > $CONDA_PREFIX) fixes this and one of my script works fine! I will come
>> > back if I find some problems with my other scripts, but so far this
>> > looks promising!
>> >
>> >
>> > Happy new year!
>> >
>> > Gergely
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Dec 20, 2019 at 8:16 PM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >
>> > > Hi everyone,
>> > >
>> > > I updated the conda packages on the cctbx-dev channel and there are
>> also now packages for macOS. The old packages have been removed so you
>> can
>> create a new environment with
>> > >
>> > > conda create -n test -c cctbx-dev cctbx python=2.7
>> > > conda activate test
>> > >
>> > > where "test" is the environment name and 2.7 is the python version.
>> For the python version, you can also pick 3.6, 3.7, or 3.8. I think
>> the
>> latest conda will install 3.8 as the default python if the python
>> argument
>> is not provided. Or install into an existing environment (with python
>> already installed) with the same command as before,
>> > >
>> > > conda install -c cctbx-dev cctbx
>> > >
>> > > Again, your .condarc file should have conda-forge as the first channel
>> so that dependencies can be pulled correctly.
>> > >
>> > > The dispatchers should work now and $LIBTBX_BUILD will be set on
>> activation of an environment (and unset on environment deactivation)
>> or
>> after installing into an existing environment, so you can do more
>> interesting things like
>> > >
>> > > iotbx.fetch_pdb --all --mtz 1kp8
>> > >
>> > > However, this package does not install databases, so
>> > >
>> > > molprobity.molprobity 1kp8.pdb 1kp8.mtz
>> > >
>> > > will fail because the database is not available (also probe and reduce
>> are not built).
>> > >
>> > > Some known issues in no particular order,
>> > >
>> > > 1) You can run "libtbx.run_tests_parallel module=cctbx nproc=Auto" to
>> run tests, but not all the test files were copied, so I got 30
>> failures
>> here. The final package will probably not have the test files.
>> > > 2) The other python dispatchers, like "cctbx.python", were not
>> updated, so those will fail. But you can just run "python" in the
>> active
>> environment.
>> > > 3) Parts of mmtbx still need to be updated for Python 3.
>> > > 4) The LIBTBX_BUILD variable should be set for sh and csh shells. Let
>> me know if it's not working for your shell (i.e. "echo $LIBTBX_BUILD"
>> should show the same thing as $CONDA_PREFIX).
>> > > 5) One future step is to enable building other CCTBX-like modules with
>> this conda package. This will require some bookkeeping updates so that
>> modules in the conda environment and the modules being built are
>> tracked
>> properly. Also, making sure that all the necessary headers exist in
>> $PREFIX/include.
>> > >
>> > > Gergely, let me know if your scripts are working.
>> > >
>> > > Happy holidays!
>> > >
>> > > --
>> > > Billy K. Poon
>> > > Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > > Lawrence Berkeley National Laboratory
>> > > 1 Cyclotron Road, M/S 33R0345
>> > > Berkeley, CA 94720
>> > > Tel: (510) 486-5709
>> > > Fax: (510) 486-5909
>> > > Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
>> > >
>> > >
>> > > On Wed, Dec 18, 2019 at 11:25 AM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >>
>> > >> Hi Gergely,
>> > >>
>> > >> That's the other information that hasn't been updated yet in the
>> conda package. :)
>> > >>
>> > >> For LIBTBX_BUILD, I can set up environment variables sort of how the
>> conda gcc package sets up environment variables. I have to double
>> check to
>> see if they're set after installation of the package and after
>> activation
>> of an environment. The variable will just be a copy of $CONDA_PREFIX.
>> > >>
>> > >> The libtbx_env file is a file that exists in $LIBTBX_BUILD that
>> basically stores configuration information. I'll have to modify the
>> contents to update the locations of the different modules since the
>> conda
>> package will not have a "modules" directory.
>> > >>
>> > >> --
>> > >> Billy K. Poon
>> > >> Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > >> Lawrence Berkeley National Laboratory
>> > >> 1 Cyclotron Road, M/S 33R0345
>> > >> Berkeley, CA 94720
>> > >> Tel: (510) 486-5709
>> > >> Fax: (510) 486-5909
>> > >> Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
>> > >>
>> > >>
>> > >> On Wed, Dec 18, 2019 at 2:08 AM Gergely Katona <gkatona(a)gmail.com>
>> wrote:
>> > >>>
>> > >>> Hi Billy,
>> > >>>
>> > >>> Conda install went fine with your instructions after rearranging the
>> > >>> channels by putting cctbx last. I removed all environmental variables
>> > >>> and previous build of cctbx.
>> > >>>
>> > >>> import sys
>> > >>> print (sys.path, sys.prefix)
>> > >>>
>> > >>> ['/home/gergely/anaconda3/lib/https://urldefense.proofpoint.com/v2/url?u=http-3A__python36.zip&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=cUHL8qgeloeUO3oYTQcr9d56RegJk01wJ2i-c3Zo6oE&e= ',
>> > >>> '/home/gergely/anaconda3/lib/python3.6',
>> > >>> '/home/gergely/anaconda3/lib/python3.6/lib-dynload', '',
>> > >>> '/home/gergely/anaconda3/lib/python3.6/site-packages',
>> > >>>
>> '/home/gergely/anaconda3/lib/python3.6/site-packages/IPython/extensions',
>> > >>> '/home/gergely/.ipython'] /home/gergely/anaconda3
>> > >>>
>> > >>> Many imports went fine including pymc3, but I encountered problems
>> > >>> with these three:
>> > >>>
>> > >>> from cctbx import miller
>> > >>> import iotbx.pdb
>> > >>> from iotbx import reflection_file_reader, mtz
>> > >>>
>> > >>>
>> > >>> I also got type error when handling space groups.
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> ---------------------------------------------------------------------------
>> > >>> KeyError Traceback (most recent
>> call last)
>> > >>> <ipython-input-5-cedbd4f84f3d> in <module>
>> > >>> 25 from cctbx import uctbx
>> > >>> 26 from cctbx import sgtbx
>> > >>> ---> 27 from cctbx import miller
>> > >>> 28 #from iotbx import reflection_file_reader, mtz
>> > >>> 29 sns.set_context("poster")
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/miller/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
>> <module>
>> > >>> 11
>> > >>> 12 from cctbx import crystal
>> > >>> ---> 13 from cctbx import maptbx
>> > >>> 14 from cctbx import sgtbx
>> > >>> 15 from cctbx.sgtbx import lattice_symmetry
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/maptbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
>> <module>
>> > >>> 15 from libtbx import adopt_init_args
>> > >>> 16 from libtbx.utils import Sorry
>> > >>> ---> 17 import libtbx.load_env
>> > >>> 18 import math
>> > >>> 19 import sys, os
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__load-5Fenv.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=rJbfdPr3ymiz7Ir508_VGIhspAKCOwozKhY5I6OXux0&e= in
>> <module>
>> > >>> 3 import libtbx.env_config
>> > >>> 4 import os
>> > >>> ----> 5 libtbx.env = libtbx.env_config.unpickle()
>> > >>> 6 libtbx.env.set_os_environ_all_dist()
>> > >>> 7 libtbx.env.dispatcher_name =
>> os.environ.get("LIBTBX_DISPATCHER_NAME")
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__env-5Fconfig.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=UrsFhQ0dxNNuzXmOT3D6pnGJ5lXko4mfrKceNgBaQV4&e= in
>> unpickle()
>> > >>> 2603
>> > >>> 2604 def unpickle():
>> > >>> -> 2605 build_path = os.environ["LIBTBX_BUILD"]
>> > >>> 2606
>> set_preferred_sys_prefix_and_sys_executable(build_path=build_path)
>> > >>> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/https://urldefense.proofpoint.com/v2/url?u=http-3A__os.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=hA1zo5wap0c9wK3DrlWIKqYBIqqbfKXca5jQ9aX5pYU&e= in __getitem__(self, key)
>> > >>> 667 except KeyError:
>> > >>> 668 # raise KeyError with the original key value
>> > >>> --> 669 raise KeyError(key) from None
>> > >>> 670 return self.decodevalue(value)
>> > >>> 671
>> > >>>
>> > >>> KeyError: 'LIBTBX_BUILD'
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> ---------------------------------------------------------------------------
>> > >>> ArgumentError Traceback (most recent
>> call last)
>> > >>> <ipython-input-7-2acd3a9ce26a> in <module>
>> > >>> 26 return
>> ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar
>> > >>> 27
>> > >>> ---> 28
>> ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar=initializecrystal()
>> > >>>
>> > >>> <ipython-input-7-2acd3a9ce26a> in initializecrystal()
>> > >>> 11 uc = uctbx.unit_cell(unit_cell)
>> > >>> 12 wavelength = 1.54980
>> > >>> ---> 13 xtal_symm = crystal.symmetry(unit_cell=unit_cell,
>> > >>> space_group_symbol="P 43 21 2")
>> > >>> 14
>> > >>> 15 ms =
>> > >>> miller.build_set(crystal_symmetry=xtal_symm,anomalous_flag=True,
>> > >>> d_min=1.61)
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/crystal/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
>> > >>> __init__(self, unit_cell, space_group_symbol, space_group_info,
>> > >>> space_group, correct_rhombohedral_setting_if_necessary,
>> > >>> assert_is_compatible_unit_cell,
>> raise_sorry_if_incompatible_unit_cell,
>> > >>> force_compatible_unit_cell)
>> > >>> 74 if (space_group_symbol is not None):
>> > >>> 75 self._space_group_info = sgtbx.space_group_info(
>> > >>> ---> 76 symbol=space_group_symbol)
>> > >>> 77 elif (space_group is not None):
>> > >>> 78 if (isinstance(space_group, sgtbx.space_group)):
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
>> > >>> __init__(self, symbol, table_id, group, number, space_group_t_den)
>> > >>> 100 assert group is None
>> > >>> 101 if (table_id is None):
>> > >>> --> 102 symbols = space_group_symbols(symbol)
>> > >>> 103 else:
>> > >>> 104 if (isinstance(symbol, int)): symbol = str(symbol)
>> > >>>
>> > >>> ArgumentError: Python argument types in
>> > >>> space_group_symbols.__init__(space_group_symbols, str)
>> > >>> did not match C++ signature:
>> > >>> __init__(_object*, int space_group_number)
>> > >>> __init__(_object*, int space_group_number,
>> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> > >>> std::allocator<char> > extension='')
>> > >>> __init__(_object*, int space_group_number,
>> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> > >>> std::allocator<char> > extension='', std::__cxx11::basic_string<char,
>> > >>> std::char_traits<char>, std::allocator<char> > table_id='')
>> > >>> __init__(_object*, std::__cxx11::basic_string<char,
>> > >>> std::char_traits<char>, std::allocator<char> > symbol)
>> > >>> __init__(_object*, std::__cxx11::basic_string<char,
>> > >>> std::char_traits<char>, std::allocator<char> > symbol,
>> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> > >>> std::allocator<char> > table_id='')
>> > >>>
>> > >>>
>> > >>> When I tried to set LIBTBX_BUILD to /home/gergely/anaconda3 (this is
>> > >>> sys.prefix on my system), I got other problems when importing. Can
>> > >>> LIBTBX_BUILD be set in the conda package?
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> ---------------------------------------------------------------------------
>> > >>> FileNotFoundError Traceback (most recent
>> call last)
>> > >>> <ipython-input-1-7e2b144826cc> in <module>
>> > >>> 26 from cctbx import sgtbx
>> > >>> 27 #from cctbx import miller
>> > >>> ---> 28 from iotbx import reflection_file_reader, mtz
>> > >>> 29 sns.set_context("poster")
>> > >>> 30 plt.rcParams.update({'figure.autolayout': True})
>> > >>>
>> > >>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__reflection-5Ffile-5Freader.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=U5BvLmuCgufvN_LRxQw9n78p33jNaRrXVbH1BRSXgxg&e=
>> > >>> in <module>
>> > >>> 53
>> > >>> 54 from __future__ import absolute_import, division,
>> print_function
>> > >>> ---> 55 from iotbx import mtz
>> > >>> 56 from iotbx.scalepack import merge as scalepack_merge
>> > >>> 57 from iotbx.scalepack import no_merge_original_index as
>> > >>> scalepack_no_merge
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/https://urldefense.proofpoint.com/v2/url?u=http-3A__-5F-5Finit-5F-5F.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=9NiF4LTTyEpDCh-vdb5bXVIJ4i9E6HmD6eWJg_3SjzE&e= in
>> <module>
>> > >>> 9 import iotbx_mtz_ext as ext
>> > >>> 10
>> > >>> ---> 11 from iotbx.mtz import extract_from_symmetry_lib
>> > >>> 12 from cctbx import xray
>> > >>> 13 import cctbx.xray.observation_types
>> > >>>
>> > >>>
>> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/https://urldefense.proofpoint.com/v2/url?u=http-3A__extract-5Ffrom-5Fsymmetry-5Flib.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=k3WR6VZd4-PDULPXl4O0Vwz9q3LCaocUUhK_ehTn9UE&e=
>> > >>> in <module>
>> > >>> 1 from __future__ import absolute_import, division,
>> print_function
>> > >>> 2 from cctbx import sgtbx
>> > >>> ----> 3 import libtbx.load_env
>> > >>> 4 import os.path as op
>> > >>> 5 from six.moves import range
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__load-5Fenv.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=rJbfdPr3ymiz7Ir508_VGIhspAKCOwozKhY5I6OXux0&e= in
>> <module>
>> > >>> 3 import libtbx.env_config
>> > >>> 4 import os
>> > >>> ----> 5 libtbx.env = libtbx.env_config.unpickle()
>> > >>> 6 libtbx.env.set_os_environ_all_dist()
>> > >>> 7 libtbx.env.dispatcher_name =
>> os.environ.get("LIBTBX_DISPATCHER_NAME")
>> > >>>
>> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__env-5Fconfig.py&d=DwICAg&c=VjzId-SM5S6aVB_cCGQ0d3uo9UfKByQ3sI6Audoy6dY&r=k0gMbcsdOcdbPUNV5tW66KQSZfXL0ewVDPVBp7tqbks&m=9IjF95nqsrMq_R_HHQ4nsTpGpSpiU2e4ynF8kN3qTHY&s=UrsFhQ0dxNNuzXmOT3D6pnGJ5lXko4mfrKceNgBaQV4&e= in
>> unpickle()
>> > >>> 2605 build_path = os.environ["LIBTBX_BUILD"]
>> > >>> 2606
>> set_preferred_sys_prefix_and_sys_executable(build_path=build_path)
>> > >>> -> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")
>> > >>> 2608 env = pickle.load(libtbx_env)
>> > >>> 2609 if (env.python_version_major_minor !=
>> sys.version_info[:2]):
>> > >>>
>> > >>> FileNotFoundError: [Errno 2] No such file or directory:
>> > >>> '/home/gergely/anaconda3/libtbx_env'
>> > >>>
>> > >>> On Wed, Dec 18, 2019 at 9:10 AM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >>> >
>> > >>> > Hi Gergely,
>> > >>> >
>> > >>> > I've uploaded linux packages to a new channel, cctbx-dev, and you
>> can install it with
>> > >>> >
>> > >>> > conda install -c cctbx-dev cctbx
>> > >>> >
>> > >>> > in your current environment. You should set your ~/.condarc file
>> to pull the other dependencies from the conda-forge channel first, so
>> put
>> conda-forge above cctbx. Mine looks like
>> > >>> >
>> > >>> > channels:
>> > >>> > - conda-forge
>> > >>> > - defaults
>> > >>> > - cctbx
>> > >>> >
>> > >>> > Do you need dxtbx for your scripts? This package does not build
>> that part. I think the plan is to build a separate conda package for
>> dxtbx
>> so that it can be updated more frequently. I can rebuild the packages
>> to
>> include it for testing, but the one being submitted to conda-forge
>> will not
>> have it.
>> > >>> >
>> > >>> > Also, your error message is probably due to version of HDF5 that
>> the development build installs. The
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__bootstrap.py&d=DwICAg&c…
>> script will install
>> 1.10.4, but your other dependency is looking for 1.10.5. Installing
>> this
>> cctbx conda package should install 1.10.5, which should fix the issue.
>> Also, I'm updating those environments and 1.10.5 will be the new
>> default
>> version.
>> > >>> >
>> > >>> > Lastly, the dispatchers will not work in these packages because
>> the old paths during the build process are still in them and many of
>> them
>> expect some additional information that has not been updated in the
>> packages yet. I'm in the process of doing that and will update the
>> cctbx-dev channel when that's done. However, by starting python, you
>> can
>> import cctbx modules. So you can run commands like
>> > >>> >
>> > >>> > from scitbx.array_family import flex
>> > >>> > a = flex.random_double(1000000)
>> > >>> > b = flex.min_max_mean_double(a)
>> > >>> > b.min
>> > >>> > b.max
>> > >>> > b.mean
>> > >>> >
>> > >>> > Thanks!
>> > >>> >
>> > >>> > --
>> > >>> > Billy K. Poon
>> > >>> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > >>> > Lawrence Berkeley National Laboratory
>> > >>> > 1 Cyclotron Road, M/S 33R0345
>> > >>> > Berkeley, CA 94720
>> > >>> > Tel: (510) 486-5709
>> > >>> > Fax: (510) 486-5909
>> > >>> > Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
>> > >>> >
>> > >>> >
>> > >>> > On Tue, Dec 17, 2019 at 5:39 AM Gergely Katona <gkatona(a)gmail.com>
>> wrote:
>> > >>> >>
>> > >>> >> Dear Billy,
>> > >>> >>
>> > >>> >> Thank you for the detailed explanation, I look forward to do
>> further
>> > >>> >> testing! I aim to pool all modules under the same environment, if
>> > >>> >> there are conflicts then I just try to reshuffle the order of
>> imports.
>> > >>> >> So far this did not cause problems for me even when I was using
>> system
>> > >>> >> python, but of course it is not the most prudent thing to do. With
>> > >>> >> anaconda everything is much more standardized and isolated
>> already and
>> > >>> >> without being superuser I can have my familiar environment at any
>> > >>> >> synchrotron based cluster. It is great that cctbx will be an
>> integral
>> > >>> >> part of this ecosystem and this was also the last thing holding me
>> > >>> >> back from adopting python3. About hdf5, pymc3 requires it and
>> > >>> >> importing causes a kernel restart with the following error
>> messages,
>> > >>> >> curiously if I import h5py first this can be avoided.
>> > >>> >>
>> > >>> >> Best wishes,
>> > >>> >>
>> > >>> >> Gergely
>> > >>> >>
>> > >>> >> import pymc3 as pm
>> > >>> >>
>> > >>> >> Warning! ***HDF5 library version mismatched error***
>> > >>> >> The HDF5 header files used to compile this application do not
>> match
>> > >>> >> the version used by the HDF5 library to which this application is
>> linked.
>> > >>> >> Data corruption or segmentation faults may occur if the
>> application continues.
>> > >>> >> This can happen when an application was compiled by one version
>> of HDF5 but
>> > >>> >> linked with a different version of static or shared HDF5 library.
>> > >>> >> You should recompile the application or check your shared library
>> related
>> > >>> >> settings such as 'LD_LIBRARY_PATH'.
>> > >>> >> You can, at your own risk, disable this warning by setting the
>> environment
>> > >>> >> variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.
>> > >>> >> Setting it to 2 or higher will suppress the warning messages
>> totally.
>> > >>> >> Headers are 1.10.4, library is 1.10.5
>> > >>> >> SUMMARY OF THE HDF5 CONFIGURATION
>> > >>> >> =================================
>> > >>> >>
>> > >>> >> General Information:
>> > >>> >> -------------------
>> > >>> >> HDF5 Version: 1.10.5
>> > >>> >> Configured on: Tue Oct 22 12:02:13 UTC 2019
>> > >>> >> Configured by: conda@16247e67ecd5
>> > >>> >> Host system: x86_64-conda_cos6-linux-gnu
>> > >>> >> Uname information: Linux 16247e67ecd5
>> 4.15.0-1059-azure
>> > >>> >> #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64
>> > >>> >> GNU/Linux
>> > >>> >> Byte sex: little-endian
>> > >>> >> Installation point: /home/gergely/anaconda3
>> > >>> >>
>> > >>> >> Compiling Options:
>> > >>> >> ------------------
>> > >>> >> Build Mode: production
>> > >>> >> Debugging Symbols: no
>> > >>> >> Asserts: no
>> > >>> >> Profiling: no
>> > >>> >> Optimization Level: high
>> > >>> >>
>> > >>> >> Linking Options:
>> > >>> >> ----------------
>> > >>> >> Libraries: static, shared
>> > >>> >> Statically Linked Executables:
>> > >>> >> LDFLAGS: -Wl,-O2 -Wl,--sort-common
>> > >>> >> -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags
>> > >>> >> -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib
>> > >>> >> -Wl,-rpath-link,/home/gergely/anaconda3/lib
>> > >>> >> -L/home/gergely/anaconda3/lib
>> > >>> >> H5_LDFLAGS:
>> > >>> >> AM_LDFLAGS: -L/home/gergely/anaconda3/lib
>> > >>> >> Extra libraries: -lrt -lpthread -lz -ldl -lm
>> > >>> >> Archiver:
>> > >>> >>
>> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar
>> > >>> >> AR_FLAGS: cr
>> > >>> >> Ranlib:
>> > >>> >>
>> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib
>> > >>> >>
>> > >>> >> Languages:
>> > >>> >> ----------
>> > >>> >> C: yes
>> > >>> >> C Compiler:
>> > >>> >>
>> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc
>> > >>> >> CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2
>> > >>> >> -I/home/gergely/anaconda3/include
>> > >>> >> H5_CPPFLAGS: -D_GNU_SOURCE
>> > >>> >> -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API
>> > >>> >> AM_CPPFLAGS:
>> -I/home/gergely/anaconda3/include
>> > >>> >> C Flags: -march=nocona -mtune=haswell
>> > >>> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
>> > >>> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include
>> > >>> >>
>> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5
>> > >>> >>
>> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix
>> > >>> >> H5 C Flags: -std=c99 -pedantic -Wall
>> -Wextra
>> > >>> >> -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual
>> -Wconversion
>> > >>> >> -Wdeclaration-after-statement -Wdisabled-optimization
>> -Wfloat-equal
>> > >>> >> -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations
>> > >>> >> -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs
>> > >>> >> -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls
>> > >>> >> -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum
>> -Wundef
>> > >>> >> -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings
>> > >>> >> -finline-functions -s -Wno-inline -Wno-aggregate-return
>> > >>> >> -Wno-missing-format-attribute -Wno-missing-noreturn -O
>> > >>> >> AM C Flags:
>> > >>> >> Shared C Library: yes
>> > >>> >> Static C Library: yes
>> > >>> >>
>> > >>> >>
>> > >>> >> Fortran: yes
>> > >>> >> Fortran Compiler:
>> > >>> >>
>> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran
>> > >>> >> Fortran Flags:
>> > >>> >> H5 Fortran Flags: -pedantic -Wall -Wextra
>> -Wunderflow
>> > >>> >> -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2
>> > >>> >> AM Fortran Flags:
>> > >>> >> Shared Fortran Library: yes
>> > >>> >> Static Fortran Library: yes
>> > >>> >>
>> > >>> >> C++: yes
>> > >>> >> C++ Compiler:
>> > >>> >>
>> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++
>> > >>> >> C++ Flags: -fvisibility-inlines-hidden
>> > >>> >> -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell
>> > >>> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
>> > >>> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include
>> > >>> >>
>> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5
>> > >>> >>
>> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix
>> > >>> >> H5 C++ Flags: -pedantic -Wall -W -Wundef
>> -Wshadow
>> > >>> >> -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings
>> -Wconversion
>> > >>> >> -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual
>> > >>> >> -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor
>> > >>> >> -Wctor-dtor-privacy -Wabi -finline-functions -s -O
>> > >>> >> AM C++ Flags:
>> > >>> >> Shared C++ Library: yes
>> > >>> >> Static C++ Library: yes
>> > >>> >>
>> > >>> >> Java: no
>> > >>> >>
>> > >>> >>
>> > >>> >> Features:
>> > >>> >> ---------
>> > >>> >> Parallel HDF5: no
>> > >>> >> Parallel Filtered Dataset Writes: no
>> > >>> >> Large Parallel I/O: no
>> > >>> >> High-level library: yes
>> > >>> >> Threadsafety: yes
>> > >>> >> Default API mapping: v110
>> > >>> >> With deprecated public symbols: yes
>> > >>> >> I/O filters (external): deflate(zlib)
>> > >>> >> MPE: no
>> > >>> >> Direct VFD: no
>> > >>> >> dmalloc: no
>> > >>> >> Packages w/ extra debug output: none
>> > >>> >> API tracing: no
>> > >>> >> Using memory checker: yes
>> > >>> >> Memory allocation sanity checks: no
>> > >>> >> Function stack tracing: no
>> > >>> >> Strict file format checks: no
>> > >>> >> Optimization instrumentation: no
>> > >>> >>
>> > >>> >> On Tue, Dec 17, 2019 at 8:35 AM Billy Poon <BKPoon(a)lbl.gov>
>> wrote:
>> > >>> >> >
>> > >>> >> > Hi Gergely,
>> > >>> >> >
>> > >>> >> > Let me build the test package tomorrow. All the gritty details
>> for building with conda is being finalized and the official
>> documentation
>> will be updated to describe the steps. It would be too confusing to
>> keep
>> changing the documentation as the process evolves. You do have the
>> general
>> process, though, which is summarized as follows.
>> > >>> >> >
>> > >>> >> > 1) Installing dependencies. The cctbx_dependencies metapackage
>> was an initial approach for managing the CCTBX dependencies, but after
>> contacting the conda-forge folks, they recommended using the
>> --only-deps
>> flag. So when the CCTBX conda package is available, you'll be able to
>> get a
>> set of dependencies with,
>> > >>> >> >
>> > >>> >> > conda install -c conda-forge --only-deps cctbx
>> > >>> >> >
>> > >>> >> > By default, the https://urldefense.proofpoint.com/v2/url?u=http-3A__bootstrap.py&d=DwICAg&c… file will automatically install a
>> set of dependencies in the "conda_base" directory (and a conda
>> installation
>> if one is not found). It just uses standard conda environment files
>> located
>> in libtbx/auto_build/conda_envs, so you do not need to install
>> cctbx_dependencies as a separate step. The environment files avoid
>> channel
>> issues by explicitly defining the channel to pull the packages from
>> and the
>> cctbx channel just stores copies of packages from conda-forge. There
>> were
>> issues earlier where the conda-forge packages would sometimes be moved
>> to a
>> different label. The --use-conda flag also accepts a path to
>> $CONDA_PREFIX
>> if you want to use a specific environment for testing.
>> > >>> >> >
>> > >>> >> > 2) Building. The https://urldefense.proofpoint.com/v2/url?u=http-3A__bootstrap.py&d=DwICAg&c… file handles that with SCons.
>> > >>> >> >
>> > >>> >> > 3) Running. After building, there should be a https://urldefense.proofpoint.com/v2/url?u=http-3A__setpaths.sh&d=DwICAg&c=… (and
>> .csh) file that adds build/bin to your path. The build/bin directory
>> has
>> our dispatchers, which are just scripts that set up the environment
>> variables for you. This prevents other programs from loading our
>> libraries,
>> whose versions may conflict. You should see that there is a "python"
>> dispatcher, which is a convenience for developers. Otherwise, you can
>> use
>> libtbx.python, which will be able to import CCTBX modules.
>> > >>> >> >
>> > >>> >> > The running part is where the conda package for CCTBX will be
>> different than this build. Since our Python files and extensions
>> modules
>> will be in the "site-packages" directory for Python, the PYTHONPATH
>> variable will not be needed (and conda suggests that that variable not
>> be
>> set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so
>> LD_LIBRARY_PATH is not needed. In an active environment, PATH will
>> already
>> be modified. And then our LIBTBX_BUILD directory can be set to
>> sys.prefix.
>> > >>> >> >
>> > >>> >> > So with the conda package, you would only need to activate your
>> conda environment and CCTBX should integrate with other conda
>> packages.
>> What is the conflict with hdf5? That's something that should be fixed.
>> Thanks!
>> > >>> >> >
>> > >>> >> > --
>> > >>> >> > Billy K. Poon
>> > >>> >> > Research Scientist, Molecular Biophysics and Integrated
>> Bioimaging
>> > >>> >> > Lawrence Berkeley National Laboratory
>> > >>> >> > 1 Cyclotron Road, M/S 33R0345
>> > >>> >> > Berkeley, CA 94720
>> > >>> >> > Tel: (510) 486-5709
>> > >>> >> > Fax: (510) 486-5909
>> > >>> >> > Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
>> > >>> >> >
>> > >>> >> >
>> > >>> >> > On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona <
>> gkatona(a)gmail.com> wrote:
>> > >>> >> >>
>> > >>> >> >> Dear Billy,
>> > >>> >> >>
>> > >>> >> >> Thank you for this update and for your efforts! I found a
>> solution and
>> > >>> >> >> indeed most things already work in anaconda3. The steps I took
>> (even
>> > >>> >> >> if these instructions will have short expiry date):
>> > >>> >> >>
>> > >>> >> >> Modified .condarc with:
>> > >>> >> >> channels:
>> > >>> >> >> - cctbx
>> > >>> >> >> - conda-forge
>> > >>> >> >> - defaults
>> > >>> >> >>
>> > >>> >> >> run
>> > >>> >> >> conda conda install cctbx_dependencies python=3.6
>> > >>> >> >>
>> > >>> >> >> I expect this will work without python=3.6 in the near future.
>> > >>> >> >>
>> > >>> >> >> Then compiling cctbx with anaconda3 python went without any
>> problem
>> > >>> >> >> when using these flags:
>> > >>> >> >> python https://urldefense.proofpoint.com/v2/url?u=http-3A__bootstrap.py&d=DwICAg&c… --use-conda --python3 --nproc=12
>> > >>> >> >>
>> > >>> >> >> Then I probably did the unorthodox thing and sourced these
>> directories:
>> > >>> >> >>
>> > >>> >> >> setenv LIBTBX_BUILD /home/gergely/cctbx/build
>> > >>> >> >> setenv PATH ${PATH}:/home/gergely/cctbx/build/bin
>> > >>> >> >> setenv PYTHONPATH
>> > >>> >> >>
>> /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH
>> > >>> >> >> setenv LD_LIBRARY_PATH
>> > >>> >> >>
>> /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH
>> > >>> >> >>
>> > >>> >> >> There is probably a better way to put this into conda
>> environment.
>> > >>> >> >> With these steps I could run one of my scripts depending on
>> cctbx. The
>> > >>> >> >> only problem I found is that hdf5 library had conflict with
>> another
>> > >>> >> >> package in conda, but I does not find this as a showstopper.
>> > >>> >> >>
>> > >>> >> >> Best wishes,
>> > >>> >> >>
>> > >>> >> >> Gergely
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >> On Fri, Dec 13, 2019 at 7:23 PM Billy Poon <BKPoon(a)lbl.gov>
>> wrote:
>> > >>> >> >> >
>> > >>> >> >> > Hi Gergely,
>> > >>> >> >> >
>> > >>> >> >> > It's still a work in progress. I'm sorting out some Windows
>> issues right now. I can probably build a test package on a separate
>> channel
>> for people that want to test it (let's say next week?). I'll provide
>> instructions, but basically, the test conda package will be in its own
>> separate channel and the dependencies will be pulled from the
>> conda-forge
>> channel. I want most things to be working correctly on Python 2.7,
>> 3.6,
>> 3.7, and 3.8 on all 3 platforms.
>> > >>> >> >> >
>> > >>> >> >> > Thanks!
>> > >>> >> >> >
>> > >>> >> >> > --
>> > >>> >> >> > Billy K. Poon
>> > >>> >> >> > Research Scientist, Molecular Biophysics and Integrated
>> Bioimaging
>> > >>> >> >> > Lawrence Berkeley National Laboratory
>> > >>> >> >> > 1 Cyclotron Road, M/S 33R0345
>> > >>> >> >> > Berkeley, CA 94720
>> > >>> >> >> > Tel: (510) 486-5709
>> > >>> >> >> > Fax: (510) 486-5909
>> > >>> >> >> > Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
>> > >>> >> >> >
>> > >>> >> >> >
>> > >>> >> >> > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona <
>> gkatona(a)gmail.com> wrote:
>> > >>> >> >> >>
>> > >>> >> >> >> Dear Billy,
>> > >>> >> >> >>
>> > >>> >> >> >> This sounds very promising and exciting. I am not sure if
>> cctbx is
>> > >>> >> >> >> already functional as a conda package in anaconda3 (Linux)
>> or this is
>> > >>> >> >> >> still work in progress. My technical expertise does not
>> allow me to
>> > >>> >> >> >> tell the difference. What I tried:
>> > >>> >> >> >>
>> > >>> >> >> >> Fresh install of anaconda3. Adding - cctbx and -
>> conda-forge to
>> > >>> >> >> >> .condarc . Installing pyside2 with conda. Running conda
>> install
>> > >>> >> >> >> conda_dependencies . I get a lot package version conflict,
>> and I
>> > >>> >> >> >> cannot import cctbx or iotbx to anaconda python. Am I
>> following the
>> > >>> >> >> >> right instructions? Or it is too early to expect that cctbx
>> works when
>> > >>> >> >> >> installed through conda?
>> > >>> >> >> >>
>> > >>> >> >> >> Best wishes,
>> > >>> >> >> >>
>> > >>> >> >> >> Gergely
>> > >>> >> >> >>
>> > >>> >> >> >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon <BKPoon(a)lbl.gov>
>> wrote:
>> > >>> >> >> >> >
>> > >>> >> >> >> > Hi all,
>> > >>> >> >> >> >
>> > >>> >> >> >> > For a brief update, I have submitted a recipe for
>> cctbxlite to conda-forge (
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_conda-2Dfor…
>> ) and support for
>> Python 3.7 and 3.8 is being added (
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_cctbx_cctbx…
>> ). With some fixes for
>> Windows
>> (https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_cctbx_cctbx…
>> ), all platforms
>> (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and
>> 3.8.
>> Some additional changes will be needed to get Windows to work with
>> Python 3
>> and for tests to pass with Boost 1.70.0. That will enable the
>> conda-forge
>> recipe to build for all platforms and for all supported versions of
>> Python.
>> > >>> >> >> >> >
>> > >>> >> >> >> > Currently, the conda-forge recipe will install into the
>> conda python and cctbx imports can be done without sourcing the
>> environment
>> scripts. It looks like a lot of the environment variables being set in
>> the
>> dispatchers can be removed since the Python files and C++ extensions
>> are in
>> the right places. I'll update the libtbx_env file so that commands
>> that
>> load the environment can work correctly.
>> > >>> >> >> >> >
>> > >>> >> >> >> > --
>> > >>> >> >> >> > Billy K. Poon
>> > >>> >> >> >> > Research Scientist, Molecular Biophysics and Integrated
>> Bioimaging
>> > >>> >> >> >> > Lawrence Berkeley National Laboratory
>> > >>> >> >> >> > 1 Cyclotron Road, M/S 33R0345
>> > >>> >> >> >> > Berkeley, CA 94720
>> > >>> >> >> >> > Tel: (510) 486-5709
>> > >>> >> >> >> > Fax: (510) 486-5909
>> > >>> >> >> >> > Web: https://urldefense.proofpoint.com/v2/url?u=https-3A__phenix-2Donline.org&d=…
>> > >>> >> >> >> >
>> > >>> >> >> >> >
>> > >>> >> >> >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll <
>> tic20(a)cam.ac.uk> wrote:
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> Hi Luc,
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> That sounds promising. From there, I?d need to work out
>> how to make a fully-packaged installer (basically a modified wheel
>> file)
>> for the ChimeraX ToolShed - the aim is for the end user to not have to
>> worry about any of this. That adds a couple of complications - e.g.
>> $LIBTBX_BUILD would need to be set dynamically before first import -
>> but
>> doesn?t seem insurmountable.
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> Thanks,
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> Tristan
>> > >>> >> >> >> >>
>> > >>> >> >> >> >>
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> Tristan Croll
>> > >>> >> >> >> >> Research Fellow
>> > >>> >> >> >> >> Cambridge Institute for Medical Research
>> > >>> >> >> >> >> University of Cambridge CB2 0XY
>> > >>> >> >> >> >>
>> > >>> >> >> >> >>
>> > >>> >> >> >> >>
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis <
>> luc_j_bourhis(a)mac.com> wrote:
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> > Hi Tristan,
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> > cctbx could be built to use your ChimeraX python, now
>> that cctbx is moving to Python 3. The option ?with-python is there for
>> that
>> with the bootstrap script. The specific environment setup boil down to
>> setting two environment variable LIBTBX_BUILD and either
>> LD_LIBRARY_PATH on
>> Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work
>> within a
>> framework such as ChimeraX, that should not be difficult to ensure
>> those
>> two variables are set.
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> > Best wishes,
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> > Luc
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll <
>> tic20(a)cam.ac.uk> wrote:
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >> To add my two cents on this: probably the second-most
>> common question I've had about ISOLDE's implementation is, "why didn't
>> you
>> use CCTBX?". The honest answer to that is, "I didn't know how."
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >> Still don't, really - although the current
>> developments are rather promising. The problem I've faced is that
>> CCTBX was
>> designed as its own self-contained Python (2.7, until very recently)
>> environment, with its own interpreter and a lot of very specific
>> environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which
>> is
>> *also* its own self-contained Python (3.7) environment. To plug one
>> into
>> the other in that form... well, I don't think I'm a good enough
>> programmer
>> to really know where to start.
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >> The move to Conda and a more modular CCTBX
>> architecture should make a lot more possible in that direction. Pip
>> would
>> be even better for me personally (ChimeraX can install directly from
>> the
>> PyPI, but doesn't interact with Conda) - but I understand pretty well
>> the
>> substantial challenge that would amount to (not least being that the
>> PyPI
>> imposes a limit - around 100MB from memory? - on the size of an
>> individual
>> package).
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >> Best regards,
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >> Tristan
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote:
>> > >>> >> >> >> >> >>> Hi Graeme,
>> > >>> >> >> >> >> >>> Yes, I know. But ?black" is a program doing a very
>> particular task
>> > >>> >> >> >> >> >>> (code formatting from the top of my head). Requiring
>> to use a wrapper
>> > >>> >> >> >> >> >>> for python itself is another level. But ok, I think
>> I am mellowing to
>> > >>> >> >> >> >> >>> the idea after all! Talking with people around me,
>> and extrapolating,
>> > >>> >> >> >> >> >>> I would bet that, right now, a great majority of
>> people interested by
>> > >>> >> >> >> >> >>> cctbx in pip have already used the cctbx, so they
>> know about the
>> > >>> >> >> >> >> >>> Python wrapper, and they would not be too sanguine
>> about that. My
>> > >>> >> >> >> >> >>> concern is for the future, when pip will be the
>> first time some people
>> > >>> >> >> >> >> >>> use cctbx. Big fat warning notices on PyPI page and
>> a better error
>> > >>> >> >> >> >> >>> message when cctbx fails because LIBTBX_BUILD is not
>> set would be
>> > >>> >> >> >> >> >>> needed but that could be all right.
>> > >>> >> >> >> >> >>> If we do a pip installer, we should aim at a minimal
>> install: cctbx,
>> > >>> >> >> >> >> >>> iotbx and their dependencies, and that?s it.
>> > >>> >> >> >> >> >>> Best wishes,
>> > >>> >> >> >> >> >>> Luc
>> > >>> >> >> >> >> >>>> On 23 Aug 2019, at 07:17,
>> Graeme.Winter(a)Diamond.ac.uk <Graeme.Winter(a)diamond.ac.uk> wrote:
>> > >>> >> >> >> >> >>>> Without discussing the merits of this or whether we
>> _choose_ to make the move to supporting PIP, I am certain it would be
>> _possible_ - many other packages make dispatcher scripts when you pip
>> install them e.g.
>> > >>> >> >> >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat
>> $(which black)
>> > >>> >> >> >> >> >>>>
>> /Library/Frameworks/Python.framework/Versions/3.6/bin/black
>> > >>> >> >> >> >> >>>>
>> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
>> > >>> >> >> >> >> >>>> # -*- coding: utf-8 -*-
>> > >>> >> >> >> >> >>>> import re
>> > >>> >> >> >> >> >>>> import sys
>> > >>> >> >> >> >> >>>> from black import main
>> > >>> >> >> >> >> >>>> if __name__ == '__main__':
>> > >>> >> >> >> >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$',
>> '', sys.argv[0])
>> > >>> >> >> >> >> >>>> sys.exit(main())
>> > >>> >> >> >> >> >>>> So we _could_ work around the absence of
>> LIBTBX_BUILD etc. in the system. Whether or not we elect to do the
>> work is
>> a different question, and it seems clear that here are very mixed
>> opinions
>> on this.
>> > >>> >> >> >> >> >>>> Best wishes Graeme
>> > >>> >> >> >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis <
>> luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote:
>> > >>> >> >> >> >> >>>> Hi,
>> > >>> >> >> >> >> >>>> Even if we managed to ship our the boost dynamic
>> libraries with pip, it would still not be pip-like, as we would still
>> need
>> our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal
>> pip
>> packages work with the standard python exe. LD_LIBRARY_PATH, we could
>> get
>> around that by changing the way we compile, using -Wl,-R, which is the
>> runtime equivalent of build time -L. That?s a significant change that
>> would
>> need to be tested. But there is no way around setting LIBTBX_BUILD
>> right
>> now. Leaving that to the user is horrible. Perhaps there is a way to
>> hack
>> libtbx/https://urldefense.proofpoint.com/v2/url?u=http-3A__env-5Fconfig.py&…
>> so that we can hardwire LIBTBX_BUILD in there when pip
>> installs?
>> > >>> >> >> >> >> >>>> Best wishes,
>> > >>> >> >> >> >> >>>> Luc
>> > >>> >> >> >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis <
>> luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote:
>> > >>> >> >> >> >> >>>> Hi,
>> > >>> >> >> >> >> >>>> I did look into that many years ago, and even toyed
>> with building a pip installer. What stopped me is the exact conclusion
>> you
>> reached too: the user would not have the pip experience he expects.
>> You are
>> right that it is a lot of effort but is it worth it? Considering that
>> remark, I don?t think so. Now, Conda was created specifically to go
>> beyond
>> pip pure-python-only support. Since cctbx has garnered support for
>> Conda,
>> the best avenue imho is to go the extra length to have a package on
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__Anaconda.org&d=DwICAg&c…
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__anaconda.org_&d=DwICAg&…
>> >, and then to advertise it hard to
>> every potential user out there.
>> > >>> >> >> >> >> >>>> Best wishes,
>> > >>> >> >> >> >> >>>> Luc
>> > >>> >> >> >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster <
>> asbrewster(a)lbl.gov<mailto:[email protected]>> wrote:
>> > >>> >> >> >> >> >>>> Hi, to avoid clouding Dorothee's documentation
>> email thread, which I think is a highly useful enterprise, here's some
>> thoughts about putting cctbx into pip. Pip doesn't install non-python
>> dependencies well. I don't think boost is available as a package on
>> pip
>> (at least the package version we use). wxPython4 isn't portable
>> through
>> pip (
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.wxpython.org_How-…
>> ).
>> MPI libraries are system dependent. If cctbx were a pure python
>> package,
>> pip would be fine, but cctbx is not.
>> > >>> >> >> >> >> >>>> All that said, we could build a manylinux1 version
>> of cctbx and upload it to PyPi (I'm just learning about this). For a
>> pip
>> package to be portable (which is a requirement for cctbx), it needs to
>> conform to PEP513, the manylinux1 standard (
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.python.org_dev_pep…
>> ). For example, numpy is built
>> according to this standard (see
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__pypi.org_project_numpy…
>> ,
>> where you'll see the manylinux1 wheel). Note, the manylinux1 standard
>> is
>> built with Centos 5.11 which we no longer support.
>> > >>> >> >> >> >> >>>> There is also a manylinux2010 standard, which is
>> based on Centos 6
>> (https://urldefense.proofpoint.com/v2/url?u=https-3A__www.python.org_dev_pep…
>> ). This is
>> likely a more attainable target (note though by default C++11 is not
>> supported on Centos 6).
>> > >>> >> >> >> >> >>>> If we built a manylinuxX version of cctbx and
>> uploaded it to PyPi, the user would need all the non-python
>> dependencies.
>> There's no way to specify these in pip. For example, cctbx requires
>> boost
>> 1.63 or better. The user will need to have it in a place their python
>> can
>> find it, or we could package it ourselves and supply it, similar to
>> how the
>> pip h5py package now comes with an hd5f library, or how the pip numpy
>> package includes an openblas library. We'd have to do the same for
>> any
>> packages we depend on that aren't on pip using the manylinux
>> standards,
>> such as wxPython4.
>> > >>> >> >> >> >> >>>> Further, we need to think about how dials and other
>> cctbx-based packages interact. If pip install cctbx is set up, how
>> does
>> pip install dials work, such that any dials shared libraries can find
>> the
>> cctbx libraries? Can shared libraries from one pip package link
>> against
>> libraries in another pip package? Would each package need to supply
>> its
>> own boost? Possibly this is well understood in the pip field, but not
>> by
>> me :)
>> > >>> >> >> >> >> >>>> Finally, there's the option of providing a source
>> pip package. This would require the full compiler toolchain for any
>> given
>> platform (macOS, linux, windows). These are likely available for
>> developers, but not for general users.
>> > >>> >> >> >> >> >>>> Anyway, these are some of the obstacles. Not
>> saying it isn't possible, it's just a lot of effort.
>> > >>> >> >> >> >> >>>> Thanks,
>> > >>> >> >> >> >> >>>> -Aaron
>> > >>> >> >> >> >> >>>> _______________________________________________
>> > >>> >> >> >> >> >>>> cctbxbb mailing list
>> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:
>> cctbxbb(a)phenix-online.org>
>> > >>> >> >> >> >> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >> >> >>>> _______________________________________________
>> > >>> >> >> >> >> >>>> cctbxbb mailing list
>> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:
>> cctbxbb(a)phenix-online.org>
>> > >>> >> >> >> >> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >> >> >>>> _______________________________________________
>> > >>> >> >> >> >> >>>> cctbxbb mailing list
>> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:
>> cctbxbb(a)phenix-online.org>
>> > >>> >> >> >> >> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >> >> >>>> --
>> > >>> >> >> >> >> >>>> This e-mail and any attachments may contain
>> confidential, copyright and or privileged material, and are for the
>> use of
>> the intended addressee only. If you are not the intended addressee or
>> an
>> authorised recipient of the addressee please notify us of receipt by
>> returning the e-mail and do not use, copy, retain, distribute or
>> disclose
>> the information in or attached to the e-mail.
>> > >>> >> >> >> >> >>>> Any opinions expressed within this e-mail are those
>> of the individual and not necessarily of Diamond Light Source Ltd.
>> > >>> >> >> >> >> >>>> Diamond Light Source Ltd. cannot guarantee that
>> this e-mail or any attachments are free from viruses and we cannot
>> accept
>> liability for any damage which you may sustain as a result of software
>> viruses which may be transmitted in or with the message.
>> > >>> >> >> >> >> >>>> Diamond Light Source Limited (company no. 4375679).
>> Registered in England and Wales with its registered office at Diamond
>> House, Harwell Science and Innovation Campus, Didcot, Oxfordshire,
>> OX11
>> 0DE, United Kingdom
>> > >>> >> >> >> >> >>>> _______________________________________________
>> > >>> >> >> >> >> >>>> cctbxbb mailing list
>> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org
>> > >>> >> >> >> >> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >> >> >>> _______________________________________________
>> > >>> >> >> >> >> >>> cctbxbb mailing list
>> > >>> >> >> >> >> >>> cctbxbb(a)phenix-online.org
>> > >>> >> >> >> >> >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >>
>> > >>> >> >> >> >> >> _______________________________________________
>> > >>> >> >> >> >> >> cctbxbb mailing list
>> > >>> >> >> >> >> >> cctbxbb(a)phenix-online.org
>> > >>> >> >> >> >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> >
>> > >>> >> >> >> >> > _______________________________________________
>> > >>> >> >> >> >> > cctbxbb mailing list
>> > >>> >> >> >> >> > cctbxbb(a)phenix-online.org
>> > >>> >> >> >> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >> >>
>> > >>> >> >> >> >>
>> > >>> >> >> >> >> _______________________________________________
>> > >>> >> >> >> >> cctbxbb mailing list
>> > >>> >> >> >> >> cctbxbb(a)phenix-online.org
>> > >>> >> >> >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >> >
>> > >>> >> >> >> > _______________________________________________
>> > >>> >> >> >> > cctbxbb mailing list
>> > >>> >> >> >> > cctbxbb(a)phenix-online.org
>> > >>> >> >> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >>
>> > >>> >> >> >>
>> > >>> >> >> >>
>> > >>> >> >> >> --
>> > >>> >> >> >> Gergely Katona, PhD
>> > >>> >> >> >> Associate Professor
>> > >>> >> >> >> Department of Chemistry and Molecular Biology, University
>> of Gothenburg
>> > >>> >> >> >> Box 462, 40530 G?teborg, Sweden
>> > >>> >> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax:
>> +46-31-786-3910
>> > >>> >> >> >> Web: https://urldefense.proofpoint.com/v2/url?u=http-3A__katonalab.eu&d=DwICAg&c… , Email: gergely.katona(a)gu.se
>> > >>> >> >> >>
>> > >>> >> >> >> _______________________________________________
>> > >>> >> >> >> cctbxbb mailing list
>> > >>> >> >> >> cctbxbb(a)phenix-online.org
>> > >>> >> >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >> >
>> > >>> >> >> > _______________________________________________
>> > >>> >> >> > cctbxbb mailing list
>> > >>> >> >> > cctbxbb(a)phenix-online.org
>> > >>> >> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >> --
>> > >>> >> >> Gergely Katona, PhD
>> > >>> >> >> Associate Professor
>> > >>> >> >> Department of Chemistry and Molecular Biology, University of
>> Gothenburg
>> > >>> >> >> Box 462, 40530 G?teborg, Sweden
>> > >>> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax:
>> +46-31-786-3910
>> > >>> >> >> Web: https://urldefense.proofpoint.com/v2/url?u=http-3A__katonalab.eu&d=DwICAg&c… , Email: gergely.katona(a)gu.se
>> > >>> >> >>
>> > >>> >> >> _______________________________________________
>> > >>> >> >> cctbxbb mailing list
>> > >>> >> >> cctbxbb(a)phenix-online.org
>> > >>> >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >> >
>> > >>> >> > _______________________________________________
>> > >>> >> > cctbxbb mailing list
>> > >>> >> > cctbxbb(a)phenix-online.org
>> > >>> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >>
>> > >>> >>
>> > >>> >>
>> > >>> >> --
>> > >>> >> Gergely Katona, PhD
>> > >>> >> Associate Professor
>> > >>> >> Department of Chemistry and Molecular Biology, University of
>> Gothenburg
>> > >>> >> Box 462, 40530 G?teborg, Sweden
>> > >>> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> > >>> >> Web: https://urldefense.proofpoint.com/v2/url?u=http-3A__katonalab.eu&d=DwICAg&c… , Email: gergely.katona(a)gu.se
>> > >>> >>
>> > >>> >> _______________________________________________
>> > >>> >> cctbxbb mailing list
>> > >>> >> cctbxbb(a)phenix-online.org
>> > >>> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>> >
>> > >>> > _______________________________________________
>> > >>> > cctbxbb mailing list
>> > >>> > cctbxbb(a)phenix-online.org
>> > >>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >>>
>> > >>>
>> > >>>
>> > >>> --
>> > >>> Gergely Katona, PhD
>> > >>> Associate Professor
>> > >>> Department of Chemistry and Molecular Biology, University of
>> Gothenburg
>> > >>> Box 462, 40530 G?teborg, Sweden
>> > >>> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> > >>> Web: https://urldefense.proofpoint.com/v2/url?u=http-3A__katonalab.eu&d=DwICAg&c… , Email: gergely.katona(a)gu.se
>> > >>>
>> > >>> _______________________________________________
>> > >>> cctbxbb mailing list
>> > >>> cctbxbb(a)phenix-online.org
>> > >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> > >
>> > > _______________________________________________
>> > > cctbxbb mailing list
>> > > cctbxbb(a)phenix-online.org
>> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>> >
>> >
>> >
>> > --
>> > Gergely Katona, Professor
>> > Department of Chemistry and Molecular Biology, University of Gothenburg
>> > Box 462, 40530 G?teborg, Sweden
>> > Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> > Web: https://urldefense.proofpoint.com/v2/url?u=http-3A__katonalab.eu&d=DwICAg&c… , Email: gergely.katona(a)gu.se
>>
>>
>>
>> --
>> Gergely Katona, Professor
>> Department of Chemistry and Molecular Biology, University of
>> Gothenburg
>> Box 462, 40530 G?teborg, Sweden
>> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> Web:
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__katonalab.eu&d=DwICAg&c…
>> , Email: gergely.katona(a)gu.se
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mai…
>>
>
6 years, 1 month
Re: [cctbxbb] some thoughts on cctbx and pip
by Gergely Katona
Ah, an there is no sign of the mentioned file in
PYTHON_VERSION_MAJOR_MINOR in /home/gergely/anaconda3/lib before I
forget.
Gergely
On Tue, Jan 7, 2020 at 9:37 AM Gergely Katona <gkatona(a)gmail.com> wrote:
>
> Hi Billy,
>
> Here come the package lists from conda. I will do more testing with
> the cleaned condas installation. Indeed, the boost packages came from
> the conda-forge channel but for some odd reason in the test
> environment not in the base.
>
> Best wishes,
>
> Gergely
>
> On Fri, Jan 3, 2020 at 9:05 PM Billy Poon <BKPoon(a)lbl.gov> wrote:
> >
> > Hi Gergely,
> >
> > Happy New Year!
> >
> > Can you provide the output to "conda list" for each environment?
> >
> > I suspect that the old version of cctbx might still be around. Is there a file named PYTHON_VERSION_MAJOR_MINOR in "/home/gergely/anaconda3/lib"? If so, delete that file. To make sure you are downloading a new copy of the package, clear the cache where conda keeps copies of all the packages. To do that, you can run "conda clean -y --all". You can remove the "-y" flag if you want to see what will be deleted. The hash for the new linux package should be h8b68381_0.
> >
> > The last error looks like an incompatible version of boost or gcc library was installed. The output from "conda list" should show the boost and boost-cpp packages coming from conda-forge. I would avoid installing into the root or base environment because that usually pulls from the default channels. All the dependencies should be pulled from the conda-forge channel.
> >
> > --
> > Billy K. Poon
> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> > Lawrence Berkeley National Laboratory
> > 1 Cyclotron Road, M/S 33R0345
> > Berkeley, CA 94720
> > Tel: (510) 486-5709
> > Fax: (510) 486-5909
> > Web: https://phenix-online.org
> >
> >
> > On Fri, Jan 3, 2020 at 2:20 PM Gergely Katona <gkatona(a)gmail.com> wrote:
> >>
> >> I found some other errors when using the test environment.
> >>
> >>
> >> ---------------------------------------------------------------------------
> >> RuntimeError Traceback (most recent call last)
> >> <ipython-input-1-fe308b345b3f> in <module>
> >> 10 import scipy as sp
> >> 11 from scipy import linalg
> >> ---> 12 import iotbx.pdb
> >> 13 import cctbx
> >> 14 import pandas as pd
> >>
> >> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/__init__.py
> >> in <module>
> >> 8
> >> 9 import iotbx.pdb.records
> >> ---> 10 import iotbx.pdb.hierarchy
> >> 11 from scitbx import matrix
> >> 12
> >>
> >> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/hierarchy.py
> >> in <module>
> >> 14 from six.moves import cStringIO as StringIO
> >> 15 from iotbx.pdb import hy36encode, hy36decode
> >> ---> 16 import iotbx.cif.model
> >> 17 from cctbx import crystal
> >> 18 from libtbx import group_args
> >>
> >> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/cif/__init__.py
> >> in <module>
> >> 19
> >> 20 from cctbx.array_family import flex
> >> ---> 21 from cctbx import miller
> >> 22 from iotbx.cif import model, builders, geometry
> >> 23 from libtbx.containers import OrderedDict
> >>
> >> ~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/miller/__init__.py
> >> in <module>
> >> 11
> >> 12 from cctbx import crystal
> >> ---> 13 from cctbx import maptbx
> >> 14 from cctbx import sgtbx
> >> 15 from cctbx.sgtbx import lattice_symmetry
> >>
> >> ~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/maptbx/__init__.py
> >> in <module>
> >> 15 from libtbx import adopt_init_args
> >> 16 from libtbx.utils import Sorry
> >> ---> 17 import libtbx.load_env
> >> 18 import math
> >> 19 import sys, os
> >>
> >> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/load_env.py in <module>
> >> 3 import libtbx.env_config
> >> 4 import os
> >> ----> 5 libtbx.env = libtbx.env_config.unpickle()
> >> 6 libtbx.env.set_os_environ_all_dist()
> >> 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME")
> >>
> >> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/env_config.py
> >> in unpickle()
> >> 2736 env = pickle.load(libtbx_env)
> >> 2737 if (env.python_version_major_minor != sys.version_info[:2]):
> >> -> 2738 env.raise_python_version_incompatible()
> >> 2739 if (op.realpath(build_path) != op.realpath(abs(env.build_path))):
> >> 2740 env.build_path.reset(build_path)
> >>
> >> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/env_config.py
> >> in raise_python_version_incompatible(self, prev_pvmm)
> >> 469 if (prev_pvmm is None):
> >> 470 prev_pvmm = "%d.%d" % self.python_version_major_minor
> >> --> 471 raise RuntimeError("Python version incompatible with this build:\n"
> >> 472 + " Build directory: %s\n" % show_string(abs(self.build_path))
> >> 473 + " Python version used initially: %s\n" % prev_pvmm
> >>
> >> RuntimeError: Python version incompatible with this build:
> >> Build directory: "/home/gergely/anaconda3"
> >> Python version used initially: 3.6
> >> Python version in use now: 3.8
> >>
> >>
> >> Interestingly in the base environment all import worked, but I got an error at:
> >>
> >>
> >> ArgumentError Traceback (most recent call last)
> >> <ipython-input-4-5bc71c36e121> in <module>
> >> 33 atm_odd.set_b(0)
> >> 34
> >> ---> 35 odd.write_pdb_file("bdiff_in_even_000.pdb") #skriver ny pdb
> >> med dessa inkluderade
> >> 36 dfWBvals=pd.DataFrame(rows_list)
> >>
> >> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in
> >> write_pdb_file(self, file_name, open_append, crystal_symmetry,
> >> cryst1_z, write_scale_records, append_end, atom_hetatm, sigatm,
> >> anisou, siguij)
> >> 930 siguij=True):
> >> 931 if (crystal_symmetry is Auto):
> >> --> 932 crystal_symmetry = self.crystal_symmetry()
> >> 933 if (cryst1_z is Auto):
> >> 934 cryst1_z = self.extract_cryst1_z_columns()
> >>
> >> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in
> >> crystal_symmetry(self, crystal_symmetry, weak_symmetry)
> >> 1218 crystal_symmetry=None,
> >> 1219 weak_symmetry=False):
> >> -> 1220 self_symmetry = self.crystal_symmetry_from_cryst1()
> >> 1221 if (self_symmetry is None):
> >> 1222 self_symmetry = self._crystal_symmetry_from_cns_remark_sg()
> >>
> >> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in
> >> crystal_symmetry_from_cryst1(self)
> >> 1195 for line in self.crystallographic_section():
> >> 1196 if (line.startswith("CRYST1")):
> >> -> 1197 return
> >> cryst1_interpretation.crystal_symmetry(cryst1_record=line)
> >> 1198 return None
> >> 1199
> >>
> >> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/cryst1_interpretation.py
> >> in crystal_symmetry(cryst1_record)
> >> 136 space_group_info=None)
> >> 137 space_group_info = categorize(cryst1_record.sgroup).space_group_info(
> >> --> 138 unit_cell=u)
> >> 139 return crystal.symmetry(unit_cell=u,
> >> space_group_info=space_group_info)
> >>
> >> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/cryst1_interpretation.py
> >> in space_group_info(self, unit_cell)
> >> 85 if (self.symbol is None): return None
> >> 86 if (self.category is None):
> >> ---> 87 try: return sgtbx.space_group_info(self.symbol)
> >> 88 except RuntimeError: return None
> >> 89 if (isinstance(unit_cell, uctbx.ext.unit_cell)):
> >>
> >> ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in
> >> __init__(self, symbol, table_id, group, number, space_group_t_den)
> >> 100 assert group is None
> >> 101 if (table_id is None):
> >> --> 102 symbols = space_group_symbols(symbol)
> >> 103 else:
> >> 104 if (isinstance(symbol, int)): symbol = str(symbol)
> >>
> >> ArgumentError: Python argument types in
> >> space_group_symbols.__init__(space_group_symbols, str)
> >> did not match C++ signature:
> >> __init__(_object*, int space_group_number)
> >> __init__(_object*, int space_group_number,
> >> std::__cxx11::basic_string<char, std::char_traits<char>,
> >> std::allocator<char> > extension='')
> >> __init__(_object*, int space_group_number,
> >> std::__cxx11::basic_string<char, std::char_traits<char>,
> >> std::allocator<char> > extension='', std::__cxx11::basic_string<char,
> >> std::char_traits<char>, std::allocator<char> > table_id='')
> >> __init__(_object*, std::__cxx11::basic_string<char,
> >> std::char_traits<char>, std::allocator<char> > symbol)
> >> __init__(_object*, std::__cxx11::basic_string<char,
> >> std::char_traits<char>, std::allocator<char> > symbol,
> >> std::__cxx11::basic_string<char, std::char_traits<char>,
> >> std::allocator<char> > table_id='')
> >>
> >>
> >> Gergely
> >>
> >>
> >>
> >> On Fri, Jan 3, 2020 at 1:44 PM Gergely Katona <gkatona(a)gmail.com> wrote:
> >> >
> >> > Dear Billy,
> >> >
> >> > I did some tests installing in my base conda was not troublefree in
> >> > python 3.8. Running iotbx.fetch_pdb --all --mtz 1kp8 resulted in boost
> >> > related error when converting to mtz. Installing into a test
> >> > environment went fine with iotbx.fetch_pdb --all --mtz 1kp8 running
> >> > successfully. I could add pymc3, jupyter and seaborn and iotbx still
> >> > did not break. My LIBTBX_BUILD is not set however in the test
> >> > environment, which breaks miller when I was trying to import it.
> >> > Manually setting the LIBTBX_BUILD to CONDA_PREFIX (setenv LIBTBX_BUILD
> >> > $CONDA_PREFIX) fixes this and one of my script works fine! I will come
> >> > back if I find some problems with my other scripts, but so far this
> >> > looks promising!
> >> >
> >> >
> >> > Happy new year!
> >> >
> >> > Gergely
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Fri, Dec 20, 2019 at 8:16 PM Billy Poon <BKPoon(a)lbl.gov> wrote:
> >> > >
> >> > > Hi everyone,
> >> > >
> >> > > I updated the conda packages on the cctbx-dev channel and there are also now packages for macOS. The old packages have been removed so you can create a new environment with
> >> > >
> >> > > conda create -n test -c cctbx-dev cctbx python=2.7
> >> > > conda activate test
> >> > >
> >> > > where "test" is the environment name and 2.7 is the python version. For the python version, you can also pick 3.6, 3.7, or 3.8. I think the latest conda will install 3.8 as the default python if the python argument is not provided. Or install into an existing environment (with python already installed) with the same command as before,
> >> > >
> >> > > conda install -c cctbx-dev cctbx
> >> > >
> >> > > Again, your .condarc file should have conda-forge as the first channel so that dependencies can be pulled correctly.
> >> > >
> >> > > The dispatchers should work now and $LIBTBX_BUILD will be set on activation of an environment (and unset on environment deactivation) or after installing into an existing environment, so you can do more interesting things like
> >> > >
> >> > > iotbx.fetch_pdb --all --mtz 1kp8
> >> > >
> >> > > However, this package does not install databases, so
> >> > >
> >> > > molprobity.molprobity 1kp8.pdb 1kp8.mtz
> >> > >
> >> > > will fail because the database is not available (also probe and reduce are not built).
> >> > >
> >> > > Some known issues in no particular order,
> >> > >
> >> > > 1) You can run "libtbx.run_tests_parallel module=cctbx nproc=Auto" to run tests, but not all the test files were copied, so I got 30 failures here. The final package will probably not have the test files.
> >> > > 2) The other python dispatchers, like "cctbx.python", were not updated, so those will fail. But you can just run "python" in the active environment.
> >> > > 3) Parts of mmtbx still need to be updated for Python 3.
> >> > > 4) The LIBTBX_BUILD variable should be set for sh and csh shells. Let me know if it's not working for your shell (i.e. "echo $LIBTBX_BUILD" should show the same thing as $CONDA_PREFIX).
> >> > > 5) One future step is to enable building other CCTBX-like modules with this conda package. This will require some bookkeeping updates so that modules in the conda environment and the modules being built are tracked properly. Also, making sure that all the necessary headers exist in $PREFIX/include.
> >> > >
> >> > > Gergely, let me know if your scripts are working.
> >> > >
> >> > > Happy holidays!
> >> > >
> >> > > --
> >> > > Billy K. Poon
> >> > > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> >> > > Lawrence Berkeley National Laboratory
> >> > > 1 Cyclotron Road, M/S 33R0345
> >> > > Berkeley, CA 94720
> >> > > Tel: (510) 486-5709
> >> > > Fax: (510) 486-5909
> >> > > Web: https://phenix-online.org
> >> > >
> >> > >
> >> > > On Wed, Dec 18, 2019 at 11:25 AM Billy Poon <BKPoon(a)lbl.gov> wrote:
> >> > >>
> >> > >> Hi Gergely,
> >> > >>
> >> > >> That's the other information that hasn't been updated yet in the conda package. :)
> >> > >>
> >> > >> For LIBTBX_BUILD, I can set up environment variables sort of how the conda gcc package sets up environment variables. I have to double check to see if they're set after installation of the package and after activation of an environment. The variable will just be a copy of $CONDA_PREFIX.
> >> > >>
> >> > >> The libtbx_env file is a file that exists in $LIBTBX_BUILD that basically stores configuration information. I'll have to modify the contents to update the locations of the different modules since the conda package will not have a "modules" directory.
> >> > >>
> >> > >> --
> >> > >> Billy K. Poon
> >> > >> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> >> > >> Lawrence Berkeley National Laboratory
> >> > >> 1 Cyclotron Road, M/S 33R0345
> >> > >> Berkeley, CA 94720
> >> > >> Tel: (510) 486-5709
> >> > >> Fax: (510) 486-5909
> >> > >> Web: https://phenix-online.org
> >> > >>
> >> > >>
> >> > >> On Wed, Dec 18, 2019 at 2:08 AM Gergely Katona <gkatona(a)gmail.com> wrote:
> >> > >>>
> >> > >>> Hi Billy,
> >> > >>>
> >> > >>> Conda install went fine with your instructions after rearranging the
> >> > >>> channels by putting cctbx last. I removed all environmental variables
> >> > >>> and previous build of cctbx.
> >> > >>>
> >> > >>> import sys
> >> > >>> print (sys.path, sys.prefix)
> >> > >>>
> >> > >>> ['/home/gergely/anaconda3/lib/python36.zip',
> >> > >>> '/home/gergely/anaconda3/lib/python3.6',
> >> > >>> '/home/gergely/anaconda3/lib/python3.6/lib-dynload', '',
> >> > >>> '/home/gergely/anaconda3/lib/python3.6/site-packages',
> >> > >>> '/home/gergely/anaconda3/lib/python3.6/site-packages/IPython/extensions',
> >> > >>> '/home/gergely/.ipython'] /home/gergely/anaconda3
> >> > >>>
> >> > >>> Many imports went fine including pymc3, but I encountered problems
> >> > >>> with these three:
> >> > >>>
> >> > >>> from cctbx import miller
> >> > >>> import iotbx.pdb
> >> > >>> from iotbx import reflection_file_reader, mtz
> >> > >>>
> >> > >>>
> >> > >>> I also got type error when handling space groups.
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> ---------------------------------------------------------------------------
> >> > >>> KeyError Traceback (most recent call last)
> >> > >>> <ipython-input-5-cedbd4f84f3d> in <module>
> >> > >>> 25 from cctbx import uctbx
> >> > >>> 26 from cctbx import sgtbx
> >> > >>> ---> 27 from cctbx import miller
> >> > >>> 28 #from iotbx import reflection_file_reader, mtz
> >> > >>> 29 sns.set_context("poster")
> >> > >>>
> >> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/miller/__init__.py in <module>
> >> > >>> 11
> >> > >>> 12 from cctbx import crystal
> >> > >>> ---> 13 from cctbx import maptbx
> >> > >>> 14 from cctbx import sgtbx
> >> > >>> 15 from cctbx.sgtbx import lattice_symmetry
> >> > >>>
> >> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/maptbx/__init__.py in <module>
> >> > >>> 15 from libtbx import adopt_init_args
> >> > >>> 16 from libtbx.utils import Sorry
> >> > >>> ---> 17 import libtbx.load_env
> >> > >>> 18 import math
> >> > >>> 19 import sys, os
> >> > >>>
> >> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module>
> >> > >>> 3 import libtbx.env_config
> >> > >>> 4 import os
> >> > >>> ----> 5 libtbx.env = libtbx.env_config.unpickle()
> >> > >>> 6 libtbx.env.set_os_environ_all_dist()
> >> > >>> 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME")
> >> > >>>
> >> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle()
> >> > >>> 2603
> >> > >>> 2604 def unpickle():
> >> > >>> -> 2605 build_path = os.environ["LIBTBX_BUILD"]
> >> > >>> 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path)
> >> > >>> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")
> >> > >>>
> >> > >>> ~/anaconda3/lib/python3.6/os.py in __getitem__(self, key)
> >> > >>> 667 except KeyError:
> >> > >>> 668 # raise KeyError with the original key value
> >> > >>> --> 669 raise KeyError(key) from None
> >> > >>> 670 return self.decodevalue(value)
> >> > >>> 671
> >> > >>>
> >> > >>> KeyError: 'LIBTBX_BUILD'
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> ---------------------------------------------------------------------------
> >> > >>> ArgumentError Traceback (most recent call last)
> >> > >>> <ipython-input-7-2acd3a9ce26a> in <module>
> >> > >>> 26 return ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar
> >> > >>> 27
> >> > >>> ---> 28 ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar=initializecrystal()
> >> > >>>
> >> > >>> <ipython-input-7-2acd3a9ce26a> in initializecrystal()
> >> > >>> 11 uc = uctbx.unit_cell(unit_cell)
> >> > >>> 12 wavelength = 1.54980
> >> > >>> ---> 13 xtal_symm = crystal.symmetry(unit_cell=unit_cell,
> >> > >>> space_group_symbol="P 43 21 2")
> >> > >>> 14
> >> > >>> 15 ms =
> >> > >>> miller.build_set(crystal_symmetry=xtal_symm,anomalous_flag=True,
> >> > >>> d_min=1.61)
> >> > >>>
> >> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/crystal/__init__.py in
> >> > >>> __init__(self, unit_cell, space_group_symbol, space_group_info,
> >> > >>> space_group, correct_rhombohedral_setting_if_necessary,
> >> > >>> assert_is_compatible_unit_cell, raise_sorry_if_incompatible_unit_cell,
> >> > >>> force_compatible_unit_cell)
> >> > >>> 74 if (space_group_symbol is not None):
> >> > >>> 75 self._space_group_info = sgtbx.space_group_info(
> >> > >>> ---> 76 symbol=space_group_symbol)
> >> > >>> 77 elif (space_group is not None):
> >> > >>> 78 if (isinstance(space_group, sgtbx.space_group)):
> >> > >>>
> >> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in
> >> > >>> __init__(self, symbol, table_id, group, number, space_group_t_den)
> >> > >>> 100 assert group is None
> >> > >>> 101 if (table_id is None):
> >> > >>> --> 102 symbols = space_group_symbols(symbol)
> >> > >>> 103 else:
> >> > >>> 104 if (isinstance(symbol, int)): symbol = str(symbol)
> >> > >>>
> >> > >>> ArgumentError: Python argument types in
> >> > >>> space_group_symbols.__init__(space_group_symbols, str)
> >> > >>> did not match C++ signature:
> >> > >>> __init__(_object*, int space_group_number)
> >> > >>> __init__(_object*, int space_group_number,
> >> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
> >> > >>> std::allocator<char> > extension='')
> >> > >>> __init__(_object*, int space_group_number,
> >> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
> >> > >>> std::allocator<char> > extension='', std::__cxx11::basic_string<char,
> >> > >>> std::char_traits<char>, std::allocator<char> > table_id='')
> >> > >>> __init__(_object*, std::__cxx11::basic_string<char,
> >> > >>> std::char_traits<char>, std::allocator<char> > symbol)
> >> > >>> __init__(_object*, std::__cxx11::basic_string<char,
> >> > >>> std::char_traits<char>, std::allocator<char> > symbol,
> >> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
> >> > >>> std::allocator<char> > table_id='')
> >> > >>>
> >> > >>>
> >> > >>> When I tried to set LIBTBX_BUILD to /home/gergely/anaconda3 (this is
> >> > >>> sys.prefix on my system), I got other problems when importing. Can
> >> > >>> LIBTBX_BUILD be set in the conda package?
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> ---------------------------------------------------------------------------
> >> > >>> FileNotFoundError Traceback (most recent call last)
> >> > >>> <ipython-input-1-7e2b144826cc> in <module>
> >> > >>> 26 from cctbx import sgtbx
> >> > >>> 27 #from cctbx import miller
> >> > >>> ---> 28 from iotbx import reflection_file_reader, mtz
> >> > >>> 29 sns.set_context("poster")
> >> > >>> 30 plt.rcParams.update({'figure.autolayout': True})
> >> > >>>
> >> > >>> ~/anaconda3/lib/python3.6/site-packages/iotbx/reflection_file_reader.py
> >> > >>> in <module>
> >> > >>> 53
> >> > >>> 54 from __future__ import absolute_import, division, print_function
> >> > >>> ---> 55 from iotbx import mtz
> >> > >>> 56 from iotbx.scalepack import merge as scalepack_merge
> >> > >>> 57 from iotbx.scalepack import no_merge_original_index as
> >> > >>> scalepack_no_merge
> >> > >>>
> >> > >>> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/__init__.py in <module>
> >> > >>> 9 import iotbx_mtz_ext as ext
> >> > >>> 10
> >> > >>> ---> 11 from iotbx.mtz import extract_from_symmetry_lib
> >> > >>> 12 from cctbx import xray
> >> > >>> 13 import cctbx.xray.observation_types
> >> > >>>
> >> > >>> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/extract_from_symmetry_lib.py
> >> > >>> in <module>
> >> > >>> 1 from __future__ import absolute_import, division, print_function
> >> > >>> 2 from cctbx import sgtbx
> >> > >>> ----> 3 import libtbx.load_env
> >> > >>> 4 import os.path as op
> >> > >>> 5 from six.moves import range
> >> > >>>
> >> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module>
> >> > >>> 3 import libtbx.env_config
> >> > >>> 4 import os
> >> > >>> ----> 5 libtbx.env = libtbx.env_config.unpickle()
> >> > >>> 6 libtbx.env.set_os_environ_all_dist()
> >> > >>> 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME")
> >> > >>>
> >> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle()
> >> > >>> 2605 build_path = os.environ["LIBTBX_BUILD"]
> >> > >>> 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path)
> >> > >>> -> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")
> >> > >>> 2608 env = pickle.load(libtbx_env)
> >> > >>> 2609 if (env.python_version_major_minor != sys.version_info[:2]):
> >> > >>>
> >> > >>> FileNotFoundError: [Errno 2] No such file or directory:
> >> > >>> '/home/gergely/anaconda3/libtbx_env'
> >> > >>>
> >> > >>> On Wed, Dec 18, 2019 at 9:10 AM Billy Poon <BKPoon(a)lbl.gov> wrote:
> >> > >>> >
> >> > >>> > Hi Gergely,
> >> > >>> >
> >> > >>> > I've uploaded linux packages to a new channel, cctbx-dev, and you can install it with
> >> > >>> >
> >> > >>> > conda install -c cctbx-dev cctbx
> >> > >>> >
> >> > >>> > in your current environment. You should set your ~/.condarc file to pull the other dependencies from the conda-forge channel first, so put conda-forge above cctbx. Mine looks like
> >> > >>> >
> >> > >>> > channels:
> >> > >>> > - conda-forge
> >> > >>> > - defaults
> >> > >>> > - cctbx
> >> > >>> >
> >> > >>> > Do you need dxtbx for your scripts? This package does not build that part. I think the plan is to build a separate conda package for dxtbx so that it can be updated more frequently. I can rebuild the packages to include it for testing, but the one being submitted to conda-forge will not have it.
> >> > >>> >
> >> > >>> > Also, your error message is probably due to version of HDF5 that the development build installs. The bootstrap.py script will install 1.10.4, but your other dependency is looking for 1.10.5. Installing this cctbx conda package should install 1.10.5, which should fix the issue. Also, I'm updating those environments and 1.10.5 will be the new default version.
> >> > >>> >
> >> > >>> > Lastly, the dispatchers will not work in these packages because the old paths during the build process are still in them and many of them expect some additional information that has not been updated in the packages yet. I'm in the process of doing that and will update the cctbx-dev channel when that's done. However, by starting python, you can import cctbx modules. So you can run commands like
> >> > >>> >
> >> > >>> > from scitbx.array_family import flex
> >> > >>> > a = flex.random_double(1000000)
> >> > >>> > b = flex.min_max_mean_double(a)
> >> > >>> > b.min
> >> > >>> > b.max
> >> > >>> > b.mean
> >> > >>> >
> >> > >>> > Thanks!
> >> > >>> >
> >> > >>> > --
> >> > >>> > Billy K. Poon
> >> > >>> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> >> > >>> > Lawrence Berkeley National Laboratory
> >> > >>> > 1 Cyclotron Road, M/S 33R0345
> >> > >>> > Berkeley, CA 94720
> >> > >>> > Tel: (510) 486-5709
> >> > >>> > Fax: (510) 486-5909
> >> > >>> > Web: https://phenix-online.org
> >> > >>> >
> >> > >>> >
> >> > >>> > On Tue, Dec 17, 2019 at 5:39 AM Gergely Katona <gkatona(a)gmail.com> wrote:
> >> > >>> >>
> >> > >>> >> Dear Billy,
> >> > >>> >>
> >> > >>> >> Thank you for the detailed explanation, I look forward to do further
> >> > >>> >> testing! I aim to pool all modules under the same environment, if
> >> > >>> >> there are conflicts then I just try to reshuffle the order of imports.
> >> > >>> >> So far this did not cause problems for me even when I was using system
> >> > >>> >> python, but of course it is not the most prudent thing to do. With
> >> > >>> >> anaconda everything is much more standardized and isolated already and
> >> > >>> >> without being superuser I can have my familiar environment at any
> >> > >>> >> synchrotron based cluster. It is great that cctbx will be an integral
> >> > >>> >> part of this ecosystem and this was also the last thing holding me
> >> > >>> >> back from adopting python3. About hdf5, pymc3 requires it and
> >> > >>> >> importing causes a kernel restart with the following error messages,
> >> > >>> >> curiously if I import h5py first this can be avoided.
> >> > >>> >>
> >> > >>> >> Best wishes,
> >> > >>> >>
> >> > >>> >> Gergely
> >> > >>> >>
> >> > >>> >> import pymc3 as pm
> >> > >>> >>
> >> > >>> >> Warning! ***HDF5 library version mismatched error***
> >> > >>> >> The HDF5 header files used to compile this application do not match
> >> > >>> >> the version used by the HDF5 library to which this application is linked.
> >> > >>> >> Data corruption or segmentation faults may occur if the application continues.
> >> > >>> >> This can happen when an application was compiled by one version of HDF5 but
> >> > >>> >> linked with a different version of static or shared HDF5 library.
> >> > >>> >> You should recompile the application or check your shared library related
> >> > >>> >> settings such as 'LD_LIBRARY_PATH'.
> >> > >>> >> You can, at your own risk, disable this warning by setting the environment
> >> > >>> >> variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.
> >> > >>> >> Setting it to 2 or higher will suppress the warning messages totally.
> >> > >>> >> Headers are 1.10.4, library is 1.10.5
> >> > >>> >> SUMMARY OF THE HDF5 CONFIGURATION
> >> > >>> >> =================================
> >> > >>> >>
> >> > >>> >> General Information:
> >> > >>> >> -------------------
> >> > >>> >> HDF5 Version: 1.10.5
> >> > >>> >> Configured on: Tue Oct 22 12:02:13 UTC 2019
> >> > >>> >> Configured by: conda@16247e67ecd5
> >> > >>> >> Host system: x86_64-conda_cos6-linux-gnu
> >> > >>> >> Uname information: Linux 16247e67ecd5 4.15.0-1059-azure
> >> > >>> >> #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64
> >> > >>> >> GNU/Linux
> >> > >>> >> Byte sex: little-endian
> >> > >>> >> Installation point: /home/gergely/anaconda3
> >> > >>> >>
> >> > >>> >> Compiling Options:
> >> > >>> >> ------------------
> >> > >>> >> Build Mode: production
> >> > >>> >> Debugging Symbols: no
> >> > >>> >> Asserts: no
> >> > >>> >> Profiling: no
> >> > >>> >> Optimization Level: high
> >> > >>> >>
> >> > >>> >> Linking Options:
> >> > >>> >> ----------------
> >> > >>> >> Libraries: static, shared
> >> > >>> >> Statically Linked Executables:
> >> > >>> >> LDFLAGS: -Wl,-O2 -Wl,--sort-common
> >> > >>> >> -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags
> >> > >>> >> -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib
> >> > >>> >> -Wl,-rpath-link,/home/gergely/anaconda3/lib
> >> > >>> >> -L/home/gergely/anaconda3/lib
> >> > >>> >> H5_LDFLAGS:
> >> > >>> >> AM_LDFLAGS: -L/home/gergely/anaconda3/lib
> >> > >>> >> Extra libraries: -lrt -lpthread -lz -ldl -lm
> >> > >>> >> Archiver:
> >> > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar
> >> > >>> >> AR_FLAGS: cr
> >> > >>> >> Ranlib:
> >> > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib
> >> > >>> >>
> >> > >>> >> Languages:
> >> > >>> >> ----------
> >> > >>> >> C: yes
> >> > >>> >> C Compiler:
> >> > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc
> >> > >>> >> CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2
> >> > >>> >> -I/home/gergely/anaconda3/include
> >> > >>> >> H5_CPPFLAGS: -D_GNU_SOURCE
> >> > >>> >> -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API
> >> > >>> >> AM_CPPFLAGS: -I/home/gergely/anaconda3/include
> >> > >>> >> C Flags: -march=nocona -mtune=haswell
> >> > >>> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
> >> > >>> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include
> >> > >>> >> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5
> >> > >>> >> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix
> >> > >>> >> H5 C Flags: -std=c99 -pedantic -Wall -Wextra
> >> > >>> >> -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion
> >> > >>> >> -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal
> >> > >>> >> -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations
> >> > >>> >> -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs
> >> > >>> >> -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls
> >> > >>> >> -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef
> >> > >>> >> -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings
> >> > >>> >> -finline-functions -s -Wno-inline -Wno-aggregate-return
> >> > >>> >> -Wno-missing-format-attribute -Wno-missing-noreturn -O
> >> > >>> >> AM C Flags:
> >> > >>> >> Shared C Library: yes
> >> > >>> >> Static C Library: yes
> >> > >>> >>
> >> > >>> >>
> >> > >>> >> Fortran: yes
> >> > >>> >> Fortran Compiler:
> >> > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran
> >> > >>> >> Fortran Flags:
> >> > >>> >> H5 Fortran Flags: -pedantic -Wall -Wextra -Wunderflow
> >> > >>> >> -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2
> >> > >>> >> AM Fortran Flags:
> >> > >>> >> Shared Fortran Library: yes
> >> > >>> >> Static Fortran Library: yes
> >> > >>> >>
> >> > >>> >> C++: yes
> >> > >>> >> C++ Compiler:
> >> > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++
> >> > >>> >> C++ Flags: -fvisibility-inlines-hidden
> >> > >>> >> -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell
> >> > >>> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
> >> > >>> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include
> >> > >>> >> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5
> >> > >>> >> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix
> >> > >>> >> H5 C++ Flags: -pedantic -Wall -W -Wundef -Wshadow
> >> > >>> >> -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion
> >> > >>> >> -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual
> >> > >>> >> -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor
> >> > >>> >> -Wctor-dtor-privacy -Wabi -finline-functions -s -O
> >> > >>> >> AM C++ Flags:
> >> > >>> >> Shared C++ Library: yes
> >> > >>> >> Static C++ Library: yes
> >> > >>> >>
> >> > >>> >> Java: no
> >> > >>> >>
> >> > >>> >>
> >> > >>> >> Features:
> >> > >>> >> ---------
> >> > >>> >> Parallel HDF5: no
> >> > >>> >> Parallel Filtered Dataset Writes: no
> >> > >>> >> Large Parallel I/O: no
> >> > >>> >> High-level library: yes
> >> > >>> >> Threadsafety: yes
> >> > >>> >> Default API mapping: v110
> >> > >>> >> With deprecated public symbols: yes
> >> > >>> >> I/O filters (external): deflate(zlib)
> >> > >>> >> MPE: no
> >> > >>> >> Direct VFD: no
> >> > >>> >> dmalloc: no
> >> > >>> >> Packages w/ extra debug output: none
> >> > >>> >> API tracing: no
> >> > >>> >> Using memory checker: yes
> >> > >>> >> Memory allocation sanity checks: no
> >> > >>> >> Function stack tracing: no
> >> > >>> >> Strict file format checks: no
> >> > >>> >> Optimization instrumentation: no
> >> > >>> >>
> >> > >>> >> On Tue, Dec 17, 2019 at 8:35 AM Billy Poon <BKPoon(a)lbl.gov> wrote:
> >> > >>> >> >
> >> > >>> >> > Hi Gergely,
> >> > >>> >> >
> >> > >>> >> > Let me build the test package tomorrow. All the gritty details for building with conda is being finalized and the official documentation will be updated to describe the steps. It would be too confusing to keep changing the documentation as the process evolves. You do have the general process, though, which is summarized as follows.
> >> > >>> >> >
> >> > >>> >> > 1) Installing dependencies. The cctbx_dependencies metapackage was an initial approach for managing the CCTBX dependencies, but after contacting the conda-forge folks, they recommended using the --only-deps flag. So when the CCTBX conda package is available, you'll be able to get a set of dependencies with,
> >> > >>> >> >
> >> > >>> >> > conda install -c conda-forge --only-deps cctbx
> >> > >>> >> >
> >> > >>> >> > By default, the bootstrap.py file will automatically install a set of dependencies in the "conda_base" directory (and a conda installation if one is not found). It just uses standard conda environment files located in libtbx/auto_build/conda_envs, so you do not need to install cctbx_dependencies as a separate step. The environment files avoid channel issues by explicitly defining the channel to pull the packages from and the cctbx channel just stores copies of packages from conda-forge. There were issues earlier where the conda-forge packages would sometimes be moved to a different label. The --use-conda flag also accepts a path to $CONDA_PREFIX if you want to use a specific environment for testing.
> >> > >>> >> >
> >> > >>> >> > 2) Building. The bootstrap.py file handles that with SCons.
> >> > >>> >> >
> >> > >>> >> > 3) Running. After building, there should be a setpaths.sh (and .csh) file that adds build/bin to your path. The build/bin directory has our dispatchers, which are just scripts that set up the environment variables for you. This prevents other programs from loading our libraries, whose versions may conflict. You should see that there is a "python" dispatcher, which is a convenience for developers. Otherwise, you can use libtbx.python, which will be able to import CCTBX modules.
> >> > >>> >> >
> >> > >>> >> > The running part is where the conda package for CCTBX will be different than this build. Since our Python files and extensions modules will be in the "site-packages" directory for Python, the PYTHONPATH variable will not be needed (and conda suggests that that variable not be set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so LD_LIBRARY_PATH is not needed. In an active environment, PATH will already be modified. And then our LIBTBX_BUILD directory can be set to sys.prefix.
> >> > >>> >> >
> >> > >>> >> > So with the conda package, you would only need to activate your conda environment and CCTBX should integrate with other conda packages. What is the conflict with hdf5? That's something that should be fixed. Thanks!
> >> > >>> >> >
> >> > >>> >> > --
> >> > >>> >> > Billy K. Poon
> >> > >>> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> >> > >>> >> > Lawrence Berkeley National Laboratory
> >> > >>> >> > 1 Cyclotron Road, M/S 33R0345
> >> > >>> >> > Berkeley, CA 94720
> >> > >>> >> > Tel: (510) 486-5709
> >> > >>> >> > Fax: (510) 486-5909
> >> > >>> >> > Web: https://phenix-online.org
> >> > >>> >> >
> >> > >>> >> >
> >> > >>> >> > On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona <gkatona(a)gmail.com> wrote:
> >> > >>> >> >>
> >> > >>> >> >> Dear Billy,
> >> > >>> >> >>
> >> > >>> >> >> Thank you for this update and for your efforts! I found a solution and
> >> > >>> >> >> indeed most things already work in anaconda3. The steps I took (even
> >> > >>> >> >> if these instructions will have short expiry date):
> >> > >>> >> >>
> >> > >>> >> >> Modified .condarc with:
> >> > >>> >> >> channels:
> >> > >>> >> >> - cctbx
> >> > >>> >> >> - conda-forge
> >> > >>> >> >> - defaults
> >> > >>> >> >>
> >> > >>> >> >> run
> >> > >>> >> >> conda conda install cctbx_dependencies python=3.6
> >> > >>> >> >>
> >> > >>> >> >> I expect this will work without python=3.6 in the near future.
> >> > >>> >> >>
> >> > >>> >> >> Then compiling cctbx with anaconda3 python went without any problem
> >> > >>> >> >> when using these flags:
> >> > >>> >> >> python bootstrap.py --use-conda --python3 --nproc=12
> >> > >>> >> >>
> >> > >>> >> >> Then I probably did the unorthodox thing and sourced these directories:
> >> > >>> >> >>
> >> > >>> >> >> setenv LIBTBX_BUILD /home/gergely/cctbx/build
> >> > >>> >> >> setenv PATH ${PATH}:/home/gergely/cctbx/build/bin
> >> > >>> >> >> setenv PYTHONPATH
> >> > >>> >> >> /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH
> >> > >>> >> >> setenv LD_LIBRARY_PATH
> >> > >>> >> >> /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH
> >> > >>> >> >>
> >> > >>> >> >> There is probably a better way to put this into conda environment.
> >> > >>> >> >> With these steps I could run one of my scripts depending on cctbx. The
> >> > >>> >> >> only problem I found is that hdf5 library had conflict with another
> >> > >>> >> >> package in conda, but I does not find this as a showstopper.
> >> > >>> >> >>
> >> > >>> >> >> Best wishes,
> >> > >>> >> >>
> >> > >>> >> >> Gergely
> >> > >>> >> >>
> >> > >>> >> >>
> >> > >>> >> >>
> >> > >>> >> >>
> >> > >>> >> >>
> >> > >>> >> >>
> >> > >>> >> >> On Fri, Dec 13, 2019 at 7:23 PM Billy Poon <BKPoon(a)lbl.gov> wrote:
> >> > >>> >> >> >
> >> > >>> >> >> > Hi Gergely,
> >> > >>> >> >> >
> >> > >>> >> >> > It's still a work in progress. I'm sorting out some Windows issues right now. I can probably build a test package on a separate channel for people that want to test it (let's say next week?). I'll provide instructions, but basically, the test conda package will be in its own separate channel and the dependencies will be pulled from the conda-forge channel. I want most things to be working correctly on Python 2.7, 3.6, 3.7, and 3.8 on all 3 platforms.
> >> > >>> >> >> >
> >> > >>> >> >> > Thanks!
> >> > >>> >> >> >
> >> > >>> >> >> > --
> >> > >>> >> >> > Billy K. Poon
> >> > >>> >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> >> > >>> >> >> > Lawrence Berkeley National Laboratory
> >> > >>> >> >> > 1 Cyclotron Road, M/S 33R0345
> >> > >>> >> >> > Berkeley, CA 94720
> >> > >>> >> >> > Tel: (510) 486-5709
> >> > >>> >> >> > Fax: (510) 486-5909
> >> > >>> >> >> > Web: https://phenix-online.org
> >> > >>> >> >> >
> >> > >>> >> >> >
> >> > >>> >> >> > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona <gkatona(a)gmail.com> wrote:
> >> > >>> >> >> >>
> >> > >>> >> >> >> Dear Billy,
> >> > >>> >> >> >>
> >> > >>> >> >> >> This sounds very promising and exciting. I am not sure if cctbx is
> >> > >>> >> >> >> already functional as a conda package in anaconda3 (Linux) or this is
> >> > >>> >> >> >> still work in progress. My technical expertise does not allow me to
> >> > >>> >> >> >> tell the difference. What I tried:
> >> > >>> >> >> >>
> >> > >>> >> >> >> Fresh install of anaconda3. Adding - cctbx and - conda-forge to
> >> > >>> >> >> >> .condarc . Installing pyside2 with conda. Running conda install
> >> > >>> >> >> >> conda_dependencies . I get a lot package version conflict, and I
> >> > >>> >> >> >> cannot import cctbx or iotbx to anaconda python. Am I following the
> >> > >>> >> >> >> right instructions? Or it is too early to expect that cctbx works when
> >> > >>> >> >> >> installed through conda?
> >> > >>> >> >> >>
> >> > >>> >> >> >> Best wishes,
> >> > >>> >> >> >>
> >> > >>> >> >> >> Gergely
> >> > >>> >> >> >>
> >> > >>> >> >> >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon <BKPoon(a)lbl.gov> wrote:
> >> > >>> >> >> >> >
> >> > >>> >> >> >> > Hi all,
> >> > >>> >> >> >> >
> >> > >>> >> >> >> > For a brief update, I have submitted a recipe for cctbxlite to conda-forge (https://github.com/conda-forge/staged-recipes/pull/10021) and support for Python 3.7 and 3.8 is being added (https://github.com/cctbx/cctbx_project/pull/409). With some fixes for Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8. Some additional changes will be needed to get Windows to work with Python 3 and for tests to pass with Boost 1.70.0. That will enable the conda-forge recipe to build for all platforms and for all supported versions of Python.
> >> > >>> >> >> >> >
> >> > >>> >> >> >> > Currently, the conda-forge recipe will install into the conda python and cctbx imports can be done without sourcing the environment scripts. It looks like a lot of the environment variables being set in the dispatchers can be removed since the Python files and C++ extensions are in the right places. I'll update the libtbx_env file so that commands that load the environment can work correctly.
> >> > >>> >> >> >> >
> >> > >>> >> >> >> > --
> >> > >>> >> >> >> > Billy K. Poon
> >> > >>> >> >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> >> > >>> >> >> >> > Lawrence Berkeley National Laboratory
> >> > >>> >> >> >> > 1 Cyclotron Road, M/S 33R0345
> >> > >>> >> >> >> > Berkeley, CA 94720
> >> > >>> >> >> >> > Tel: (510) 486-5709
> >> > >>> >> >> >> > Fax: (510) 486-5909
> >> > >>> >> >> >> > Web: https://phenix-online.org
> >> > >>> >> >> >> >
> >> > >>> >> >> >> >
> >> > >>> >> >> >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll <tic20(a)cam.ac.uk> wrote:
> >> > >>> >> >> >> >>
> >> > >>> >> >> >> >> Hi Luc,
> >> > >>> >> >> >> >>
> >> > >>> >> >> >> >> That sounds promising. From there, I’d need to work out how to make a fully-packaged installer (basically a modified wheel file) for the ChimeraX ToolShed - the aim is for the end user to not have to worry about any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would need to be set dynamically before first import - but doesn’t seem insurmountable.
> >> > >>> >> >> >> >>
> >> > >>> >> >> >> >> Thanks,
> >> > >>> >> >> >> >>
> >> > >>> >> >> >> >> Tristan
> >> > >>> >> >> >> >>
> >> > >>> >> >> >> >>
> >> > >>> >> >> >> >>
> >> > >>> >> >> >> >> Tristan Croll
> >> > >>> >> >> >> >> Research Fellow
> >> > >>> >> >> >> >> Cambridge Institute for Medical Research
> >> > >>> >> >> >> >> University of Cambridge CB2 0XY
> >> > >>> >> >> >> >>
> >> > >>> >> >> >> >>
> >> > >>> >> >> >> >>
> >> > >>> >> >> >> >>
> >> > >>> >> >> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis <luc_j_bourhis(a)mac.com> wrote:
> >> > >>> >> >> >> >> >
> >> > >>> >> >> >> >> > Hi Tristan,
> >> > >>> >> >> >> >> >
> >> > >>> >> >> >> >> > cctbx could be built to use your ChimeraX python, now that cctbx is moving to Python 3. The option —with-python is there for that with the bootstrap script. The specific environment setup boil down to setting two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a framework such as ChimeraX, that should not be difficult to ensure those two variables are set.
> >> > >>> >> >> >> >> >
> >> > >>> >> >> >> >> > Best wishes,
> >> > >>> >> >> >> >> >
> >> > >>> >> >> >> >> > Luc
> >> > >>> >> >> >> >> >
> >> > >>> >> >> >> >> >
> >> > >>> >> >> >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll <tic20(a)cam.ac.uk> wrote:
> >> > >>> >> >> >> >> >>
> >> > >>> >> >> >> >> >> To add my two cents on this: probably the second-most common question I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". The honest answer to that is, "I didn't know how."
> >> > >>> >> >> >> >> >>
> >> > >>> >> >> >> >> >> Still don't, really - although the current developments are rather promising. The problem I've faced is that CCTBX was designed as its own self-contained Python (2.7, until very recently) environment, with its own interpreter and a lot of very specific environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is *also* its own self-contained Python (3.7) environment. To plug one into the other in that form... well, I don't think I'm a good enough programmer to really know where to start.
> >> > >>> >> >> >> >> >>
> >> > >>> >> >> >> >> >> The move to Conda and a more modular CCTBX architecture should make a lot more possible in that direction. Pip would be even better for me personally (ChimeraX can install directly from the PyPI, but doesn't interact with Conda) - but I understand pretty well the substantial challenge that would amount to (not least being that the PyPI imposes a limit - around 100MB from memory? - on the size of an individual package).
> >> > >>> >> >> >> >> >>
> >> > >>> >> >> >> >> >> Best regards,
> >> > >>> >> >> >> >> >>
> >> > >>> >> >> >> >> >> Tristan
> >> > >>> >> >> >> >> >>
> >> > >>> >> >> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote:
> >> > >>> >> >> >> >> >>> Hi Graeme,
> >> > >>> >> >> >> >> >>> Yes, I know. But “black" is a program doing a very particular task
> >> > >>> >> >> >> >> >>> (code formatting from the top of my head). Requiring to use a wrapper
> >> > >>> >> >> >> >> >>> for python itself is another level. But ok, I think I am mellowing to
> >> > >>> >> >> >> >> >>> the idea after all! Talking with people around me, and extrapolating,
> >> > >>> >> >> >> >> >>> I would bet that, right now, a great majority of people interested by
> >> > >>> >> >> >> >> >>> cctbx in pip have already used the cctbx, so they know about the
> >> > >>> >> >> >> >> >>> Python wrapper, and they would not be too sanguine about that. My
> >> > >>> >> >> >> >> >>> concern is for the future, when pip will be the first time some people
> >> > >>> >> >> >> >> >>> use cctbx. Big fat warning notices on PyPI page and a better error
> >> > >>> >> >> >> >> >>> message when cctbx fails because LIBTBX_BUILD is not set would be
> >> > >>> >> >> >> >> >>> needed but that could be all right.
> >> > >>> >> >> >> >> >>> If we do a pip installer, we should aim at a minimal install: cctbx,
> >> > >>> >> >> >> >> >>> iotbx and their dependencies, and that’s it.
> >> > >>> >> >> >> >> >>> Best wishes,
> >> > >>> >> >> >> >> >>> Luc
> >> > >>> >> >> >> >> >>>> On 23 Aug 2019, at 07:17, Graeme.Winter(a)Diamond.ac.uk <Graeme.Winter(a)diamond.ac.uk> wrote:
> >> > >>> >> >> >> >> >>>> Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g.
> >> > >>> >> >> >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat $(which black)
> >> > >>> >> >> >> >> >>>> /Library/Frameworks/Python.framework/Versions/3.6/bin/black
> >> > >>> >> >> >> >> >>>> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
> >> > >>> >> >> >> >> >>>> # -*- coding: utf-8 -*-
> >> > >>> >> >> >> >> >>>> import re
> >> > >>> >> >> >> >> >>>> import sys
> >> > >>> >> >> >> >> >>>> from black import main
> >> > >>> >> >> >> >> >>>> if __name__ == '__main__':
> >> > >>> >> >> >> >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
> >> > >>> >> >> >> >> >>>> sys.exit(main())
> >> > >>> >> >> >> >> >>>> So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this.
> >> > >>> >> >> >> >> >>>> Best wishes Graeme
> >> > >>> >> >> >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote:
> >> > >>> >> >> >> >> >>>> Hi,
> >> > >>> >> >> >> >> >>>> Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs?
> >> > >>> >> >> >> >> >>>> Best wishes,
> >> > >>> >> >> >> >> >>>> Luc
> >> > >>> >> >> >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote:
> >> > >>> >> >> >> >> >>>> Hi,
> >> > >>> >> >> >> >> >>>> I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.org<http://anaconda.org/>, and then to advertise it hard to every potential user out there.
> >> > >>> >> >> >> >> >>>> Best wishes,
> >> > >>> >> >> >> >> >>>> Luc
> >> > >>> >> >> >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster <asbrewster(a)lbl.gov<mailto:[email protected]>> wrote:
> >> > >>> >> >> >> >> >>>> Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython…). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not.
> >> > >>> >> >> >> >> >>>> All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support.
> >> > >>> >> >> >> >> >>>> There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6).
> >> > >>> >> >> >> >> >>>> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4.
> >> > >>> >> >> >> >> >>>> Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :)
> >> > >>> >> >> >> >> >>>> Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users.
> >> > >>> >> >> >> >> >>>> Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort.
> >> > >>> >> >> >> >> >>>> Thanks,
> >> > >>> >> >> >> >> >>>> -Aaron
> >> > >>> >> >> >> >> >>>> _______________________________________________
> >> > >>> >> >> >> >> >>>> cctbxbb mailing list
> >> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
> >> > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> > >>> >> >> >> >> >>>> _______________________________________________
> >> > >>> >> >> >> >> >>>> cctbxbb mailing list
> >> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
> >> > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> > >>> >> >> >> >> >>>> _______________________________________________
> >> > >>> >> >> >> >> >>>> cctbxbb mailing list
> >> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
> >> > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> > >>> >> >> >> >> >>>> --
> >> > >>> >> >> >> >> >>>> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
> >> > >>> >> >> >> >> >>>> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
> >> > >>> >> >> >> >> >>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
> >> > >>> >> >> >> >> >>>> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
> >> > >>> >> >> >> >> >>>> _______________________________________________
> >> > >>> >> >> >> >> >>>> cctbxbb mailing list
> >> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org
> >> > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> > >>> >> >> >> >> >>> _______________________________________________
> >> > >>> >> >> >> >> >>> cctbxbb mailing list
> >> > >>> >> >> >> >> >>> cctbxbb(a)phenix-online.org
> >> > >>> >> >> >> >> >>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> > >>> >> >> >> >> >>
> >> > >>> >> >> >> >> >>
> >> > >>> >> >> >> >> >> _______________________________________________
> >> > >>> >> >> >> >> >> cctbxbb mailing list
> >> > >>> >> >> >> >> >> cctbxbb(a)phenix-online.org
> >> > >>> >> >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> > >>> >> >> >> >> >
> >> > >>> >> >> >> >> >
> >> > >>> >> >> >> >> > _______________________________________________
> >> > >>> >> >> >> >> > cctbxbb mailing list
> >> > >>> >> >> >> >> > cctbxbb(a)phenix-online.org
> >> > >>> >> >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
> >> > >>> >> >> >> >>
> >> > >>> >> >> >> >>
> >> > >>> >> >> >> >> _______________________________________________
> >> > >>> >> >> >> >> cctbxbb mailing list
> >> > >>> >> >> >> >> cctbxbb(a)phenix-online.org
> >> > >>> >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> > >>> >> >> >> >
> >> > >>> >> >> >> > _______________________________________________
> >> > >>> >> >> >> > cctbxbb mailing list
> >> > >>> >> >> >> > cctbxbb(a)phenix-online.org
> >> > >>> >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
> >> > >>> >> >> >>
> >> > >>> >> >> >>
> >> > >>> >> >> >>
> >> > >>> >> >> >> --
> >> > >>> >> >> >> Gergely Katona, PhD
> >> > >>> >> >> >> Associate Professor
> >> > >>> >> >> >> Department of Chemistry and Molecular Biology, University of Gothenburg
> >> > >>> >> >> >> Box 462, 40530 Göteborg, Sweden
> >> > >>> >> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> >> > >>> >> >> >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
> >> > >>> >> >> >>
> >> > >>> >> >> >> _______________________________________________
> >> > >>> >> >> >> cctbxbb mailing list
> >> > >>> >> >> >> cctbxbb(a)phenix-online.org
> >> > >>> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> > >>> >> >> >
> >> > >>> >> >> > _______________________________________________
> >> > >>> >> >> > cctbxbb mailing list
> >> > >>> >> >> > cctbxbb(a)phenix-online.org
> >> > >>> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
> >> > >>> >> >>
> >> > >>> >> >>
> >> > >>> >> >>
> >> > >>> >> >> --
> >> > >>> >> >> Gergely Katona, PhD
> >> > >>> >> >> Associate Professor
> >> > >>> >> >> Department of Chemistry and Molecular Biology, University of Gothenburg
> >> > >>> >> >> Box 462, 40530 Göteborg, Sweden
> >> > >>> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> >> > >>> >> >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
> >> > >>> >> >>
> >> > >>> >> >> _______________________________________________
> >> > >>> >> >> cctbxbb mailing list
> >> > >>> >> >> cctbxbb(a)phenix-online.org
> >> > >>> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> > >>> >> >
> >> > >>> >> > _______________________________________________
> >> > >>> >> > cctbxbb mailing list
> >> > >>> >> > cctbxbb(a)phenix-online.org
> >> > >>> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
> >> > >>> >>
> >> > >>> >>
> >> > >>> >>
> >> > >>> >> --
> >> > >>> >> Gergely Katona, PhD
> >> > >>> >> Associate Professor
> >> > >>> >> Department of Chemistry and Molecular Biology, University of Gothenburg
> >> > >>> >> Box 462, 40530 Göteborg, Sweden
> >> > >>> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> >> > >>> >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
> >> > >>> >>
> >> > >>> >> _______________________________________________
> >> > >>> >> cctbxbb mailing list
> >> > >>> >> cctbxbb(a)phenix-online.org
> >> > >>> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> > >>> >
> >> > >>> > _______________________________________________
> >> > >>> > cctbxbb mailing list
> >> > >>> > cctbxbb(a)phenix-online.org
> >> > >>> > http://phenix-online.org/mailman/listinfo/cctbxbb
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> --
> >> > >>> Gergely Katona, PhD
> >> > >>> Associate Professor
> >> > >>> Department of Chemistry and Molecular Biology, University of Gothenburg
> >> > >>> Box 462, 40530 Göteborg, Sweden
> >> > >>> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> >> > >>> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
> >> > >>>
> >> > >>> _______________________________________________
> >> > >>> cctbxbb mailing list
> >> > >>> cctbxbb(a)phenix-online.org
> >> > >>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> > >
> >> > > _______________________________________________
> >> > > cctbxbb mailing list
> >> > > cctbxbb(a)phenix-online.org
> >> > > http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >
> >> >
> >> >
> >> > --
> >> > Gergely Katona, Professor
> >> > Department of Chemistry and Molecular Biology, University of Gothenburg
> >> > Box 462, 40530 Göteborg, Sweden
> >> > Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> >> > Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
> >>
> >>
> >>
> >> --
> >> Gergely Katona, Professor
> >> Department of Chemistry and Molecular Biology, University of Gothenburg
> >> Box 462, 40530 Göteborg, Sweden
> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
> >>
> >> _______________________________________________
> >> cctbxbb mailing list
> >> cctbxbb(a)phenix-online.org
> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >
> > _______________________________________________
> > cctbxbb mailing list
> > cctbxbb(a)phenix-online.org
> > http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
>
> --
> Gergely Katona, Professor
> Department of Chemistry and Molecular Biology, University of Gothenburg
> Box 462, 40530 Göteborg, Sweden
> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
--
Gergely Katona, Professor
Department of Chemistry and Molecular Biology, University of Gothenburg
Box 462, 40530 Göteborg, Sweden
Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
6 years, 1 month
Re: [cctbxbb] some thoughts on cctbx and pip
by Gergely Katona
Hi Billy,
I am using tcsh, the conda activate.d directory contains a sh and csh
script for cctbx with a single line assigning the CONDA_PREFIX to the
LIBTBX_BUILD environmental variable, it looks proper, I do not know
why these are not executed/working. CONDA_PREFIX only get set when I
use conda python, not when I open a terminal.
Best wishes,
Gergely
On Fri, Jan 10, 2020 at 5:40 PM Billy Poon <BKPoon(a)lbl.gov> wrote:
>
> Hi Gergely,
>
> So for your base environment, the std::__cxx11::basic_string error is caused by the boost package coming from the cctbx channel. I built that copy of boost to support older compilers (developer environments may use GCC versions that are different than the one used for building conda-forge packages), but I'll just move it to another channel so it's less confusing. I'm surprised that the dependency resolution step kept that package instead of replacing it with the version from conda-forge.
>
> What shell are you using? The LIBTBX_BUILD environment variable should be set for sh and csh type shells. What files exist in $CONDA_PREFIX/etc/conda/activate.d ?
>
> Thanks for testing!
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709
> Fax: (510) 486-5909
> Web: https://phenix-online.org
>
>
> On Tue, Jan 7, 2020 at 12:54 AM Gergely Katona <gkatona(a)gmail.com> wrote:
>>
>> Ah, an there is no sign of the mentioned file in
>> PYTHON_VERSION_MAJOR_MINOR in /home/gergely/anaconda3/lib before I
>> forget.
>>
>> Gergely
>>
>> On Tue, Jan 7, 2020 at 9:37 AM Gergely Katona <gkatona(a)gmail.com> wrote:
>> >
>> > Hi Billy,
>> >
>> > Here come the package lists from conda. I will do more testing with
>> > the cleaned condas installation. Indeed, the boost packages came from
>> > the conda-forge channel but for some odd reason in the test
>> > environment not in the base.
>> >
>> > Best wishes,
>> >
>> > Gergely
>> >
>> > On Fri, Jan 3, 2020 at 9:05 PM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >
>> > > Hi Gergely,
>> > >
>> > > Happy New Year!
>> > >
>> > > Can you provide the output to "conda list" for each environment?
>> > >
>> > > I suspect that the old version of cctbx might still be around. Is there a file named PYTHON_VERSION_MAJOR_MINOR in "/home/gergely/anaconda3/lib"? If so, delete that file. To make sure you are downloading a new copy of the package, clear the cache where conda keeps copies of all the packages. To do that, you can run "conda clean -y --all". You can remove the "-y" flag if you want to see what will be deleted. The hash for the new linux package should be h8b68381_0.
>> > >
>> > > The last error looks like an incompatible version of boost or gcc library was installed. The output from "conda list" should show the boost and boost-cpp packages coming from conda-forge. I would avoid installing into the root or base environment because that usually pulls from the default channels. All the dependencies should be pulled from the conda-forge channel.
>> > >
>> > > --
>> > > Billy K. Poon
>> > > Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > > Lawrence Berkeley National Laboratory
>> > > 1 Cyclotron Road, M/S 33R0345
>> > > Berkeley, CA 94720
>> > > Tel: (510) 486-5709
>> > > Fax: (510) 486-5909
>> > > Web: https://phenix-online.org
>> > >
>> > >
>> > > On Fri, Jan 3, 2020 at 2:20 PM Gergely Katona <gkatona(a)gmail.com> wrote:
>> > >>
>> > >> I found some other errors when using the test environment.
>> > >>
>> > >>
>> > >> ---------------------------------------------------------------------------
>> > >> RuntimeError Traceback (most recent call last)
>> > >> <ipython-input-1-fe308b345b3f> in <module>
>> > >> 10 import scipy as sp
>> > >> 11 from scipy import linalg
>> > >> ---> 12 import iotbx.pdb
>> > >> 13 import cctbx
>> > >> 14 import pandas as pd
>> > >>
>> > >> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/__init__.py
>> > >> in <module>
>> > >> 8
>> > >> 9 import iotbx.pdb.records
>> > >> ---> 10 import iotbx.pdb.hierarchy
>> > >> 11 from scitbx import matrix
>> > >> 12
>> > >>
>> > >> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/hierarchy.py
>> > >> in <module>
>> > >> 14 from six.moves import cStringIO as StringIO
>> > >> 15 from iotbx.pdb import hy36encode, hy36decode
>> > >> ---> 16 import iotbx.cif.model
>> > >> 17 from cctbx import crystal
>> > >> 18 from libtbx import group_args
>> > >>
>> > >> ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/cif/__init__.py
>> > >> in <module>
>> > >> 19
>> > >> 20 from cctbx.array_family import flex
>> > >> ---> 21 from cctbx import miller
>> > >> 22 from iotbx.cif import model, builders, geometry
>> > >> 23 from libtbx.containers import OrderedDict
>> > >>
>> > >> ~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/miller/__init__.py
>> > >> in <module>
>> > >> 11
>> > >> 12 from cctbx import crystal
>> > >> ---> 13 from cctbx import maptbx
>> > >> 14 from cctbx import sgtbx
>> > >> 15 from cctbx.sgtbx import lattice_symmetry
>> > >>
>> > >> ~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/maptbx/__init__.py
>> > >> in <module>
>> > >> 15 from libtbx import adopt_init_args
>> > >> 16 from libtbx.utils import Sorry
>> > >> ---> 17 import libtbx.load_env
>> > >> 18 import math
>> > >> 19 import sys, os
>> > >>
>> > >> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/load_env.py in <module>
>> > >> 3 import libtbx.env_config
>> > >> 4 import os
>> > >> ----> 5 libtbx.env = libtbx.env_config.unpickle()
>> > >> 6 libtbx.env.set_os_environ_all_dist()
>> > >> 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME")
>> > >>
>> > >> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/env_config.py
>> > >> in unpickle()
>> > >> 2736 env = pickle.load(libtbx_env)
>> > >> 2737 if (env.python_version_major_minor != sys.version_info[:2]):
>> > >> -> 2738 env.raise_python_version_incompatible()
>> > >> 2739 if (op.realpath(build_path) != op.realpath(abs(env.build_path))):
>> > >> 2740 env.build_path.reset(build_path)
>> > >>
>> > >> ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/env_config.py
>> > >> in raise_python_version_incompatible(self, prev_pvmm)
>> > >> 469 if (prev_pvmm is None):
>> > >> 470 prev_pvmm = "%d.%d" % self.python_version_major_minor
>> > >> --> 471 raise RuntimeError("Python version incompatible with this build:\n"
>> > >> 472 + " Build directory: %s\n" % show_string(abs(self.build_path))
>> > >> 473 + " Python version used initially: %s\n" % prev_pvmm
>> > >>
>> > >> RuntimeError: Python version incompatible with this build:
>> > >> Build directory: "/home/gergely/anaconda3"
>> > >> Python version used initially: 3.6
>> > >> Python version in use now: 3.8
>> > >>
>> > >>
>> > >> Interestingly in the base environment all import worked, but I got an error at:
>> > >>
>> > >>
>> > >> ArgumentError Traceback (most recent call last)
>> > >> <ipython-input-4-5bc71c36e121> in <module>
>> > >> 33 atm_odd.set_b(0)
>> > >> 34
>> > >> ---> 35 odd.write_pdb_file("bdiff_in_even_000.pdb") #skriver ny pdb
>> > >> med dessa inkluderade
>> > >> 36 dfWBvals=pd.DataFrame(rows_list)
>> > >>
>> > >> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in
>> > >> write_pdb_file(self, file_name, open_append, crystal_symmetry,
>> > >> cryst1_z, write_scale_records, append_end, atom_hetatm, sigatm,
>> > >> anisou, siguij)
>> > >> 930 siguij=True):
>> > >> 931 if (crystal_symmetry is Auto):
>> > >> --> 932 crystal_symmetry = self.crystal_symmetry()
>> > >> 933 if (cryst1_z is Auto):
>> > >> 934 cryst1_z = self.extract_cryst1_z_columns()
>> > >>
>> > >> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in
>> > >> crystal_symmetry(self, crystal_symmetry, weak_symmetry)
>> > >> 1218 crystal_symmetry=None,
>> > >> 1219 weak_symmetry=False):
>> > >> -> 1220 self_symmetry = self.crystal_symmetry_from_cryst1()
>> > >> 1221 if (self_symmetry is None):
>> > >> 1222 self_symmetry = self._crystal_symmetry_from_cns_remark_sg()
>> > >>
>> > >> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in
>> > >> crystal_symmetry_from_cryst1(self)
>> > >> 1195 for line in self.crystallographic_section():
>> > >> 1196 if (line.startswith("CRYST1")):
>> > >> -> 1197 return
>> > >> cryst1_interpretation.crystal_symmetry(cryst1_record=line)
>> > >> 1198 return None
>> > >> 1199
>> > >>
>> > >> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/cryst1_interpretation.py
>> > >> in crystal_symmetry(cryst1_record)
>> > >> 136 space_group_info=None)
>> > >> 137 space_group_info = categorize(cryst1_record.sgroup).space_group_info(
>> > >> --> 138 unit_cell=u)
>> > >> 139 return crystal.symmetry(unit_cell=u,
>> > >> space_group_info=space_group_info)
>> > >>
>> > >> ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/cryst1_interpretation.py
>> > >> in space_group_info(self, unit_cell)
>> > >> 85 if (self.symbol is None): return None
>> > >> 86 if (self.category is None):
>> > >> ---> 87 try: return sgtbx.space_group_info(self.symbol)
>> > >> 88 except RuntimeError: return None
>> > >> 89 if (isinstance(unit_cell, uctbx.ext.unit_cell)):
>> > >>
>> > >> ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in
>> > >> __init__(self, symbol, table_id, group, number, space_group_t_den)
>> > >> 100 assert group is None
>> > >> 101 if (table_id is None):
>> > >> --> 102 symbols = space_group_symbols(symbol)
>> > >> 103 else:
>> > >> 104 if (isinstance(symbol, int)): symbol = str(symbol)
>> > >>
>> > >> ArgumentError: Python argument types in
>> > >> space_group_symbols.__init__(space_group_symbols, str)
>> > >> did not match C++ signature:
>> > >> __init__(_object*, int space_group_number)
>> > >> __init__(_object*, int space_group_number,
>> > >> std::__cxx11::basic_string<char, std::char_traits<char>,
>> > >> std::allocator<char> > extension='')
>> > >> __init__(_object*, int space_group_number,
>> > >> std::__cxx11::basic_string<char, std::char_traits<char>,
>> > >> std::allocator<char> > extension='', std::__cxx11::basic_string<char,
>> > >> std::char_traits<char>, std::allocator<char> > table_id='')
>> > >> __init__(_object*, std::__cxx11::basic_string<char,
>> > >> std::char_traits<char>, std::allocator<char> > symbol)
>> > >> __init__(_object*, std::__cxx11::basic_string<char,
>> > >> std::char_traits<char>, std::allocator<char> > symbol,
>> > >> std::__cxx11::basic_string<char, std::char_traits<char>,
>> > >> std::allocator<char> > table_id='')
>> > >>
>> > >>
>> > >> Gergely
>> > >>
>> > >>
>> > >>
>> > >> On Fri, Jan 3, 2020 at 1:44 PM Gergely Katona <gkatona(a)gmail.com> wrote:
>> > >> >
>> > >> > Dear Billy,
>> > >> >
>> > >> > I did some tests installing in my base conda was not troublefree in
>> > >> > python 3.8. Running iotbx.fetch_pdb --all --mtz 1kp8 resulted in boost
>> > >> > related error when converting to mtz. Installing into a test
>> > >> > environment went fine with iotbx.fetch_pdb --all --mtz 1kp8 running
>> > >> > successfully. I could add pymc3, jupyter and seaborn and iotbx still
>> > >> > did not break. My LIBTBX_BUILD is not set however in the test
>> > >> > environment, which breaks miller when I was trying to import it.
>> > >> > Manually setting the LIBTBX_BUILD to CONDA_PREFIX (setenv LIBTBX_BUILD
>> > >> > $CONDA_PREFIX) fixes this and one of my script works fine! I will come
>> > >> > back if I find some problems with my other scripts, but so far this
>> > >> > looks promising!
>> > >> >
>> > >> >
>> > >> > Happy new year!
>> > >> >
>> > >> > Gergely
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Fri, Dec 20, 2019 at 8:16 PM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >> > >
>> > >> > > Hi everyone,
>> > >> > >
>> > >> > > I updated the conda packages on the cctbx-dev channel and there are also now packages for macOS. The old packages have been removed so you can create a new environment with
>> > >> > >
>> > >> > > conda create -n test -c cctbx-dev cctbx python=2.7
>> > >> > > conda activate test
>> > >> > >
>> > >> > > where "test" is the environment name and 2.7 is the python version. For the python version, you can also pick 3.6, 3.7, or 3.8. I think the latest conda will install 3.8 as the default python if the python argument is not provided. Or install into an existing environment (with python already installed) with the same command as before,
>> > >> > >
>> > >> > > conda install -c cctbx-dev cctbx
>> > >> > >
>> > >> > > Again, your .condarc file should have conda-forge as the first channel so that dependencies can be pulled correctly.
>> > >> > >
>> > >> > > The dispatchers should work now and $LIBTBX_BUILD will be set on activation of an environment (and unset on environment deactivation) or after installing into an existing environment, so you can do more interesting things like
>> > >> > >
>> > >> > > iotbx.fetch_pdb --all --mtz 1kp8
>> > >> > >
>> > >> > > However, this package does not install databases, so
>> > >> > >
>> > >> > > molprobity.molprobity 1kp8.pdb 1kp8.mtz
>> > >> > >
>> > >> > > will fail because the database is not available (also probe and reduce are not built).
>> > >> > >
>> > >> > > Some known issues in no particular order,
>> > >> > >
>> > >> > > 1) You can run "libtbx.run_tests_parallel module=cctbx nproc=Auto" to run tests, but not all the test files were copied, so I got 30 failures here. The final package will probably not have the test files.
>> > >> > > 2) The other python dispatchers, like "cctbx.python", were not updated, so those will fail. But you can just run "python" in the active environment.
>> > >> > > 3) Parts of mmtbx still need to be updated for Python 3.
>> > >> > > 4) The LIBTBX_BUILD variable should be set for sh and csh shells. Let me know if it's not working for your shell (i.e. "echo $LIBTBX_BUILD" should show the same thing as $CONDA_PREFIX).
>> > >> > > 5) One future step is to enable building other CCTBX-like modules with this conda package. This will require some bookkeeping updates so that modules in the conda environment and the modules being built are tracked properly. Also, making sure that all the necessary headers exist in $PREFIX/include.
>> > >> > >
>> > >> > > Gergely, let me know if your scripts are working.
>> > >> > >
>> > >> > > Happy holidays!
>> > >> > >
>> > >> > > --
>> > >> > > Billy K. Poon
>> > >> > > Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > >> > > Lawrence Berkeley National Laboratory
>> > >> > > 1 Cyclotron Road, M/S 33R0345
>> > >> > > Berkeley, CA 94720
>> > >> > > Tel: (510) 486-5709
>> > >> > > Fax: (510) 486-5909
>> > >> > > Web: https://phenix-online.org
>> > >> > >
>> > >> > >
>> > >> > > On Wed, Dec 18, 2019 at 11:25 AM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >> > >>
>> > >> > >> Hi Gergely,
>> > >> > >>
>> > >> > >> That's the other information that hasn't been updated yet in the conda package. :)
>> > >> > >>
>> > >> > >> For LIBTBX_BUILD, I can set up environment variables sort of how the conda gcc package sets up environment variables. I have to double check to see if they're set after installation of the package and after activation of an environment. The variable will just be a copy of $CONDA_PREFIX.
>> > >> > >>
>> > >> > >> The libtbx_env file is a file that exists in $LIBTBX_BUILD that basically stores configuration information. I'll have to modify the contents to update the locations of the different modules since the conda package will not have a "modules" directory.
>> > >> > >>
>> > >> > >> --
>> > >> > >> Billy K. Poon
>> > >> > >> Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > >> > >> Lawrence Berkeley National Laboratory
>> > >> > >> 1 Cyclotron Road, M/S 33R0345
>> > >> > >> Berkeley, CA 94720
>> > >> > >> Tel: (510) 486-5709
>> > >> > >> Fax: (510) 486-5909
>> > >> > >> Web: https://phenix-online.org
>> > >> > >>
>> > >> > >>
>> > >> > >> On Wed, Dec 18, 2019 at 2:08 AM Gergely Katona <gkatona(a)gmail.com> wrote:
>> > >> > >>>
>> > >> > >>> Hi Billy,
>> > >> > >>>
>> > >> > >>> Conda install went fine with your instructions after rearranging the
>> > >> > >>> channels by putting cctbx last. I removed all environmental variables
>> > >> > >>> and previous build of cctbx.
>> > >> > >>>
>> > >> > >>> import sys
>> > >> > >>> print (sys.path, sys.prefix)
>> > >> > >>>
>> > >> > >>> ['/home/gergely/anaconda3/lib/python36.zip',
>> > >> > >>> '/home/gergely/anaconda3/lib/python3.6',
>> > >> > >>> '/home/gergely/anaconda3/lib/python3.6/lib-dynload', '',
>> > >> > >>> '/home/gergely/anaconda3/lib/python3.6/site-packages',
>> > >> > >>> '/home/gergely/anaconda3/lib/python3.6/site-packages/IPython/extensions',
>> > >> > >>> '/home/gergely/.ipython'] /home/gergely/anaconda3
>> > >> > >>>
>> > >> > >>> Many imports went fine including pymc3, but I encountered problems
>> > >> > >>> with these three:
>> > >> > >>>
>> > >> > >>> from cctbx import miller
>> > >> > >>> import iotbx.pdb
>> > >> > >>> from iotbx import reflection_file_reader, mtz
>> > >> > >>>
>> > >> > >>>
>> > >> > >>> I also got type error when handling space groups.
>> > >> > >>>
>> > >> > >>>
>> > >> > >>>
>> > >> > >>> ---------------------------------------------------------------------------
>> > >> > >>> KeyError Traceback (most recent call last)
>> > >> > >>> <ipython-input-5-cedbd4f84f3d> in <module>
>> > >> > >>> 25 from cctbx import uctbx
>> > >> > >>> 26 from cctbx import sgtbx
>> > >> > >>> ---> 27 from cctbx import miller
>> > >> > >>> 28 #from iotbx import reflection_file_reader, mtz
>> > >> > >>> 29 sns.set_context("poster")
>> > >> > >>>
>> > >> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/miller/__init__.py in <module>
>> > >> > >>> 11
>> > >> > >>> 12 from cctbx import crystal
>> > >> > >>> ---> 13 from cctbx import maptbx
>> > >> > >>> 14 from cctbx import sgtbx
>> > >> > >>> 15 from cctbx.sgtbx import lattice_symmetry
>> > >> > >>>
>> > >> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/maptbx/__init__.py in <module>
>> > >> > >>> 15 from libtbx import adopt_init_args
>> > >> > >>> 16 from libtbx.utils import Sorry
>> > >> > >>> ---> 17 import libtbx.load_env
>> > >> > >>> 18 import math
>> > >> > >>> 19 import sys, os
>> > >> > >>>
>> > >> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module>
>> > >> > >>> 3 import libtbx.env_config
>> > >> > >>> 4 import os
>> > >> > >>> ----> 5 libtbx.env = libtbx.env_config.unpickle()
>> > >> > >>> 6 libtbx.env.set_os_environ_all_dist()
>> > >> > >>> 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME")
>> > >> > >>>
>> > >> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle()
>> > >> > >>> 2603
>> > >> > >>> 2604 def unpickle():
>> > >> > >>> -> 2605 build_path = os.environ["LIBTBX_BUILD"]
>> > >> > >>> 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path)
>> > >> > >>> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")
>> > >> > >>>
>> > >> > >>> ~/anaconda3/lib/python3.6/os.py in __getitem__(self, key)
>> > >> > >>> 667 except KeyError:
>> > >> > >>> 668 # raise KeyError with the original key value
>> > >> > >>> --> 669 raise KeyError(key) from None
>> > >> > >>> 670 return self.decodevalue(value)
>> > >> > >>> 671
>> > >> > >>>
>> > >> > >>> KeyError: 'LIBTBX_BUILD'
>> > >> > >>>
>> > >> > >>>
>> > >> > >>>
>> > >> > >>> ---------------------------------------------------------------------------
>> > >> > >>> ArgumentError Traceback (most recent call last)
>> > >> > >>> <ipython-input-7-2acd3a9ce26a> in <module>
>> > >> > >>> 26 return ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar
>> > >> > >>> 27
>> > >> > >>> ---> 28 ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar=initializecrystal()
>> > >> > >>>
>> > >> > >>> <ipython-input-7-2acd3a9ce26a> in initializecrystal()
>> > >> > >>> 11 uc = uctbx.unit_cell(unit_cell)
>> > >> > >>> 12 wavelength = 1.54980
>> > >> > >>> ---> 13 xtal_symm = crystal.symmetry(unit_cell=unit_cell,
>> > >> > >>> space_group_symbol="P 43 21 2")
>> > >> > >>> 14
>> > >> > >>> 15 ms =
>> > >> > >>> miller.build_set(crystal_symmetry=xtal_symm,anomalous_flag=True,
>> > >> > >>> d_min=1.61)
>> > >> > >>>
>> > >> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/crystal/__init__.py in
>> > >> > >>> __init__(self, unit_cell, space_group_symbol, space_group_info,
>> > >> > >>> space_group, correct_rhombohedral_setting_if_necessary,
>> > >> > >>> assert_is_compatible_unit_cell, raise_sorry_if_incompatible_unit_cell,
>> > >> > >>> force_compatible_unit_cell)
>> > >> > >>> 74 if (space_group_symbol is not None):
>> > >> > >>> 75 self._space_group_info = sgtbx.space_group_info(
>> > >> > >>> ---> 76 symbol=space_group_symbol)
>> > >> > >>> 77 elif (space_group is not None):
>> > >> > >>> 78 if (isinstance(space_group, sgtbx.space_group)):
>> > >> > >>>
>> > >> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in
>> > >> > >>> __init__(self, symbol, table_id, group, number, space_group_t_den)
>> > >> > >>> 100 assert group is None
>> > >> > >>> 101 if (table_id is None):
>> > >> > >>> --> 102 symbols = space_group_symbols(symbol)
>> > >> > >>> 103 else:
>> > >> > >>> 104 if (isinstance(symbol, int)): symbol = str(symbol)
>> > >> > >>>
>> > >> > >>> ArgumentError: Python argument types in
>> > >> > >>> space_group_symbols.__init__(space_group_symbols, str)
>> > >> > >>> did not match C++ signature:
>> > >> > >>> __init__(_object*, int space_group_number)
>> > >> > >>> __init__(_object*, int space_group_number,
>> > >> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> > >> > >>> std::allocator<char> > extension='')
>> > >> > >>> __init__(_object*, int space_group_number,
>> > >> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> > >> > >>> std::allocator<char> > extension='', std::__cxx11::basic_string<char,
>> > >> > >>> std::char_traits<char>, std::allocator<char> > table_id='')
>> > >> > >>> __init__(_object*, std::__cxx11::basic_string<char,
>> > >> > >>> std::char_traits<char>, std::allocator<char> > symbol)
>> > >> > >>> __init__(_object*, std::__cxx11::basic_string<char,
>> > >> > >>> std::char_traits<char>, std::allocator<char> > symbol,
>> > >> > >>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> > >> > >>> std::allocator<char> > table_id='')
>> > >> > >>>
>> > >> > >>>
>> > >> > >>> When I tried to set LIBTBX_BUILD to /home/gergely/anaconda3 (this is
>> > >> > >>> sys.prefix on my system), I got other problems when importing. Can
>> > >> > >>> LIBTBX_BUILD be set in the conda package?
>> > >> > >>>
>> > >> > >>>
>> > >> > >>>
>> > >> > >>> ---------------------------------------------------------------------------
>> > >> > >>> FileNotFoundError Traceback (most recent call last)
>> > >> > >>> <ipython-input-1-7e2b144826cc> in <module>
>> > >> > >>> 26 from cctbx import sgtbx
>> > >> > >>> 27 #from cctbx import miller
>> > >> > >>> ---> 28 from iotbx import reflection_file_reader, mtz
>> > >> > >>> 29 sns.set_context("poster")
>> > >> > >>> 30 plt.rcParams.update({'figure.autolayout': True})
>> > >> > >>>
>> > >> > >>> ~/anaconda3/lib/python3.6/site-packages/iotbx/reflection_file_reader.py
>> > >> > >>> in <module>
>> > >> > >>> 53
>> > >> > >>> 54 from __future__ import absolute_import, division, print_function
>> > >> > >>> ---> 55 from iotbx import mtz
>> > >> > >>> 56 from iotbx.scalepack import merge as scalepack_merge
>> > >> > >>> 57 from iotbx.scalepack import no_merge_original_index as
>> > >> > >>> scalepack_no_merge
>> > >> > >>>
>> > >> > >>> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/__init__.py in <module>
>> > >> > >>> 9 import iotbx_mtz_ext as ext
>> > >> > >>> 10
>> > >> > >>> ---> 11 from iotbx.mtz import extract_from_symmetry_lib
>> > >> > >>> 12 from cctbx import xray
>> > >> > >>> 13 import cctbx.xray.observation_types
>> > >> > >>>
>> > >> > >>> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/extract_from_symmetry_lib.py
>> > >> > >>> in <module>
>> > >> > >>> 1 from __future__ import absolute_import, division, print_function
>> > >> > >>> 2 from cctbx import sgtbx
>> > >> > >>> ----> 3 import libtbx.load_env
>> > >> > >>> 4 import os.path as op
>> > >> > >>> 5 from six.moves import range
>> > >> > >>>
>> > >> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module>
>> > >> > >>> 3 import libtbx.env_config
>> > >> > >>> 4 import os
>> > >> > >>> ----> 5 libtbx.env = libtbx.env_config.unpickle()
>> > >> > >>> 6 libtbx.env.set_os_environ_all_dist()
>> > >> > >>> 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME")
>> > >> > >>>
>> > >> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle()
>> > >> > >>> 2605 build_path = os.environ["LIBTBX_BUILD"]
>> > >> > >>> 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path)
>> > >> > >>> -> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")
>> > >> > >>> 2608 env = pickle.load(libtbx_env)
>> > >> > >>> 2609 if (env.python_version_major_minor != sys.version_info[:2]):
>> > >> > >>>
>> > >> > >>> FileNotFoundError: [Errno 2] No such file or directory:
>> > >> > >>> '/home/gergely/anaconda3/libtbx_env'
>> > >> > >>>
>> > >> > >>> On Wed, Dec 18, 2019 at 9:10 AM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >> > >>> >
>> > >> > >>> > Hi Gergely,
>> > >> > >>> >
>> > >> > >>> > I've uploaded linux packages to a new channel, cctbx-dev, and you can install it with
>> > >> > >>> >
>> > >> > >>> > conda install -c cctbx-dev cctbx
>> > >> > >>> >
>> > >> > >>> > in your current environment. You should set your ~/.condarc file to pull the other dependencies from the conda-forge channel first, so put conda-forge above cctbx. Mine looks like
>> > >> > >>> >
>> > >> > >>> > channels:
>> > >> > >>> > - conda-forge
>> > >> > >>> > - defaults
>> > >> > >>> > - cctbx
>> > >> > >>> >
>> > >> > >>> > Do you need dxtbx for your scripts? This package does not build that part. I think the plan is to build a separate conda package for dxtbx so that it can be updated more frequently. I can rebuild the packages to include it for testing, but the one being submitted to conda-forge will not have it.
>> > >> > >>> >
>> > >> > >>> > Also, your error message is probably due to version of HDF5 that the development build installs. The bootstrap.py script will install 1.10.4, but your other dependency is looking for 1.10.5. Installing this cctbx conda package should install 1.10.5, which should fix the issue. Also, I'm updating those environments and 1.10.5 will be the new default version.
>> > >> > >>> >
>> > >> > >>> > Lastly, the dispatchers will not work in these packages because the old paths during the build process are still in them and many of them expect some additional information that has not been updated in the packages yet. I'm in the process of doing that and will update the cctbx-dev channel when that's done. However, by starting python, you can import cctbx modules. So you can run commands like
>> > >> > >>> >
>> > >> > >>> > from scitbx.array_family import flex
>> > >> > >>> > a = flex.random_double(1000000)
>> > >> > >>> > b = flex.min_max_mean_double(a)
>> > >> > >>> > b.min
>> > >> > >>> > b.max
>> > >> > >>> > b.mean
>> > >> > >>> >
>> > >> > >>> > Thanks!
>> > >> > >>> >
>> > >> > >>> > --
>> > >> > >>> > Billy K. Poon
>> > >> > >>> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > >> > >>> > Lawrence Berkeley National Laboratory
>> > >> > >>> > 1 Cyclotron Road, M/S 33R0345
>> > >> > >>> > Berkeley, CA 94720
>> > >> > >>> > Tel: (510) 486-5709
>> > >> > >>> > Fax: (510) 486-5909
>> > >> > >>> > Web: https://phenix-online.org
>> > >> > >>> >
>> > >> > >>> >
>> > >> > >>> > On Tue, Dec 17, 2019 at 5:39 AM Gergely Katona <gkatona(a)gmail.com> wrote:
>> > >> > >>> >>
>> > >> > >>> >> Dear Billy,
>> > >> > >>> >>
>> > >> > >>> >> Thank you for the detailed explanation, I look forward to do further
>> > >> > >>> >> testing! I aim to pool all modules under the same environment, if
>> > >> > >>> >> there are conflicts then I just try to reshuffle the order of imports.
>> > >> > >>> >> So far this did not cause problems for me even when I was using system
>> > >> > >>> >> python, but of course it is not the most prudent thing to do. With
>> > >> > >>> >> anaconda everything is much more standardized and isolated already and
>> > >> > >>> >> without being superuser I can have my familiar environment at any
>> > >> > >>> >> synchrotron based cluster. It is great that cctbx will be an integral
>> > >> > >>> >> part of this ecosystem and this was also the last thing holding me
>> > >> > >>> >> back from adopting python3. About hdf5, pymc3 requires it and
>> > >> > >>> >> importing causes a kernel restart with the following error messages,
>> > >> > >>> >> curiously if I import h5py first this can be avoided.
>> > >> > >>> >>
>> > >> > >>> >> Best wishes,
>> > >> > >>> >>
>> > >> > >>> >> Gergely
>> > >> > >>> >>
>> > >> > >>> >> import pymc3 as pm
>> > >> > >>> >>
>> > >> > >>> >> Warning! ***HDF5 library version mismatched error***
>> > >> > >>> >> The HDF5 header files used to compile this application do not match
>> > >> > >>> >> the version used by the HDF5 library to which this application is linked.
>> > >> > >>> >> Data corruption or segmentation faults may occur if the application continues.
>> > >> > >>> >> This can happen when an application was compiled by one version of HDF5 but
>> > >> > >>> >> linked with a different version of static or shared HDF5 library.
>> > >> > >>> >> You should recompile the application or check your shared library related
>> > >> > >>> >> settings such as 'LD_LIBRARY_PATH'.
>> > >> > >>> >> You can, at your own risk, disable this warning by setting the environment
>> > >> > >>> >> variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.
>> > >> > >>> >> Setting it to 2 or higher will suppress the warning messages totally.
>> > >> > >>> >> Headers are 1.10.4, library is 1.10.5
>> > >> > >>> >> SUMMARY OF THE HDF5 CONFIGURATION
>> > >> > >>> >> =================================
>> > >> > >>> >>
>> > >> > >>> >> General Information:
>> > >> > >>> >> -------------------
>> > >> > >>> >> HDF5 Version: 1.10.5
>> > >> > >>> >> Configured on: Tue Oct 22 12:02:13 UTC 2019
>> > >> > >>> >> Configured by: conda@16247e67ecd5
>> > >> > >>> >> Host system: x86_64-conda_cos6-linux-gnu
>> > >> > >>> >> Uname information: Linux 16247e67ecd5 4.15.0-1059-azure
>> > >> > >>> >> #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64
>> > >> > >>> >> GNU/Linux
>> > >> > >>> >> Byte sex: little-endian
>> > >> > >>> >> Installation point: /home/gergely/anaconda3
>> > >> > >>> >>
>> > >> > >>> >> Compiling Options:
>> > >> > >>> >> ------------------
>> > >> > >>> >> Build Mode: production
>> > >> > >>> >> Debugging Symbols: no
>> > >> > >>> >> Asserts: no
>> > >> > >>> >> Profiling: no
>> > >> > >>> >> Optimization Level: high
>> > >> > >>> >>
>> > >> > >>> >> Linking Options:
>> > >> > >>> >> ----------------
>> > >> > >>> >> Libraries: static, shared
>> > >> > >>> >> Statically Linked Executables:
>> > >> > >>> >> LDFLAGS: -Wl,-O2 -Wl,--sort-common
>> > >> > >>> >> -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags
>> > >> > >>> >> -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib
>> > >> > >>> >> -Wl,-rpath-link,/home/gergely/anaconda3/lib
>> > >> > >>> >> -L/home/gergely/anaconda3/lib
>> > >> > >>> >> H5_LDFLAGS:
>> > >> > >>> >> AM_LDFLAGS: -L/home/gergely/anaconda3/lib
>> > >> > >>> >> Extra libraries: -lrt -lpthread -lz -ldl -lm
>> > >> > >>> >> Archiver:
>> > >> > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar
>> > >> > >>> >> AR_FLAGS: cr
>> > >> > >>> >> Ranlib:
>> > >> > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib
>> > >> > >>> >>
>> > >> > >>> >> Languages:
>> > >> > >>> >> ----------
>> > >> > >>> >> C: yes
>> > >> > >>> >> C Compiler:
>> > >> > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc
>> > >> > >>> >> CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2
>> > >> > >>> >> -I/home/gergely/anaconda3/include
>> > >> > >>> >> H5_CPPFLAGS: -D_GNU_SOURCE
>> > >> > >>> >> -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API
>> > >> > >>> >> AM_CPPFLAGS: -I/home/gergely/anaconda3/include
>> > >> > >>> >> C Flags: -march=nocona -mtune=haswell
>> > >> > >>> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
>> > >> > >>> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include
>> > >> > >>> >> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5
>> > >> > >>> >> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix
>> > >> > >>> >> H5 C Flags: -std=c99 -pedantic -Wall -Wextra
>> > >> > >>> >> -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion
>> > >> > >>> >> -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal
>> > >> > >>> >> -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations
>> > >> > >>> >> -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs
>> > >> > >>> >> -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls
>> > >> > >>> >> -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef
>> > >> > >>> >> -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings
>> > >> > >>> >> -finline-functions -s -Wno-inline -Wno-aggregate-return
>> > >> > >>> >> -Wno-missing-format-attribute -Wno-missing-noreturn -O
>> > >> > >>> >> AM C Flags:
>> > >> > >>> >> Shared C Library: yes
>> > >> > >>> >> Static C Library: yes
>> > >> > >>> >>
>> > >> > >>> >>
>> > >> > >>> >> Fortran: yes
>> > >> > >>> >> Fortran Compiler:
>> > >> > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran
>> > >> > >>> >> Fortran Flags:
>> > >> > >>> >> H5 Fortran Flags: -pedantic -Wall -Wextra -Wunderflow
>> > >> > >>> >> -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2
>> > >> > >>> >> AM Fortran Flags:
>> > >> > >>> >> Shared Fortran Library: yes
>> > >> > >>> >> Static Fortran Library: yes
>> > >> > >>> >>
>> > >> > >>> >> C++: yes
>> > >> > >>> >> C++ Compiler:
>> > >> > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++
>> > >> > >>> >> C++ Flags: -fvisibility-inlines-hidden
>> > >> > >>> >> -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell
>> > >> > >>> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
>> > >> > >>> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include
>> > >> > >>> >> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5
>> > >> > >>> >> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix
>> > >> > >>> >> H5 C++ Flags: -pedantic -Wall -W -Wundef -Wshadow
>> > >> > >>> >> -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion
>> > >> > >>> >> -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual
>> > >> > >>> >> -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor
>> > >> > >>> >> -Wctor-dtor-privacy -Wabi -finline-functions -s -O
>> > >> > >>> >> AM C++ Flags:
>> > >> > >>> >> Shared C++ Library: yes
>> > >> > >>> >> Static C++ Library: yes
>> > >> > >>> >>
>> > >> > >>> >> Java: no
>> > >> > >>> >>
>> > >> > >>> >>
>> > >> > >>> >> Features:
>> > >> > >>> >> ---------
>> > >> > >>> >> Parallel HDF5: no
>> > >> > >>> >> Parallel Filtered Dataset Writes: no
>> > >> > >>> >> Large Parallel I/O: no
>> > >> > >>> >> High-level library: yes
>> > >> > >>> >> Threadsafety: yes
>> > >> > >>> >> Default API mapping: v110
>> > >> > >>> >> With deprecated public symbols: yes
>> > >> > >>> >> I/O filters (external): deflate(zlib)
>> > >> > >>> >> MPE: no
>> > >> > >>> >> Direct VFD: no
>> > >> > >>> >> dmalloc: no
>> > >> > >>> >> Packages w/ extra debug output: none
>> > >> > >>> >> API tracing: no
>> > >> > >>> >> Using memory checker: yes
>> > >> > >>> >> Memory allocation sanity checks: no
>> > >> > >>> >> Function stack tracing: no
>> > >> > >>> >> Strict file format checks: no
>> > >> > >>> >> Optimization instrumentation: no
>> > >> > >>> >>
>> > >> > >>> >> On Tue, Dec 17, 2019 at 8:35 AM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >> > >>> >> >
>> > >> > >>> >> > Hi Gergely,
>> > >> > >>> >> >
>> > >> > >>> >> > Let me build the test package tomorrow. All the gritty details for building with conda is being finalized and the official documentation will be updated to describe the steps. It would be too confusing to keep changing the documentation as the process evolves. You do have the general process, though, which is summarized as follows.
>> > >> > >>> >> >
>> > >> > >>> >> > 1) Installing dependencies. The cctbx_dependencies metapackage was an initial approach for managing the CCTBX dependencies, but after contacting the conda-forge folks, they recommended using the --only-deps flag. So when the CCTBX conda package is available, you'll be able to get a set of dependencies with,
>> > >> > >>> >> >
>> > >> > >>> >> > conda install -c conda-forge --only-deps cctbx
>> > >> > >>> >> >
>> > >> > >>> >> > By default, the bootstrap.py file will automatically install a set of dependencies in the "conda_base" directory (and a conda installation if one is not found). It just uses standard conda environment files located in libtbx/auto_build/conda_envs, so you do not need to install cctbx_dependencies as a separate step. The environment files avoid channel issues by explicitly defining the channel to pull the packages from and the cctbx channel just stores copies of packages from conda-forge. There were issues earlier where the conda-forge packages would sometimes be moved to a different label. The --use-conda flag also accepts a path to $CONDA_PREFIX if you want to use a specific environment for testing.
>> > >> > >>> >> >
>> > >> > >>> >> > 2) Building. The bootstrap.py file handles that with SCons.
>> > >> > >>> >> >
>> > >> > >>> >> > 3) Running. After building, there should be a setpaths.sh (and .csh) file that adds build/bin to your path. The build/bin directory has our dispatchers, which are just scripts that set up the environment variables for you. This prevents other programs from loading our libraries, whose versions may conflict. You should see that there is a "python" dispatcher, which is a convenience for developers. Otherwise, you can use libtbx.python, which will be able to import CCTBX modules.
>> > >> > >>> >> >
>> > >> > >>> >> > The running part is where the conda package for CCTBX will be different than this build. Since our Python files and extensions modules will be in the "site-packages" directory for Python, the PYTHONPATH variable will not be needed (and conda suggests that that variable not be set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so LD_LIBRARY_PATH is not needed. In an active environment, PATH will already be modified. And then our LIBTBX_BUILD directory can be set to sys.prefix.
>> > >> > >>> >> >
>> > >> > >>> >> > So with the conda package, you would only need to activate your conda environment and CCTBX should integrate with other conda packages. What is the conflict with hdf5? That's something that should be fixed. Thanks!
>> > >> > >>> >> >
>> > >> > >>> >> > --
>> > >> > >>> >> > Billy K. Poon
>> > >> > >>> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > >> > >>> >> > Lawrence Berkeley National Laboratory
>> > >> > >>> >> > 1 Cyclotron Road, M/S 33R0345
>> > >> > >>> >> > Berkeley, CA 94720
>> > >> > >>> >> > Tel: (510) 486-5709
>> > >> > >>> >> > Fax: (510) 486-5909
>> > >> > >>> >> > Web: https://phenix-online.org
>> > >> > >>> >> >
>> > >> > >>> >> >
>> > >> > >>> >> > On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona <gkatona(a)gmail.com> wrote:
>> > >> > >>> >> >>
>> > >> > >>> >> >> Dear Billy,
>> > >> > >>> >> >>
>> > >> > >>> >> >> Thank you for this update and for your efforts! I found a solution and
>> > >> > >>> >> >> indeed most things already work in anaconda3. The steps I took (even
>> > >> > >>> >> >> if these instructions will have short expiry date):
>> > >> > >>> >> >>
>> > >> > >>> >> >> Modified .condarc with:
>> > >> > >>> >> >> channels:
>> > >> > >>> >> >> - cctbx
>> > >> > >>> >> >> - conda-forge
>> > >> > >>> >> >> - defaults
>> > >> > >>> >> >>
>> > >> > >>> >> >> run
>> > >> > >>> >> >> conda conda install cctbx_dependencies python=3.6
>> > >> > >>> >> >>
>> > >> > >>> >> >> I expect this will work without python=3.6 in the near future.
>> > >> > >>> >> >>
>> > >> > >>> >> >> Then compiling cctbx with anaconda3 python went without any problem
>> > >> > >>> >> >> when using these flags:
>> > >> > >>> >> >> python bootstrap.py --use-conda --python3 --nproc=12
>> > >> > >>> >> >>
>> > >> > >>> >> >> Then I probably did the unorthodox thing and sourced these directories:
>> > >> > >>> >> >>
>> > >> > >>> >> >> setenv LIBTBX_BUILD /home/gergely/cctbx/build
>> > >> > >>> >> >> setenv PATH ${PATH}:/home/gergely/cctbx/build/bin
>> > >> > >>> >> >> setenv PYTHONPATH
>> > >> > >>> >> >> /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH
>> > >> > >>> >> >> setenv LD_LIBRARY_PATH
>> > >> > >>> >> >> /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH
>> > >> > >>> >> >>
>> > >> > >>> >> >> There is probably a better way to put this into conda environment.
>> > >> > >>> >> >> With these steps I could run one of my scripts depending on cctbx. The
>> > >> > >>> >> >> only problem I found is that hdf5 library had conflict with another
>> > >> > >>> >> >> package in conda, but I does not find this as a showstopper.
>> > >> > >>> >> >>
>> > >> > >>> >> >> Best wishes,
>> > >> > >>> >> >>
>> > >> > >>> >> >> Gergely
>> > >> > >>> >> >>
>> > >> > >>> >> >>
>> > >> > >>> >> >>
>> > >> > >>> >> >>
>> > >> > >>> >> >>
>> > >> > >>> >> >>
>> > >> > >>> >> >> On Fri, Dec 13, 2019 at 7:23 PM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >> > >>> >> >> >
>> > >> > >>> >> >> > Hi Gergely,
>> > >> > >>> >> >> >
>> > >> > >>> >> >> > It's still a work in progress. I'm sorting out some Windows issues right now. I can probably build a test package on a separate channel for people that want to test it (let's say next week?). I'll provide instructions, but basically, the test conda package will be in its own separate channel and the dependencies will be pulled from the conda-forge channel. I want most things to be working correctly on Python 2.7, 3.6, 3.7, and 3.8 on all 3 platforms.
>> > >> > >>> >> >> >
>> > >> > >>> >> >> > Thanks!
>> > >> > >>> >> >> >
>> > >> > >>> >> >> > --
>> > >> > >>> >> >> > Billy K. Poon
>> > >> > >>> >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > >> > >>> >> >> > Lawrence Berkeley National Laboratory
>> > >> > >>> >> >> > 1 Cyclotron Road, M/S 33R0345
>> > >> > >>> >> >> > Berkeley, CA 94720
>> > >> > >>> >> >> > Tel: (510) 486-5709
>> > >> > >>> >> >> > Fax: (510) 486-5909
>> > >> > >>> >> >> > Web: https://phenix-online.org
>> > >> > >>> >> >> >
>> > >> > >>> >> >> >
>> > >> > >>> >> >> > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona <gkatona(a)gmail.com> wrote:
>> > >> > >>> >> >> >>
>> > >> > >>> >> >> >> Dear Billy,
>> > >> > >>> >> >> >>
>> > >> > >>> >> >> >> This sounds very promising and exciting. I am not sure if cctbx is
>> > >> > >>> >> >> >> already functional as a conda package in anaconda3 (Linux) or this is
>> > >> > >>> >> >> >> still work in progress. My technical expertise does not allow me to
>> > >> > >>> >> >> >> tell the difference. What I tried:
>> > >> > >>> >> >> >>
>> > >> > >>> >> >> >> Fresh install of anaconda3. Adding - cctbx and - conda-forge to
>> > >> > >>> >> >> >> .condarc . Installing pyside2 with conda. Running conda install
>> > >> > >>> >> >> >> conda_dependencies . I get a lot package version conflict, and I
>> > >> > >>> >> >> >> cannot import cctbx or iotbx to anaconda python. Am I following the
>> > >> > >>> >> >> >> right instructions? Or it is too early to expect that cctbx works when
>> > >> > >>> >> >> >> installed through conda?
>> > >> > >>> >> >> >>
>> > >> > >>> >> >> >> Best wishes,
>> > >> > >>> >> >> >>
>> > >> > >>> >> >> >> Gergely
>> > >> > >>> >> >> >>
>> > >> > >>> >> >> >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon <BKPoon(a)lbl.gov> wrote:
>> > >> > >>> >> >> >> >
>> > >> > >>> >> >> >> > Hi all,
>> > >> > >>> >> >> >> >
>> > >> > >>> >> >> >> > For a brief update, I have submitted a recipe for cctbxlite to conda-forge (https://github.com/conda-forge/staged-recipes/pull/10021) and support for Python 3.7 and 3.8 is being added (https://github.com/cctbx/cctbx_project/pull/409). With some fixes for Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8. Some additional changes will be needed to get Windows to work with Python 3 and for tests to pass with Boost 1.70.0. That will enable the conda-forge recipe to build for all platforms and for all supported versions of Python.
>> > >> > >>> >> >> >> >
>> > >> > >>> >> >> >> > Currently, the conda-forge recipe will install into the conda python and cctbx imports can be done without sourcing the environment scripts. It looks like a lot of the environment variables being set in the dispatchers can be removed since the Python files and C++ extensions are in the right places. I'll update the libtbx_env file so that commands that load the environment can work correctly.
>> > >> > >>> >> >> >> >
>> > >> > >>> >> >> >> > --
>> > >> > >>> >> >> >> > Billy K. Poon
>> > >> > >>> >> >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > >> > >>> >> >> >> > Lawrence Berkeley National Laboratory
>> > >> > >>> >> >> >> > 1 Cyclotron Road, M/S 33R0345
>> > >> > >>> >> >> >> > Berkeley, CA 94720
>> > >> > >>> >> >> >> > Tel: (510) 486-5709
>> > >> > >>> >> >> >> > Fax: (510) 486-5909
>> > >> > >>> >> >> >> > Web: https://phenix-online.org
>> > >> > >>> >> >> >> >
>> > >> > >>> >> >> >> >
>> > >> > >>> >> >> >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll <tic20(a)cam.ac.uk> wrote:
>> > >> > >>> >> >> >> >>
>> > >> > >>> >> >> >> >> Hi Luc,
>> > >> > >>> >> >> >> >>
>> > >> > >>> >> >> >> >> That sounds promising. From there, I’d need to work out how to make a fully-packaged installer (basically a modified wheel file) for the ChimeraX ToolShed - the aim is for the end user to not have to worry about any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would need to be set dynamically before first import - but doesn’t seem insurmountable.
>> > >> > >>> >> >> >> >>
>> > >> > >>> >> >> >> >> Thanks,
>> > >> > >>> >> >> >> >>
>> > >> > >>> >> >> >> >> Tristan
>> > >> > >>> >> >> >> >>
>> > >> > >>> >> >> >> >>
>> > >> > >>> >> >> >> >>
>> > >> > >>> >> >> >> >> Tristan Croll
>> > >> > >>> >> >> >> >> Research Fellow
>> > >> > >>> >> >> >> >> Cambridge Institute for Medical Research
>> > >> > >>> >> >> >> >> University of Cambridge CB2 0XY
>> > >> > >>> >> >> >> >>
>> > >> > >>> >> >> >> >>
>> > >> > >>> >> >> >> >>
>> > >> > >>> >> >> >> >>
>> > >> > >>> >> >> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis <luc_j_bourhis(a)mac.com> wrote:
>> > >> > >>> >> >> >> >> >
>> > >> > >>> >> >> >> >> > Hi Tristan,
>> > >> > >>> >> >> >> >> >
>> > >> > >>> >> >> >> >> > cctbx could be built to use your ChimeraX python, now that cctbx is moving to Python 3. The option —with-python is there for that with the bootstrap script. The specific environment setup boil down to setting two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a framework such as ChimeraX, that should not be difficult to ensure those two variables are set.
>> > >> > >>> >> >> >> >> >
>> > >> > >>> >> >> >> >> > Best wishes,
>> > >> > >>> >> >> >> >> >
>> > >> > >>> >> >> >> >> > Luc
>> > >> > >>> >> >> >> >> >
>> > >> > >>> >> >> >> >> >
>> > >> > >>> >> >> >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll <tic20(a)cam.ac.uk> wrote:
>> > >> > >>> >> >> >> >> >>
>> > >> > >>> >> >> >> >> >> To add my two cents on this: probably the second-most common question I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". The honest answer to that is, "I didn't know how."
>> > >> > >>> >> >> >> >> >>
>> > >> > >>> >> >> >> >> >> Still don't, really - although the current developments are rather promising. The problem I've faced is that CCTBX was designed as its own self-contained Python (2.7, until very recently) environment, with its own interpreter and a lot of very specific environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is *also* its own self-contained Python (3.7) environment. To plug one into the other in that form... well, I don't think I'm a good enough programmer to really know where to start.
>> > >> > >>> >> >> >> >> >>
>> > >> > >>> >> >> >> >> >> The move to Conda and a more modular CCTBX architecture should make a lot more possible in that direction. Pip would be even better for me personally (ChimeraX can install directly from the PyPI, but doesn't interact with Conda) - but I understand pretty well the substantial challenge that would amount to (not least being that the PyPI imposes a limit - around 100MB from memory? - on the size of an individual package).
>> > >> > >>> >> >> >> >> >>
>> > >> > >>> >> >> >> >> >> Best regards,
>> > >> > >>> >> >> >> >> >>
>> > >> > >>> >> >> >> >> >> Tristan
>> > >> > >>> >> >> >> >> >>
>> > >> > >>> >> >> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote:
>> > >> > >>> >> >> >> >> >>> Hi Graeme,
>> > >> > >>> >> >> >> >> >>> Yes, I know. But “black" is a program doing a very particular task
>> > >> > >>> >> >> >> >> >>> (code formatting from the top of my head). Requiring to use a wrapper
>> > >> > >>> >> >> >> >> >>> for python itself is another level. But ok, I think I am mellowing to
>> > >> > >>> >> >> >> >> >>> the idea after all! Talking with people around me, and extrapolating,
>> > >> > >>> >> >> >> >> >>> I would bet that, right now, a great majority of people interested by
>> > >> > >>> >> >> >> >> >>> cctbx in pip have already used the cctbx, so they know about the
>> > >> > >>> >> >> >> >> >>> Python wrapper, and they would not be too sanguine about that. My
>> > >> > >>> >> >> >> >> >>> concern is for the future, when pip will be the first time some people
>> > >> > >>> >> >> >> >> >>> use cctbx. Big fat warning notices on PyPI page and a better error
>> > >> > >>> >> >> >> >> >>> message when cctbx fails because LIBTBX_BUILD is not set would be
>> > >> > >>> >> >> >> >> >>> needed but that could be all right.
>> > >> > >>> >> >> >> >> >>> If we do a pip installer, we should aim at a minimal install: cctbx,
>> > >> > >>> >> >> >> >> >>> iotbx and their dependencies, and that’s it.
>> > >> > >>> >> >> >> >> >>> Best wishes,
>> > >> > >>> >> >> >> >> >>> Luc
>> > >> > >>> >> >> >> >> >>>> On 23 Aug 2019, at 07:17, Graeme.Winter(a)Diamond.ac.uk <Graeme.Winter(a)diamond.ac.uk> wrote:
>> > >> > >>> >> >> >> >> >>>> Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g.
>> > >> > >>> >> >> >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat $(which black)
>> > >> > >>> >> >> >> >> >>>> /Library/Frameworks/Python.framework/Versions/3.6/bin/black
>> > >> > >>> >> >> >> >> >>>> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
>> > >> > >>> >> >> >> >> >>>> # -*- coding: utf-8 -*-
>> > >> > >>> >> >> >> >> >>>> import re
>> > >> > >>> >> >> >> >> >>>> import sys
>> > >> > >>> >> >> >> >> >>>> from black import main
>> > >> > >>> >> >> >> >> >>>> if __name__ == '__main__':
>> > >> > >>> >> >> >> >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
>> > >> > >>> >> >> >> >> >>>> sys.exit(main())
>> > >> > >>> >> >> >> >> >>>> So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this.
>> > >> > >>> >> >> >> >> >>>> Best wishes Graeme
>> > >> > >>> >> >> >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote:
>> > >> > >>> >> >> >> >> >>>> Hi,
>> > >> > >>> >> >> >> >> >>>> Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs?
>> > >> > >>> >> >> >> >> >>>> Best wishes,
>> > >> > >>> >> >> >> >> >>>> Luc
>> > >> > >>> >> >> >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote:
>> > >> > >>> >> >> >> >> >>>> Hi,
>> > >> > >>> >> >> >> >> >>>> I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.org<http://anaconda.org/>, and then to advertise it hard to every potential user out there.
>> > >> > >>> >> >> >> >> >>>> Best wishes,
>> > >> > >>> >> >> >> >> >>>> Luc
>> > >> > >>> >> >> >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster <asbrewster(a)lbl.gov<mailto:[email protected]>> wrote:
>> > >> > >>> >> >> >> >> >>>> Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython…). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not.
>> > >> > >>> >> >> >> >> >>>> All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support.
>> > >> > >>> >> >> >> >> >>>> There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6).
>> > >> > >>> >> >> >> >> >>>> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4.
>> > >> > >>> >> >> >> >> >>>> Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :)
>> > >> > >>> >> >> >> >> >>>> Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users.
>> > >> > >>> >> >> >> >> >>>> Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort.
>> > >> > >>> >> >> >> >> >>>> Thanks,
>> > >> > >>> >> >> >> >> >>>> -Aaron
>> > >> > >>> >> >> >> >> >>>> _______________________________________________
>> > >> > >>> >> >> >> >> >>>> cctbxbb mailing list
>> > >> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> > >> > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >> > >>> >> >> >> >> >>>> _______________________________________________
>> > >> > >>> >> >> >> >> >>>> cctbxbb mailing list
>> > >> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> > >> > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >> > >>> >> >> >> >> >>>> _______________________________________________
>> > >> > >>> >> >> >> >> >>>> cctbxbb mailing list
>> > >> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> > >> > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >> > >>> >> >> >> >> >>>> --
>> > >> > >>> >> >> >> >> >>>> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
>> > >> > >>> >> >> >> >> >>>> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
>> > >> > >>> >> >> >> >> >>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
>> > >> > >>> >> >> >> >> >>>> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>> > >> > >>> >> >> >> >> >>>> _______________________________________________
>> > >> > >>> >> >> >> >> >>>> cctbxbb mailing list
>> > >> > >>> >> >> >> >> >>>> cctbxbb(a)phenix-online.org
>> > >> > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >> > >>> >> >> >> >> >>> _______________________________________________
>> > >> > >>> >> >> >> >> >>> cctbxbb mailing list
>> > >> > >>> >> >> >> >> >>> cctbxbb(a)phenix-online.org
>> > >> > >>> >> >> >> >> >>> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >> > >>> >> >> >> >> >>
>> > >> > >>> >> >> >> >> >>
>> > >> > >>> >> >> >> >> >> _______________________________________________
>> > >> > >>> >> >> >> >> >> cctbxbb mailing list
>> > >> > >>> >> >> >> >> >> cctbxbb(a)phenix-online.org
>> > >> > >>> >> >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >> > >>> >> >> >> >> >
>> > >> > >>> >> >> >> >> >
>> > >> > >>> >> >> >> >> > _______________________________________________
>> > >> > >>> >> >> >> >> > cctbxbb mailing list
>> > >> > >>> >> >> >> >> > cctbxbb(a)phenix-online.org
>> > >> > >>> >> >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >> > >>> >> >> >> >>
>> > >> > >>> >> >> >> >>
>> > >> > >>> >> >> >> >> _______________________________________________
>> > >> > >>> >> >> >> >> cctbxbb mailing list
>> > >> > >>> >> >> >> >> cctbxbb(a)phenix-online.org
>> > >> > >>> >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >> > >>> >> >> >> >
>> > >> > >>> >> >> >> > _______________________________________________
>> > >> > >>> >> >> >> > cctbxbb mailing list
>> > >> > >>> >> >> >> > cctbxbb(a)phenix-online.org
>> > >> > >>> >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >> > >>> >> >> >>
>> > >> > >>> >> >> >>
>> > >> > >>> >> >> >>
>> > >> > >>> >> >> >> --
>> > >> > >>> >> >> >> Gergely Katona, PhD
>> > >> > >>> >> >> >> Associate Professor
>> > >> > >>> >> >> >> Department of Chemistry and Molecular Biology, University of Gothenburg
>> > >> > >>> >> >> >> Box 462, 40530 Göteborg, Sweden
>> > >> > >>> >> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> > >> > >>> >> >> >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
>> > >> > >>> >> >> >>
>> > >> > >>> >> >> >> _______________________________________________
>> > >> > >>> >> >> >> cctbxbb mailing list
>> > >> > >>> >> >> >> cctbxbb(a)phenix-online.org
>> > >> > >>> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >> > >>> >> >> >
>> > >> > >>> >> >> > _______________________________________________
>> > >> > >>> >> >> > cctbxbb mailing list
>> > >> > >>> >> >> > cctbxbb(a)phenix-online.org
>> > >> > >>> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >> > >>> >> >>
>> > >> > >>> >> >>
>> > >> > >>> >> >>
>> > >> > >>> >> >> --
>> > >> > >>> >> >> Gergely Katona, PhD
>> > >> > >>> >> >> Associate Professor
>> > >> > >>> >> >> Department of Chemistry and Molecular Biology, University of Gothenburg
>> > >> > >>> >> >> Box 462, 40530 Göteborg, Sweden
>> > >> > >>> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> > >> > >>> >> >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
>> > >> > >>> >> >>
>> > >> > >>> >> >> _______________________________________________
>> > >> > >>> >> >> cctbxbb mailing list
>> > >> > >>> >> >> cctbxbb(a)phenix-online.org
>> > >> > >>> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >> > >>> >> >
>> > >> > >>> >> > _______________________________________________
>> > >> > >>> >> > cctbxbb mailing list
>> > >> > >>> >> > cctbxbb(a)phenix-online.org
>> > >> > >>> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >> > >>> >>
>> > >> > >>> >>
>> > >> > >>> >>
>> > >> > >>> >> --
>> > >> > >>> >> Gergely Katona, PhD
>> > >> > >>> >> Associate Professor
>> > >> > >>> >> Department of Chemistry and Molecular Biology, University of Gothenburg
>> > >> > >>> >> Box 462, 40530 Göteborg, Sweden
>> > >> > >>> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> > >> > >>> >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
>> > >> > >>> >>
>> > >> > >>> >> _______________________________________________
>> > >> > >>> >> cctbxbb mailing list
>> > >> > >>> >> cctbxbb(a)phenix-online.org
>> > >> > >>> >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >> > >>> >
>> > >> > >>> > _______________________________________________
>> > >> > >>> > cctbxbb mailing list
>> > >> > >>> > cctbxbb(a)phenix-online.org
>> > >> > >>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >> > >>>
>> > >> > >>>
>> > >> > >>>
>> > >> > >>> --
>> > >> > >>> Gergely Katona, PhD
>> > >> > >>> Associate Professor
>> > >> > >>> Department of Chemistry and Molecular Biology, University of Gothenburg
>> > >> > >>> Box 462, 40530 Göteborg, Sweden
>> > >> > >>> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> > >> > >>> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
>> > >> > >>>
>> > >> > >>> _______________________________________________
>> > >> > >>> cctbxbb mailing list
>> > >> > >>> cctbxbb(a)phenix-online.org
>> > >> > >>> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >> > >
>> > >> > > _______________________________________________
>> > >> > > cctbxbb mailing list
>> > >> > > cctbxbb(a)phenix-online.org
>> > >> > > http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > Gergely Katona, Professor
>> > >> > Department of Chemistry and Molecular Biology, University of Gothenburg
>> > >> > Box 462, 40530 Göteborg, Sweden
>> > >> > Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> > >> > Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Gergely Katona, Professor
>> > >> Department of Chemistry and Molecular Biology, University of Gothenburg
>> > >> Box 462, 40530 Göteborg, Sweden
>> > >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> > >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
>> > >>
>> > >> _______________________________________________
>> > >> cctbxbb mailing list
>> > >> cctbxbb(a)phenix-online.org
>> > >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> > >
>> > > _______________________________________________
>> > > cctbxbb mailing list
>> > > cctbxbb(a)phenix-online.org
>> > > http://phenix-online.org/mailman/listinfo/cctbxbb
>> >
>> >
>> >
>> > --
>> > Gergely Katona, Professor
>> > Department of Chemistry and Molecular Biology, University of Gothenburg
>> > Box 462, 40530 Göteborg, Sweden
>> > Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> > Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
>>
>>
>>
>> --
>> Gergely Katona, Professor
>> Department of Chemistry and Molecular Biology, University of Gothenburg
>> Box 462, 40530 Göteborg, Sweden
>> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
>> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org
>> http://phenix-online.org/mailman/listinfo/cctbxbb
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
--
Gergely Katona, Professor
Department of Chemistry and Molecular Biology, University of Gothenburg
Box 462, 40530 Göteborg, Sweden
Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
6 years