[cctbxbb] DYLD_FALLBACK_LIBRARY_PATH
Billy Poon
BKPoon at lbl.gov
Tue Jan 4 09:49:00 PST 2022
Hi Graeme,
Hope you had a good break! It's been cold and rainy in California which has
been good for the drought.
For napari to work in the conda environment, you do not need to set any
environment variables when building DIALS. That's how I originally followed
your instructions and was not able to reproduce the crash. I was only able
to get the error by manually setting DYLD_FALLBACK_LIBRARY_PATH before
running bootstrap.py. The dispatcher template fix is only necessary if you
build DIALS with DYLD_FALLBACK_LIBRARY_PATH set. A fresh environment
created from conda-list-e-napari.txt does not have
DYLD_FALLBACK_LIBRARY_PATH set when activated. Maybe the variable is from
other software you have installed?
--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Fax: (510) 486-5909
Web: https://phenix-online.org
On Tue, Jan 4, 2022 at 1:24 AM Winter, Graeme (DLSLtd,RAL,LSCI) <
Graeme.Winter at diamond.ac.uk> wrote:
> OK, I can confirm that the dispatcher_thing_workaround.sh works - I cannot
> work out where the need for the environment variable comes from but I
> suspect someone who has built some of the packages napari depends on (ohai
> ctypes) had it set…
>
> Cheers G
>
> On 4 Jan 2022, at 08:54, Winter, Graeme (DLSLtd,RAL,LSCI) <
> Graeme.Winter at diamond.ac.uk> wrote:
>
> Hi Billy
>
> Sorry for the slow reply - long down time over the festive break (much
> needed)
>
> Reloading the pages now
>
> One thing which occurs to me is that this is a build-dependent solution -
> so would not be portable to the general end user in the event that we ship
> something which depends on this (e.g. in this case, napari)
>
> I will see if I can get a “clean build” which works without this being set
> in the first place then investigate some more. I have no idea where the
> variable is coming from…
>
> Cheers Graeme
>
> On 17 Dec 2021, at 20:12, Billy Poon <bkpoon at lbl.gov> wrote:
>
> Hi Graeme,
>
> Are you normally setting DYLD_FALLBACK_LIBRARY_PATH in your environment?
> When I run
>
> python3 -c 'import os; print(os.environ["DYLD_FALLBACK_LIBRARY_PATH"])'
>
> I get a KeyError since I do not have anything defined.
>
> If you are setting DYLD_FALLBACK_LIBRARY_PATH normally for builds, then
> this does require special handling and the general solution is to add those
> paths to your dispatchers using the "dispatcher_include_template.sh" in
> your build directory. I generally would not recommend defining that
> environment variable since it will override some of the libraries that are
> linked during the build phase (also depends on the Xcode version since
> /usr/lib for Xcode 13 is very different from Xcode 11). This is especially
> important if you are using conda packages for dependencies since the conda
> environment is essentially self-contained.
>
> I was only able to reproduce the crash by setting
> DYLD_FALLBACK_LIBRARY_PATH before building. To make the import work, I have
> attached a modified "dispatcher_include_template.sh" which defines
> DYLD_FALLBACK_LIBRARY_PATH=/usr/X11/lib:/usr/lib prior to the check for
> DYLD_FALLBACK_LIBRARY_PATH. Put this file in your build directory and run
> "libtbx.refresh" to regenerate your dispatchers with the new setting. You
> should be able to see the change in the dispatcher for cctbx.python. It
> should look like this
>
> LIBTBX_DISPATCHER_NAME="cctbx.python"
> export LIBTBX_DISPATCHER_NAME
> #
> ----------------------------------------------------------------------------------------
> # included from
> /Users/bkpoon/software/dials/build/dispatcher_include_DYLD_LIBRARY_PATH.sh
> # Commands to be executed at the start of the
> # auto-generated dispatchers in bin.
> export DYLD_FALLBACK_LIBRARY_PATH=/usr/X11/lib:/usr/lib
> #
> #
> ----------------------------------------------------------------------------------------
>
> We use this approach for linking certain libraries on HPC builds since
> those systems usually have some optimized libraries that we want to use.
> The "disptacher_include_template.sh" approach also allows you to choose
> where in the dispatcher to add the changes so that the order of the paths
> can be customized.
>
> Let me know if that does not work.
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Fax: (510) 486-5909
> Web: https://phenix-online.org
>
>
> On Thu, Dec 16, 2021 at 2:46 AM Winter, Graeme (DLSLtd,RAL,LSCI) <
> Graeme.Winter at diamond.ac.uk> wrote:
>
>> OK, this is to do with macOS and SIP - I suspect you may have disabled
>> this at some point in the past perhaps?
>>
>> Anyhow, I found a workaround, PR in
>> https://github.com/cctbx/cctbx_project/pull/702
>>
>> To my eyeballs at least it seems relatively harmless since it is largely
>> restoring the behaviour we would have expected anyway
>>
>> Thanks and cheerio Graeme
>>
>> On 16 Dec 2021, at 07:03, Winter, Graeme (DLSLtd,RAL,LSCI) <
>> graeme.winter at diamond.ac.uk> wrote:
>>
>> Hi Billy
>>
>> Sorry for the slow response - needed to be near the right computer &
>> currently bouncing between office and home for work
>>
>> Rebuilt base to give a clean starting point, on default Python (3.9) ->
>> this is the package list
>>
>>
>> I conda install -c c-f napari’d - getting me to here -
>>
>>
>> Then
>>
>> silver-surfer viewer :) $ dials.python viewer.py
>> /Users/graeme/data/i04-1-2021-run3-ins/Insulin_6_2.nxs
>> Traceback (most recent call last):
>> File "/Users/graeme/work/viewer/viewer.py", line 24, in <module>
>> viewer = napari.view_image(d, title="DIALS Image Viewer",
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/_lazy.py",
>> line 47, in __getattr__
>> submod = import_module(f'{module_name}.{attr_to_modules[name]}')
>> File "/Users/graeme/git/dials/conda_base/python.app/Contents/lib/python3.9/importlib/__init__.py", line
>> 127, in import_module
>> return _bootstrap._gcd_import(name[level:], package, level)
>> File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
>> File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
>> File "<frozen importlib._bootstrap>", line 986, in
>> _find_and_load_unlocked
>> File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
>> File "<frozen importlib._bootstrap_external>", line 850, in exec_module
>> File "<frozen importlib._bootstrap>", line 228, in
>> _call_with_frames_removed
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/view_layers.py", line
>> 17, in <module>
>> from .viewer import Viewer
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/viewer.py",
>> line 5, in <module>
>> from .components.viewer_model import ViewerModel
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/components/__init__.py",
>> line 19, in <module>
>> from .layerlist import LayerList
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/components/layerlist.py",
>> line 8, in <module>
>> from ..layers import Image, Labels, Layer
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/layers/__init__.py",
>> line 12, in <module>
>> from .image import Image
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/layers/image/__init__.py",
>> line 1, in <module>
>> from .image import Image
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/layers/image/image.py",
>> line 12, in <module>
>> from ...utils import config
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/utils/config.py", line
>> 5, in <module>
>> from ._octree import get_octree_config
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/utils/_octree.py", line
>> 11, in <module>
>> from ..settings import get_settings
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/settings/__init__.py",
>> line 6, in <module>
>> from ._napari_settings import NapariSettings
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/settings/_napari_settings.py",
>> line 9, in <module>
>> from ._appearance import AppearanceSettings
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/settings/_appearance.py",
>> line 4, in <module>
>> from ..utils.theme import available_themes
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/utils/theme.py", line
>> 20, in <module>
>> from .._vendor import darkdetect
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/_vendor/darkdetect/__init__.py",
>> line 17, in <module>
>> from ._mac_detect import *
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/_vendor/darkdetect/_mac_detect.py",
>> line 18, in <module>
>> objc = ctypes.cdll.LoadLibrary('libobjc.dylib')
>> File
>> "/Users/graeme/git/dials/conda_base/python.app/Contents/lib/python3.9/ctypes/__init__.py", line
>> 460, in LoadLibrary
>> return self._dlltype(name)
>> File
>> "/Users/graeme/git/dials/conda_base/python.app/Contents/lib/python3.9/ctypes/__init__.py", line
>> 382, in __init__
>> self._handle = _dlopen(self._name, mode)
>> OSError: dlopen(libobjc.dylib, 6): image not found
>>
>> And
>>
>> silver-surfer viewer :( $ cctbx.python -c "from napari import viewer"
>> Traceback (most recent call last):
>> File "<string>", line 1, in <module>
>> File "<frozen importlib._bootstrap>", line 1055, in _handle_fromlist
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/_lazy.py",
>> line 45, in __getattr__
>> return import_module(f'{module_name}.{name}')
>> File "/Users/graeme/git/dials/conda_base/python.app/Contents/lib/python3.9/importlib/__init__.py", line
>> 127, in import_module
>> return _bootstrap._gcd_import(name[level:], package, level)
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/viewer.py",
>> line 5, in <module>
>> from .components.viewer_model import ViewerModel
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/components/__init__.py",
>> line 19, in <module>
>> from .layerlist import LayerList
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/components/layerlist.py",
>> line 8, in <module>
>> from ..layers import Image, Labels, Layer
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/layers/__init__.py",
>> line 12, in <module>
>> from .image import Image
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/layers/image/__init__.py",
>> line 1, in <module>
>> from .image import Image
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/layers/image/image.py",
>> line 12, in <module>
>> from ...utils import config
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/utils/config.py", line
>> 5, in <module>
>> from ._octree import get_octree_config
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/utils/_octree.py", line
>> 11, in <module>
>> from ..settings import get_settings
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/settings/__init__.py",
>> line 6, in <module>
>> from ._napari_settings import NapariSettings
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/settings/_napari_settings.py",
>> line 9, in <module>
>> from ._appearance import AppearanceSettings
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/settings/_appearance.py",
>> line 4, in <module>
>> from ..utils.theme import available_themes
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/utils/theme.py", line
>> 20, in <module>
>> from .._vendor import darkdetect
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/_vendor/darkdetect/__init__.py",
>> line 17, in <module>
>> from ._mac_detect import *
>> File
>> "/Users/graeme/git/dials/conda_base/lib/python3.9/site-packages/napari/_vendor/darkdetect/_mac_detect.py",
>> line 18, in <module>
>> objc = ctypes.cdll.LoadLibrary('libobjc.dylib')
>> File
>> "/Users/graeme/git/dials/conda_base/python.app/Contents/lib/python3.9/ctypes/__init__.py", line
>> 460, in LoadLibrary
>> return self._dlltype(name)
>> File
>> "/Users/graeme/git/dials/conda_base/python.app/Contents/lib/python3.9/ctypes/__init__.py", line
>> 382, in __init__
>> self._handle = _dlopen(self._name, mode)
>> OSError: dlopen(libobjc.dylib, 6): image not found
>>
>> however
>>
>> silver-surfer viewer :( $ which python3
>> /Users/graeme/git/dials/conda_base/bin/python3
>> silver-surfer viewer :) $ python3 -c "from napari import viewer"
>> silver-surfer viewer :) $
>>
>> Following these instructions on macOS should reproduce the problem
>>
>> https://dials.github.io/documentation/installation_developer.html
>>
>> (with an additional conda install napari) - am running on Big Sur here.
>>
>> I generated this email with fresh bootstrap -> should be reproducible
>>
>> Thanks for any help you can offer!
>>
>> Graeme
>>
>>
>>
>> On 14 Dec 2021, at 19:15, Billy Poon <bkpoon at lbl.gov> wrote:
>>
>> Hi Graeme,
>>
>> The DYLD_FALLBACK_LIBRARY_PATH is prepended because we want cctbx
>> libraries to come before any other user defined libraries when running
>> cctbx code. Otherwise, there is no way to guarantee that any libraries that
>> the user overrides in their environment will actually work with cctbx code.
>>
>>
>> Are you using conda? How did you construct your conda environment? You
>> should be able to use "conda install napari" in your conda_base directory
>> or some other conda environment that you use for building. Can you list
>> your environment (i.e. conda list)? And the explicit one so that it can be
>> reproduced (conda list -e --explicit --md5)? Also, steps to reproduce your
>> build would be helpful in reproducing the error.
>>
>> I tested "conda install -c conda-forge napari" in my Python 3.7
>> conda_base directory and the import works without any errors from
>> cctbx.python.
>>
>> --
>> Billy K. Poon
>> Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> Lawrence Berkeley National Laboratory
>> 1 Cyclotron Road, M/S 33R0345
>> Berkeley, CA 94720
>> Fax: (510) 486-5909
>> Web: https://phenix-online.org
>>
>>
>> On Tue, Dec 14, 2021 at 9:30 AM Winter, Graeme (DLSLtd,RAL,LSCI) <
>> Graeme.Winter at diamond.ac.uk> wrote:
>>
>> Hi Folks,
>>
>> By default cctbx dispatchers on macOS clobber DYLD_FALLBACK_LIBRARY_PATH
>> with some stuff which means that
>>
>> from napari import viewer
>>
>> fails with a huge stack trace ending in
>>
>> OSError: dlopen(libobjc.dylib, 6): image not found
>>
>> If I comment out the clobbering in the dispatcher then everything works
>> fine so I am fairly sure this is the problem
>>
>> Some questions -
>>
>> - why do we do this?
>> - is there a way I can make a dispatcher not to this? or maybe leave the
>> system DYLD… as it was and _append_ to rather than prepend?
>>
>> Seems like a thing I am surprised I have not broken a toe on before now
>>
>> Thanks Graeme
>>
>>
>>
>> --
>> This e-mail and any attachments may contain confidential, copyright and
>> or privileged material, and are for the use of the intended addressee only.
>> If you are not the intended addressee or an authorised recipient of the
>> addressee please notify us of receipt by returning the e-mail and do not
>> use, copy, retain, distribute or disclose the information in or attached to
>> the e-mail.
>> Any opinions expressed within this e-mail are those of the individual and
>> not necessarily of Diamond Light Source Ltd.
>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
>> attachments are free from viruses and we cannot accept liability for any
>> damage which you may sustain as a result of software viruses which may be
>> transmitted in or with the message.
>> Diamond Light Source Limited (company no. 4375679). Registered in England
>> and Wales with its registered office at Diamond House, Harwell Science and
>> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb at phenix-online.org
>> http://phenix-online.org/mailman/listinfo/cctbxbb
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb at phenix-online.org
>> http://phenix-online.org/mailman/listinfo/cctbxbb
>>
>>
>>
>>
>> --
>>
>> This e-mail and any attachments may contain confidential, copyright and
>> or privileged material, and are for the use of the intended addressee only.
>> If you are not the intended addressee or an authorised recipient of the
>> addressee please notify us of receipt by returning the e-mail and do not
>> use, copy, retain, distribute or disclose the information in or attached to
>> the e-mail.
>> Any opinions expressed within this e-mail are those of the individual and
>> not necessarily of Diamond Light Source Ltd.
>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
>> attachments are free from viruses and we cannot accept liability for any
>> damage which you may sustain as a result of software viruses which may be
>> transmitted in or with the message.
>> Diamond Light Source Limited (company no. 4375679). Registered in England
>> and Wales with its registered office at Diamond House, Harwell Science and
>> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>>
>> <conda-list-e.txt><conda-list-e-napari.txt>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb at phenix-online.org
>> http://phenix-online.org/mailman/listinfo/cctbxbb
>>
>>
>>
>>
>> --
>>
>> This e-mail and any attachments may contain confidential, copyright and
>> or privileged material, and are for the use of the intended addressee only.
>> If you are not the intended addressee or an authorised recipient of the
>> addressee please notify us of receipt by returning the e-mail and do not
>> use, copy, retain, distribute or disclose the information in or attached to
>> the e-mail.
>> Any opinions expressed within this e-mail are those of the individual and
>> not necessarily of Diamond Light Source Ltd.
>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
>> attachments are free from viruses and we cannot accept liability for any
>> damage which you may sustain as a result of software viruses which may be
>> transmitted in or with the message.
>> Diamond Light Source Limited (company no. 4375679). Registered in England
>> and Wales with its registered office at Diamond House, Harwell Science and
>> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb at phenix-online.org
>> http://phenix-online.org/mailman/listinfo/cctbxbb
>>
> <dispatcher_include_DYLD_LIBRARY_PATH.sh>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
>
>
> --
>
> This e-mail and any attachments may contain confidential, copyright and or
> privileged material, and are for the use of the intended addressee only. If
> you are not the intended addressee or an authorised recipient of the
> addressee please notify us of receipt by returning the e-mail and do not
> use, copy, retain, distribute or disclose the information in or attached to
> the e-mail.
> Any opinions expressed within this e-mail are those of the individual and
> not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
> attachments are free from viruses and we cannot accept liability for any
> damage which you may sustain as a result of software viruses which may be
> transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in England
> and Wales with its registered office at Diamond House, Harwell Science and
> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
>
>
> --
>
> This e-mail and any attachments may contain confidential, copyright and or
> privileged material, and are for the use of the intended addressee only. If
> you are not the intended addressee or an authorised recipient of the
> addressee please notify us of receipt by returning the e-mail and do not
> use, copy, retain, distribute or disclose the information in or attached to
> the e-mail.
> Any opinions expressed within this e-mail are those of the individual and
> not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
> attachments are free from viruses and we cannot accept liability for any
> damage which you may sustain as a result of software viruses which may be
> transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in England
> and Wales with its registered office at Diamond House, Harwell Science and
> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb at phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20220104/7503fb24/attachment.htm>
More information about the cctbxbb
mailing list