I think this is a problem caused by the latest Xcode 7.3 update, released 2016-03-21 (the date the update was applied according to the App Store update history coincides with the time our mac builds became broken): https://developer.apple.com/library/ios/releasenotes/DeveloperTools/RN-Xcode... I have reproduced the issue with the standard cctbx build using the bootstrap script as follows: $ svn export svn://svn.code.sf.net/p/cctbx/code/trunk/libtbx/auto_build/bootstrap.py $ python bootstrap.py --builder=cctbx --nproc=8 $ source build/setpaths.sh $ libtbx.python ../modules/cctbx_project/cctbx/xray/boost_python/tst_xray.py show_stack(1): ../modules/cctbx_project/cctbx/xray/boost_python/tst_xray.py(381) exercise_conversions show_stack(2): ../modules/cctbx_project/cctbx/xray/boost_python/tst_xray.py(1993) run show_stack(3): ../modules/cctbx_project/cctbx/xray/boost_python/tst_xray.py(2009) <module> libc backtrace (30 frames, most recent call last): 31 ??? 0x0000000000000003 0x0 + 3 30 Python 0x0000000100000d71 Python + 3441 29 Python 0x0000000100000e58 Python + 3672 28 Python 0x00000001000ef08f Py_Main + 3135 27 Python 0x00000001000d78de PyRun_SimpleFileExFlags + 702 26 Python 0x00000001000d7da4 PyRun_FileExFlags + 164 25 Python 0x00000001000aea26 PyEval_EvalCode + 54 24 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 23 Python 0x00000001000b25e8 PyEval_EvalFrameEx + 12280 22 Python 0x00000001000b6ad6 fast_function + 118 21 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 20 Python 0x00000001000b25e8 PyEval_EvalFrameEx + 12280 19 Python 0x00000001000b6ad6 fast_function + 118 18 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 17 Python 0x00000001000b41b8 PyEval_EvalFrameEx + 19400 16 Python 0x0000000100011043 PyObject_Call + 99 15 Python 0x000000010006adab type_call + 347 14 Python 0x00000001000708cf slot_tp_init + 175 13 Python 0x0000000100011043 PyObject_Call + 99 12 Python 0x000000010001f0f8 instancemethod_call + 232 11 Python 0x0000000100011043 PyObject_Call + 99 10 libboost_python.dylib 0x0000000101a88663 boost::python::objects::function_call(_object*, _object*, _object*) + 83 9 libboost_python.dylib 0x0000000101a8ce71 boost::python::handle_exception_impl(boost::function0<void>) + 81 8 libboost_python.dylib 0x0000000101a889ca boost::detail::function::void_function_ref_invoker0::invoke(boost::detail::function::function_buffer&) + 26 7 libboost_python.dylib 0x0000000101a8693a boost::python::objects::function::call(_object*, _object*) const + 874 6 cctbx_xray_ext.so 0x000000010748ba55 boost::python::detail::caller_arity<3u>::impl const&, scitbx::af::const_ref const&), boost::python::default_call_policies, boost::mpl::vector4 const&, scitbx::af::const_ref const&> >::operator()(_object*, _object*) + 229 5 cctbx_xray_ext.so 0x000000010748f7d4 boost::python::objects::make_holder<2>::apply >, boost::mpl::joint_view const&, scitbx::af::const_ref const&, boost::python::optional, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_> >, boost::python::optional > >::execute(_object*, scitbx::af::const_ref const&, scitbx::af::const_ref const&) + 100 4 libunwind.dylib 0x00007fff86be8404 _Unwind_GetRegionStart + 17 3 libsystem_platform.dylib 0x00007fff89b59eaa _sigtramp + 26 2 boost_python_meta_ext.so 0x0000000101a413d0 initboost_python_meta_ext + 0 Floating-point error (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. I have narrowed this down to what looks like a case of compiler over-optimisation at line 41 of cctbx/xray/conversions.h. Replacing: if (f_sq > 0) f = std::sqrt(f_sq); else f = 0; with: if (f_sq <= 0) { f = 0; } else { CCTBX_ASSERT(f_sq > 0); f = std::sqrt(f_sq); } Causes the error to disappear. However there are a number of other errors that are potentially more concerning, including some that result in different output rather than a floating point exception, which will likely be harder to debug: $ libtbx.run_tests_parallel module=scitbx module=cctbx module=iotbx nproc=8 run_in_tmp_dir=True Error: the following jobs returned non-zero exit codes or suspicious stderr output: libtbx.python "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/scitbx/lbfgs/tst_lbfgs_fem.py" [FAIL] Time: 14.01 Return code: 1 OKs: 0 Standard error: Traceback (most recent call last): File "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/scitbx/lbfgs/tst_lbfgs_fem.py", line 159, in <module> run(sys.argv[1:]) File "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/scitbx/lbfgs/tst_lbfgs_fem.py", line 154, in run compare_implementations() File "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/scitbx/lbfgs/tst_lbfgs_fem.py", line 130, in compare_implementations this_script=this_script, n=n, m=m, iprint=[iprint1, iprint2]) File "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/scitbx/lbfgs/tst_lbfgs_fem.py", line 117, in run_and_compare_implementations assert not show_diff(a, b) AssertionError libtbx.python "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/xray/boost_python/tst_xray.py" [FAIL] Time: 1.04 Return code: 1 OKs: 0 Standard error: show_stack(1): /Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/xray/boost_python/tst_xray.py(381) exercise_conversions show_stack(2): /Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/xray/boost_python/tst_xray.py(1993) run show_stack(3): /Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/xray/boost_python/tst_xray.py(2009) <module> libc backtrace (30 frames, most recent call last): 31 ??? 0x0000000000000003 0x0 + 3 30 Python 0x0000000100000d71 Python + 3441 29 Python 0x0000000100000e58 Python + 3672 28 Python 0x00000001000ef08f Py_Main + 3135 27 Python 0x00000001000d78de PyRun_SimpleFileExFlags + 702 26 Python 0x00000001000d7da4 PyRun_FileExFlags + 164 25 Python 0x00000001000aea26 PyEval_EvalCode + 54 24 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 23 Python 0x00000001000b25e8 PyEval_EvalFrameEx + 12280 22 Python 0x00000001000b6ad6 fast_function + 118 21 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 20 Python 0x00000001000b25e8 PyEval_EvalFrameEx + 12280 19 Python 0x00000001000b6ad6 fast_function + 118 18 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 17 Python 0x00000001000b41b8 PyEval_EvalFrameEx + 19400 16 Python 0x0000000100011043 PyObject_Call + 99 15 Python 0x000000010006adab type_call + 347 14 Python 0x00000001000708cf slot_tp_init + 175 13 Python 0x0000000100011043 PyObject_Call + 99 12 Python 0x000000010001f0f8 instancemethod_call + 232 11 Python 0x0000000100011043 PyObject_Call + 99 10 libboost_python.dylib 0x0000000101e85663 boost::python::objects::function_call(_object*, _object*, _object*) + 83 9 libboost_python.dylib 0x0000000101e89e71 boost::python::handle_exception_impl(boost::function0<void>) + 81 8 libboost_python.dylib 0x0000000101e859ca boost::detail::function::void_function_ref_invoker0::invoke(boost::detail::function::function_buffer&) + 26 7 libboost_python.dylib 0x0000000101e8393a boost::python::objects::function::call(_object*, _object*) const + 874 6 cctbx_xray_ext.so 0x0000000107c4da55 boost::python::detail::caller_arity<3u>::impl const&, scitbx::af::const_ref const&), boost::python::default_call_policies, boost::mpl::vector4 const&, scitbx::af::const_ref const&> >::operator()(_object*, _object*) + 229 5 cctbx_xray_ext.so 0x0000000107c517d4 boost::python::objects::make_holder<2>::apply >, boost::mpl::joint_view const&, scitbx::af::const_ref const&, boost::python::optional, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_> >, boost::python::optional > >::execute(_object*, scitbx::af::const_ref const&, scitbx::af::const_ref const&) + 100 4 ??? 0x0000000000000005 0x0 + 5 3 libsystem_platform.dylib 0x00007fff89b59eaa _sigtramp + 26 2 boost_python_meta_ext.so 0x0000000101e3e3d0 initboost_python_meta_ext + 0 Floating-point error (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. libtbx.python "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/maptbx/boost_python/tst_maptbx.py" [FAIL] Time: 4.96 Return code: 1 OKs: 0 Standard error: Traceback (most recent call last): File "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/maptbx/boost_python/tst_maptbx.py", line 1395, in <module> run(args=sys.argv[1:]) File "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/maptbx/boost_python/tst_maptbx.py", line 1373, in run exercise_misc() File "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/maptbx/boost_python/tst_maptbx.py", line 365, in exercise_misc assert (approx_equal(stats.min, 8.3, eps=0.1)) AssertionError libtbx.python "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/regression/tst_miller.py" P31 [FAIL] Time: 12.97 Return code: 1 OKs: 0 Standard error: show_stack(1): /Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/miller/__init__.py(2300) f_sq_as_f show_stack(2): /Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/miller/__init__.py(2343) as_amplitude_array show_stack(3): /Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/miller/__init__.py(2276) enforce_positive_amplitudes show_stack(4): /Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/regression/tst_miller.py(197) exercise_enforce_positive_amplitudes show_stack(5): /Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/regression/tst_miller.py(2398) run show_stack(6): /Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/regression/tst_miller.py(2415) <module> libc backtrace (38 frames, most recent call last): 39 Python 0x0000000100000d71 Python + 3441 38 Python 0x0000000100000e58 Python + 3672 37 Python 0x00000001000ef08f Py_Main + 3135 36 Python 0x00000001000d78de PyRun_SimpleFileExFlags + 702 35 Python 0x00000001000d7da4 PyRun_FileExFlags + 164 34 Python 0x00000001000aea26 PyEval_EvalCode + 54 33 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 32 Python 0x00000001000b25e8 PyEval_EvalFrameEx + 12280 31 Python 0x00000001000b6ad6 fast_function + 118 30 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 29 Python 0x00000001000b25e8 PyEval_EvalFrameEx + 12280 28 Python 0x00000001000b6ad6 fast_function + 118 27 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 26 Python 0x00000001000b25e8 PyEval_EvalFrameEx + 12280 25 Python 0x00000001000b6ad6 fast_function + 118 24 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 23 Python 0x00000001000b25e8 PyEval_EvalFrameEx + 12280 22 Python 0x00000001000b6ad6 fast_function + 118 21 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 20 Python 0x00000001000b25e8 PyEval_EvalFrameEx + 12280 19 Python 0x00000001000b6ad6 fast_function + 118 18 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 17 Python 0x00000001000b41b8 PyEval_EvalFrameEx + 19400 16 Python 0x0000000100011043 PyObject_Call + 99 15 Python 0x000000010006adab type_call + 347 14 Python 0x00000001000708cf slot_tp_init + 175 13 Python 0x0000000100011043 PyObject_Call + 99 12 Python 0x000000010001f0f8 instancemethod_call + 232 11 Python 0x0000000100011043 PyObject_Call + 99 10 libboost_python.dylib 0x000000010270f663 boost::python::objects::function_call(_object*, _object*, _object*) + 83 9 libboost_python.dylib 0x0000000102713e71 boost::python::handle_exception_impl(boost::function0<void>) + 81 8 libboost_python.dylib 0x000000010270f9ca boost::detail::function::void_function_ref_invoker0::invoke(boost::detail::function::function_buffer&) + 26 7 libboost_python.dylib 0x000000010270d93a boost::python::objects::function::call(_object*, _object*) const + 874 6 cctbx_xray_ext.so 0x00000001083e7f7b boost::python::detail::caller_arity<4u>::impl const&, scitbx::af::const_ref const&, double const&), boost::python::default_call_policies, boost::mpl::vector5 const&, scitbx::af::const_ref const&, double const&> >::operator()(_object*, _object*) + 347 5 cctbx_xray_ext.so 0x00000001083e7d47 boost::python::objects::make_holder<3>::apply >, boost::mpl::joint_view const&, scitbx::af::const_ref const&, boost::python::optional, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_> >, boost::python::optional > >::execute(_object*, scitbx::af::const_ref const&, scitbx::af::const_ref const&, double const&) + 87 4 ??? 0x0000000000000000 0x0 + 0 3 libsystem_platform.dylib 0x00007fff89b59eaa _sigtramp + 26 2 boost_python_meta_ext.so 0x00000001025813d0 initboost_python_meta_ext + 0 Floating-point error (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. libtbx.python "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/iotbx/regression/tst_reflection_file_editor.py" [FAIL] Time: 1.64 Return code: 1 OKs: 0 Standard error: show_stack(1): /Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/miller/__init__.py(2300) f_sq_as_f show_stack(2): /Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/iotbx/reflection_file_editor.py(1167) modify_experimental_data_array show_stack(3): /Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/iotbx/reflection_file_editor.py(1034) modify_array show_stack(4): /Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/iotbx/reflection_file_editor.py(525) __init__ show_stack(5): /Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/iotbx/regression/tst_reflection_file_editor.py(60) run_and_reload show_stack(6): /Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/iotbx/regression/tst_reflection_file_editor.py(67) exercise_basic show_stack(7): /Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/iotbx/regression/tst_reflection_file_editor.py(901) <module> libc backtrace (48 frames, most recent call last): 49 ??? 0x0000000000000003 0x0 + 3 48 Python 0x0000000100000d71 Python + 3441 47 Python 0x0000000100000e58 Python + 3672 46 Python 0x00000001000ef08f Py_Main + 3135 45 Python 0x00000001000d78de PyRun_SimpleFileExFlags + 702 44 Python 0x00000001000d7da4 PyRun_FileExFlags + 164 43 Python 0x00000001000aea26 PyEval_EvalCode + 54 42 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 41 Python 0x00000001000b25e8 PyEval_EvalFrameEx + 12280 40 Python 0x00000001000b6ad6 fast_function + 118 39 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 38 Python 0x00000001000b25e8 PyEval_EvalFrameEx + 12280 37 Python 0x00000001000b6ad6 fast_function + 118 36 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 35 Python 0x00000001000b41b8 PyEval_EvalFrameEx + 19400 34 Python 0x0000000100011043 PyObject_Call + 99 33 Python 0x000000010006adab type_call + 347 32 Python 0x00000001000708cf slot_tp_init + 175 31 Python 0x0000000100011043 PyObject_Call + 99 30 Python 0x000000010001f0f8 instancemethod_call + 232 29 Python 0x0000000100011043 PyObject_Call + 99 28 Python 0x0000000100037b7b function_call + 363 27 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 26 Python 0x00000001000b25e8 PyEval_EvalFrameEx + 12280 25 Python 0x00000001000b6ad6 fast_function + 118 24 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 23 Python 0x00000001000b25e8 PyEval_EvalFrameEx + 12280 22 Python 0x00000001000b6ad6 fast_function + 118 21 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 20 Python 0x00000001000b25e8 PyEval_EvalFrameEx + 12280 19 Python 0x00000001000b6ad6 fast_function + 118 18 Python 0x00000001000af264 PyEval_EvalCodeEx + 2100 17 Python 0x00000001000b41b8 PyEval_EvalFrameEx + 19400 16 Python 0x0000000100011043 PyObject_Call + 99 15 Python 0x000000010006adab type_call + 347 14 Python 0x00000001000708cf slot_tp_init + 175 13 Python 0x0000000100011043 PyObject_Call + 99 12 Python 0x000000010001f0f8 instancemethod_call + 232 11 Python 0x0000000100011043 PyObject_Call + 99 10 libboost_python.dylib 0x0000000101b0f663 boost::python::objects::function_call(_object*, _object*, _object*) + 83 9 libboost_python.dylib 0x0000000101b13e71 boost::python::handle_exception_impl(boost::function0<void>) + 81 8 libboost_python.dylib 0x0000000101b0f9ca boost::detail::function::void_function_ref_invoker0::invoke(boost::detail::function::function_buffer&) + 26 7 libboost_python.dylib 0x0000000101b0d93a boost::python::objects::function::call(_object*, _object*) const + 874 6 cctbx_xray_ext.so 0x00000001084eef7b boost::python::detail::caller_arity<4u>::impl const&, scitbx::af::const_ref const&, double const&), boost::python::default_call_policies, boost::mpl::vector5 const&, scitbx::af::const_ref const&, double const&> >::operator()(_object*, _object*) + 347 5 cctbx_xray_ext.so 0x00000001084eed47 boost::python::objects::make_holder<3>::apply >, boost::mpl::joint_view const&, scitbx::af::const_ref const&, boost::python::optional, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_> >, boost::python::optional > >::execute(_object*, scitbx::af::const_ref const&, scitbx::af::const_ref const&, double const&) + 87 4 ??? 0x000000010a526948 0x0 + 4468140360 3 libsystem_platform.dylib 0x00007fff89b59eaa _sigtramp + 26 2 boost_python_meta_ext.so 0x00000001019413d0 initboost_python_meta_ext + 0 Floating-point error (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. Please verify these tests manually. Summary: Tests run : 336 Failures : 5 Warnings (possible failures) : 0 Stderr output (discouraged) : 5 See run_tests_parallel_zlog for full output. In particular, there is different output for the lbfgs comparisons to the fortran implementation (fortran output remains same, both raw and raw_reference cpp outputs differ from fortran): $ scitbx.python "/Users/rjgildea/tmp/tst_bootstrap/modules/cctbx_project/scitbx/lbfgs/tst_lbfgs_fem.py" fortran 14 2 2 0 ************************************************* N= 14 NUMBER OF CORRECTIONS= 2 INITIAL VALUES F= 1.694D+02 GNORM= 6.161D+02 ************************************************* I NFN FUNC GNORM STEPLENGTH 1 2 1.476D+02 4.302D+02 1.623D-03 3 4 2.908D+01 2.874D+01 1.000D+00 5 8 2.826D+01 9.851D+00 2.100D+01 7 10 2.494D+01 4.485D+01 1.000D+00 9 17 5.456D+00 6.382D+00 1.406D+00 11 20 4.682D+00 4.658D+00 1.000D+00 13 23 3.360D+00 2.641D+01 1.000D+00 15 25 1.727D+00 2.826D+00 1.000D+00 17 28 1.291D+00 1.311D+01 1.000D+00 19 31 5.774D-01 7.928D+00 4.048D-01 21 33 3.185D-01 1.783D+01 1.000D+00 23 35 6.065D-02 3.459D+00 1.000D+00 25 37 2.849D-02 3.223D+00 1.000D+00 27 40 1.306D-02 1.305D-01 6.670D-02 29 42 2.974D-04 6.588D-01 1.000D+00 31 45 1.823D-06 1.125D-02 1.000D+00 33 48 5.604D-10 1.497D-04 2.447D-02 34 49 5.059D-15 4.741D-07 1.000D+00 THE MINIMIZATION TERMINATED WITHOUT DETECTING ERRORS. IFLAG = 0 $ scitbx.python "/Users/rjgildea/tmp/tst_bootstrap/modules/cctbx_project/scitbx/lbfgs/tst_lbfgs_fem.py" raw 14 2 2 0 ************************************************* N= 14 NUMBER OF CORRECTIONS= 2 INITIAL VALUES F= 1.694E+02 GNORM= 6.161E+02 ************************************************* I NFN FUNC GNORM STEPLENGTH 1 2 1.476E+02 4.302E+02 1.623E-03 3 4 2.908E+01 2.874E+01 1.000E+00 5 8 2.826E+01 9.851E+00 2.100E+01 7 10 2.494E+01 4.485E+01 1.000E+00 9 17 5.456E+00 6.382E+00 1.406E+00 11 20 4.682E+00 4.658E+00 1.000E+00 13 23 3.360E+00 2.641E+01 1.000E+00 15 25 1.727E+00 2.826E+00 1.000E+00 17 28 1.291E+00 1.311E+01 1.000E+00 19 31 5.774E-01 7.928E+00 4.048E-01 21 33 3.185E-01 1.783E+01 1.000E+00 23 35 6.065E-02 3.459E+00 1.000E+00 25 37 2.849E-02 3.223E+00 1.000E+00 27 40 1.306E-02 1.305E-01 6.670E-02 29 42 2.974E-04 6.588E-01 1.000E+00 31 45 1.823E-06 1.125E-02 1.000E+00 33 48 5.603E-10 1.498E-04 2.447E-02 34 49 5.060E-15 4.757E-07 1.000E+00 THE MINIMIZATION TERMINATED WITHOUT DETECTING ERRORS. IFLAG = 0 $ scitbx.python "/Users/rjgildea/tmp/tst_bootstrap/modules/cctbx_project/scitbx/lbfgs/tst_lbfgs_fem.py" raw_reference 14 2 2 0 ************************************************* N= 14 NUMBER OF CORRECTIONS= 2 INITIAL VALUES F= 1.694E+02 GNORM= 6.161E+02 ************************************************* I NFN FUNC GNORM STEPLENGTH 1 2 1.476E+02 4.302E+02 1.623E-03 3 4 2.908E+01 2.874E+01 1.000E+00 5 8 2.826E+01 9.851E+00 2.100E+01 7 10 2.494E+01 4.485E+01 1.000E+00 9 17 5.456E+00 6.382E+00 1.406E+00 11 20 4.682E+00 4.658E+00 1.000E+00 13 23 3.360E+00 2.641E+01 1.000E+00 15 25 1.727E+00 2.826E+00 1.000E+00 17 28 1.291E+00 1.311E+01 1.000E+00 19 31 5.774E-01 7.928E+00 4.048E-01 21 33 3.185E-01 1.783E+01 1.000E+00 23 35 6.065E-02 3.459E+00 1.000E+00 25 37 2.849E-02 3.223E+00 1.000E+00 27 40 1.306E-02 1.305E-01 6.670E-02 29 42 2.974E-04 6.588E-01 1.000E+00 31 45 1.823E-06 1.125E-02 1.000E+00 33 48 5.603E-10 1.498E-04 2.447E-02 34 49 5.060E-15 4.757E-07 1.000E+00 THE MINIMIZATION TERMINATED WITHOUT DETECTING ERRORS. IFLAG = 0 Testing on Linux, the raw and raw_reference give identical output to the fortran output above. Does anyone have any ideas how best to resolve these issues? Cheers, Richard On 24 Mar 2016, at 15:28, Nigel Moriarty mailto:[email protected]> wrote: Markus It seems a little odd that Boost is the problem because it has not changed for a very long time. In my experience, this usually means that the code using Boost has a memory leak. Cheers Nigel --- Nigel W. Moriarty Building 33R0349, Physical Biosciences Division Lawrence Berkeley National Laboratory Berkeley, CA 94720-8235 Phone : 510-486-5709tel:510-486-5709 Email : [email protected]mailto:[email protected] Fax : 510-486-5909tel:510-486-5909 Web : CCI.LBL.govhttp://cci.lbl.gov/ On Thu, Mar 24, 2016 at 8:05 AM, mailto:[email protected]> wrote: Dials west is currently observing test failures on Mac related to Boost. To exclude any errors on our side I have run things twice clean from bootstrap: One of a number of affected tests: libtbx.python cctbx/xray/boost_python/tst_xray.py show_stack(1): /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/cctbx/xray/boost_python/tst_xray.py(381) exercise_conversions show_stack(2): /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/cctbx/xray/boost_python/tst_xray.py(1993) run show_stack(3): /Users/dlshudson/jenkins_slave/workspace/dials_package_macosx/build_dials/modules/cctbx_project/cctbx/xray/boost_python/tst_xray.py(2009) <module> libc backtrace (29 frames, most recent call last): 30 Python 0x0000000100000d71 Python + 3441 29 Python 0x0000000100000e58 Python + 3672 28 Python 0x00000001000ef06f Py_Main + 3135 27 Python 0x00000001000d78be PyRun_SimpleFileExFlags + 702 26 Python 0x00000001000d7d84 PyRun_FileExFlags + 164 25 Python 0x00000001000aea06 PyEval_EvalCode + 54 24 Python 0x00000001000af244 PyEval_EvalCodeEx + 2100 23 Python 0x00000001000b25c8 PyEval_EvalFrameEx + 12280 22 Python 0x00000001000b6ab6 fast_function + 118 21 Python 0x00000001000af244 PyEval_EvalCodeEx + 2100 20 Python 0x00000001000b25c8 PyEval_EvalFrameEx + 12280 19 Python 0x00000001000b6ab6 fast_function + 118 18 Python 0x00000001000af244 PyEval_EvalCodeEx + 2100 17 Python 0x00000001000b4198 PyEval_EvalFrameEx + 19400 16 Python 0x0000000100011023 PyObject_Call + 99 15 Python 0x000000010006ad8b type_call + 347 14 Python 0x00000001000708af slot_tp_init + 175 13 Python 0x0000000100011023 PyObject_Call + 99 12 Python 0x000000010001f0d8 instancemethod_call + 232 11 Python 0x0000000100011023 PyObject_Call + 99 10 libboost_python.dylib 0x0000000101888663 boost::python::objects::function_call(_object*, _object*, _object*) + 83 9 libboost_python.dylib 0x000000010188ce71 boost::python::handle_exception_impl(boost::function0<void>) + 81 8 libboost_python.dylib 0x00000001018889ca boost::detail::function::void_function_ref_invoker0::invoke(boost::detail::function::function_buffer&) + 26 7 libboost_python.dylib 0x000000010188693a boost::python::objects::function::call(_object*, _object*) const + 874 6 cctbx_xray_ext.so 0x0000000107c8e1c5 boost::python::detail::caller_arity<3u>::impl const&, scitbx::af::const_ref const&), boost::python::default_call_policies, boost::mpl::vector4 const&, scitbx::af::const_ref const&> >::operator()(_object*, _object*) + 229 5 cctbx_xray_ext.so 0x0000000107c91f44 boost::python::objects::make_holder<2>::apply >, boost::mpl::joint_view const&, scitbx::af::const_ref const&, boost::python::optional, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_, mpl_::void_> >, boost::python::optional > >::execute(_object*, scitbx::af::const_ref const&, scitbx::af::const_ref const&) + 100 4 ??? 0x0000000000000002 0x0 + 2 3 libsystem_platform.dylib 0x00007fff8caabeaa _sigtramp + 26 2 boost_python_meta_ext.so 0x0000000101841370 initboost_python_meta_ext + 0 Floating-point error (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. Any ideas? Best wishes, -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

On 6 Apr 2016, at 17:20, [email protected] wrote:

I think this is a problem caused by the latest Xcode 7.3 update, released 2016-03-21 (the date the update was applied according to the App Store update history coincides with the time our mac builds became broken):

I have reproduced the issue with the standard cctbx build using the bootstrap script as follows:

Xcode 7.3 has no issue with Boost 1.56 for the record, which is older than the revision 86799 used by the bootstrap script if I am not mistaken. Best wishes, Luc

I don't think this has anything to do with boost (as Markus initially thought). This smells like over-aggressive compiler optimization after the xcode 7.3 update, I am just testing recompilation with -O2 instead of -O3 optimization.

It is indeed likely a bug in clang/llvm, which is be triggered by a particular stack of function calls. But at least, we know the said bug is not triggered when using Boost 1.56. If I had a bit of courage, I would try the trunk of clang/llvm, and then perhaps bisect to find where the problem came from.

Do you happen to know what clang/llvm version was used with the previous version of XCode (perhaps someone who has yet to update to XCode could tell us)?

Yes, thanks to this nice guy: https://gist.github.com/yamaya/2924292 https://gist.github.com/yamaya/2924292 !!

Do you happen to know what clang/llvm version was used with the previous version of XCode

probably clang-700.1.81 (that's still the latest version on my osx 10.10). But I don't think it will help you, Apple compiler is based on open-source Clang but it's not the same, and versioning is quite different (current Clang is 3.8). A couple years ago Apple "clang --version" had a note on which Clang version it is based, but then Apple stopped including it as their compiler doesn't correspond to any particular version of the open-source compiler.

On 6 Apr 2016, at 18:10, Luc Bourhis wrote:

...

I don't think this has anything to do with boost (as Markus initially thought). This smells like over-aggressive compiler optimization after the xcode 7.3 update, I am just testing recompilation with -O2 instead of -O3 optimization.

It is indeed likely a bug in clang/llvm, which is be triggered by a particular stack of function calls. But at least, we know the said bug is not triggered when using Boost 1.56. If I had a bit of courage, I would try the trunk of clang/llvm, and then perhaps bisect to find where the problem came from.

Segfault with Boost 1.60, just to confirm that this is indeed a bug in clang/llvm.

On Wed, Apr 6, 2016 at 10:05 PM, Luc Bourhis wrote:

removal of -ffast-math even more important on OS X. Now it is to be seen how much speed we actually loose…

Are Phenix official binaries built with Clang on Mac? If there is no other workaround you could remove -ffast-math partially, it's a combination of a few more specific math-related options. Actually adding this option was quite an important decision, given that it violates language standards, disallows NaNs, etc. And all this is propagated to other projects that are built with libtbx/scons.

Hi Luc, I don't think your fix will help with the other two tests that failed (note failed, not crashed) with the -ffast-math flag. In some ways these are more concerning than the floating point crash as I don't even know where to begin to track down the problematic lines of code. Could you confirm whether the following two tests fail for you after your latest change? libtbx.python "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/maptbx/boost_python/tst_maptbx.py" [FAIL] Time: 4.96 Return code: 1 OKs: 0 Standard error: Traceback (most recent call last): File "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/maptbx/boost_python/tst_maptbx.py", line 1395, in <module> run(args=sys.argv[1:]) File "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/maptbx/boost_python/tst_maptbx.py", line 1373, in run exercise_misc() File "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/maptbx/boost_python/tst_maptbx.py", line 365, in exercise_misc assert (approx_equal(stats.min, 8.3, eps=0.1)) AssertionError libtbx.python "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/scitbx/lbfgs/tst_lbfgs_fem.py" [FAIL] Time: 14.01 Return code: 1 OKs: 0 Standard error: Traceback (most recent call last): File "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/scitbx/lbfgs/tst_lbfgs_fem.py", line 159, in <module> run(sys.argv[1:]) File "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/scitbx/lbfgs/tst_lbfgs_fem.py", line 154, in run compare_implementations() File "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/scitbx/lbfgs/tst_lbfgs_fem.py", line 130, in compare_implementations this_script=this_script, n=n, m=m, iprint=[iprint1, iprint2]) File "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/scitbx/lbfgs/tst_lbfgs_fem.py", line 117, in run_and_compare_implementations assert not show_diff(a, b) AssertionError Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE ________________________________ From: [email protected] [[email protected]] on behalf of Luc Bourhis [[email protected]] Sent: 07 April 2016 02:31 To: cctbx mailing list Subject: Re: [cctbxbb] Boost broken on mac? Morning Marcin, On 7 Apr 2016, at 02:15, Marcin Wojdyr mailto:[email protected]> wrote: If there is no other workaround you could remove -ffast-math partially, it's a combination of a few more specific math-related options. Correct for gcc whose documentation clearly lists all the flags but then moving to clang is a guess work… Anyway, I tried another approach: identifying the exact point of failure and coding a workaround: commit 24207. Actually adding this option was quite an important decision, given that it violates language standards, disallows NaNs, etc. And all this is propagated to other projects that are built with libtbx/scons. By default, the cctbx will crash on you if NaN’s are produced, not because of -ffast-math but because of the way we set the FP flags. Do you unset them in the CCP4? Best wishes, Luc -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

Strangely for my development build, the build system appears convinced that it is using clang 6.1.0... Editing around line 784 of libtbx/SConscript, where clang_version is determined, to read: from subprocess import call print call(["which", "clang"]) print call(["clang", "--version"]) flag, output = conf.TryRun("\n".join(test_code), extension='.cpp') conf.Finish() assert flag output = output.replace(":,", ":None,") print output I get the following output: /usr/bin/clang 0 Apple LLVM version 7.3.0 (clang-703.0.29) Target: x86_64-apple-darwin15.3.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin 0 {"llvm":1, "clang":1, "clang_major":6, "clang_minor":1, "clang_patchlevel":0, "GNUC":4, "GNUC_MINOR":2, "GNUC_PATCHLEVEL":1, "clang_version": "6.1.0 (clang-602.0.53)", "VERSION": "4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)", } On MacOS, using clang 6.1.0 If I wipe the build directory (or rather, renamed so not wiped completely) then it is now happy that it is using clang 7.3.0: /usr/bin/clang 0 Apple LLVM version 7.3.0 (clang-703.0.29) Target: x86_64-apple-darwin15.3.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin 0 {"llvm":1, "clang":1, "clang_major":7, "clang_minor":3, "clang_patchlevel":0, "GNUC":4, "GNUC_MINOR":2, "GNUC_PATCHLEVEL":1, "clang_version": "7.3.0 (clang-703.0.29)", "VERSION": "4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.29)", } On MacOS, using clang 7.3.0 Any idea what is going on here? Why does it think it is somehow using an older version of the clang to what is actually installed?! Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE ________________________________ From: [email protected] [[email protected]] on behalf of Luc Bourhis [[email protected]] Sent: 07 April 2016 10:23 To: cctbx mailing list Subject: Re: [cctbxbb] Boost broken on mac? Could you confirm whether the following two tests fail for you after your latest change? libtbx.python "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/maptbx/boost_python/tst_maptbx.py" [FAIL] […] libtbx.python "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/scitbx/lbfgs/tst_lbfgs_fem.py" [FAIL] […] I see those too. Tracking them is too much effort indeed. I followed the route already suggested by Marcin: break -ffast-math in finer options and find the one breaking our code. After a round-trip through Clang source code (I wish they had a proper documentation of *all* their command-line options as it exists for gcc), I came with a fix which keeps all useful optimisations enabled while the 3 failing tests do now pass. Gonna commit shortly. Best wishes, Luc -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

Hi Luc, All the cctbx tests now appear to pass, however I am getting a couple of floating point exceptions running the dials tests: $ libtbx.python $(libtbx.find_in_repositories dials/test/algorithms/refinement/tst_multi_experiment_refinement.py) show_stack(1): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/outlier_detection/mcd.py(54) __init__ show_stack(2): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/outlier_detection/outlier_base.py(296) from_parameters_and_colnames show_stack(3): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/refiner.py(1194) config_refman show_stack(4): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/refiner.py(461) _build_components show_stack(5): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/refiner.py(400) from_parameters_data_experiments show_stack(6): /Users/rjgildea/software/cctbx/modules/dials/test/algorithms/refinement/tst_multi_experiment_refinement.py(269) <module> libc backtrace (41 frames, most recent call last): 42 ??? 0x0000000000000003 0x0 + 3 41 Python 0x0000000100000d71 Python + 3441 40 Python 0x0000000100000e58 Python + 3672 39 Python 0x00000001000eb6aa Py_Main + 3178 38 Python 0x00000001000d424a PyRun_SimpleFileExFlags + 698 37 Python 0x00000001000d4714 PyRun_FileExFlags + 164 36 Python 0x00000001000aba26 PyEval_EvalCode + 54 35 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690 34 Python 0x00000001000af860 PyEval_EvalFrameEx + 13472 33 Python 0x00000001000b3486 fast_function + 118 32 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690 31 Python 0x00000001000af860 PyEval_EvalFrameEx + 13472 30 Python 0x00000001000b3486 fast_function + 118 29 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690 28 Python 0x00000001000af860 PyEval_EvalFrameEx + 13472 27 Python 0x00000001000b3486 fast_function + 118 26 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690 25 Python 0x00000001000af860 PyEval_EvalFrameEx + 13472 24 Python 0x00000001000b3486 fast_function + 118 23 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690 22 Python 0x00000001000afab2 PyEval_EvalFrameEx + 14066 21 Python 0x0000000100011223 PyObject_Call + 99 20 Python 0x00000001000684ab type_call + 347 19 Python 0x000000010006dfef slot_tp_init + 175 18 Python 0x0000000100011223 PyObject_Call + 99 17 Python 0x000000010001e138 instancemethod_call + 232 16 Python 0x0000000100011223 PyObject_Call + 99 15 Python 0x0000000100036b4c function_call + 364 14 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690 13 Python 0x00000001000af11d PyEval_EvalFrameEx + 11613 12 Python 0x0000000100011223 PyObject_Call + 99 11 libboost_python.dylib 0x0000000102492663 boost::python::objects::function_call(_object*, _object*, _object*) + 83 10 libboost_python.dylib 0x0000000102496e71 boost::python::handle_exception_impl(boost::function0<void>) + 81 9 libboost_python.dylib 0x00000001024929ca boost::detail::function::void_function_ref_invoker0::invoke(boost::detail::function::function_buffer&) + 26 8 libboost_python.dylib 0x000000010249093a boost::python::objects::function::call(_object*, _object*) const + 874 7 dials_refinement_helpers_ext.so 0x000000010649a093 boost::python::detail::caller_arity<2u>::impl >::operator()(_object*, _object*) + 179 6 dials_refinement_helpers_ext.so 0x000000010648312d dials::refinement::qchisq(double, double) + 189 5 dials_refinement_helpers_ext.so 0x000000010648dbe6 double boost::math::detail::gamma_p_inv_imp >(double, double, boost::math::policies::policy const&) + 374 4 Python 0x000000010004c160 PyDict_DelItemString + 64 3 libsystem_platform.dylib 0x00007fff89b59eaa _sigtramp + 26 2 boost_python_meta_ext.so 0x000000010244b3f0 initboost_python_meta_ext + 0 Floating-point error (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. Richards-MacBook-Pro:build rjgildea$ Richards-MacBook-Pro:build rjgildea$ libtbx.python $(libtbx.find_in_repositories dials/test/algorithms/refinement/tst_multi_experiment_refinement.py) show_stack(1): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/outlier_detection/mcd.py(54) __init__ show_stack(2): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/outlier_detection/outlier_base.py(296) from_parameters_and_colnames show_stack(3): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/refiner.py(1194) config_refman show_stack(4): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/refiner.py(461) _build_components show_stack(5): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/refiner.py(400) from_parameters_data_experiments show_stack(6): /Users/rjgildea/software/cctbx/modules/dials/test/algorithms/refinement/tst_multi_experiment_refinement.py(269) <module> libc backtrace (41 frames, most recent call last): 42 ??? 0x0000000000000003 0x0 + 3 41 Python 0x0000000100000d71 Python + 3441 40 Python 0x0000000100000e58 Python + 3672 39 Python 0x00000001000eb6aa Py_Main + 3178 38 Python 0x00000001000d424a PyRun_SimpleFileExFlags + 698 37 Python 0x00000001000d4714 PyRun_FileExFlags + 164 36 Python 0x00000001000aba26 PyEval_EvalCode + 54 35 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690 34 Python 0x00000001000af860 PyEval_EvalFrameEx + 13472 33 Python 0x00000001000b3486 fast_function + 118 32 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690 31 Python 0x00000001000af860 PyEval_EvalFrameEx + 13472 30 Python 0x00000001000b3486 fast_function + 118 29 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690 28 Python 0x00000001000af860 PyEval_EvalFrameEx + 13472 27 Python 0x00000001000b3486 fast_function + 118 26 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690 25 Python 0x00000001000af860 PyEval_EvalFrameEx + 13472 24 Python 0x00000001000b3486 fast_function + 118 23 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690 22 Python 0x00000001000afab2 PyEval_EvalFrameEx + 14066 21 Python 0x0000000100011223 PyObject_Call + 99 20 Python 0x00000001000684ab type_call + 347 19 Python 0x000000010006dfef slot_tp_init + 175 18 Python 0x0000000100011223 PyObject_Call + 99 17 Python 0x000000010001e138 instancemethod_call + 232 16 Python 0x0000000100011223 PyObject_Call + 99 15 Python 0x0000000100036b4c function_call + 364 14 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690 13 Python 0x00000001000af11d PyEval_EvalFrameEx + 11613 12 Python 0x0000000100011223 PyObject_Call + 99 11 libboost_python.dylib 0x0000000101e0f663 boost::python::objects::function_call(_object*, _object*, _object*) + 83 10 libboost_python.dylib 0x0000000101e13e71 boost::python::handle_exception_impl(boost::function0<void>) + 81 9 libboost_python.dylib 0x0000000101e0f9ca boost::detail::function::void_function_ref_invoker0::invoke(boost::detail::function::function_buffer&) + 26 8 libboost_python.dylib 0x0000000101e0d93a boost::python::objects::function::call(_object*, _object*) const + 874 7 dials_refinement_helpers_ext.so 0x00000001087a8093 boost::python::detail::caller_arity<2u>::impl >::operator()(_object*, _object*) + 179 6 dials_refinement_helpers_ext.so 0x000000010879112d dials::refinement::qchisq(double, double) + 189 5 dials_refinement_helpers_ext.so 0x000000010879bbe6 double boost::math::detail::gamma_p_inv_imp >(double, double, boost::math::policies::policy const&) + 374 4 Python 0x000000010004c160 PyDict_DelItemString + 64 3 libsystem_platform.dylib 0x00007fff89b59eaa _sigtramp + 26 2 boost_python_meta_ext.so 0x0000000101c803f0 initboost_python_meta_ext + 0 Floating-point error (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. Richards-MacBook-Pro:build rjgildea$ libtbx.python $(libtbx.find_in_repositories dials/test/algorithms/statistics/tst_fast_mcd.py) OK show_stack(1): /Users/rjgildea/software/cctbx/modules/dials/algorithms/statistics/fast_mcd.py(165) __init__ show_stack(2): /Users/rjgildea/software/cctbx/modules/dials/test/algorithms/statistics/tst_fast_mcd.py(146) test_fast_mcd_small show_stack(3): /Users/rjgildea/software/cctbx/modules/dials/test/algorithms/statistics/tst_fast_mcd.py(237) <module> libc backtrace (33 frames, most recent call last): 34 ??? 0x0000000000000003 0x0 + 3 33 Python 0x0000000100000d71 Python + 3441 32 Python 0x0000000100000e58 Python + 3672 31 Python 0x00000001000eb6aa Py_Main + 3178 30 Python 0x00000001000d424a PyRun_SimpleFileExFlags + 698 29 Python 0x00000001000d4714 PyRun_FileExFlags + 164 28 Python 0x00000001000aba26 PyEval_EvalCode + 54 27 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690 26 Python 0x00000001000af860 PyEval_EvalFrameEx + 13472 25 Python 0x00000001000b3486 fast_function + 118 24 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690 23 Python 0x00000001000af11d PyEval_EvalFrameEx + 11613 22 Python 0x0000000100011223 PyObject_Call + 99 21 Python 0x00000001000684ab type_call + 347 20 Python 0x000000010006dfef slot_tp_init + 175 19 Python 0x0000000100011223 PyObject_Call + 99 18 Python 0x000000010001e138 instancemethod_call + 232 17 Python 0x0000000100011223 PyObject_Call + 99 16 Python 0x0000000100036b4c function_call + 364 15 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690 14 Python 0x00000001000af11d PyEval_EvalFrameEx + 11613 13 Python 0x0000000100011223 PyObject_Call + 99 12 libboost_python.dylib 0x000000010270f663 boost::python::objects::function_call(_object*, _object*, _object*) + 83 11 libboost_python.dylib 0x0000000102713e71 boost::python::handle_exception_impl(boost::function0<void>) + 81 10 libboost_python.dylib 0x000000010270f9ca boost::detail::function::void_function_ref_invoker0::invoke(boost::detail::function::function_buffer&) + 26 9 libboost_python.dylib 0x000000010270d93a boost::python::objects::function::call(_object*, _object*) const + 874 8 dials_refinement_helpers_ext.so 0x00000001080d4093 boost::python::detail::caller_arity<2u>::impl >::operator()(_object*, _object*) + 179 7 dials_refinement_helpers_ext.so 0x00000001080bd689 dials::refinement::mcd_consistency(double, double) + 89 6 dials_refinement_helpers_ext.so 0x00000001080bd12d dials::refinement::qchisq(double, double) + 189 5 dials_refinement_helpers_ext.so 0x00000001080c7be6 double boost::math::detail::gamma_p_inv_imp >(double, double, boost::math::policies::policy const&) + 374 4 ??? 0x00000001007ba2b0 0x0 + 4303069872 3 libsystem_platform.dylib 0x00007fff89b59eaa _sigtramp + 26 2 boost_python_meta_ext.so 0x00000001025813f0 initboost_python_meta_ext + 0 Floating-point error (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. Tracking this down further, I think the problem comes from the function qchisq in: https://github.com/dials/dials/blob/master/algorithms/refinement/outlier_det... Specifically, the line that is causing the problem is line 22, the call to quantile in the return statement: // chi^2 distribution with df d.o.f: boost::math::chi_squared_distribution<double> chisq(df); // Find x s.t. F(x; df) = p where F is the CDF of the Chi^2 dist with df d.o.f return quantile(chisq, p); Adding debug print statements shows it fails with the following values: df: 3 p: 0.52 I will try updating to boost 1.60 to see if this fixes things. Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE ________________________________________ From: [email protected] [[email protected]] on behalf of Luc Bourhis [[email protected]] Sent: 07 April 2016 11:05 To: cctbx mailing list Subject: Re: [cctbxbb] Boost broken on mac? SCons caches the result of TryRun in .sconf_temp and it gets stale sometimes. In your case, SCons did not track the fact that clang had changed I guess. I did commit my fix to libtbx/SConscript. Note that it does not fix the error in xray/conversions.h, weirdly enough. So I left my hack in there. So now, I’ve got some work cut for myself for the next rainy day: come with a reduce test case that I can submit to the clang bugzilla. The problem in xray/conversions.h is clearly the most promising to start with. But still, a lot of work.

On 7 Apr 2016, at 11:43, [email protected] wrote:

Strangely for my development build, the build system appears convinced that it is using clang 6.1.0...

Editing around line 784 of libtbx/SConscript, where clang_version is determined, to read:

from subprocess import call print call(["which", "clang"]) print call(["clang", "--version"]) flag, output = conf.TryRun("\n".join(test_code), extension='.cpp') conf.Finish() assert flag output = output.replace(":,", ":None,") print output

I get the following output:

/usr/bin/clang

0

Apple LLVM version 7.3.0 (clang-703.0.29)

Target: x86_64-apple-darwin15.3.0

Thread model: posix

InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

0

{"llvm":1, "clang":1, "clang_major":6, "clang_minor":1, "clang_patchlevel":0, "GNUC":4, "GNUC_MINOR":2, "GNUC_PATCHLEVEL":1, "clang_version": "6.1.0 (clang-602.0.53)", "VERSION": "4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)", }

On MacOS, using clang 6.1.0

If I wipe the build directory (or rather, renamed so not wiped completely) then it is now happy that it is using clang 7.3.0:

/usr/bin/clang

0

Apple LLVM version 7.3.0 (clang-703.0.29)

Target: x86_64-apple-darwin15.3.0

Thread model: posix

InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

0

{"llvm":1, "clang":1, "clang_major":7, "clang_minor":3, "clang_patchlevel":0, "GNUC":4, "GNUC_MINOR":2, "GNUC_PATCHLEVEL":1, "clang_version": "7.3.0 (clang-703.0.29)", "VERSION": "4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.29)", }

On MacOS, using clang 7.3.0

Any idea what is going on here? Why does it think it is somehow using an older version of the clang to what is actually installed?!

Cheers,

Richard

Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078

Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE ________________________________ From: [email protected] [[email protected]] on behalf of Luc Bourhis [[email protected]] Sent: 07 April 2016 10:23 To: cctbx mailing list Subject: Re: [cctbxbb] Boost broken on mac?

Could you confirm whether the following two tests fail for you after your latest change?

libtbx.python "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/maptbx/boost_python/tst_maptbx.py" [FAIL] […]

libtbx.python "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/scitbx/lbfgs/tst_lbfgs_fem.py" [FAIL] […]

I see those too. Tracking them is too much effort indeed. I followed the route already suggested by Marcin: break -ffast-math in finer options and find the one breaking our code. After a round-trip through Clang source code (I wish they had a proper documentation of *all* their command-line options as it exists for gcc), I came with a fix which keeps all useful optimisations enabled while the 3 failing tests do now pass.

Gonna commit shortly.

Best wishes,

Luc

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi Luc, Making chisq volatile as you suggest changes the floating point exception to a RuntimeError: $ libtbx.python "/Users/rjgildea/software/cctbx/modules/dials/test/algorithms/statistics/tst_fast_mcd.py" OK Traceback (most recent call last): File "/Users/rjgildea/software/cctbx/modules/dials/test/algorithms/statistics/tst_fast_mcd.py", line 237, in <module> test_fast_mcd_small() File "/Users/rjgildea/software/cctbx/modules/dials/test/algorithms/statistics/tst_fast_mcd.py", line 146, in test_fast_mcd_small fast_mcd = FastMCD([x1, x2, x3]) File "/Users/rjgildea/software/cctbx/modules/dials/algorithms/statistics/fast_mcd.py", line 165, in __init__ self._consistency_fac = mcd_consistency(self._p, self._h / self._n) RuntimeError: Error in function boost::math::gamma_p<e>(e, e): Argument x to the incomplete gamma function must be >= 0 (got x=-7.63324507713441681615e-4653). The line of dials code that is now failing is (in the same file): double calpha_inv = cdf(gamma, qalpha/2) / alpha; https://github.com/dials/dials/blob/master/algorithms/refinement/outlier_det... After a process of elimination, removing the compiler flag -fno-honor-nans means that the test now passes without error. Do you have any alternative suggestions to try other than removing said flag? Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE ________________________________ From: [email protected] [[email protected]] on behalf of Luc Bourhis [[email protected]] Sent: 07 April 2016 11:55 To: cctbx mailing list Subject: Re: [cctbxbb] Boost broken on mac? On 7 Apr 2016, at 12:46, [email protected]mailto:[email protected] wrote: Specifically, the line that is causing the problem is line 22, the call to quantile in the return statement: // chi^2 distribution with df d.o.f: boost::math::chi_squared_distribution<double> chisq(df); // Find x s.t. F(x; df) = p where F is the CDF of the Chi^2 dist with df d.o.f return quantile(chisq, p); I don’t have dials set up, unfortunately. Suggestion: make chisq volative… -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

It turns out we need both the 'volatile' trick on line 19 and removing the -fno-honor-nans compiler flag in order to work around the problem at line 40. Neither workaround is sufficient on their own.

I had started to test removing/adding flags one by one too for cctbx/xray/convert.h. Just had a look at the outcomes: in that case, removing -fno-honor-nans is enough; the volatile hack can be removed. It makes no sense to me but that’s usually the case with optimiser bugs. Well, anyway, we are walking on a tight rope with Xcode 7.3, and perhaps even onward since clang trunk brings the same havoc.

There does seem to be something really quite wrong with the latest clang :-(

I would love to give you a hand with dials but the bootstrap.py fails to compile Python. Specifically it fails to compile the Mac toolbox glue: /System/Library/Frameworks/CoreGraphics.framework/Headers/CGFont.h:53:40: error: initializer element is not constant static const CGFontIndex kCGGlyphMax = kCGFontIndexMax; How did you manage?

On Thu, Apr 7, 2016 at 7:30 PM, Luc Bourhis wrote:

Is there a release of Phenix or CCP4 looming on the horizon? That would be a good reason to go the extra mile and only disable optimisation where necessary.

I don't know about Phenix, but CCP4 official Mac binaries are compiled with icc, so it wouldn't matter. Cheers Marcin

On 12 Apr 2016, at 11:54, [email protected] wrote:

./bin/libtbx.configure . --clear-scons-memory

I can’t find where it is defined…

Line 1500 of libtbx/env_config.py: https://sourceforge.net/p/cctbx/code/HEAD/tree/trunk/libtbx/env_config.py#l1... def clear_scons_memory(self): (self.build_path / ".sconsign.dblite").remove() (self.build_path / ".sconf_temp").remove_tree() Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE ________________________________ From: [email protected] [[email protected]] on behalf of Luc Bourhis [[email protected]] Sent: 12 April 2016 13:16 To: cctbx mailing list Subject: Re: [cctbxbb] Boost broken on mac? On 12 Apr 2016, at 14:13, Luc Bourhis mailto:[email protected]> wrote: On 12 Apr 2016, at 11:54, [email protected]mailto:[email protected] wrote: ./bin/libtbx.configure . --clear-scons-memory I can’t find where it is defined… No, ok, stupid, I found it. Yes, that will remove all traces of the configuration tests performed by SCons. So good idea… -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom