Search results for query "look through"
- 527 messages
Re: [phenixbb] Problematic dataset
by Peter Zwart
Hi Lucas,
Your P21 just looks twinned. Have a look at the last section of the
xtriage output:
-------------------------------------------------------------------------------
Twinning and intensity statistics summary (acentric data):
Statistics independent of twin laws
- <I^2>/<I>^2 : 1.915
- <F>^2/<F^2> : 0.821
- <|E^2-1|> : 0.677
- <|L|>, <L^2>: 0.442, 0.267
Multivariate Z score L-test: 3.623
The multivariate Z score is a quality measure of the given
spread in intensities. Good to reasonable data is expected
to have a Z score lower than 3.5.
Large values can indicate twinning, but small values do not
neccesarily exclude it.
Statistics depending on twin laws
------------------------------------------------------------------
| Operator | type | R obs. | Britton alpha | H alpha | ML alpha |
------------------------------------------------------------------
| h,-k,-h-l | PM | 0.090 | 0.406 | 0.417 | 0.370 |
------------------------------------------------------------------
Patterson analyses
- Largest peak height : 6.379
(correpsonding p value : 6.273e-01)
The largest off-origin peak in the Patterson function is 6.38% of the
height of the origin peak. No significant pseudotranslation is detected.
The results of the L-test indicate that the intensity statistics
are significantly different then is expected from good to reasonable,
untwinned data.
As there are twin laws possible given the crystal symmetry, twinning could
be the reason for the departure of the intensity statistics from normality.
It might be worthwhile carrying out refinement with a twin specific
target function.
-------------------------------------------------------------------------------
You might have NCS parallel to the twin axis. If you run
mmtbx.xtwin_map_utils on your P21 data, you get a mtzfile called
obs_and_calc.mtz
push that through xtriage (make sure to specify labels for observed and
calculated data) to get the RvsR statistic. When you do that, please
also provide the keyword
'perform=True' on the xtriage command line for my pleasure and send me
the resulting log file.
I guess the twin refinement will push the r values down a bit.
Note that for futurte reference, xtriage produces a logfile as well,
usually named logfile.log unless you ask it to be a different name.
logfile.log contains some ccp4 style graphs you can view with loggraph
(such as the L, NZ, Britton and H curves).
If the above is too cryptic and you need more info, please let me know.
HTH
Peter
Lucas Bleicher wrote:
> Hi, Peter!
>
> I am sending the xtriage output for both space groups.
>
> So, the correct approach would be to use the MTZ
> processed in P21 with the twin law from the xtriage
> output from this MTZ file?
>
> Thanks in advance,
> Lucas
>
>
> Alertas do Yahoo! Mail em seu celular. Saiba mais em http://br.mobile.yahoo.com/mailalertas/
> ------------------------------------------------------------------------
>
> #############################################################
> ## mmtbx.xtriage ##
> ## ##
> ## P.H. Zwart, R.W. Grosse-Kunstleve & P.D. Adams ##
> ## ##
> #############################################################
> Date 2007-07-31 Time 16:43:49 BRT -0300
>
>
> ##-------------------------------------------##
> ## WARNING: ##
> ## Number of residues unspecified ##
> ##-------------------------------------------##
>
>
> Effective parameters:
> scaling.input {
> parameters {
> asu_contents {
> n_residues = None
> n_bases = None
> n_copies_per_asu = None
> }
> misc_twin_parameters {
> missing_symmetry {
> tanh_location = 0.08
> tanh_slope = 50
> }
> twinning_with_ncs {
> perform_analyses = False
> n_bins = 7
> }
> twin_test_cuts {
> low_resolution = 10
> high_resolution = None
> isigi_cut = 3
> completeness_cut = 0.85
> }
> }
> reporting {
> verbose = 1
> log = "logfile.log"
> ccp4_style_graphs = True
> }
> }
> xray_data {
> file_name = "LamNatP1High_scala1.mtz"
> obs_labels = "MEAN_lnls,SIGIMEAN_lnls"
> calc_labels = None
> unit_cell = 52.16379929 64.59190369 108.2256012 89.99259949 89.99089813 \
> 66.10500336
> space_group = "P 1"
> high_resolution = None
> low_resolution = None
> }
> }
> <scitbx_array_family_flex_ext.double object at 0xb5e96dac>
>
>
> Symmetry, cell and reflection file content summary
>
> Miller array info: LamNatP1High_scala1.mtz:IMEAN_lnls,SIGIMEAN_lnls
> Observation type: xray.amplitude
> Type of data: double, size=94082
> Type of sigmas: double, size=94082
> Number of Miller indices: 94082
> Anomalous flag: False
> Unit cell: (52.1638, 64.5919, 108.226, 89.9926, 89.9909, 66.105)
> Space group: P 1 (No. 1)
> Systematic absences: 0
> Centric reflections: 0
> Resolution range: 26.0819 1.85
> Completeness in resolution range: 0.854134
> Completeness with d_max=infinity: 0.853839
>
> ##----------------------------------------------------##
> ## Basic statistics ##
> ##----------------------------------------------------##
>
>
> Matthews coefficient and Solvent content statistics
>
> Number of residues unknown, assuming 50% solvent content
>
> ----------------------------------------------------------------
> | Best guess : 1220 residues in the asu |
> ----------------------------------------------------------------
>
>
> Completeness and data strength analyses
>
> The following table lists the completeness in various resolution
> ranges, after applying a I/sigI cut. Miller indices for which
> individual I/sigI values are larger than the value specified in
> the top row of the table, are retained, while other intensities
> are discarded. The resulting completeness profiles are an indication
> of the strength of the data.
>
> ----------------------------------------------------------------------------------------
> | Res. Range | I/sigI>1 | I/sigI>2 | I/sigI>3 | I/sigI>5 | I/sigI>10 | I/sigI>15 |
> ----------------------------------------------------------------------------------------
> | 26.08 - 4.55 | 97.0% | 96.3% | 95.5% | 93.5% | 87.3% | 80.3% |
> | 4.55 - 3.62 | 96.5% | 96.1% | 95.4% | 93.8% | 87.8% | 81.2% |
> | 3.62 - 3.16 | 95.7% | 94.5% | 93.2% | 89.4% | 78.1% | 66.3% |
> | 3.16 - 2.87 | 93.1% | 90.7% | 87.1% | 79.1% | 60.8% | 44.5% |
> | 2.87 - 2.67 | 88.4% | 84.0% | 78.5% | 66.7% | 44.0% | 27.9% |
> | 2.67 - 2.51 | 85.5% | 78.6% | 71.1% | 57.7% | 32.6% | 17.9% |
> | 2.51 - 2.38 | 82.4% | 74.1% | 65.3% | 49.3% | 24.8% | 12.2% |
> | 2.38 - 2.28 | 81.6% | 72.3% | 62.4% | 46.0% | 20.4% | 9.0% |
> | 2.28 - 2.19 | 78.5% | 67.6% | 55.1% | 37.2% | 14.8% | 5.7% |
> | 2.19 - 2.12 | 75.5% | 61.9% | 49.0% | 30.7% | 9.9% | 3.5% |
> | 2.12 - 2.05 | 72.1% | 55.6% | 41.8% | 23.7% | 6.3% | 1.9% |
> | 2.05 - 1.99 | 68.6% | 50.2% | 35.8% | 17.9% | 4.1% | 1.0% |
> | 1.99 - 1.94 | 63.2% | 42.9% | 28.5% | 13.7% | 3.1% | 0.5% |
> | 1.94 - 1.89 | 56.6% | 36.7% | 23.2% | 10.4% | 1.7% | 0.3% |
> ----------------------------------------------------------------------------------------
>
> The completeness of data for which I/sig(I)>3.00, exceeds 85% for
> for resolution ranges lower than 2.87A.
> The data is cut at this resolution for the potential twin tests
> and intensity statistics.
>
>
> Maximum likelihood isotropic Wilson scaling
> ML estimate of overall B value of LamNatP1High_scala1.mtz:IMEAN_lnls,SIGIMEAN_lnls:
> 15.59 A**(-2)
> Estimated -log of scale factor of LamNatP1High_scala1.mtz:IMEAN_lnls,SIGIMEAN_lnls:
> -2.45
>
>
> Maximum likelihood anisotropic Wilson scaling
> ML estimate of overall B_cart value of LamNatP1High_scala1.mtz:IMEAN_lnls,SIGIMEAN_lnls:
> 15.61, 1.39, -1.33
> 11.40, 0.27
> 21.10
> Equivalent representation as U_cif:
> 0.18, -0.04, -0.02
> 0.14, 0.00
> 0.27
>
> ML estimate of -log of scale factor of LamNatP1High_scala1.mtz:IMEAN_lnls,SIGIMEAN_lnls:
> -2.45
> Correcting for anisotropy in the data
>
> Some basic intensity statistics follow.
>
>
>
> Low resolution completeness analyses
>
> The following table shows the completeness
> of the data to 5 Angstrom.
> unused: - 26.0822 [ 0/38 ] 0.000
> bin 1: 26.0822 - 10.5555 [528/541] 0.976
> bin 2: 10.5555 - 8.4726 [551/564] 0.977
> bin 3: 8.4726 - 7.4299 [530/547] 0.969
> bin 4: 7.4299 - 6.7636 [568/582] 0.976
> bin 5: 6.7636 - 6.2860 [525/540] 0.972
> bin 6: 6.2860 - 5.9200 [547/559] 0.979
> bin 7: 5.9200 - 5.6266 [530/546] 0.971
> bin 8: 5.6266 - 5.3839 [519/532] 0.976
> bin 9: 5.3839 - 5.1783 [568/584] 0.973
> bin 10: 5.1783 - 5.0009 [516/532] 0.970
> unused: 5.0009 - [ 0/0 ]
>
>
>
> Mean intensity analyses
> Analyses of the mean intensity.
> Inspired by: Morris et al. (2004). J. Synch. Rad.11, 56-59.
> The following resolution shells are worrisome:
> ------------------------------------------------
> | d_spacing | z_score | compl. | <Iobs>/<Iexp> |
> ------------------------------------------------
> | 9.993 | 7.15 | 0.97 | 0.426 |
> | 8.448 | 7.29 | 0.98 | 0.559 |
> | 7.451 | 6.09 | 0.97 | 0.676 |
> | 4.303 | 4.64 | 0.97 | 1.223 |
> | 3.410 | 4.52 | 0.96 | 1.176 |
> | 3.333 | 6.63 | 0.96 | 1.262 |
> | 3.261 | 5.36 | 0.96 | 1.205 |
> | 3.194 | 5.48 | 0.96 | 1.203 |
> | 3.071 | 4.87 | 0.94 | 1.183 |
> | 2.962 | 6.43 | 0.94 | 1.245 |
> | 2.911 | 4.61 | 0.93 | 1.160 |
> | 2.454 | 4.96 | 0.87 | 0.872 |
> | 2.425 | 6.89 | 0.87 | 0.825 |
> | 2.397 | 4.68 | 0.87 | 0.873 |
> | 2.182 | 6.79 | 0.84 | 0.830 |
> | 2.142 | 5.54 | 0.84 | 0.856 |
> | 1.870 | 5.46 | 0.60 | 1.205 |
> | 1.857 | 11.65 | 0.54 | 1.570 |
> ------------------------------------------------
>
> Possible reasons for the presence of the reported
> unexpected low or elevated mean intensity in
> a given resolution bin are :
> - missing overloaded or weak reflections
> - suboptimal data processing
> - satelite (ice) crystals
> - NCS
> - translational pseudo symmetry (detected elsewhere)
> - outliers (detected elsewhere)
> - ice rings (detected elsewhere)
> - other problems
> Note that the presence of abnormalities
> in a certain region of reciprocal space might
> confuse the data validation algorithm throughout
> a large region of reciprocal space, even though
> the data is acceptable in those areas.
>
>
> Possible outliers
> Inspired by: Read, Acta Cryst. (1999). D55, 1759-1764
>
> Acentric reflections:
>
> -----------------------------------------------------------------
> | d_space | H K L | |E| | p(wilson) | p(extreme) |
> -----------------------------------------------------------------
> | 2.114 | -18, -9, 35 | 3.79 | 5.57e-07 | 5.08e-02 |
> | 3.066 | -16, -8, 12 | 3.93 | 1.90e-07 | 1.77e-02 |
> | 3.066 | 16, 8, 12 | 3.95 | 1.69e-07 | 1.57e-02 |
> | 2.115 | 18, 9, 35 | 3.79 | 5.65e-07 | 5.15e-02 |
> -----------------------------------------------------------------
>
> p(wilson) : 1-(1-exp[-|E|^2])
> p(extreme) : 1-(1-exp[-|E|^2])^(n_acentrics)
> p(wilson) is the probability that an E-value of the specified
> value would be observed when it would selected at random from
> the given data set.
> p(extreme) is the probability that the largest |E| value is
> larger or equal then the observed largest |E| value.
>
> Both measures can be used for outlier detection. p(extreme)
> takes into account the size of the dataset.
>
>
> Centric reflections:
>
> None
>
>
> Ice ring related problems
>
> The following statistics were obtained from ice-ring
> insensitive resolution ranges
> mean bin z_score : 3.46
> ( rms deviation : 1.89 )
> mean bin completeness : 0.89
> ( rms deviation : 0.09 )
>
> The following table shows the z-scores
> and completeness in ice-ring sensitive areas.
> Large z-scores and high completeness in these
> resolution ranges might be a reason to re-assess
> your data processsing if ice rings were present.
>
> ------------------------------------------------
> | d_spacing | z_score | compl. | Rel. Ice int. |
> ------------------------------------------------
> | 3.897 | 0.51 | 0.97 | 1.000 |
> | 3.669 | 1.87 | 0.97 | 0.750 |
> | 3.441 | 4.52 | 0.96 | 0.530 |
> | 2.671 | 3.84 | 0.89 | 0.170 |
> | 2.249 | 3.61 | 0.87 | 0.390 |
> | 2.072 | 3.23 | 0.82 | 0.300 |
> | 1.948 | 0.62 | 0.76 | 0.040 |
> | 1.918 | 0.44 | 0.73 | 0.180 |
> | 1.883 | 5.46 | 0.60 | 0.030 |
> ------------------------------------------------
>
> Abnormalities in mean intensity or completeness at
> resolution ranges with a relative ice ring intensity
> lower then 0.10 will be ignored.
>
> No ice ring related problems detected.
> If ice rings were present, the data does not look
> worse at ice ring related d_spacings as compared
> to the rest of the data set
>
>
>
>
> Basic analyses completed
>
> ##----------------------------------------------------##
> ## Twinning Analyses ##
> ##----------------------------------------------------##
>
>
>
> Using data between 10.00 to 2.87 Angstrom.
>
> Determining possible twin laws.
>
> The following twin laws have been found:
>
> ---------------------------------------------------------------------------------
> | Type | Axis | R metric (%) | delta (le Page) | delta (Lebedev) | Twin law |
> ---------------------------------------------------------------------------------
> | PM | 2-fold | 0.009 | 0.010 | 0.000 | -h,-k,l |
> | PM | 2-fold | 0.090 | 0.079 | 0.001 | -h,-h+k,-l |
> | PM | 2-fold | 0.099 | 0.080 | 0.001 | h,h-k,-l |
> ---------------------------------------------------------------------------------
> M: Merohedral twin law
> PM: Pseudomerohedral twin law
>
> 0 merohedral twin operators found
> 3 pseudo-merohedral twin operators found
> In total, 3 twin operator were found
>
>
> Number of centrics : 0
> Number of acentrics : 27599
>
> Largest patterson peak with length larger then 15 Angstrom
>
> Frac. coord. : 0.119 -0.276 -0.000
> Distance to origin : 16.337
> Height (origin=100) : 5.893
> p_value(height) : 7.365e-01
>
> The reported p_value has the following meaning:
> The probability that a peak of the specified height
> or larger is found in a Patterson function of a
> macro molecule that does not have any translational
> pseudo symmetry is equal to 7.365e-01
> p_values smaller then 0.05 might indicate
> weak translation pseudo symmetry, or the self vector of
> a large anomalous scatterer such as Hg, whereas values
> smaller then 1e-3 are a very strong indication for
> the presence of translational pseudo symmetry.
>
>
>
> Wilson ratio and moments
>
> Acentric reflections
> <I^2>/<I>^2 :1.932 (untwinned: 2.000; perfect twin 1.500)
> <F>^2/<F^2> :0.820 (untwinned: 0.785; perfect twin 0.885)
> <|E^2 - 1|> :0.678 (untwinned: 0.736; perfect twin 0.541)
>
>
>
> NZ test (0<=z<1) to detect twinning and possible translational NCS
>
>
> -----------------------------------------------
> | Z | Nac_obs | Nac_theo | Nc_obs | Nc_theo |
> -----------------------------------------------
> | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 |
> | 0.1 | 0.046 | 0.095 | 0.000 | 0.248 |
> | 0.2 | 0.127 | 0.181 | 0.000 | 0.345 |
> | 0.3 | 0.210 | 0.259 | 0.000 | 0.419 |
> | 0.4 | 0.287 | 0.330 | 0.000 | 0.474 |
> | 0.5 | 0.361 | 0.394 | 0.000 | 0.520 |
> | 0.6 | 0.427 | 0.451 | 0.000 | 0.561 |
> | 0.7 | 0.486 | 0.503 | 0.000 | 0.597 |
> | 0.8 | 0.544 | 0.551 | 0.000 | 0.629 |
> | 0.9 | 0.597 | 0.593 | 0.000 | 0.657 |
> | 1.0 | 0.643 | 0.632 | 0.000 | 0.683 |
> -----------------------------------------------
> | Maximum deviation acentric : 0.055 |
> | Maximum deviation centric : 0.683 |
> | |
> | <NZ(obs)-NZ(twinned)>_acentric : -0.024 |
> | <NZ(obs)-NZ(twinned)>_centric : -0.467 |
> -----------------------------------------------
>
>
> L test for acentric data
>
> using difference vectors (dh,dk,dl) of the form:
> (2hp,2kp,2lp)
> where hp, kp, and lp are random signed integers such that
> 2 <= |dh| + |dk| + |dl| <= 8
>
> Mean |L| :0.441 (untwinned: 0.500; perfect twin: 0.375)
> Mean L^2 :0.267 (untwinned: 0.333; perfect twin: 0.200)
>
> The distribution of |L| values indicates a twin fraction of
> 0.08. Note that this estimate is not as reliable as obtained
> via a Britton plot or H-test if twin laws are available.
>
>
>
> ---------------------------------------------
> Analysing possible twin law : -h,-k,l
> ---------------------------------------------
>
>
> Results of the H-test on a-centric data:
>
> (Only 50.0% of the strongest twin pairs were used)
>
> mean |H| : 0.053 (0.50: untwinned; 0.0: 50% twinned)
> mean H^2 : 0.005 (0.33: untwinned; 0.0: 50% twinned)
> Estimation of twin fraction via mean |H|: 0.447
> Estimation of twin fraction via cum. dist. of H: 0.449
>
>
>
> Britton analyses
>
> Extrapolation performed on 0.47 < alpha < 0.495
> Estimated twin fraction: 0.442
> Correlation: 0.9953
>
> R vs R statistic:
> R_abs_twin = <|I1-I2|>/<|I1+I2|>
> Lebedev, Vagin, Murshudov. Acta Cryst. (2006). D62, 83-95
>
> R_abs_twin observed data : 0.054
>
> R_sq_twin = <(I1-I2)^2>/<(I1+I2)^2>
> R_sq_twin observed data : 0.004
> No calculated data available.
> R_twin for calculated data not determined.
>
>
> Maximum Likelihood twin fraction determination
> Zwart, Read, Grosse-Kunstleve & Adams, to be published.
>
>
> The estimated twin fraction is equal to 0.415
>
>
> ---------------------------------------------
> Analysing possible twin law : -h,-h+k,-l
> ---------------------------------------------
>
>
> Results of the H-test on a-centric data:
>
> (Only 50.0% of the strongest twin pairs were used)
>
> mean |H| : 0.093 (0.50: untwinned; 0.0: 50% twinned)
> mean H^2 : 0.015 (0.33: untwinned; 0.0: 50% twinned)
> Estimation of twin fraction via mean |H|: 0.407
> Estimation of twin fraction via cum. dist. of H: 0.407
>
>
>
> Britton analyses
>
> Extrapolation performed on 0.44 < alpha < 0.495
> Estimated twin fraction: 0.399
> Correlation: 0.9959
>
> R vs R statistic:
> R_abs_twin = <|I1-I2|>/<|I1+I2|>
> Lebedev, Vagin, Murshudov. Acta Cryst. (2006). D62, 83-95
>
> R_abs_twin observed data : 0.093
>
> R_sq_twin = <(I1-I2)^2>/<(I1+I2)^2>
> R_sq_twin observed data : 0.011
> No calculated data available.
> R_twin for calculated data not determined.
>
>
> Maximum Likelihood twin fraction determination
> Zwart, Read, Grosse-Kunstleve & Adams, to be published.
>
>
> The estimated twin fraction is equal to 0.361
>
>
> ---------------------------------------------
> Analysing possible twin law : h,h-k,-l
> ---------------------------------------------
>
>
> Results of the H-test on a-centric data:
>
> (Only 50.0% of the strongest twin pairs were used)
>
> mean |H| : 0.085 (0.50: untwinned; 0.0: 50% twinned)
> mean H^2 : 0.013 (0.33: untwinned; 0.0: 50% twinned)
> Estimation of twin fraction via mean |H|: 0.415
> Estimation of twin fraction via cum. dist. of H: 0.417
>
>
>
> Britton analyses
>
> Extrapolation performed on 0.46 < alpha < 0.495
> Estimated twin fraction: 0.411
> Correlation: 0.9961
>
> R vs R statistic:
> R_abs_twin = <|I1-I2|>/<|I1+I2|>
> Lebedev, Vagin, Murshudov. Acta Cryst. (2006). D62, 83-95
>
> R_abs_twin observed data : 0.086
>
> R_sq_twin = <(I1-I2)^2>/<(I1+I2)^2>
> R_sq_twin observed data : 0.009
> No calculated data available.
> R_twin for calculated data not determined.
>
>
> Maximum Likelihood twin fraction determination
> Zwart, Read, Grosse-Kunstleve & Adams, to be published.
>
>
> The estimated twin fraction is equal to 0.350
>
>
>
> Exploring higher metric symmetry
>
> The point group of data as dictated by the space group is P 1
> the point group in the niggli setting is P 1
> The point group of the lattice is Hall: C 2 2 (x+y,2*y,z)
> A summary of R values for various possible point groups follow.
>
> -----------------------------------------------------------------------------------------------
> | Point group | mean R_used | max R_used | mean R_unused | min R_unused | choice |
> -----------------------------------------------------------------------------------------------
> | Hall: C 2y (x-y,2*x,z) | 0.086 | 0.086 | 0.054 | 0.054 | |
> | P 1 | None | None | 0.070 | 0.054 | |
> | Hall: C 2y (x+y,2*y,z) | 0.093 | 0.093 | 0.054 | 0.054 | |
> | Hall: C 2 2 (x+y,2*y,z) | 0.070 | 0.086 | None | None | |
> | P 1 1 2 | 0.054 | 0.054 | 0.093 | 0.093 | <--- |
> -----------------------------------------------------------------------------------------------
>
> R_used: mean and maximum R value for symmetry operators *used* in this point group
> R_unused: mean and minimum R value for symmetry operators *not used* in this point group
> The likely point group of the data is: P 1 1 2
>
> Possible space groups in this point groups are:
> Unit cell: (52.1638, 108.226, 64.5258, 90, 113.762, 90)
> Space group: P 1 2 1 (No. 3)
>
> Unit cell: (52.1638, 108.226, 64.5258, 90, 113.762, 90)
> Space group: P 1 21 1 (No. 4)
>
> Note that this analyses does not take into account the effects of twinning.
> If the data is (allmost) perfectly twinned, the symmetry will appear to be
> higher than it actually is.
>
>
>
>
>
> -------------------------------------------------------------------------------
> Twinning and intensity statistics summary (acentric data):
>
> Statistics independent of twin laws
> - <I^2>/<I>^2 : 1.932
> - <F>^2/<F^2> : 0.820
> - <|E^2-1|> : 0.678
> - <|L|>, <L^2>: 0.441, 0.267
> Multivariate Z score L-test: 3.807
> The multivariate Z score is a quality measure of the given
> spread in intensities. Good to reasonable data is expected
> to have a Z score lower than 3.5.
> Large values can indicate twinning, but small values do not
> neccesarily exclude it.
>
>
> Statistics depending on twin laws
> -------------------------------------------------------------------
> | Operator | type | R obs. | Britton alpha | H alpha | ML alpha |
> -------------------------------------------------------------------
> | -h,-k,l | PM | 0.054 | 0.442 | 0.449 | 0.415 |
> | -h,-h+k,-l | PM | 0.093 | 0.399 | 0.407 | 0.361 |
> | h,h-k,-l | PM | 0.086 | 0.411 | 0.417 | 0.350 |
> -------------------------------------------------------------------
>
> Patterson analyses
> - Largest peak height : 5.893
> (correpsonding p value : 7.365e-01)
>
>
> The largest off-origin peak in the Patterson function is 5.89% of the
> height of the origin peak. No significant pseudotranslation is detected.
>
> The results of the L-test indicate that the intensity statistics
> are significantly different then is expected from good to reasonable,
> untwinned data.
> As there are twin laws possible given the crystal symmetry, twinning could
> be the reason for the departure of the intensity statistics from normality.
> It might be worthwhile carrying refinement with a twin specific target function.
>
> Note that the symmetry of the intensities suggest that the assumed space group
> is too low. As twinning is however suspected, it is not immediuatly clear if this
> is the case. Carefull reprocessing and (twin)refinement for all cases might resolve
> this question.
>
> -------------------------------------------------------------------------------
18 years, 6 months
Re: [cctbxbb] Boost Python 1.56
by Graeme.Winter＠diamond.ac.uk
Colleagues
For those of us who are elderly and hard of thinking 😉 please could you send a relatively complete set of instructions on how best to update i.e. Using bootstrap or whatever.
5 line shell is fine
My usual recipe of redo from start is kind of annoying
Thanks Graeme
On 12 Apr 2017, at 21:48, Billy Poon <bkpoon(a)lbl.gov<mailto:[email protected]>> wrote:
Hi everyone,
Boost 1.56 is now available and the tests have been updated. To update an existing installtion, you have to delete the "build" directory first. If you don't, I guess some files are not recompiled with the new Boost headers, which will cause cctbx_project/scitbx/random/tests/tst_random.py to expect the old values. Then run bootstrap.py with hot, update, and build to rebuild.
cd <installation directory>
rm -fr build
python bootstrap.py hot update build --builder=cctbx
Let me know if there are any issues. Thanks!
--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709
Fax: (510) 486-5909
Web: https://phenix-online.org
On Fri, Apr 7, 2017 at 1:08 PM, Billy Poon <bkpoon(a)lbl.gov<mailto:[email protected]>> wrote:
I can do the switch to Boost 1.56 next Wednesday now that Dials 1.5 has been released. I'm just double-checking some other Phenix tests.
Once the update is live, bootstrap.py should delete the existing modules/boost directory and replace it with the new one. And then scons should recompile anything that depends on boost. If in doubt, you can manually delete the module/boost and build directories.
--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709<tel:(510)%20486-5709>
Fax: (510) 486-5909<tel:(510)%20486-5909>
Web: https://phenix-online.org
On Thu, Apr 6, 2017 at 9:32 AM, Billy Poon <bkpoon(a)lbl.gov<mailto:[email protected]>> wrote:
Hi James,
I agree with Graeme's tests and I would add
cctbx_regression.test_nightly
That command is a shortcut for running the test modules for libtbx, boost_adaptbx, scitbx, cctbx, iotbx, dxtbx, and smtbx. The mmtbx module is also tested if chem_data is available. Should we add rstbx to the shortcut (cctbx_project/cctbx/command_line/cctbx_test_nightly.py)?
--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709<tel:(510)%20486-5709>
Fax: (510) 486-5909<tel:(510)%20486-5909>
Web: https://phenix-online.org
On Thu, Apr 6, 2017 at 1:37 AM, <richard.gildea(a)diamond.ac.uk<mailto:[email protected]>> wrote:
I've made a start on transcribing this document here:
https://github.com/cctbx/cctbx_project/wiki/cctbx-Developer-Guidance
It probably still needs cleaning up a bit (e.g. I couldn't figure out quickly how to do 3-level list nesting, 1.a.i etc.) and updating to reflect current practice (e.g. git not svn).
Cheers,
Richard
Dr Richard Gildea
Data Analysis Scientist
Tel: +441235 77 8078<tel:%2B441235%2077%208078>
Diamond Light Source Ltd.
Diamond House
Harwell Science & Innovation Campus
Didcot
Oxfordshire
OX11 0DE
________________________________________
From: cctbxbb-bounces(a)phenix-online.org<mailto:[email protected]> [cctbxbb-bounces(a)phenix-online.org<mailto:[email protected]>] on behalf of Pavel Afonine [pafonine(a)lbl.gov<mailto:[email protected]>]
Sent: 06 April 2017 09:07
To: cctbx mailing list
Subject: Re: [cctbxbb] Boost Python 1.56
Hi Graeme,
hm.. this is a good question. We've been through back-and-forth
iterations of editing this file and I think the latest I have is from
Paul. But I can't find a non-PDF version of it. Paul: do you have an
editable version of this file?
Thanks,
Pavel
On 4/6/17 13:45, Graeme.Winter(a)diamond.ac.uk<mailto:[email protected]> wrote:
> Hi Pavel
>
> These all seem sensible
>
> If you have the original non pdf document it may be easier to transcribe this over..
>
> I also note that it lacks the actual detail on how to run tests! However would be happy to add this once on wiki
>
> Best wishes Graeme
>
> On 6 Apr 2017, at 04:00, Pavel Afonine <pafonine(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote:
>
> Not sure if that answers your questions but once upon a time we here at Berkeley tried to write a some sort of document that was supposed to answer questions like this. Attached. By no means it is complete, up-to-date, etc, but it might be worth reading for anyone who contributes to cctbx. (Even not sure if I'm sending the latest version).
> Unfortunately, nobody bothered to put it in some central place.
>
> Pavel
>
> On 4/6/17 10:51, James Holton wrote:
> Hey Billy,
>
> On a related note. How do I run these regression tests before committing something into Git? Is there a document on dials regression testing I can't find?
>
> -James
>
> On Apr 5, 2017, at 3:38 PM, Billy Poon <bkpoon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote:
>
> Hi all,
>
> I tested Boost 1.56 on our buildbot servers and got some new test failures with
>
> cctbx_project/scitbx/array_family/boost_python/tst_flex.py
> cctbx_project/scitbx/random/tests/tst_random.py
>
> The full log for CentOS 6 can be found at
>
> http://cci-vm-6.lbl.gov:8010/builders/phenix-nightly-intel-linux-2.6-x86_64…
>
> It looks like the errors are related to random number generation. For a given seed, would the sequence of numbers change when Boost is changed? I did a diff between Boost 1.56 and the current Boost and could not see any changes that immediately stood out as being related to random numbers.
>
> Are these tests failing for others as well?
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709<tel:%28510%29%20486-5709>
> Fax: (510) 486-5909<tel:%28510%29%20486-5909>
> Web: https://phenix-online.org<https://phenix-online.org/>
>
> On Wed, Apr 5, 2017 at 8:12 AM, Charles Ballard <charles.ballard(a)stfc.ac.uk<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote:
> FYI, we (CCP4) have been using 1.56 for building cctbx/phaser/dials for the last while with no issues. Don't know about 1.60, but 1.59 causes issues with the boost python make_getter and make_setter (initialisation of none const reference if the passed type is a temporary).
>
> Charles
>
> On 3 Apr 2017, at 14:31, Luc Bourhis wrote:
>
> Hi all,
>
> everybody seemed to agree but then it was proposed to move straight to Boost 1.60, and this caused troubles. Could we consider again to move to at least 1.56? As far as I can tell, this does not cause any issue and as stated one year ago, it would help me and Olex 2.
>
> Thanks,
>
> Luc
>
> On 10 Feb 2016, at 15:17, Nicholas Sauter <nksauter(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote:
>
> Nigel, Billy & Aaron,
>
> I completely endorse this move to Boost 1.56. Can we update our build?
>
> Nick
>
> Nicholas K. Sauter, Ph. D.
> Computer Staff Scientist, Molecular Biophysics and Integrated Bioimaging Division
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Rd., Bldg. 33R0345
> Berkeley, CA 94720
> (510) 486-5713<tel:%28510%29%20486-5713><tel:%28510%29%20486-5713>
>
> On Wed, Feb 10, 2016 at 2:41 PM, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote:
> Hi,
>
> I have improvements to the smtbx on their way to be committed which require Boost version 1.56. This is related to Boost.Threads, whose support I re-activated a few months ago on Nick’s request. I need the function boost::thread::physical_concurrency which returns the number of physical cores on the machine, as opposed to virtual cores when hyperthreading is enabled (which it is by default on any Intel machine). That function is not available in Boost 1.55 which is the version currently used in the nightly tests: it appeared in 1.56.
>
> So, would it be possible to move to Boost 1.56? Otherwise, I will need to backport that function. Not too difficult but not thrilling.
>
> Best wishes,
>
> Luc
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> <cctbx-developer-guidance-08-2015.pdf>_______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
_______________________________________________
cctbxbb mailing list
cctbxbb(a)phenix-online.org<mailto:[email protected]>
http://phenix-online.org/mailman/listinfo/cctbxbb
--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
_______________________________________________
cctbxbb mailing list
cctbxbb(a)phenix-online.org<mailto:[email protected]>
http://phenix-online.org/mailman/listinfo/cctbxbb
_______________________________________________
cctbxbb mailing list
cctbxbb(a)phenix-online.org<mailto:[email protected]>
http://phenix-online.org/mailman/listinfo/cctbxbb
8 years, 9 months
Re: [cctbxbb] some thoughts on cctbx and pip
by Tristan Croll
Hi Billy,
Sorry, I wasn’t clear: ChimeraX can install packages *from* Pip, but isn’t installable that way itself. It’s built and distributed as a self-contained application. Would certainly make my life easier if they were to migrate to Conda, but I don’t see that happening in the near future.
Best regards,
Tristan
> On 24 Aug 2019, at 02:16, Billy Poon <BKPoon(a)lbl.gov> wrote:
>
> Hi all,
>
> Sorry for the delayed response, I was getting the message digests to test some bulletin board settings, so I wasn't getting the individual messages for replying.
>
> For a brief update on conda,
>
> 1) External Boost libraries can now be used to build on all platforms (linux, macOS, Windows) and I'm testing updated environments with Boost 1.68 and 1.70. This is for Python 2.7, 3.6, and some initial testing on 3.7. Once 3.7 is ready, Azure Pipelines will be updated to also build 3.7. I'll do the updated dependencies as a pull request since there will be changes that stop the downloading of modules/boost. Mixing Boost versions can cause seg faults or unexpected behavior. I also did some brief testing with 1.65 since that is the minimum version that the latest versions of CUDA require.
> 2) The next step, like Aaron mentioned, is to get our build system to put things in more standard locations (e.g. dispatchers/binaries in $PREFIX/bin, shared libraries in $PREFIX/lib, Python source/extension modules in $PREFIX/lib/python<version>/site-packages) as a "make install" command. conda on Windows has a similar hierarchy that's specific to conda, so for Windows, this will be conda-specific.
> 3) Update the conda_compiler branch (https://github.com/cctbx/cctbx_project/tree/conda_compiler). This lets the build process use the standard compilers in conda's build system.
>
> Then a CCTBX conda package can be built and added to the conda-forge channel. The conda environments for building CCTBX are already based on conda-forge dependencies.
>
> 4) With the ability to build a conda package, conda constructor (https://github.com/conda/constructor) can be used for building our installers. This would be the more official way of packaging CCTBX with conda dependencies as a stand-alone installer. One of the roadblocks to using conda constructor was supporting noarch packages. It looks like that has been fixed (https://www.anaconda.com/announcing-the-anaconda-2019-07-release/).
>
> So a goal for this work with conda is to make CCTBX-based programs installable as conda packages so that installing and running the software is consistent across platforms and operating system versions while also making it easier for users to add their tools to the software stack.
>
> We do have a Phenix release coming up, so I won't be able to spend much effort on this until later in September.
>
> Tristan,
>
> If ChimeraX is installable via pip, it can be installed into a conda environment. And once 2) is complete, you would be able to access CCTBX from ChimeraX. Also, this would make installing ChimeraX pretty easy for Phenix. Interestingly, the latest open source PyMOL can be built with dependencies from Schrodinger's conda channel, so it should be possible to build a conda package for PyMOL that integrates with our conda environment.
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709
> Fax: (510) 486-5909
> Web: https://phenix-online.org
>
>
>> On Fri, Aug 23, 2019 at 10:03 AM Tristan Croll <tic20(a)cam.ac.uk> wrote:
>> To add my two cents on this: probably the second-most common question
>> I've had about ISOLDE's implementation is, "why didn't you use CCTBX?".
>> The honest answer to that is, "I didn't know how."
>>
>> Still don't, really - although the current developments are rather
>> promising. The problem I've faced is that CCTBX was designed as its own
>> self-contained Python (2.7, until very recently) environment, with its
>> own interpreter and a lot of very specific environment setup. Meanwhile
>> I'm developing ISOLDE in ChimeraX, which is *also* its own
>> self-contained Python (3.7) environment. To plug one into the other in
>> that form... well, I don't think I'm a good enough programmer to really
>> know where to start.
>>
>> The move to Conda and a more modular CCTBX architecture should make a
>> lot more possible in that direction. Pip would be even better for me
>> personally (ChimeraX can install directly from the PyPI, but doesn't
>> interact with Conda) - but I understand pretty well the substantial
>> challenge that would amount to (not least being that the PyPI imposes a
>> limit - around 100MB from memory? - on the size of an individual
>> package).
>>
>> Best regards,
>>
>> Tristan
>>
>> On 2019-08-23 09:28, Luc Bourhis wrote:
>> > Hi Graeme,
>> >
>> > Yes, I know. But “black" is a program doing a very particular task
>> > (code formatting from the top of my head). Requiring to use a wrapper
>> > for python itself is another level. But ok, I think I am mellowing to
>> > the idea after all! Talking with people around me, and extrapolating,
>> > I would bet that, right now, a great majority of people interested by
>> > cctbx in pip have already used the cctbx, so they know about the
>> > Python wrapper, and they would not be too sanguine about that. My
>> > concern is for the future, when pip will be the first time some people
>> > use cctbx. Big fat warning notices on PyPI page and a better error
>> > message when cctbx fails because LIBTBX_BUILD is not set would be
>> > needed but that could be all right.
>> >
>> > If we do a pip installer, we should aim at a minimal install: cctbx,
>> > iotbx and their dependencies, and that’s it.
>> >
>> > Best wishes,
>> >
>> > Luc
>> >
>> >
>> >> On 23 Aug 2019, at 07:17, Graeme.Winter(a)Diamond.ac.uk
>> >> <Graeme.Winter(a)diamond.ac.uk> wrote:
>> >>
>> >> Without discussing the merits of this or whether we _choose_ to make
>> >> the move to supporting PIP, I am certain it would be _possible_ - many
>> >> other packages make dispatcher scripts when you pip install them e.g.
>> >>
>> >> Silver-Surfer rescale_f2 :) $ which black; cat $(which black)
>> >> /Library/Frameworks/Python.framework/Versions/3.6/bin/black
>> >> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
>> >>
>> >> # -*- coding: utf-8 -*-
>> >> import re
>> >> import sys
>> >>
>> >> from black import main
>> >>
>> >> if __name__ == '__main__':
>> >> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
>> >> sys.exit(main())
>> >>
>> >> So we _could_ work around the absence of LIBTBX_BUILD etc. in the
>> >> system. Whether or not we elect to do the work is a different
>> >> question, and it seems clear that here are very mixed opinions on
>> >> this.
>> >>
>> >> Best wishes Graeme
>> >>
>> >>
>> >> On 23 Aug 2019, at 01:21, Luc Bourhis
>> >> <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote:
>> >>
>> >> Hi,
>> >>
>> >> Even if we managed to ship our the boost dynamic libraries with pip,
>> >> it would still not be pip-like, as we would still need our python
>> >> wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages
>> >> work with the standard python exe. LD_LIBRARY_PATH, we could get
>> >> around that by changing the way we compile, using -Wl,-R, which is the
>> >> runtime equivalent of build time -L. That’s a significant change that
>> >> would need to be tested. But there is no way around setting
>> >> LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps
>> >> there is a way to hack libtbx/env_config.py so that we can hardwire
>> >> LIBTBX_BUILD in there when pip installs?
>> >>
>> >> Best wishes,
>> >>
>> >> Luc
>> >>
>> >>
>> >> On 16 Aug 2019, at 22:47, Luc Bourhis
>> >> <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote:
>> >>
>> >> Hi,
>> >>
>> >> I did look into that many years ago, and even toyed with building a
>> >> pip installer. What stopped me is the exact conclusion you reached
>> >> too: the user would not have the pip experience he expects. You are
>> >> right that it is a lot of effort but is it worth it? Considering that
>> >> remark, I don’t think so. Now, Conda was created specifically to go
>> >> beyond pip pure-python-only support. Since cctbx has garnered support
>> >> for Conda, the best avenue imho is to go the extra length to have a
>> >> package on Anaconda.org<http://anaconda.org/>, and then to advertise
>> >> it hard to every potential user out there.
>> >>
>> >> Best wishes,
>> >>
>> >> Luc
>> >>
>> >>
>> >> On 16 Aug 2019, at 21:45, Aaron Brewster
>> >> <asbrewster(a)lbl.gov<mailto:[email protected]>> wrote:
>> >>
>> >> Hi, to avoid clouding Dorothee's documentation email thread, which I
>> >> think is a highly useful enterprise, here's some thoughts about
>> >> putting cctbx into pip. Pip doesn't install non-python dependencies
>> >> well. I don't think boost is available as a package on pip (at least
>> >> the package version we use). wxPython4 isn't portable through pip
>> >> (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython…).
>> >> MPI libraries are system dependent. If cctbx were a pure python
>> >> package, pip would be fine, but cctbx is not.
>> >>
>> >> All that said, we could build a manylinux1 version of cctbx and upload
>> >> it to PyPi (I'm just learning about this). For a pip package to be
>> >> portable (which is a requirement for cctbx), it needs to conform to
>> >> PEP513, the manylinux1 standard
>> >> (https://www.python.org/dev/peps/pep-0513/). For example, numpy is
>> >> built according to this standard (see
>> >> https://pypi.org/project/numpy/#files, where you'll see the manylinux1
>> >> wheel). Note, the manylinux1 standard is built with Centos 5.11 which
>> >> we no longer support.
>> >>
>> >> There is also a manylinux2010 standard, which is based on Centos 6
>> >> (https://www.python.org/dev/peps/pep-0571/). This is likely a more
>> >> attainable target (note though by default C++11 is not supported on
>> >> Centos 6).
>> >>
>> >> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the
>> >> user would need all the non-python dependencies. There's no way to
>> >> specify these in pip. For example, cctbx requires boost 1.63 or
>> >> better. The user will need to have it in a place their python can
>> >> find it, or we could package it ourselves and supply it, similar to
>> >> how the pip h5py package now comes with an hd5f library, or how the
>> >> pip numpy package includes an openblas library. We'd have to do the
>> >> same for any packages we depend on that aren't on pip using the
>> >> manylinux standards, such as wxPython4.
>> >>
>> >> Further, we need to think about how dials and other cctbx-based
>> >> packages interact. If pip install cctbx is set up, how does pip
>> >> install dials work, such that any dials shared libraries can find the
>> >> cctbx libraries? Can shared libraries from one pip package link
>> >> against libraries in another pip package? Would each package need to
>> >> supply its own boost? Possibly this is well understood in the pip
>> >> field, but not by me :)
>> >>
>> >> Finally, there's the option of providing a source pip package. This
>> >> would require the full compiler toolchain for any given platform
>> >> (macOS, linux, windows). These are likely available for developers,
>> >> but not for general users.
>> >>
>> >> Anyway, these are some of the obstacles. Not saying it isn't
>> >> possible, it's just a lot of effort.
>> >>
>> >> Thanks,
>> >> -Aaron
>> >>
>> >> _______________________________________________
>> >> cctbxbb mailing list
>> >> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> >>
>> >> _______________________________________________
>> >> cctbxbb mailing list
>> >> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> >>
>> >> _______________________________________________
>> >> cctbxbb mailing list
>> >> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> >>
>> >>
>> >> --
>> >> This e-mail and any attachments may contain confidential, copyright
>> >> and or privileged material, and are for the use of the intended
>> >> addressee only. If you are not the intended addressee or an authorised
>> >> recipient of the addressee please notify us of receipt by returning
>> >> the e-mail and do not use, copy, retain, distribute or disclose the
>> >> information in or attached to the e-mail.
>> >> Any opinions expressed within this e-mail are those of the individual
>> >> and not necessarily of Diamond Light Source Ltd.
>> >> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
>> >> attachments are free from viruses and we cannot accept liability for
>> >> any damage which you may sustain as a result of software viruses which
>> >> may be transmitted in or with the message.
>> >> Diamond Light Source Limited (company no. 4375679). Registered in
>> >> England and Wales with its registered office at Diamond House, Harwell
>> >> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United
>> >> Kingdom
>> >>
>> >> _______________________________________________
>> >> cctbxbb mailing list
>> >> cctbxbb(a)phenix-online.org
>> >> http://phenix-online.org/mailman/listinfo/cctbxbb
>> >
>> >
>> > _______________________________________________
>> > cctbxbb mailing list
>> > cctbxbb(a)phenix-online.org
>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>>
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org
>> http://phenix-online.org/mailman/listinfo/cctbxbb
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
6 years, 5 months
Re: [phenixbb] help with phenix.ligand_identification
by Edward A. Berry
Thanks, Li-wei,
That makes sense, I'll try those suggestions.
Meanwhile the foreach loop is up to ligand 79 and found a few with CC > 0.75.
But it won't provide the analysis that ligand_identification will.
Ed
On 09/28/2017 12:47 AM, Li-Wei Hung wrote:
> Hi Ed,
>
> Sorry for problems in ligand_identification. The default is to use
> LigandFit processes (slower, but more thorough), so the search center
> syntax will be under ligandfit.search_target.search_center. You can on
> the other hand use the faster ligand_id process with
> use_ligandfit=False, the your search_center should work. I noticed that
> you are using map coefficients so mtz_type=diffmap should be set. I
> short, you can either
>
> phenix.ligand_identification mtz_in=sqr2803or13_031.mtz
> input_labels="2FOFCWT PH2FOFCWT" mtz_type=diffmap\
> model=sqr2803or13_031.pdb ligandfit.search_target.ligand_near_chain=S
> ligandfit.search_target.ligand_near_res=2063 nproc=2 ....
> (the two ligandfit.search_targets.ligand_near_xxx, and be replaced by
> ligandfit.search_target.search_center="n.n n.n n.n")
> or
> phenix.ligand_identification mtz_in=sqr2803or13_031.mtz
> input_labels="2FOFCWT PH2FOFCWT" mtz_type=diffmap\
> model=sqr2803or13_031.pdb use_ligandfit=false ligand_near_res="chain S
> and resid 2063" (or search_center="n.n n.n n.n" )nproc=2
>
> If you have further problems or questions on ligand_identification,
> please do not hesitates to email me with input files (off list), and
> I'll help straighten problems.
> In any event, I will update the documentation to add examples, and
> probably change one of the search_center keywords to make things clearer.
>
> Best regards,
>
> Li-Wei
>
>
>
>
> Edward A. Berry wrote:
>> Ligandfit itself has more helpful error message:
>> Sorry the string 'S2063' cannot be interpreted as a residue number?
>> (Duh!)
>> and does actually place the ligand in the requested blob with
>> "search_center="
>> So I can run ligandfit in a foreach loop with each of the 180 ligands
>> left by ligand_identification
>> =================
>>
>> On 09/26/2017 03:32 PM, Edward A. Berry wrote:
>>> I'm having some problems using ligand_identification.
>>> I would like to restrict the search to a specific density peak, even
>>> if it is not the highest unmodeled peak or the highest Fo-Fc peak in
>>> the map.
>>> I tried using options:
>>> search_center="67.5 18.3 11.8"
>>> or
>>> ligand_near_res=S2063,
>>> Will the search be restricted to that region, or if a particuar
>>> ligand doesn't fit that blob,
>>> will it search through the rest of the map? If it is restricted, in
>>> what radius?
>>> Is this radius affected by the "search_dist" or "local_search"
>>> parameters?
>>> and local_search = True is default?
>>> Is there a threshold level for density level, below which building a
>>> ligand in a blob will not be attempted?
>>>
>>> I've tried with both search_center= and ligand_near_res=, and
>>> something gets
>>> built far away from that site. But there were errors, so I may have
>>> something wrong:
>>>
>>>
>>> phenix.ligand_identification mtz_in=sqr2803or13_031.mtz
>>> input_labels="2FOFCWT PH2FOFCWT" \
>>> model=sqr2803or13_031.pdb ligand_near_res=S2063 nproc=2
>>>
>>> After preparing the ligand library, then:
>>> Running LigandFit process 1...
>>>
>>> Number of atoms in ligand suc.pdb is 23
>>> Running job sequence 1, ligand 2, in
>>> /tb/sb/usr20c/berry/ref/sqrdep/sqr2803dep2/phenix2/Temp_1...
>>> Evaluating all ligands in ligand-lib now...and placing fittedligand
>>> ### in resolve_ligand_###.pdb
>>>
>>> Number of atoms in ligand 2pe.pdb is 28
>>> Running job sequence 0, ligand 1, in
>>> /tb/sb/usr20c/berry/ref/sqrdep/sqr2803dep2/phenix2/Temp_0...
>>> Process Process-2:
>>> Traceback (most recent call last):
>>> File
>>> "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/multiprocessing/process.py",
>>> line 258, in _bootstrap
>>> self.run()
>>> File
>>> "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/multiprocessing/process.py",
>>> line 114, in run
>>> self._target(*self._args, **self._kwargs)
>>> File
>>> "/sw/lnx/phenix-1.12-2829/build/../modules/phenix/phenix/command_line/ligand_identification.py",
>>> line 1382, in RunLigandFit
>>> shutil.rmtree(ligandfit_dir)
>>> File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/shutil.py", line
>>> 247, in rmtree
>>> rmtree(fullname, ignore_errors, onerror)
>>> File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/shutil.py", line
>>> 256, in rmtree
>>> onerror(os.rmdir, path, sys.exc_info())
>>> File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/shutil.py", line
>>> 254, in rmtree
>>> os.rmdir(path)
>>> OSError: [Errno 39] Directory not empty:
>>> '/tb/sb/usr20c/berry/ref/sqrdep/sqr2803dep2/phenix2/Temp_1/LigandFit_run_1_/TEMP0'
>>> Process Process-1:
>>> Traceback (most recent call last):
>>> File
>>> "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/multiprocessing/process.py",
>>> line 258, in _bootstrap
>>> self.run()
>>> File
>>> "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/multiprocessing/process.py",
>>> line 114, in run
>>> self._target(*self._args, **self._kwargs)
>>> File
>>> "/sw/lnx/phenix-1.12-2829/build/../modules/phenix/phenix/command_line/ligand_identification.py",
>>> line 1382, in RunLigandFit
>>> shutil.rmtree(ligandfit_dir)
>>> File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/shutil.py", line
>>> 247, in rmtree
>>> rmtree(fullname, ignore_errors, onerror)
>>> File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/shutil.py", line
>>> 256, in rmtree
>>> onerror(os.rmdir, path, sys.exc_info())
>>> File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/shutil.py", line
>>> 254, in rmtree
>>> os.rmdir(path)
>>> OSError: [Errno 39] Directory not empty:
>>> '/tb/sb/usr20c/berry/ref/sqrdep/sqr2803dep2/phenix2/Temp_0/LigandFit_run_1_/TEMP0'
>>>
>>> Evaluating LigandFit results ...
>>>
>>> The run continues, but it does not test any more ligands after those
>>> first two but goes on to evaluate the results.
>>> With nproc = 1, only 1 ligand gets tested. In all cases the first
>>> ligand in the library (2PE.pdb) is evaluated as the best.
>>> It is placed in density, but density that has already been built out
>>> with (and looks more like) a string of water molecules.
>>> And this is far from the selected residue or coordinates specified.
>>>
>>> The directory that raised the error when attempting to be deleted
>>> does eventually get removed: after the run there is no TEMP_N in the
>>> parent directory.
>>>
>>> Any suggestions would be welcome.
>>> Ed
>>>
>>> P.S.
>>> - running with .eff file:
>>>
>>>
>>> ['--show_defaults']
>>> ligand_identification {
>>> mtz_in = sqr2803or13_031.mtz
>>> mtz_type = *F diffmap
>>> model = sqr2803or13_031.pdb
>>> ncpu = 1
>>> n_indiv_tries_min = 30
>>> n_indiv_tries_max = 300
>>> n_group_search = 4
>>> search_dist = 10
>>> local_search = True
>>> search_center = "67.5 18.3 11.8"
>>> # ligand_near_res = S2063
>>> verbose = False
>>> debug = False
>>> use_ligandfit = True
>>> search_mode = *default LigandFit
>>> temp_dir = Auto
>>> dry_run = False
>>> # number_of_ligands = 1
>>> cc_min = 0.75
>>> open_in_coot = False
>>> non_bonded = True
>>> keep_all_files = False
>>> # cif_def_file_list =
>>> real_space_target_weight = 10
>>> # job_title = None
>>> ligandfit {
>>> }
>>> }
>>>
>>> gives:
>>>
>>> [['2pe.pdb', 'suc.pdb', . . . 'upl.pdb']]
>>>
>>> /tb/sb/usr20c/berry/ref/sqrdep/sqr2803dep2/phenix2
>>>
>>> *******************************************************************************
>>>
>>>
>>> Sorry, the protein model file None does not seem to exist?
>>>
>>> *******************************************************************************
>>>
>>>
>>>
>>> Running LigandFit process 0...
>>>
>>> Process Process-1:
>>> Traceback (most recent call last):
>>> File
>>> "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/multiprocessing/process.py",
>>> line 258, in _bootstrap
>>> self.run()
>>> File
>>> "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/multiprocessing/process.py",
>>> line 114, in run
>>> self._target(*self._args, **self._kwargs)
>>> File
>>> "/sw/lnx/phenix-1.12-2829/build/../modules/phenix/phenix/command_line/ligand_identification.py",
>>> line 1122, in RunLigandFit
>>> shutil.copyfile(mtz_in,data_local)
>>> File "/sw/lnx/phenix-1.12-2829/base/lib/python2.7/shutil.py", line
>>> 82, in copyfile
>>> with open(src, 'rb') as fsrc:
>>> IOError: [Errno 2] No such file or directory: 'None'
>>>
>>> Evaluating LigandFit results ...
>>>
>>> Lig_seq Placed/total cc_all cc cc_adj score Code HBscore
>>>
>>> Cannot find overall_ligand_scores.log0. This could mean that none of
>>> that (sub)set of ligand fitted well.
>>>
>>>
>>>
>>> None of the ligand fit the difference desity well enough. Please try
>>> the following --
>>> 1) if you input a custom library, try to use the default library (no
>>> extra keywords needed), or
>>> 2) if you used the default library already, ususlly this means that
>>> the density is too small (> 6 atome or more is needed.)
>>> Exiting ......
>>>
>>>
>>>
>>> No good ligand found.
>>>
>>>
>>> _______________________________________________
>>> phenixbb mailing list
>>> phenixbb(a)phenix-online.org
>>> http://phenix-online.org/mailman/listinfo/phenixbb
>>> Unsubscribe: phenixbb-leave(a)phenix-online.org
>>>
>> _______________________________________________
>> phenixbb mailing list
>> phenixbb(a)phenix-online.org
>> http://phenix-online.org/mailman/listinfo/phenixbb
>> Unsubscribe: phenixbb-leave(a)phenix-online.org
>
>
8 years, 4 months
Re: [cctbxbb] some thoughts on cctbx and pip
by Billy Poon
Hi Gergely,
I've uploaded linux packages to a new channel, cctbx-dev, and you can
install it with
conda install -c cctbx-dev cctbx
in your current environment. You should set your ~/.condarc file to pull
the other dependencies from the conda-forge channel first, so put
conda-forge above cctbx. Mine looks like
channels:
- conda-forge
- defaults
- cctbx
Do you need dxtbx for your scripts? This package does not build that part.
I think the plan is to build a separate conda package for dxtbx so that it
can be updated more frequently. I can rebuild the packages to include it
for testing, but the one being submitted to conda-forge will not have it.
Also, your error message is probably due to version of HDF5 that the
development build installs. The bootstrap.py script will install 1.10.4,
but your other dependency is looking for 1.10.5. Installing this cctbx
conda package should install 1.10.5, which should fix the issue. Also, I'm
updating those environments and 1.10.5 will be the new default version.
Lastly, the dispatchers will not work in these packages because the old
paths during the build process are still in them and many of them expect
some additional information that has not been updated in the packages yet.
I'm in the process of doing that and will update the cctbx-dev channel when
that's done. However, by starting python, you can import cctbx modules. So
you can run commands like
from scitbx.array_family import flex
a = flex.random_double(1000000)
b = flex.min_max_mean_double(a)
b.min
b.max
b.mean
Thanks!
--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709
Fax: (510) 486-5909
Web: https://phenix-online.org
On Tue, Dec 17, 2019 at 5:39 AM Gergely Katona <gkatona(a)gmail.com> wrote:
> Dear Billy,
>
> Thank you for the detailed explanation, I look forward to do further
> testing! I aim to pool all modules under the same environment, if
> there are conflicts then I just try to reshuffle the order of imports.
> So far this did not cause problems for me even when I was using system
> python, but of course it is not the most prudent thing to do. With
> anaconda everything is much more standardized and isolated already and
> without being superuser I can have my familiar environment at any
> synchrotron based cluster. It is great that cctbx will be an integral
> part of this ecosystem and this was also the last thing holding me
> back from adopting python3. About hdf5, pymc3 requires it and
> importing causes a kernel restart with the following error messages,
> curiously if I import h5py first this can be avoided.
>
> Best wishes,
>
> Gergely
>
> import pymc3 as pm
>
> Warning! ***HDF5 library version mismatched error***
> The HDF5 header files used to compile this application do not match
> the version used by the HDF5 library to which this application is linked.
> Data corruption or segmentation faults may occur if the application
> continues.
> This can happen when an application was compiled by one version of HDF5 but
> linked with a different version of static or shared HDF5 library.
> You should recompile the application or check your shared library related
> settings such as 'LD_LIBRARY_PATH'.
> You can, at your own risk, disable this warning by setting the environment
> variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.
> Setting it to 2 or higher will suppress the warning messages totally.
> Headers are 1.10.4, library is 1.10.5
> SUMMARY OF THE HDF5 CONFIGURATION
> =================================
>
> General Information:
> -------------------
> HDF5 Version: 1.10.5
> Configured on: Tue Oct 22 12:02:13 UTC 2019
> Configured by: conda@16247e67ecd5
> Host system: x86_64-conda_cos6-linux-gnu
> Uname information: Linux 16247e67ecd5 4.15.0-1059-azure
> #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64
> GNU/Linux
> Byte sex: little-endian
> Installation point: /home/gergely/anaconda3
>
> Compiling Options:
> ------------------
> Build Mode: production
> Debugging Symbols: no
> Asserts: no
> Profiling: no
> Optimization Level: high
>
> Linking Options:
> ----------------
> Libraries: static, shared
> Statically Linked Executables:
> LDFLAGS: -Wl,-O2 -Wl,--sort-common
> -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags
> -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib
> -Wl,-rpath-link,/home/gergely/anaconda3/lib
> -L/home/gergely/anaconda3/lib
> H5_LDFLAGS:
> AM_LDFLAGS: -L/home/gergely/anaconda3/lib
> Extra libraries: -lrt -lpthread -lz -ldl -lm
> Archiver:
>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar
> AR_FLAGS: cr
> Ranlib:
>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib
>
> Languages:
> ----------
> C: yes
> C Compiler:
>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc
> CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2
> -I/home/gergely/anaconda3/include
> H5_CPPFLAGS: -D_GNU_SOURCE
> -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API
> AM_CPPFLAGS: -I/home/gergely/anaconda3/include
> C Flags: -march=nocona -mtune=haswell
> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
> -ffunction-sections -pipe -I/home/gergely/anaconda3/include
>
> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5
> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix
> H5 C Flags: -std=c99 -pedantic -Wall -Wextra
> -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion
> -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal
> -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations
> -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs
> -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls
> -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef
> -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings
> -finline-functions -s -Wno-inline -Wno-aggregate-return
> -Wno-missing-format-attribute -Wno-missing-noreturn -O
> AM C Flags:
> Shared C Library: yes
> Static C Library: yes
>
>
> Fortran: yes
> Fortran Compiler:
>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran
> Fortran Flags:
> H5 Fortran Flags: -pedantic -Wall -Wextra -Wunderflow
> -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2
> AM Fortran Flags:
> Shared Fortran Library: yes
> Static Fortran Library: yes
>
> C++: yes
> C++ Compiler:
>
> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++
> C++ Flags: -fvisibility-inlines-hidden
> -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell
> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
> -ffunction-sections -pipe -I/home/gergely/anaconda3/include
>
> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5
> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix
> H5 C++ Flags: -pedantic -Wall -W -Wundef -Wshadow
> -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion
> -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual
> -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor
> -Wctor-dtor-privacy -Wabi -finline-functions -s -O
> AM C++ Flags:
> Shared C++ Library: yes
> Static C++ Library: yes
>
> Java: no
>
>
> Features:
> ---------
> Parallel HDF5: no
> Parallel Filtered Dataset Writes: no
> Large Parallel I/O: no
> High-level library: yes
> Threadsafety: yes
> Default API mapping: v110
> With deprecated public symbols: yes
> I/O filters (external): deflate(zlib)
> MPE: no
> Direct VFD: no
> dmalloc: no
> Packages w/ extra debug output: none
> API tracing: no
> Using memory checker: yes
> Memory allocation sanity checks: no
> Function stack tracing: no
> Strict file format checks: no
> Optimization instrumentation: no
>
> On Tue, Dec 17, 2019 at 8:35 AM Billy Poon <BKPoon(a)lbl.gov> wrote:
> >
> > Hi Gergely,
> >
> > Let me build the test package tomorrow. All the gritty details for
> building with conda is being finalized and the official documentation will
> be updated to describe the steps. It would be too confusing to keep
> changing the documentation as the process evolves. You do have the general
> process, though, which is summarized as follows.
> >
> > 1) Installing dependencies. The cctbx_dependencies metapackage was an
> initial approach for managing the CCTBX dependencies, but after contacting
> the conda-forge folks, they recommended using the --only-deps flag. So when
> the CCTBX conda package is available, you'll be able to get a set of
> dependencies with,
> >
> > conda install -c conda-forge --only-deps cctbx
> >
> > By default, the bootstrap.py file will automatically install a set of
> dependencies in the "conda_base" directory (and a conda installation if one
> is not found). It just uses standard conda environment files located in
> libtbx/auto_build/conda_envs, so you do not need to install
> cctbx_dependencies as a separate step. The environment files avoid channel
> issues by explicitly defining the channel to pull the packages from and the
> cctbx channel just stores copies of packages from conda-forge. There were
> issues earlier where the conda-forge packages would sometimes be moved to a
> different label. The --use-conda flag also accepts a path to $CONDA_PREFIX
> if you want to use a specific environment for testing.
> >
> > 2) Building. The bootstrap.py file handles that with SCons.
> >
> > 3) Running. After building, there should be a setpaths.sh (and .csh)
> file that adds build/bin to your path. The build/bin directory has our
> dispatchers, which are just scripts that set up the environment variables
> for you. This prevents other programs from loading our libraries, whose
> versions may conflict. You should see that there is a "python" dispatcher,
> which is a convenience for developers. Otherwise, you can use
> libtbx.python, which will be able to import CCTBX modules.
> >
> > The running part is where the conda package for CCTBX will be different
> than this build. Since our Python files and extensions modules will be in
> the "site-packages" directory for Python, the PYTHONPATH variable will not
> be needed (and conda suggests that that variable not be set). The other
> CCTBX libraries will be in $CONDA_PREFIX/lib, so LD_LIBRARY_PATH is not
> needed. In an active environment, PATH will already be modified. And then
> our LIBTBX_BUILD directory can be set to sys.prefix.
> >
> > So with the conda package, you would only need to activate your conda
> environment and CCTBX should integrate with other conda packages. What is
> the conflict with hdf5? That's something that should be fixed. Thanks!
> >
> > --
> > Billy K. Poon
> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> > Lawrence Berkeley National Laboratory
> > 1 Cyclotron Road, M/S 33R0345
> > Berkeley, CA 94720
> > Tel: (510) 486-5709
> > Fax: (510) 486-5909
> > Web: https://phenix-online.org
> >
> >
> > On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona <gkatona(a)gmail.com>
> wrote:
> >>
> >> Dear Billy,
> >>
> >> Thank you for this update and for your efforts! I found a solution and
> >> indeed most things already work in anaconda3. The steps I took (even
> >> if these instructions will have short expiry date):
> >>
> >> Modified .condarc with:
> >> channels:
> >> - cctbx
> >> - conda-forge
> >> - defaults
> >>
> >> run
> >> conda conda install cctbx_dependencies python=3.6
> >>
> >> I expect this will work without python=3.6 in the near future.
> >>
> >> Then compiling cctbx with anaconda3 python went without any problem
> >> when using these flags:
> >> python bootstrap.py --use-conda --python3 --nproc=12
> >>
> >> Then I probably did the unorthodox thing and sourced these directories:
> >>
> >> setenv LIBTBX_BUILD /home/gergely/cctbx/build
> >> setenv PATH ${PATH}:/home/gergely/cctbx/build/bin
> >> setenv PYTHONPATH
> >>
> /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH
> >> setenv LD_LIBRARY_PATH
> >>
> /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH
> >>
> >> There is probably a better way to put this into conda environment.
> >> With these steps I could run one of my scripts depending on cctbx. The
> >> only problem I found is that hdf5 library had conflict with another
> >> package in conda, but I does not find this as a showstopper.
> >>
> >> Best wishes,
> >>
> >> Gergely
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Fri, Dec 13, 2019 at 7:23 PM Billy Poon <BKPoon(a)lbl.gov> wrote:
> >> >
> >> > Hi Gergely,
> >> >
> >> > It's still a work in progress. I'm sorting out some Windows issues
> right now. I can probably build a test package on a separate channel for
> people that want to test it (let's say next week?). I'll provide
> instructions, but basically, the test conda package will be in its own
> separate channel and the dependencies will be pulled from the conda-forge
> channel. I want most things to be working correctly on Python 2.7, 3.6,
> 3.7, and 3.8 on all 3 platforms.
> >> >
> >> > Thanks!
> >> >
> >> > --
> >> > Billy K. Poon
> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> >> > Lawrence Berkeley National Laboratory
> >> > 1 Cyclotron Road, M/S 33R0345
> >> > Berkeley, CA 94720
> >> > Tel: (510) 486-5709
> >> > Fax: (510) 486-5909
> >> > Web: https://phenix-online.org
> >> >
> >> >
> >> > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona <gkatona(a)gmail.com>
> wrote:
> >> >>
> >> >> Dear Billy,
> >> >>
> >> >> This sounds very promising and exciting. I am not sure if cctbx is
> >> >> already functional as a conda package in anaconda3 (Linux) or this is
> >> >> still work in progress. My technical expertise does not allow me to
> >> >> tell the difference. What I tried:
> >> >>
> >> >> Fresh install of anaconda3. Adding - cctbx and - conda-forge to
> >> >> .condarc . Installing pyside2 with conda. Running conda install
> >> >> conda_dependencies . I get a lot package version conflict, and I
> >> >> cannot import cctbx or iotbx to anaconda python. Am I following the
> >> >> right instructions? Or it is too early to expect that cctbx works
> when
> >> >> installed through conda?
> >> >>
> >> >> Best wishes,
> >> >>
> >> >> Gergely
> >> >>
> >> >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon <BKPoon(a)lbl.gov> wrote:
> >> >> >
> >> >> > Hi all,
> >> >> >
> >> >> > For a brief update, I have submitted a recipe for cctbxlite to
> conda-forge (https://github.com/conda-forge/staged-recipes/pull/10021)
> and support for Python 3.7 and 3.8 is being added (
> https://github.com/cctbx/cctbx_project/pull/409). With some fixes for
> Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms
> (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8.
> Some additional changes will be needed to get Windows to work with Python 3
> and for tests to pass with Boost 1.70.0. That will enable the conda-forge
> recipe to build for all platforms and for all supported versions of Python.
> >> >> >
> >> >> > Currently, the conda-forge recipe will install into the conda
> python and cctbx imports can be done without sourcing the environment
> scripts. It looks like a lot of the environment variables being set in the
> dispatchers can be removed since the Python files and C++ extensions are in
> the right places. I'll update the libtbx_env file so that commands that
> load the environment can work correctly.
> >> >> >
> >> >> > --
> >> >> > Billy K. Poon
> >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
> >> >> > Lawrence Berkeley National Laboratory
> >> >> > 1 Cyclotron Road, M/S 33R0345
> >> >> > Berkeley, CA 94720
> >> >> > Tel: (510) 486-5709
> >> >> > Fax: (510) 486-5909
> >> >> > Web: https://phenix-online.org
> >> >> >
> >> >> >
> >> >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll <tic20(a)cam.ac.uk>
> wrote:
> >> >> >>
> >> >> >> Hi Luc,
> >> >> >>
> >> >> >> That sounds promising. From there, I’d need to work out how to
> make a fully-packaged installer (basically a modified wheel file) for the
> ChimeraX ToolShed - the aim is for the end user to not have to worry about
> any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would
> need to be set dynamically before first import - but doesn’t seem
> insurmountable.
> >> >> >>
> >> >> >> Thanks,
> >> >> >>
> >> >> >> Tristan
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> Tristan Croll
> >> >> >> Research Fellow
> >> >> >> Cambridge Institute for Medical Research
> >> >> >> University of Cambridge CB2 0XY
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis <luc_j_bourhis(a)mac.com>
> wrote:
> >> >> >> >
> >> >> >> > Hi Tristan,
> >> >> >> >
> >> >> >> > cctbx could be built to use your ChimeraX python, now that
> cctbx is moving to Python 3. The option —with-python is there for that with
> the bootstrap script. The specific environment setup boil down to setting
> two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux,
> PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a
> framework such as ChimeraX, that should not be difficult to ensure those
> two variables are set.
> >> >> >> >
> >> >> >> > Best wishes,
> >> >> >> >
> >> >> >> > Luc
> >> >> >> >
> >> >> >> >
> >> >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll <tic20(a)cam.ac.uk>
> wrote:
> >> >> >> >>
> >> >> >> >> To add my two cents on this: probably the second-most common
> question I've had about ISOLDE's implementation is, "why didn't you use
> CCTBX?". The honest answer to that is, "I didn't know how."
> >> >> >> >>
> >> >> >> >> Still don't, really - although the current developments are
> rather promising. The problem I've faced is that CCTBX was designed as its
> own self-contained Python (2.7, until very recently) environment, with its
> own interpreter and a lot of very specific environment setup. Meanwhile I'm
> developing ISOLDE in ChimeraX, which is *also* its own self-contained
> Python (3.7) environment. To plug one into the other in that form... well,
> I don't think I'm a good enough programmer to really know where to start.
> >> >> >> >>
> >> >> >> >> The move to Conda and a more modular CCTBX architecture should
> make a lot more possible in that direction. Pip would be even better for me
> personally (ChimeraX can install directly from the PyPI, but doesn't
> interact with Conda) - but I understand pretty well the substantial
> challenge that would amount to (not least being that the PyPI imposes a
> limit - around 100MB from memory? - on the size of an individual package).
> >> >> >> >>
> >> >> >> >> Best regards,
> >> >> >> >>
> >> >> >> >> Tristan
> >> >> >> >>
> >> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote:
> >> >> >> >>> Hi Graeme,
> >> >> >> >>> Yes, I know. But “black" is a program doing a very particular
> task
> >> >> >> >>> (code formatting from the top of my head). Requiring to use a
> wrapper
> >> >> >> >>> for python itself is another level. But ok, I think I am
> mellowing to
> >> >> >> >>> the idea after all! Talking with people around me, and
> extrapolating,
> >> >> >> >>> I would bet that, right now, a great majority of people
> interested by
> >> >> >> >>> cctbx in pip have already used the cctbx, so they know about
> the
> >> >> >> >>> Python wrapper, and they would not be too sanguine about
> that. My
> >> >> >> >>> concern is for the future, when pip will be the first time
> some people
> >> >> >> >>> use cctbx. Big fat warning notices on PyPI page and a better
> error
> >> >> >> >>> message when cctbx fails because LIBTBX_BUILD is not set
> would be
> >> >> >> >>> needed but that could be all right.
> >> >> >> >>> If we do a pip installer, we should aim at a minimal install:
> cctbx,
> >> >> >> >>> iotbx and their dependencies, and that’s it.
> >> >> >> >>> Best wishes,
> >> >> >> >>> Luc
> >> >> >> >>>> On 23 Aug 2019, at 07:17, Graeme.Winter(a)Diamond.ac.uk <
> Graeme.Winter(a)diamond.ac.uk> wrote:
> >> >> >> >>>> Without discussing the merits of this or whether we _choose_
> to make the move to supporting PIP, I am certain it would be _possible_ -
> many other packages make dispatcher scripts when you pip install them e.g.
> >> >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat $(which black)
> >> >> >> >>>> /Library/Frameworks/Python.framework/Versions/3.6/bin/black
> >> >> >> >>>>
> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
> >> >> >> >>>> # -*- coding: utf-8 -*-
> >> >> >> >>>> import re
> >> >> >> >>>> import sys
> >> >> >> >>>> from black import main
> >> >> >> >>>> if __name__ == '__main__':
> >> >> >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '',
> sys.argv[0])
> >> >> >> >>>> sys.exit(main())
> >> >> >> >>>> So we _could_ work around the absence of LIBTBX_BUILD etc.
> in the system. Whether or not we elect to do the work is a different
> question, and it seems clear that here are very mixed opinions on this.
> >> >> >> >>>> Best wishes Graeme
> >> >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis <luc_j_bourhis(a)mac.com
> <mailto:[email protected]>> wrote:
> >> >> >> >>>> Hi,
> >> >> >> >>>> Even if we managed to ship our the boost dynamic libraries
> with pip, it would still not be pip-like, as we would still need our python
> wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work
> with the standard python exe. LD_LIBRARY_PATH, we could get around that by
> changing the way we compile, using -Wl,-R, which is the runtime equivalent
> of build time -L. That’s a significant change that would need to be tested.
> But there is no way around setting LIBTBX_BUILD right now. Leaving that to
> the user is horrible. Perhaps there is a way to hack libtbx/env_config.py
> so that we can hardwire LIBTBX_BUILD in there when pip installs?
> >> >> >> >>>> Best wishes,
> >> >> >> >>>> Luc
> >> >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis <luc_j_bourhis(a)mac.com
> <mailto:[email protected]>> wrote:
> >> >> >> >>>> Hi,
> >> >> >> >>>> I did look into that many years ago, and even toyed with
> building a pip installer. What stopped me is the exact conclusion you
> reached too: the user would not have the pip experience he expects. You are
> right that it is a lot of effort but is it worth it? Considering that
> remark, I don’t think so. Now, Conda was created specifically to go beyond
> pip pure-python-only support. Since cctbx has garnered support for Conda,
> the best avenue imho is to go the extra length to have a package on
> Anaconda.org<http://anaconda.org/>, and then to advertise it hard to
> every potential user out there.
> >> >> >> >>>> Best wishes,
> >> >> >> >>>> Luc
> >> >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster <asbrewster(a)lbl.gov
> <mailto:[email protected]>> wrote:
> >> >> >> >>>> Hi, to avoid clouding Dorothee's documentation email thread,
> which I think is a highly useful enterprise, here's some thoughts about
> putting cctbx into pip. Pip doesn't install non-python dependencies well.
> I don't think boost is available as a package on pip (at least the package
> version we use). wxPython4 isn't portable through pip (
> https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython…).
> MPI libraries are system dependent. If cctbx were a pure python package,
> pip would be fine, but cctbx is not.
> >> >> >> >>>> All that said, we could build a manylinux1 version of cctbx
> and upload it to PyPi (I'm just learning about this). For a pip package to
> be portable (which is a requirement for cctbx), it needs to conform to
> PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/).
> For example, numpy is built according to this standard (see
> https://pypi.org/project/numpy/#files, where you'll see the manylinux1
> wheel). Note, the manylinux1 standard is built with Centos 5.11 which we
> no longer support.
> >> >> >> >>>> There is also a manylinux2010 standard, which is based on
> Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a
> more attainable target (note though by default C++11 is not supported on
> Centos 6).
> >> >> >> >>>> If we built a manylinuxX version of cctbx and uploaded it to
> PyPi, the user would need all the non-python dependencies. There's no way
> to specify these in pip. For example, cctbx requires boost 1.63 or
> better. The user will need to have it in a place their python can find it,
> or we could package it ourselves and supply it, similar to how the pip h5py
> package now comes with an hd5f library, or how the pip numpy package
> includes an openblas library. We'd have to do the same for any packages we
> depend on that aren't on pip using the manylinux standards, such as
> wxPython4.
> >> >> >> >>>> Further, we need to think about how dials and other
> cctbx-based packages interact. If pip install cctbx is set up, how does
> pip install dials work, such that any dials shared libraries can find the
> cctbx libraries? Can shared libraries from one pip package link against
> libraries in another pip package? Would each package need to supply its
> own boost? Possibly this is well understood in the pip field, but not by
> me :)
> >> >> >> >>>> Finally, there's the option of providing a source pip
> package. This would require the full compiler toolchain for any given
> platform (macOS, linux, windows). These are likely available for
> developers, but not for general users.
> >> >> >> >>>> Anyway, these are some of the obstacles. Not saying it
> isn't possible, it's just a lot of effort.
> >> >> >> >>>> Thanks,
> >> >> >> >>>> -Aaron
> >> >> >> >>>> _______________________________________________
> >> >> >> >>>> cctbxbb mailing list
> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >> >>>> _______________________________________________
> >> >> >> >>>> cctbxbb mailing list
> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >> >>>> _______________________________________________
> >> >> >> >>>> cctbxbb mailing list
> >> >> >> >>>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >> >>>> --
> >> >> >> >>>> This e-mail and any attachments may contain confidential,
> copyright and or privileged material, and are for the use of the intended
> addressee only. If you are not the intended addressee or an authorised
> recipient of the addressee please notify us of receipt by returning the
> e-mail and do not use, copy, retain, distribute or disclose the information
> in or attached to the e-mail.
> >> >> >> >>>> Any opinions expressed within this e-mail are those of the
> individual and not necessarily of Diamond Light Source Ltd.
> >> >> >> >>>> Diamond Light Source Ltd. cannot guarantee that this e-mail
> or any attachments are free from viruses and we cannot accept liability for
> any damage which you may sustain as a result of software viruses which may
> be transmitted in or with the message.
> >> >> >> >>>> Diamond Light Source Limited (company no. 4375679).
> Registered in England and Wales with its registered office at Diamond
> House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11
> 0DE, United Kingdom
> >> >> >> >>>> _______________________________________________
> >> >> >> >>>> cctbxbb mailing list
> >> >> >> >>>> cctbxbb(a)phenix-online.org
> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >> >>> _______________________________________________
> >> >> >> >>> cctbxbb mailing list
> >> >> >> >>> cctbxbb(a)phenix-online.org
> >> >> >> >>> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> _______________________________________________
> >> >> >> >> cctbxbb mailing list
> >> >> >> >> cctbxbb(a)phenix-online.org
> >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >> >
> >> >> >> >
> >> >> >> > _______________________________________________
> >> >> >> > cctbxbb mailing list
> >> >> >> > cctbxbb(a)phenix-online.org
> >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >>
> >> >> >>
> >> >> >> _______________________________________________
> >> >> >> cctbxbb mailing list
> >> >> >> cctbxbb(a)phenix-online.org
> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >> >
> >> >> > _______________________________________________
> >> >> > cctbxbb mailing list
> >> >> > cctbxbb(a)phenix-online.org
> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Gergely Katona, PhD
> >> >> Associate Professor
> >> >> Department of Chemistry and Molecular Biology, University of
> Gothenburg
> >> >> Box 462, 40530 Göteborg, Sweden
> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> >> >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
> >> >>
> >> >> _______________________________________________
> >> >> cctbxbb mailing list
> >> >> cctbxbb(a)phenix-online.org
> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >> >
> >> > _______________________________________________
> >> > cctbxbb mailing list
> >> > cctbxbb(a)phenix-online.org
> >> > http://phenix-online.org/mailman/listinfo/cctbxbb
> >>
> >>
> >>
> >> --
> >> Gergely Katona, PhD
> >> Associate Professor
> >> Department of Chemistry and Molecular Biology, University of Gothenburg
> >> Box 462, 40530 Göteborg, Sweden
> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> >> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
> >>
> >> _______________________________________________
> >> cctbxbb mailing list
> >> cctbxbb(a)phenix-online.org
> >> http://phenix-online.org/mailman/listinfo/cctbxbb
> >
> > _______________________________________________
> > cctbxbb mailing list
> > cctbxbb(a)phenix-online.org
> > http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
>
> --
> Gergely Katona, PhD
> Associate Professor
> Department of Chemistry and Molecular Biology, University of Gothenburg
> Box 462, 40530 Göteborg, Sweden
> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910
> Web: http://katonalab.eu, Email: gergely.katona(a)gu.se
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
6 years, 1 month
Re: [cctbxbb] Boost Python 1.56
by Billy Poon
I can do the switch to Boost 1.56 next Wednesday now that Dials 1.5 has
been released. I'm just double-checking some other Phenix tests.
Once the update is live, bootstrap.py should delete the existing
modules/boost directory and replace it with the new one. And then scons
should recompile anything that depends on boost. If in doubt, you can
manually delete the module/boost and build directories.
--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709
Fax: (510) 486-5909
Web: https://phenix-online.org
On Thu, Apr 6, 2017 at 9:32 AM, Billy Poon <bkpoon(a)lbl.gov> wrote:
> Hi James,
>
> I agree with Graeme's tests and I would add
>
> cctbx_regression.test_nightly
>
> That command is a shortcut for running the test modules for libtbx,
> boost_adaptbx, scitbx, cctbx, iotbx, dxtbx, and smtbx. The mmtbx module is
> also tested if chem_data is available. Should we add rstbx to the shortcut
> (cctbx_project/cctbx/command_line/cctbx_test_nightly.py)?
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709
> Fax: (510) 486-5909
> Web: https://phenix-online.org
>
> On Thu, Apr 6, 2017 at 1:37 AM, <richard.gildea(a)diamond.ac.uk> wrote:
>
>> I've made a start on transcribing this document here:
>>
>> https://github.com/cctbx/cctbx_project/wiki/cctbx-Developer-Guidance
>>
>> It probably still needs cleaning up a bit (e.g. I couldn't figure out
>> quickly how to do 3-level list nesting, 1.a.i etc.) and updating to reflect
>> current practice (e.g. git not svn).
>>
>> Cheers,
>>
>> Richard
>>
>> Dr Richard Gildea
>> Data Analysis Scientist
>> Tel: +441235 77 8078
>>
>> Diamond Light Source Ltd.
>> Diamond House
>> Harwell Science & Innovation Campus
>> Didcot
>> Oxfordshire
>> OX11 0DE
>>
>> ________________________________________
>> From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces@phenix-online
>> .org] on behalf of Pavel Afonine [pafonine(a)lbl.gov]
>> Sent: 06 April 2017 09:07
>> To: cctbx mailing list
>> Subject: Re: [cctbxbb] Boost Python 1.56
>>
>> Hi Graeme,
>>
>> hm.. this is a good question. We've been through back-and-forth
>> iterations of editing this file and I think the latest I have is from
>> Paul. But I can't find a non-PDF version of it. Paul: do you have an
>> editable version of this file?
>>
>> Thanks,
>> Pavel
>>
>> On 4/6/17 13:45, Graeme.Winter(a)diamond.ac.uk wrote:
>> > Hi Pavel
>> >
>> > These all seem sensible
>> >
>> > If you have the original non pdf document it may be easier to
>> transcribe this over..
>> >
>> > I also note that it lacks the actual detail on how to run tests!
>> However would be happy to add this once on wiki
>> >
>> > Best wishes Graeme
>> >
>> > On 6 Apr 2017, at 04:00, Pavel Afonine <pafonine(a)lbl.gov<mailto:pafon
>> ine(a)lbl.gov>> wrote:
>> >
>> > Not sure if that answers your questions but once upon a time we here at
>> Berkeley tried to write a some sort of document that was supposed to answer
>> questions like this. Attached. By no means it is complete, up-to-date, etc,
>> but it might be worth reading for anyone who contributes to cctbx. (Even
>> not sure if I'm sending the latest version).
>> > Unfortunately, nobody bothered to put it in some central place.
>> >
>> > Pavel
>> >
>> > On 4/6/17 10:51, James Holton wrote:
>> > Hey Billy,
>> >
>> > On a related note. How do I run these regression tests before
>> committing something into Git? Is there a document on dials regression
>> testing I can't find?
>> >
>> > -James
>> >
>> > On Apr 5, 2017, at 3:38 PM, Billy Poon <bkpoon(a)lbl.gov<mailto:bkpoon@
>> lbl.gov>> wrote:
>> >
>> > Hi all,
>> >
>> > I tested Boost 1.56 on our buildbot servers and got some new test
>> failures with
>> >
>> > cctbx_project/scitbx/array_family/boost_python/tst_flex.py
>> > cctbx_project/scitbx/random/tests/tst_random.py
>> >
>> > The full log for CentOS 6 can be found at
>> >
>> > http://cci-vm-6.lbl.gov:8010/builders/phenix-nightly-intel-l
>> inux-2.6-x86_64-centos6/builds/601/steps/test%20cctbx_regres
>> sion.test_nightly/logs/stdio
>> >
>> > It looks like the errors are related to random number generation. For a
>> given seed, would the sequence of numbers change when Boost is changed? I
>> did a diff between Boost 1.56 and the current Boost and could not see any
>> changes that immediately stood out as being related to random numbers.
>> >
>> > Are these tests failing for others as well?
>> >
>> > --
>> > Billy K. Poon
>> > Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> > Lawrence Berkeley National Laboratory
>> > 1 Cyclotron Road, M/S 33R0345
>> > Berkeley, CA 94720
>> > Tel: (510) 486-5709
>> > Fax: (510) 486-5909
>> > Web: https://phenix-online.org<https://phenix-online.org/>
>> >
>> > On Wed, Apr 5, 2017 at 8:12 AM, Charles Ballard <
>> charles.ballard(a)stfc.ac.uk<mailto:[email protected]>> wrote:
>> > FYI, we (CCP4) have been using 1.56 for building cctbx/phaser/dials
>> for the last while with no issues. Don't know about 1.60, but 1.59 causes
>> issues with the boost python make_getter and make_setter (initialisation of
>> none const reference if the passed type is a temporary).
>> >
>> > Charles
>> >
>> > On 3 Apr 2017, at 14:31, Luc Bourhis wrote:
>> >
>> > Hi all,
>> >
>> > everybody seemed to agree but then it was proposed to move straight to
>> Boost 1.60, and this caused troubles. Could we consider again to move to at
>> least 1.56? As far as I can tell, this does not cause any issue and as
>> stated one year ago, it would help me and Olex 2.
>> >
>> > Thanks,
>> >
>> > Luc
>> >
>> > On 10 Feb 2016, at 15:17, Nicholas Sauter <nksauter(a)lbl.gov<mailto:
>> nksauter(a)lbl.gov>> wrote:
>> >
>> > Nigel, Billy & Aaron,
>> >
>> > I completely endorse this move to Boost 1.56. Can we update our build?
>> >
>> > Nick
>> >
>> > Nicholas K. Sauter, Ph. D.
>> > Computer Staff Scientist, Molecular Biophysics and Integrated
>> Bioimaging Division
>> > Lawrence Berkeley National Laboratory
>> > 1 Cyclotron Rd., Bldg. 33R0345
>> > Berkeley, CA 94720
>> > (510) 486-5713<tel:%28510%29%20486-5713>
>> >
>> > On Wed, Feb 10, 2016 at 2:41 PM, Luc Bourhis <luc_j_bourhis(a)mac.com
>> <mailto:[email protected]>> wrote:
>> > Hi,
>> >
>> > I have improvements to the smtbx on their way to be committed which
>> require Boost version 1.56. This is related to Boost.Threads, whose support
>> I re-activated a few months ago on Nick’s request. I need the function
>> boost::thread::physical_concurrency which returns the number of physical
>> cores on the machine, as opposed to virtual cores when hyperthreading is
>> enabled (which it is by default on any Intel machine). That function is not
>> available in Boost 1.55 which is the version currently used in the nightly
>> tests: it appeared in 1.56.
>> >
>> > So, would it be possible to move to Boost 1.56? Otherwise, I will need
>> to backport that function. Not too difficult but not thrilling.
>> >
>> > Best wishes,
>> >
>> > Luc
>> >
>> >
>> > _______________________________________________
>> > cctbxbb mailing list
>> > cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> >
>> > _______________________________________________
>> > cctbxbb mailing list
>> > cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> >
>> > _______________________________________________
>> > cctbxbb mailing list
>> > cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> >
>> >
>> > _______________________________________________
>> > cctbxbb mailing list
>> > cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> >
>> >
>> > _______________________________________________
>> > cctbxbb mailing list
>> > cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> >
>> >
>> >
>> > _______________________________________________
>> > cctbxbb mailing list
>> > cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> >
>> >
>> > <cctbx-developer-guidance-08-2015.pdf>______________________
>> _________________________
>> > cctbxbb mailing list
>> > cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> > http://phenix-online.org/mailman/listinfo/cctbxbb
>> >
>> >
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org
>> http://phenix-online.org/mailman/listinfo/cctbxbb
>>
>> --
>> This e-mail and any attachments may contain confidential, copyright and
>> or privileged material, and are for the use of the intended addressee only.
>> If you are not the intended addressee or an authorised recipient of the
>> addressee please notify us of receipt by returning the e-mail and do not
>> use, copy, retain, distribute or disclose the information in or attached to
>> the e-mail.
>> Any opinions expressed within this e-mail are those of the individual and
>> not necessarily of Diamond Light Source Ltd.
>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
>> attachments are free from viruses and we cannot accept liability for any
>> damage which you may sustain as a result of software viruses which may be
>> transmitted in or with the message.
>> Diamond Light Source Limited (company no. 4375679). Registered in England
>> and Wales with its registered office at Diamond House, Harwell Science and
>> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>>
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org
>> http://phenix-online.org/mailman/listinfo/cctbxbb
>>
>
>
8 years, 10 months
Re: [cctbxbb] Boost Python 1.56
by markus.gerstel＠diamond.ac.uk
Hi Graeme,
But Billy has essentially done this :)
rm -rv $(libtbx.show_build_path)
libtbx.bootstrap hot build --builder=dials
in your case. No warranty for anything though.
Jenkins seems to have mostly picked up the new library with no issues. A few jobs apparently have not yet picked it up, but should do so soon by themselves.
-Markus
-----Original Message-----
From: cctbxbb-bounces(a)phenix-online.org [mailto:[email protected]] On Behalf Of Graeme.Winter(a)diamond.ac.uk
Sent: 13 April 2017 08:11
To: cctbxbb(a)phenix-online.org
Subject: Re: [cctbxbb] Boost Python 1.56
Colleagues
For those of us who are elderly and hard of thinking 😉 please could you send a relatively complete set of instructions on how best to update i.e. Using bootstrap or whatever.
5 line shell is fine
My usual recipe of redo from start is kind of annoying
Thanks Graeme
On 12 Apr 2017, at 21:48, Billy Poon <bkpoon(a)lbl.gov<mailto:[email protected]>> wrote:
Hi everyone,
Boost 1.56 is now available and the tests have been updated. To update an existing installtion, you have to delete the "build" directory first. If you don't, I guess some files are not recompiled with the new Boost headers, which will cause cctbx_project/scitbx/random/tests/tst_random.py to expect the old values. Then run bootstrap.py with hot, update, and build to rebuild.
cd <installation directory>
rm -fr build
python bootstrap.py hot update build --builder=cctbx
Let me know if there are any issues. Thanks!
--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709
Fax: (510) 486-5909
Web: https://phenix-online.org
On Fri, Apr 7, 2017 at 1:08 PM, Billy Poon <bkpoon(a)lbl.gov<mailto:[email protected]>> wrote:
I can do the switch to Boost 1.56 next Wednesday now that Dials 1.5 has been released. I'm just double-checking some other Phenix tests.
Once the update is live, bootstrap.py should delete the existing modules/boost directory and replace it with the new one. And then scons should recompile anything that depends on boost. If in doubt, you can manually delete the module/boost and build directories.
--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709<tel:(510)%20486-5709>
Fax: (510) 486-5909<tel:(510)%20486-5909>
Web: https://phenix-online.org
On Thu, Apr 6, 2017 at 9:32 AM, Billy Poon <bkpoon(a)lbl.gov<mailto:[email protected]>> wrote:
Hi James,
I agree with Graeme's tests and I would add
cctbx_regression.test_nightly
That command is a shortcut for running the test modules for libtbx, boost_adaptbx, scitbx, cctbx, iotbx, dxtbx, and smtbx. The mmtbx module is also tested if chem_data is available. Should we add rstbx to the shortcut (cctbx_project/cctbx/command_line/cctbx_test_nightly.py)?
--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709<tel:(510)%20486-5709>
Fax: (510) 486-5909<tel:(510)%20486-5909>
Web: https://phenix-online.org
On Thu, Apr 6, 2017 at 1:37 AM, <richard.gildea(a)diamond.ac.uk<mailto:[email protected]>> wrote:
I've made a start on transcribing this document here:
https://github.com/cctbx/cctbx_project/wiki/cctbx-Developer-Guidance
It probably still needs cleaning up a bit (e.g. I couldn't figure out quickly how to do 3-level list nesting, 1.a.i etc.) and updating to reflect current practice (e.g. git not svn).
Cheers,
Richard
Dr Richard Gildea
Data Analysis Scientist
Tel: +441235 77 8078<tel:%2B441235%2077%208078>
Diamond Light Source Ltd.
Diamond House
Harwell Science & Innovation Campus
Didcot
Oxfordshire
OX11 0DE
________________________________________
From: cctbxbb-bounces(a)phenix-online.org<mailto:[email protected]> [cctbxbb-bounces(a)phenix-online.org<mailto:[email protected]>] on behalf of Pavel Afonine [pafonine(a)lbl.gov<mailto:[email protected]>]
Sent: 06 April 2017 09:07
To: cctbx mailing list
Subject: Re: [cctbxbb] Boost Python 1.56
Hi Graeme,
hm.. this is a good question. We've been through back-and-forth iterations of editing this file and I think the latest I have is from Paul. But I can't find a non-PDF version of it. Paul: do you have an editable version of this file?
Thanks,
Pavel
On 4/6/17 13:45, Graeme.Winter(a)diamond.ac.uk<mailto:[email protected]> wrote:
> Hi Pavel
>
> These all seem sensible
>
> If you have the original non pdf document it may be easier to transcribe this over..
>
> I also note that it lacks the actual detail on how to run tests!
> However would be happy to add this once on wiki
>
> Best wishes Graeme
>
> On 6 Apr 2017, at 04:00, Pavel Afonine <pafonine(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote:
>
> Not sure if that answers your questions but once upon a time we here at Berkeley tried to write a some sort of document that was supposed to answer questions like this. Attached. By no means it is complete, up-to-date, etc, but it might be worth reading for anyone who contributes to cctbx. (Even not sure if I'm sending the latest version).
> Unfortunately, nobody bothered to put it in some central place.
>
> Pavel
>
> On 4/6/17 10:51, James Holton wrote:
> Hey Billy,
>
> On a related note. How do I run these regression tests before committing something into Git? Is there a document on dials regression testing I can't find?
>
> -James
>
> On Apr 5, 2017, at 3:38 PM, Billy Poon <bkpoon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote:
>
> Hi all,
>
> I tested Boost 1.56 on our buildbot servers and got some new test
> failures with
>
> cctbx_project/scitbx/array_family/boost_python/tst_flex.py
> cctbx_project/scitbx/random/tests/tst_random.py
>
> The full log for CentOS 6 can be found at
>
> http://cci-vm-6.lbl.gov:8010/builders/phenix-nightly-intel-linux-2.6-x
> 86_64-centos6/builds/601/steps/test%20cctbx_regression.test_nightly/lo
> gs/stdio
>
> It looks like the errors are related to random number generation. For a given seed, would the sequence of numbers change when Boost is changed? I did a diff between Boost 1.56 and the current Boost and could not see any changes that immediately stood out as being related to random numbers.
>
> Are these tests failing for others as well?
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709<tel:%28510%29%20486-5709>
> Fax: (510) 486-5909<tel:%28510%29%20486-5909>
> Web: https://phenix-online.org<https://phenix-online.org/>
>
> On Wed, Apr 5, 2017 at 8:12 AM, Charles Ballard <charles.ballard(a)stfc.ac.uk<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote:
> FYI, we (CCP4) have been using 1.56 for building cctbx/phaser/dials for the last while with no issues. Don't know about 1.60, but 1.59 causes issues with the boost python make_getter and make_setter (initialisation of none const reference if the passed type is a temporary).
>
> Charles
>
> On 3 Apr 2017, at 14:31, Luc Bourhis wrote:
>
> Hi all,
>
> everybody seemed to agree but then it was proposed to move straight to Boost 1.60, and this caused troubles. Could we consider again to move to at least 1.56? As far as I can tell, this does not cause any issue and as stated one year ago, it would help me and Olex 2.
>
> Thanks,
>
> Luc
>
> On 10 Feb 2016, at 15:17, Nicholas Sauter <nksauter(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote:
>
> Nigel, Billy & Aaron,
>
> I completely endorse this move to Boost 1.56. Can we update our build?
>
> Nick
>
> Nicholas K. Sauter, Ph. D.
> Computer Staff Scientist, Molecular Biophysics and Integrated
> Bioimaging Division Lawrence Berkeley National Laboratory
> 1 Cyclotron Rd., Bldg. 33R0345
> Berkeley, CA 94720
> (510) 486-5713<tel:%28510%29%20486-5713><tel:%28510%29%20486-5713>
>
> On Wed, Feb 10, 2016 at 2:41 PM, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote:
> Hi,
>
> I have improvements to the smtbx on their way to be committed which require Boost version 1.56. This is related to Boost.Threads, whose support I re-activated a few months ago on Nick’s request. I need the function boost::thread::physical_concurrency which returns the number of physical cores on the machine, as opposed to virtual cores when hyperthreading is enabled (which it is by default on any Intel machine). That function is not available in Boost 1.55 which is the version currently used in the nightly tests: it appeared in 1.56.
>
> So, would it be possible to move to Boost 1.56? Otherwise, I will need to backport that function. Not too difficult but not thrilling.
>
> Best wishes,
>
> Luc
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:cct
> bxbb(a)phenix-online.org<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:cct
> bxbb(a)phenix-online.org<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:cct
> bxbb(a)phenix-online.org<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:cct
> bxbb(a)phenix-online.org<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:cct
> bxbb(a)phenix-online.org<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:cct
> bxbb(a)phenix-online.org<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> <cctbx-developer-guidance-08-2015.pdf>________________________________
> _______________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:cct
> bxbb(a)phenix-online.org<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
_______________________________________________
cctbxbb mailing list
cctbxbb(a)phenix-online.org<mailto:[email protected]>
http://phenix-online.org/mailman/listinfo/cctbxbb
--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
_______________________________________________
cctbxbb mailing list
cctbxbb(a)phenix-online.org<mailto:[email protected]>
http://phenix-online.org/mailman/listinfo/cctbxbb
_______________________________________________
cctbxbb mailing list
cctbxbb(a)phenix-online.org<mailto:[email protected]>
http://phenix-online.org/mailman/listinfo/cctbxbb
_______________________________________________
cctbxbb mailing list
cctbxbb(a)phenix-online.org
http://phenix-online.org/mailman/listinfo/cctbxbb
8 years, 9 months
Re: [cctbxbb] Boost Python 1.56
by R.D. Oeffner
Hi Billy,
As you'll know the Windows builds failed with the new boost libraries.
this is due to some filenames exceeding the 255 char limit on Windows.
It worked fine on my nightly build because the build folder isn't as
deeply nested as on pulse and flux. On pulse and flux it dies in
tarfile.TarFile._extract_member() which I have made a monkey patch for
which appears to work by ignoring the offending files.
However, as all the long filenames seem to come from html folders within
boost presumably these are just documentation and it would be better to
exclude these altogether in the cci subset of boost. People can always
go online if they want to read up on the documentation for boost.
Besides bootstrap.py is already quite bloated.
Let me know what you think,
Rob
On 12/04/2017 21:47, Billy Poon wrote:
> Hi everyone,
>
> Boost 1.56 is now available and the tests have been updated. To update an existing installtion, you have to delete the "build" directory first. If you don't, I guess some files are not recompiled with the new Boost headers, which will cause cctbx_project/scitbx/random/tests/tst_random.py to expect the old values. Then run bootstrap.py with hot, update, and build to rebuild.
>
> cd <installation directory>
> rm -fr build
> python bootstrap.py hot update build --builder=cctbx
>
> Let me know if there are any issues. Thanks!
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709
> Fax: (510) 486-5909
> Web: https://phenix-online.org [1]
>
> On Fri, Apr 7, 2017 at 1:08 PM, Billy Poon <bkpoon(a)lbl.gov> wrote:
>
> I can do the switch to Boost 1.56 next Wednesday now that Dials 1.5 has been released. I'm just double-checking some other Phenix tests.
>
> Once the update is live, bootstrap.py should delete the existing modules/boost directory and replace it with the new one. And then scons should recompile anything that depends on boost. If in doubt, you can manually delete the module/boost and build directories.
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709 [2]
> Fax: (510) 486-5909 [3]
> Web: https://phenix-online.org [1]
>
> On Thu, Apr 6, 2017 at 9:32 AM, Billy Poon <bkpoon(a)lbl.gov> wrote:
>
> Hi James,
>
> I agree with Graeme's tests and I would add
>
> cctbx_regression.test_nightly
>
> That command is a shortcut for running the test modules for libtbx, boost_adaptbx, scitbx, cctbx, iotbx, dxtbx, and smtbx. The mmtbx module is also tested if chem_data is available. Should we add rstbx to the shortcut (cctbx_project/cctbx/command_line/cctbx_test_nightly.py)?
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709 [2]
> Fax: (510) 486-5909 [3]
> Web: https://phenix-online.org [1]
>
> On Thu, Apr 6, 2017 at 1:37 AM, <richard.gildea(a)diamond.ac.uk> wrote:
> I've made a start on transcribing this document here:
>
> https://github.com/cctbx/cctbx_project/wiki/cctbx-Developer-Guidance [4]
>
> It probably still needs cleaning up a bit (e.g. I couldn't figure out quickly how to do 3-level list nesting, 1.a.i etc.) and updating to reflect current practice (e.g. git not svn).
>
> Cheers,
>
> Richard
>
> Dr Richard Gildea
> Data Analysis Scientist
> Tel: +441235 77 8078 [5]
>
> Diamond Light Source Ltd.
> Diamond House
> Harwell Science & Innovation Campus
> Didcot
> Oxfordshire
> OX11 0DE
>
> ________________________________________
> From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of Pavel Afonine [pafonine(a)lbl.gov]
> Sent: 06 April 2017 09:07
> To: cctbx mailing list
> Subject: Re: [cctbxbb] Boost Python 1.56
>
> Hi Graeme,
>
> hm.. this is a good question. We've been through back-and-forth
> iterations of editing this file and I think the latest I have is from
> Paul. But I can't find a non-PDF version of it. Paul: do you have an
> editable version of this file?
>
> Thanks,
> Pavel
>
> On 4/6/17 13:45, Graeme.Winter(a)diamond.ac.uk wrote:
>> Hi Pavel
>>
>> These all seem sensible
>>
>> If you have the original non pdf document it may be easier to transcribe this over..
>>
>> I also note that it lacks the actual detail on how to run tests! However would be happy to add this once on wiki
>>
>> Best wishes Graeme
>>
>> On 6 Apr 2017, at 04:00, Pavel Afonine <pafonine(a)lbl.gov<mailto:[email protected]>> wrote:
>>
>> Not sure if that answers your questions but once upon a time we here at Berkeley tried to write a some sort of document that was supposed to answer questions like this. Attached. By no means it is complete, up-to-date, etc, but it might be worth reading for anyone who contributes to cctbx. (Even not sure if I'm sending the latest version).
>> Unfortunately, nobody bothered to put it in some central place.
>>
>> Pavel
>>
>> On 4/6/17 10:51, James Holton wrote:
>> Hey Billy,
>>
>> On a related note. How do I run these regression tests before committing something into Git? Is there a document on dials regression testing I can't find?
>>
>> -James
>>
>> On Apr 5, 2017, at 3:38 PM, Billy Poon <bkpoon(a)lbl.gov<mailto:[email protected]>> wrote:
>>
>> Hi all,
>>
>> I tested Boost 1.56 on our buildbot servers and got some new test failures with
>>
>> cctbx_project/scitbx/array_family/boost_python/tst_flex.py
>> cctbx_project/scitbx/random/tests/tst_random.py
>>
>> The full log for CentOS 6 can be found at
>>
>> http://cci-vm-6.lbl.gov:8010/builders/phenix-nightly-intel-linux-2.6-x86_64… [6]
>>
>> It looks like the errors are related to random number generation. For a given seed, would the sequence of numbers change when Boost is changed? I did a diff between Boost 1.56 and the current Boost and could not see any changes that immediately stood out as being related to random numbers.
>>
>> Are these tests failing for others as well?
>>
>> --
>> Billy K. Poon
>> Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> Lawrence Berkeley National Laboratory
>> 1 Cyclotron Road, M/S 33R0345
>> Berkeley, CA 94720
>> Tel: (510) 486-5709 [7]
>> Fax: (510) 486-5909 [8]
>> Web: https://phenix-online.org [1]<https://phenix-online.org/ [9]>
>>
>> On Wed, Apr 5, 2017 at 8:12 AM, Charles Ballard <charles.ballard(a)stfc.ac.uk<mailto:[email protected]>> wrote:
>> FYI, we (CCP4) have been using 1.56 for building cctbx/phaser/dials for the last while with no issues. Don't know about 1.60, but 1.59 causes issues with the boost python make_getter and make_setter (initialisation of none const reference if the passed type is a temporary).
>>
>> Charles
>>
>> On 3 Apr 2017, at 14:31, Luc Bourhis wrote:
>>
>> Hi all,
>>
>> everybody seemed to agree but then it was proposed to move straight to Boost 1.60, and this caused troubles. Could we consider again to move to at least 1.56? As far as I can tell, this does not cause any issue and as stated one year ago, it would help me and Olex 2.
>>
>> Thanks,
>>
>> Luc
>>
>> On 10 Feb 2016, at 15:17, Nicholas Sauter <nksauter(a)lbl.gov<mailto:[email protected]>> wrote:
>>
>> Nigel, Billy & Aaron,
>>
>> I completely endorse this move to Boost 1.56. Can we update our build?
>>
>> Nick
>>
>> Nicholas K. Sauter, Ph. D.
>> Computer Staff Scientist, Molecular Biophysics and Integrated Bioimaging Division
>> Lawrence Berkeley National Laboratory
>> 1 Cyclotron Rd., Bldg. 33R0345
>> Berkeley, CA 94720
>> (510) 486-5713 [10]<tel:%28510%29%20486-5713>
>>
>> On Wed, Feb 10, 2016 at 2:41 PM, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote:
>> Hi,
>>
>> I have improvements to the smtbx on their way to be committed which require Boost version 1.56. This is related to Boost.Threads, whose support I re-activated a few months ago on Nick's request. I need the function boost::thread::physical_concurrency which returns the number of physical cores on the machine, as opposed to virtual cores when hyperthreading is enabled (which it is by default on any Intel machine). That function is not available in Boost 1.55 which is the version currently used in the nightly tests: it appeared in 1.56.
>>
>> So, would it be possible to move to Boost 1.56? Otherwise, I will need to backport that function. Not too difficult but not thrilling.
>>
>> Best wishes,
>>
>> Luc
>>
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> http://phenix-online.org/mailman/listinfo/cctbxbb [11]
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> http://phenix-online.org/mailman/listinfo/cctbxbb [11]
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> http://phenix-online.org/mailman/listinfo/cctbxbb [11]
>>
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> http://phenix-online.org/mailman/listinfo/cctbxbb [11]
>>
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> http://phenix-online.org/mailman/listinfo/cctbxbb [11]
>>
>>
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> http://phenix-online.org/mailman/listinfo/cctbxbb [11]
>>
>>
>> <cctbx-developer-guidance-08-2015.pdf>_______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> http://phenix-online.org/mailman/listinfo/cctbxbb [11]
>>
>>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb [11]
>
> --
> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb [11]
_______________________________________________
cctbxbb mailing list
cctbxbb(a)phenix-online.org
http://phenix-online.org/mailman/listinfo/cctbxbb [11]
Links:
------
[1] https://phenix-online.org
[2] tel:(510)%20486-5709
[3] tel:(510)%20486-5909
[4] https://github.com/cctbx/cctbx_project/wiki/cctbx-Developer-Guidance
[5] tel:%2B441235%2077%208078
[6]
http://cci-vm-6.lbl.gov:8010/builders/phenix-nightly-intel-linux-2.6-x86_64…
[7] tel:%28510%29%20486-5709
[8] tel:%28510%29%20486-5909
[9] https://phenix-online.org/
[10] tel:%28510%29%20486-5713
[11] http://phenix-online.org/mailman/listinfo/cctbxbb
8 years, 9 months
Re: [cctbxbb] Boost Python 1.56
by markus.gerstel＠diamond.ac.uk
Okay, let me correct myself here since libtbx.bootstrap fails after deleting the build directory (surprise)...
cd $(libtbx.show_build_path)/..
rm -rv build
cp modules/cctbx_project/libtbx/auto_build/bootstrap.py .
python bootstrap.py hot build --builder=dials
you may then have to do
cd build
libtbx.configure xia2 xia2_regression dials_regression # (...)
make
# and, assuming you are in the DLS network:
modules/xia2_regression/command_line/fetch_test_data_local.sh
and that should be it.
-Markus
-----Original Message-----
From: cctbxbb-bounces(a)phenix-online.org [mailto:[email protected]] On Behalf Of markus.gerstel(a)diamond.ac.uk
Sent: 13 April 2017 08:20
To: cctbxbb(a)phenix-online.org
Subject: Re: [cctbxbb] Boost Python 1.56
Hi Graeme,
But Billy has essentially done this :)
rm -rv $(libtbx.show_build_path)
libtbx.bootstrap hot build --builder=dials
in your case. No warranty for anything though.
Jenkins seems to have mostly picked up the new library with no issues. A few jobs apparently have not yet picked it up, but should do so soon by themselves.
-Markus
-----Original Message-----
From: cctbxbb-bounces(a)phenix-online.org [mailto:[email protected]] On Behalf Of Graeme.Winter(a)diamond.ac.uk
Sent: 13 April 2017 08:11
To: cctbxbb(a)phenix-online.org
Subject: Re: [cctbxbb] Boost Python 1.56
Colleagues
For those of us who are elderly and hard of thinking 😉 please could you send a relatively complete set of instructions on how best to update i.e. Using bootstrap or whatever.
5 line shell is fine
My usual recipe of redo from start is kind of annoying
Thanks Graeme
On 12 Apr 2017, at 21:48, Billy Poon <bkpoon(a)lbl.gov<mailto:[email protected]>> wrote:
Hi everyone,
Boost 1.56 is now available and the tests have been updated. To update an existing installtion, you have to delete the "build" directory first. If you don't, I guess some files are not recompiled with the new Boost headers, which will cause cctbx_project/scitbx/random/tests/tst_random.py to expect the old values. Then run bootstrap.py with hot, update, and build to rebuild.
cd <installation directory>
rm -fr build
python bootstrap.py hot update build --builder=cctbx
Let me know if there are any issues. Thanks!
--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709
Fax: (510) 486-5909
Web: https://phenix-online.org
On Fri, Apr 7, 2017 at 1:08 PM, Billy Poon <bkpoon(a)lbl.gov<mailto:[email protected]>> wrote:
I can do the switch to Boost 1.56 next Wednesday now that Dials 1.5 has been released. I'm just double-checking some other Phenix tests.
Once the update is live, bootstrap.py should delete the existing modules/boost directory and replace it with the new one. And then scons should recompile anything that depends on boost. If in doubt, you can manually delete the module/boost and build directories.
--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709<tel:(510)%20486-5709>
Fax: (510) 486-5909<tel:(510)%20486-5909>
Web: https://phenix-online.org
On Thu, Apr 6, 2017 at 9:32 AM, Billy Poon <bkpoon(a)lbl.gov<mailto:[email protected]>> wrote:
Hi James,
I agree with Graeme's tests and I would add
cctbx_regression.test_nightly
That command is a shortcut for running the test modules for libtbx, boost_adaptbx, scitbx, cctbx, iotbx, dxtbx, and smtbx. The mmtbx module is also tested if chem_data is available. Should we add rstbx to the shortcut (cctbx_project/cctbx/command_line/cctbx_test_nightly.py)?
--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709<tel:(510)%20486-5709>
Fax: (510) 486-5909<tel:(510)%20486-5909>
Web: https://phenix-online.org
On Thu, Apr 6, 2017 at 1:37 AM, <richard.gildea(a)diamond.ac.uk<mailto:[email protected]>> wrote:
I've made a start on transcribing this document here:
https://github.com/cctbx/cctbx_project/wiki/cctbx-Developer-Guidance
It probably still needs cleaning up a bit (e.g. I couldn't figure out quickly how to do 3-level list nesting, 1.a.i etc.) and updating to reflect current practice (e.g. git not svn).
Cheers,
Richard
Dr Richard Gildea
Data Analysis Scientist
Tel: +441235 77 8078<tel:%2B441235%2077%208078>
Diamond Light Source Ltd.
Diamond House
Harwell Science & Innovation Campus
Didcot
Oxfordshire
OX11 0DE
________________________________________
From: cctbxbb-bounces(a)phenix-online.org<mailto:[email protected]> [cctbxbb-bounces(a)phenix-online.org<mailto:[email protected]>] on behalf of Pavel Afonine [pafonine(a)lbl.gov<mailto:[email protected]>]
Sent: 06 April 2017 09:07
To: cctbx mailing list
Subject: Re: [cctbxbb] Boost Python 1.56
Hi Graeme,
hm.. this is a good question. We've been through back-and-forth iterations of editing this file and I think the latest I have is from Paul. But I can't find a non-PDF version of it. Paul: do you have an editable version of this file?
Thanks,
Pavel
On 4/6/17 13:45, Graeme.Winter(a)diamond.ac.uk<mailto:[email protected]> wrote:
> Hi Pavel
>
> These all seem sensible
>
> If you have the original non pdf document it may be easier to transcribe this over..
>
> I also note that it lacks the actual detail on how to run tests!
> However would be happy to add this once on wiki
>
> Best wishes Graeme
>
> On 6 Apr 2017, at 04:00, Pavel Afonine <pafonine(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote:
>
> Not sure if that answers your questions but once upon a time we here at Berkeley tried to write a some sort of document that was supposed to answer questions like this. Attached. By no means it is complete, up-to-date, etc, but it might be worth reading for anyone who contributes to cctbx. (Even not sure if I'm sending the latest version).
> Unfortunately, nobody bothered to put it in some central place.
>
> Pavel
>
> On 4/6/17 10:51, James Holton wrote:
> Hey Billy,
>
> On a related note. How do I run these regression tests before committing something into Git? Is there a document on dials regression testing I can't find?
>
> -James
>
> On Apr 5, 2017, at 3:38 PM, Billy Poon <bkpoon(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote:
>
> Hi all,
>
> I tested Boost 1.56 on our buildbot servers and got some new test
> failures with
>
> cctbx_project/scitbx/array_family/boost_python/tst_flex.py
> cctbx_project/scitbx/random/tests/tst_random.py
>
> The full log for CentOS 6 can be found at
>
> http://cci-vm-6.lbl.gov:8010/builders/phenix-nightly-intel-linux-2.6-x
> 86_64-centos6/builds/601/steps/test%20cctbx_regression.test_nightly/lo
> gs/stdio
>
> It looks like the errors are related to random number generation. For a given seed, would the sequence of numbers change when Boost is changed? I did a diff between Boost 1.56 and the current Boost and could not see any changes that immediately stood out as being related to random numbers.
>
> Are these tests failing for others as well?
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709<tel:%28510%29%20486-5709>
> Fax: (510) 486-5909<tel:%28510%29%20486-5909>
> Web: https://phenix-online.org<https://phenix-online.org/>
>
> On Wed, Apr 5, 2017 at 8:12 AM, Charles Ballard <charles.ballard(a)stfc.ac.uk<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote:
> FYI, we (CCP4) have been using 1.56 for building cctbx/phaser/dials for the last while with no issues. Don't know about 1.60, but 1.59 causes issues with the boost python make_getter and make_setter (initialisation of none const reference if the passed type is a temporary).
>
> Charles
>
> On 3 Apr 2017, at 14:31, Luc Bourhis wrote:
>
> Hi all,
>
> everybody seemed to agree but then it was proposed to move straight to Boost 1.60, and this caused troubles. Could we consider again to move to at least 1.56? As far as I can tell, this does not cause any issue and as stated one year ago, it would help me and Olex 2.
>
> Thanks,
>
> Luc
>
> On 10 Feb 2016, at 15:17, Nicholas Sauter <nksauter(a)lbl.gov<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote:
>
> Nigel, Billy & Aaron,
>
> I completely endorse this move to Boost 1.56. Can we update our build?
>
> Nick
>
> Nicholas K. Sauter, Ph. D.
> Computer Staff Scientist, Molecular Biophysics and Integrated
> Bioimaging Division Lawrence Berkeley National Laboratory
> 1 Cyclotron Rd., Bldg. 33R0345
> Berkeley, CA 94720
> (510) 486-5713<tel:%28510%29%20486-5713><tel:%28510%29%20486-5713>
>
> On Wed, Feb 10, 2016 at 2:41 PM, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote:
> Hi,
>
> I have improvements to the smtbx on their way to be committed which require Boost version 1.56. This is related to Boost.Threads, whose support I re-activated a few months ago on Nick’s request. I need the function boost::thread::physical_concurrency which returns the number of physical cores on the machine, as opposed to virtual cores when hyperthreading is enabled (which it is by default on any Intel machine). That function is not available in Boost 1.55 which is the version currently used in the nightly tests: it appeared in 1.56.
>
> So, would it be possible to move to Boost 1.56? Otherwise, I will need to backport that function. Not too difficult but not thrilling.
>
> Best wishes,
>
> Luc
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:cct
> bxbb(a)phenix-online.org<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:cct
> bxbb(a)phenix-online.org<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:cct
> bxbb(a)phenix-online.org<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:cct
> bxbb(a)phenix-online.org<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:cct
> bxbb(a)phenix-online.org<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:cct
> bxbb(a)phenix-online.org<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
> <cctbx-developer-guidance-08-2015.pdf>________________________________
> _______________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org<mailto:[email protected]><mailto:cct
> bxbb(a)phenix-online.org<mailto:[email protected]>>
> http://phenix-online.org/mailman/listinfo/cctbxbb
>
>
_______________________________________________
cctbxbb mailing list
cctbxbb(a)phenix-online.org<mailto:[email protected]>
http://phenix-online.org/mailman/listinfo/cctbxbb
--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
_______________________________________________
cctbxbb mailing list
cctbxbb(a)phenix-online.org<mailto:[email protected]>
http://phenix-online.org/mailman/listinfo/cctbxbb
_______________________________________________
cctbxbb mailing list
cctbxbb(a)phenix-online.org<mailto:[email protected]>
http://phenix-online.org/mailman/listinfo/cctbxbb
_______________________________________________
cctbxbb mailing list
cctbxbb(a)phenix-online.org
http://phenix-online.org/mailman/listinfo/cctbxbb
_______________________________________________
cctbxbb mailing list
cctbxbb(a)phenix-online.org
http://phenix-online.org/mailman/listinfo/cctbxbb
8 years, 9 months
Re: [cctbxbb] Boost Python 1.56
by R.D. Oeffner
Sorry this was to Billy only. Please ignore.
Rob
On 13/04/2017 16:00, R.D. Oeffner wrote:
> Hi Billy,
>
> As you'll know the Windows builds failed with the new boost libraries. this is due to some filenames exceeding the 255 char limit on Windows. It worked fine on my nightly build because the build folder isn't as deeply nested as on pulse and flux. On pulse and flux it dies in tarfile.TarFile._extract_member() which I have made a monkey patch for which appears to work by ignoring the offending files.
>
> However, as all the long filenames seem to come from html folders within boost presumably these are just documentation and it would be better to exclude these altogether in the cci subset of boost. People can always go online if they want to read up on the documentation for boost. Besides bootstrap.py is already quite bloated.
>
> Let me know what you think,
>
> Rob
>
> On 12/04/2017 21:47, Billy Poon wrote:
> Hi everyone,
>
> Boost 1.56 is now available and the tests have been updated. To update an existing installtion, you have to delete the "build" directory first. If you don't, I guess some files are not recompiled with the new Boost headers, which will cause cctbx_project/scitbx/random/tests/tst_random.py to expect the old values. Then run bootstrap.py with hot, update, and build to rebuild.
>
> cd <installation directory>
> rm -fr build
> python bootstrap.py hot update build --builder=cctbx
>
> Let me know if there are any issues. Thanks!
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709
> Fax: (510) 486-5909
> Web: https://phenix-online.org [1]
>
> On Fri, Apr 7, 2017 at 1:08 PM, Billy Poon <bkpoon(a)lbl.gov> wrote:
>
> I can do the switch to Boost 1.56 next Wednesday now that Dials 1.5 has been released. I'm just double-checking some other Phenix tests.
>
> Once the update is live, bootstrap.py should delete the existing modules/boost directory and replace it with the new one. And then scons should recompile anything that depends on boost. If in doubt, you can manually delete the module/boost and build directories.
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709 [2]
> Fax: (510) 486-5909 [3]
> Web: https://phenix-online.org [1]
>
> On Thu, Apr 6, 2017 at 9:32 AM, Billy Poon <bkpoon(a)lbl.gov> wrote:
>
> Hi James,
>
> I agree with Graeme's tests and I would add
>
> cctbx_regression.test_nightly
>
> That command is a shortcut for running the test modules for libtbx, boost_adaptbx, scitbx, cctbx, iotbx, dxtbx, and smtbx. The mmtbx module is also tested if chem_data is available. Should we add rstbx to the shortcut (cctbx_project/cctbx/command_line/cctbx_test_nightly.py)?
>
> --
> Billy K. Poon
> Research Scientist, Molecular Biophysics and Integrated Bioimaging
> Lawrence Berkeley National Laboratory
> 1 Cyclotron Road, M/S 33R0345
> Berkeley, CA 94720
> Tel: (510) 486-5709 [2]
> Fax: (510) 486-5909 [3]
> Web: https://phenix-online.org [1]
>
> On Thu, Apr 6, 2017 at 1:37 AM, <richard.gildea(a)diamond.ac.uk> wrote:
> I've made a start on transcribing this document here:
>
> https://github.com/cctbx/cctbx_project/wiki/cctbx-Developer-Guidance [4]
>
> It probably still needs cleaning up a bit (e.g. I couldn't figure out quickly how to do 3-level list nesting, 1.a.i etc.) and updating to reflect current practice (e.g. git not svn).
>
> Cheers,
>
> Richard
>
> Dr Richard Gildea
> Data Analysis Scientist
> Tel: +441235 77 8078 [5]
>
> Diamond Light Source Ltd.
> Diamond House
> Harwell Science & Innovation Campus
> Didcot
> Oxfordshire
> OX11 0DE
>
> ________________________________________
> From: cctbxbb-bounces(a)phenix-online.org [cctbxbb-bounces(a)phenix-online.org] on behalf of Pavel Afonine [pafonine(a)lbl.gov]
> Sent: 06 April 2017 09:07
> To: cctbx mailing list
> Subject: Re: [cctbxbb] Boost Python 1.56
>
> Hi Graeme,
>
> hm.. this is a good question. We've been through back-and-forth
> iterations of editing this file and I think the latest I have is from
> Paul. But I can't find a non-PDF version of it. Paul: do you have an
> editable version of this file?
>
> Thanks,
> Pavel
>
> On 4/6/17 13:45, Graeme.Winter(a)diamond.ac.uk wrote:
>> Hi Pavel
>>
>> These all seem sensible
>>
>> If you have the original non pdf document it may be easier to transcribe this over..
>>
>> I also note that it lacks the actual detail on how to run tests! However would be happy to add this once on wiki
>>
>> Best wishes Graeme
>>
>> On 6 Apr 2017, at 04:00, Pavel Afonine <pafonine(a)lbl.gov<mailto:[email protected]>> wrote:
>>
>> Not sure if that answers your questions but once upon a time we here at Berkeley tried to write a some sort of document that was supposed to answer questions like this. Attached. By no means it is complete, up-to-date, etc, but it might be worth reading for anyone who contributes to cctbx. (Even not sure if I'm sending the latest version).
>> Unfortunately, nobody bothered to put it in some central place.
>>
>> Pavel
>>
>> On 4/6/17 10:51, James Holton wrote:
>> Hey Billy,
>>
>> On a related note. How do I run these regression tests before committing something into Git? Is there a document on dials regression testing I can't find?
>>
>> -James
>>
>> On Apr 5, 2017, at 3:38 PM, Billy Poon <bkpoon(a)lbl.gov<mailto:[email protected]>> wrote:
>>
>> Hi all,
>>
>> I tested Boost 1.56 on our buildbot servers and got some new test failures with
>>
>> cctbx_project/scitbx/array_family/boost_python/tst_flex.py
>> cctbx_project/scitbx/random/tests/tst_random.py
>>
>> The full log for CentOS 6 can be found at
>>
>> http://cci-vm-6.lbl.gov:8010/builders/phenix-nightly-intel-linux-2.6-x86_64… [6]
>>
>> It looks like the errors are related to random number generation. For a given seed, would the sequence of numbers change when Boost is changed? I did a diff between Boost 1.56 and the current Boost and could not see any changes that immediately stood out as being related to random numbers.
>>
>> Are these tests failing for others as well?
>>
>> --
>> Billy K. Poon
>> Research Scientist, Molecular Biophysics and Integrated Bioimaging
>> Lawrence Berkeley National Laboratory
>> 1 Cyclotron Road, M/S 33R0345
>> Berkeley, CA 94720
>> Tel: (510) 486-5709 [7]
>> Fax: (510) 486-5909 [8]
>> Web: https://phenix-online.org [1]<https://phenix-online.org/ [9]>
>>
>> On Wed, Apr 5, 2017 at 8:12 AM, Charles Ballard <charles.ballard(a)stfc.ac.uk<mailto:[email protected]>> wrote:
>> FYI, we (CCP4) have been using 1.56 for building cctbx/phaser/dials for the last while with no issues. Don't know about 1.60, but 1.59 causes issues with the boost python make_getter and make_setter (initialisation of none const reference if the passed type is a temporary).
>>
>> Charles
>>
>> On 3 Apr 2017, at 14:31, Luc Bourhis wrote:
>>
>> Hi all,
>>
>> everybody seemed to agree but then it was proposed to move straight to Boost 1.60, and this caused troubles. Could we consider again to move to at least 1.56? As far as I can tell, this does not cause any issue and as stated one year ago, it would help me and Olex 2.
>>
>> Thanks,
>>
>> Luc
>>
>> On 10 Feb 2016, at 15:17, Nicholas Sauter <nksauter(a)lbl.gov<mailto:[email protected]>> wrote:
>>
>> Nigel, Billy & Aaron,
>>
>> I completely endorse this move to Boost 1.56. Can we update our build?
>>
>> Nick
>>
>> Nicholas K. Sauter, Ph. D.
>> Computer Staff Scientist, Molecular Biophysics and Integrated Bioimaging Division
>> Lawrence Berkeley National Laboratory
>> 1 Cyclotron Rd., Bldg. 33R0345
>> Berkeley, CA 94720
>> (510) 486-5713 [10]<tel:%28510%29%20486-5713>
>>
>> On Wed, Feb 10, 2016 at 2:41 PM, Luc Bourhis <luc_j_bourhis(a)mac.com<mailto:[email protected]>> wrote:
>> Hi,
>>
>> I have improvements to the smtbx on their way to be committed which require Boost version 1.56. This is related to Boost.Threads, whose support I re-activated a few months ago on Nick's request. I need the function boost::thread::physical_concurrency which returns the number of physical cores on the machine, as opposed to virtual cores when hyperthreading is enabled (which it is by default on any Intel machine). That function is not available in Boost 1.55 which is the version currently used in the nightly tests: it appeared in 1.56.
>>
>> So, would it be possible to move to Boost 1.56? Otherwise, I will need to backport that function. Not too difficult but not thrilling.
>>
>> Best wishes,
>>
>> Luc
>>
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> http://phenix-online.org/mailman/listinfo/cctbxbb [11]
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> http://phenix-online.org/mailman/listinfo/cctbxbb [11]
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> http://phenix-online.org/mailman/listinfo/cctbxbb [11]
>>
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> http://phenix-online.org/mailman/listinfo/cctbxbb [11]
>>
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> http://phenix-online.org/mailman/listinfo/cctbxbb [11]
>>
>>
>>
>> _______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> http://phenix-online.org/mailman/listinfo/cctbxbb [11]
>>
>>
>> <cctbx-developer-guidance-08-2015.pdf>_______________________________________________
>> cctbxbb mailing list
>> cctbxbb(a)phenix-online.org<mailto:[email protected]>
>> http://phenix-online.org/mailman/listinfo/cctbxbb [11]
>>
>>
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb [11]
>
> --
> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>
> _______________________________________________
> cctbxbb mailing list
> cctbxbb(a)phenix-online.org
> http://phenix-online.org/mailman/listinfo/cctbxbb [11]
_______________________________________________
cctbxbb mailing list
cctbxbb(a)phenix-online.org
http://phenix-online.org/mailman/listinfo/cctbxbb [11]
Links:
------
[1] https://phenix-online.org
[2] tel:(510)%20486-5709
[3] tel:(510)%20486-5909
[4] https://github.com/cctbx/cctbx_project/wiki/cctbx-Developer-Guidance
[5] tel:%2B441235%2077%208078
[6]
http://cci-vm-6.lbl.gov:8010/builders/phenix-nightly-intel-linux-2.6-x86_64…
[7] tel:%28510%29%20486-5709
[8] tel:%28510%29%20486-5909
[9] https://phenix-online.org/
[10] tel:%28510%29%20486-5713
[11] http://phenix-online.org/mailman/listinfo/cctbxbb
8 years, 9 months