Hi Kristoffer, If you are looking to integrate your modeling with deep-learning libraries (torch, jax, tf), you might want to have a look at SFCalculator https://github.com/Hekstra-Lab/SFcalculator. It's being developed by a graduate student, Minhuan Li, in my advisor's group. It would give you access to automatic differentiation if you ever want to innovate on the refinement target function. It also supports GPU acceleration. I would say the major downside is that algorithms like solvent masking had to be made differentiable, and it is a newer project which is not as mature as CCTBX. Cheers, Kevin On Thu, Oct 5, 2023 at 8:26 PM Pavel Afonine wrote:

Hi Kristoffer,

CCTBX is a source of building blocks for Phenix, and Phenix has real- and reciprocal-space refinement programs, phenix.real_space_refine and phenix.refine, so the answer to your question is yes.

You have not specified, but I'm assuming you are looking for gradients of reciprocal space refinement target (LS, ML, MLHL, etc) and real-space target (we have only one described here: https://pubmed.ncbi.nlm.nih.gov/29872004/ https://urldefense.proofpoint.com/v2/url?u=https-3A__pubmed.ncbi.nlm.nih.gov_29872004_&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=30D8FbpCRRhFIowTlh1rEFMmkV5i20_4fL5nmehOS68&m=Kr-Fzcpb9IFO8sPYrtkzzeMXXL9A0uaYDFlX07gIddQf-BCoe3Qe1EZPtJPOoqaP&s=YmCd41ZAFbFw7lMImQPLoEhq49GXKHWv0RoJ_-iX53o&e=) with respect to model parameters such as coordinates, ADPs (isotropic, anisotropic), occupancies, f' and f'', etc.

Here are a few places:

- reciprocal space:

cctbx_project/mmtbx/refinement

files:

minimization.py occupancies.py rigid_body.py group.py

regression/tst_xray_fast_gradients.py xray/boost_python/tst_xray.py regression/tst_xray_derivatives.py

- real space:

cctbx_project/mmtbx/refinement/real_space/individual_sites.py cctbx_project/cctbx/maptbx/tst_real_space_refinement_simple.py cctbx_project/cctbx/maptbx/target_and_gradients.h cctbx_project/cctbx/maptbx/real_space_refinement_simple.py

Note, these are examples of high-level code that uses target function and its gradients to perform various kind of refinement. You'll need to do some detective work to trace to the low-level code that actually calculates gradients. Hope that's good enough starting point for you.

Also, try grep for "gradient" and "target"! Usually tests (files tst*.py serve as good examples).

Good luck! Pavel

On 10/3/23 08:13, Kristoffer Lundgren wrote:

Hello all,

I am wondering if it is possible to use cctbx to calculate gradients from experimental data? Both in reciprocal and real space is of interest.

Are there any code examples available somewhere showcasing how this can be achieved?

Best regards

Kristoffer Lundgren

_______________________________________________ cctbxbb mailing [email protected]http://phenix-online.org/mailman/listinfo/cctbxbb https://urldefense.proofpoint.com/v2/url?u=http-3A__phenix-2Donline.org_mailman_listinfo_cctbxbb&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=30D8FbpCRRhFIowTlh1rEFMmkV5i20_4fL5nmehOS68&m=Kr-Fzcpb9IFO8sPYrtkzzeMXXL9A0uaYDFlX07gIddQf-BCoe3Qe1EZPtJPOoqaP&s=xJfn2wSoFtMiJLtJXzxFvvDrAWlujEZExgjTJihJFJ0&e=

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi Kristoffer, you got it almost right! Attached is the working example (using 1yjp model from PDB). Note: - choosing f_obs as "f_obs = miller_arrays[1]"  assumes you know it is indexed as "1" in the list of miller arrays (which isn't always the case!); - once you made fmodel you need to do "update_all_scales()" in order to be able to compute Fmodel = ktotal * (Fcalc + kmask*Fmask) because ktotal=1 and kmask=0 otherwise (the way it is done described here: https://journals.iucr.org/d/issues/2013/04/00/dz5273/dz5273.pdf); - choice of target_name means the target that you get the gradients of. Let me know if you have any questions or need any help! Pavel On 10/10/23 03:47, Kristoffer Lundgren wrote:

Hi Pavel,

Thank you for your answer and hints. Trying to just test and understand how the libraries works, this is as far as I have come:

import iotbx.pdb

import mmtbx.model

import mmtbx.f_model

from iotbx import reflection_file_reader

pdb_file = '1gwd.pdb'

mtz_file = '1gwd.mtz'

pdb_inp = iotbx.pdb.input(file_name=pdb_file)

model = mmtbx.model.manager(model_input=pdb_inp)

miller_arrays = reflection_file_reader.any_reflection_file(file_name=mtz_file).as_miller_arrays()

f_obs = miller_arrays[1]

fmodel = mmtbx.f_model.manager(f_obs=f_obs, xray_structure=model.get_xray_structure(), target_name='ls_wunit_k1')

gradients = fmodel.one_time_gradients_wrt_atomic_parameters()

print(list(gradients.d_target_d_site_cart()))

...

...

...

[(0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), […]

I would expect the gradients to not be zero for all scatterers so obviously I am doing something wrong – help on how to get further would be very much appreciated.

Best regards

Kristoffer

*From: *Pavel Afonine *Date: *Friday, 6 October 2023 at 02:35 *To: *cctbx mailing list , Kristoffer Lundgren *Subject: *Re: [cctbxbb] Experimental gradients

Hi Kristoffer,

CCTBX is a source of building blocks for Phenix, and Phenix has real- and reciprocal-space refinement programs, phenix.real_space_refine and phenix.refine, so the answer to your question is yes.

You have not specified, but I'm assuming you are looking for gradients of reciprocal space refinement target (LS, ML, MLHL, etc) and real-space target (we have only one described here: https://pubmed.ncbi.nlm.nih.gov/29872004/) with respect to model parameters such as coordinates, ADPs (isotropic, anisotropic), occupancies, f' and f'', etc.

Here are a few places:

- reciprocal space:

cctbx_project/mmtbx/refinement

files:

minimization.py occupancies.py rigid_body.py group.py

regression/tst_xray_fast_gradients.py xray/boost_python/tst_xray.py regression/tst_xray_derivatives.py

- real space:

cctbx_project/mmtbx/refinement/real_space/individual_sites.py cctbx_project/cctbx/maptbx/tst_real_space_refinement_simple.py cctbx_project/cctbx/maptbx/target_and_gradients.h cctbx_project/cctbx/maptbx/real_space_refinement_simple.py

Note, these are examples of high-level code that uses target function and its gradients to perform various kind of refinement. You'll need to do some detective work to trace to the low-level code that actually calculates gradients. Hope that's good enough starting point for you.

Also, try grep for "gradient" and "target"! Usually tests (files tst*.py serve as good examples).

Good luck! Pavel

On 10/3/23 08:13, Kristoffer Lundgren wrote:

Hello all,

I am wondering if it is possible to use cctbx to calculate gradients from experimental data? Both in reciprocal and real space is of interest.

Are there any code examples available somewhere showcasing how this can be achieved?

Best regards

Kristoffer Lundgren

_______________________________________________

cctbxbb mailing list

[email protected]

http://phenix-online.org/mailman/listinfo/cctbxbb