Hello all, I am wondering if it is possible to use cctbx to calculate gradients from experimental data? Both in reciprocal and real space is of interest. Are there any code examples available somewhere showcasing how this can be achieved? Best regards Kristoffer Lundgren
Hi Kristoffer, CCTBX is a source of building blocks for Phenix, and Phenix has real- and reciprocal-space refinement programs, phenix.real_space_refine and phenix.refine, so the answer to your question is yes. You have not specified, but I'm assuming you are looking for gradients of reciprocal space refinement target (LS, ML, MLHL, etc) and real-space target (we have only one described here: https://pubmed.ncbi.nlm.nih.gov/29872004/) with respect to model parameters such as coordinates, ADPs (isotropic, anisotropic), occupancies, f' and f'', etc. Here are a few places: - reciprocal space: cctbx_project/mmtbx/refinement files: minimization.py occupancies.py rigid_body.py group.py regression/tst_xray_fast_gradients.py xray/boost_python/tst_xray.py regression/tst_xray_derivatives.py - real space: cctbx_project/mmtbx/refinement/real_space/individual_sites.py cctbx_project/cctbx/maptbx/tst_real_space_refinement_simple.py cctbx_project/cctbx/maptbx/target_and_gradients.h cctbx_project/cctbx/maptbx/real_space_refinement_simple.py Note, these are examples of high-level code that uses target function and its gradients to perform various kind of refinement. You'll need to do some detective work to trace to the low-level code that actually calculates gradients. Hope that's good enough starting point for you. Also, try grep for "gradient" and "target"! Usually tests (files tst*.py serve as good examples). Good luck! Pavel On 10/3/23 08:13, Kristoffer Lundgren wrote:
Hi Kristoffer,
If you are looking to integrate your modeling with deep-learning libraries
(torch, jax, tf), you might want to have a look at SFCalculator
https://github.com/Hekstra-Lab/SFcalculator. It's being developed by a
graduate student, Minhuan Li, in my advisor's group. It would give you
access to automatic differentiation if you ever want to innovate on the
refinement target function. It also supports GPU acceleration. I would say
the major downside is that algorithms like solvent masking had to be made
differentiable, and it is a newer project which is not as mature as CCTBX.
Cheers,
Kevin
On Thu, Oct 5, 2023 at 8:26 PM Pavel Afonine
Hi Pavel, Thank you for your answer and hints. Trying to just test and understand how the libraries works, this is as far as I have come: import iotbx.pdb import mmtbx.model import mmtbx.f_model from iotbx import reflection_file_reader pdb_file = '1gwd.pdb' mtz_file = '1gwd.mtz' pdb_inp = iotbx.pdb.input(file_name=pdb_file) model = mmtbx.model.manager(model_input=pdb_inp) miller_arrays = reflection_file_reader.any_reflection_file(file_name=mtz_file).as_miller_arrays() f_obs = miller_arrays[1] fmodel = mmtbx.f_model.manager(f_obs=f_obs, xray_structure=model.get_xray_structure(), target_name='ls_wunit_k1') gradients = fmodel.one_time_gradients_wrt_atomic_parameters() print(list(gradients.d_target_d_site_cart()))
[(0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), […]
I would expect the gradients to not be zero for all scatterers so obviously I am doing something wrong – help on how to get further would be very much appreciated.
Best regards
Kristoffer
From: Pavel Afonine
Hi Kristoffer, you got it almost right! Attached is the working example (using 1yjp model from PDB). Note: - choosing f_obs as "f_obs = miller_arrays[1]" assumes you know it is indexed as "1" in the list of miller arrays (which isn't always the case!); - once you made fmodel you need to do "update_all_scales()" in order to be able to compute Fmodel = ktotal * (Fcalc + kmask*Fmask) because ktotal=1 and kmask=0 otherwise (the way it is done described here: https://journals.iucr.org/d/issues/2013/04/00/dz5273/dz5273.pdf); - choice of target_name means the target that you get the gradients of. Let me know if you have any questions or need any help! Pavel On 10/10/23 03:47, Kristoffer Lundgren wrote:
participants (3)
-
Dalton, Kevin -
Kristoffer Lundgren -
Pavel Afonine