Hi Maike, could you please send me input files and I will provide the solution for your issue?
1) reduce deviation from planarity restraints to very small value or zero -> ligand still gets distorted in refinement
Setting esd to very small number means huge contribution of that restrain to refinement target because corresponding weight is 1/esd**2. So I'm not surprised it did not work.
2) Set all bond angle restraints to ideal values and checked sums of angles in planar rings -> ligand still gets distorted in refinement
You need a plane and phenix.refine allows you to put any desired atoms into a plane, so no need to work-around this using angles.
3) Used phenix.elbow with - -final-geometry on a pdb of a planar ligand —> ligand still gets distorted in refinement
I hoped Nigel addresses this one.
4) Used the grade server (grade.globalphasing.org http://grade.globalphasing.org) to generate cif file from SMILES string—> phenix.refine stops with message “Unknown file format: grade-ligand.cif”.
External to Phenix tools do not guarantee to provide inputs good for Phenix.
Does anyone have another suggestion,
See above: send me files and I will make it work.
except for installing an extra QM package?
QM in this particular case would be a "cannon to kill a fly". Pavel