advice for speeding up cif_as_mtz - phenixbb - phenix-online.org

newer
Two versions of TFZ, molecular...

advice for speeding up cif_as_mtz

older
B-iso vs. B-aniso

Eric Williams

10 Sep 2012 10 Sep '12

2:38 p.m.

I'm currently running cif_as_mtz and model_vs_data on the whole PDB as part of my dissertation. Time has become a critical concern, and cif_as_mtz (dev-1141) is running very, very slowly (1-4 minutes per entry). Here's how I'm calling it: phenix.cif_as_mtz --ignore-bad-sigmas --merge --remove-systematic-absences --map-to-asu --symmetry= --output-file-name= I'm hoping that perhaps someone could help me determine which command line argument(s) could be removed to provide maximum speed-up. Thanks. :) Eric

Attachments:

attachment.html (text/html — 709 bytes)

Reply

Sign in to reply online Use email software

Show replies by date

Phil Jeffrey

10 Sep 10 Sep

7:04 p.m.

I would take a look at cif2mtz in the CCP4 package. Phil Jeffrey Princeton On 9/10/12 10:38 AM, Eric Williams wrote:

I'm currently running cif_as_mtz and model_vs_data on the whole PDB as part of my dissertation. Time has become a critical concern, and cif_as_mtz (dev-1141) is running very, very slowly (1-4 minutes per entry).

Here's how I'm calling it:

phenix.cif_as_mtz --ignore-bad-sigmas --merge --remove-systematic-absences --map-to-asu --symmetry= --output-file-name=

I'm hoping that perhaps someone could help me determine which command line argument(s) could be removed to provide maximum speed-up.

Thanks. :)

Eric

_______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb

Reply

Sign in to reply online Use email software

Nathaniel Echols

7:20 p.m.

On Mon, Sep 10, 2012 at 7:38 AM, Eric Williams wrote:

I'm currently running cif_as_mtz and model_vs_data on the whole PDB as part of my dissertation. Time has become a critical concern, and cif_as_mtz (dev-1141) is running very, very slowly (1-4 minutes per entry).

Here's how I'm calling it:

phenix.cif_as_mtz --ignore-bad-sigmas --merge --remove-systematic-absences --map-to-asu --symmetry= --output-file-name=

I'm hoping that perhaps someone could help me determine which command line argument(s) could be removed to provide maximum speed-up.

Are you running this on NFS-mounted disks by any chance? None of these should be especially time-consuming - the only big bottleneck in the program is "--use-model". I just tried running the command above on a reasonably large structure (1zy8) and it was very quick (< 15s). On the off chance that something big changed in the last couple of weeks, can you update to the latest nightly build and see if it's still running slowly? If so, send us some IDs that take a particularly long time and we'll see if we can reproduce the problem. thanks, Nat

Reply

Sign in to reply online Use email software

Eric Williams

7:31 p.m.

I'm running it on a CIFS-mounted disk. Phenix and the data are located on the same disk, though. I'll give a newer build a try. Thanks. :) Eric On Mon, Sep 10, 2012 at 3:20 PM, Nathaniel Echols wrote:

...
I'm currently running cif_as_mtz and model_vs_data on the whole PDB as

...
of my dissertation. Time has become a critical concern, and cif_as_mtz (dev-1141) is running very, very slowly (1-4 minutes per entry).

Here's how I'm calling it:

phenix.cif_as_mtz --ignore-bad-sigmas --merge --remove-systematic-absences --map-to-asu --symmetry= --output-file-name=

I'm hoping that perhaps someone could help me determine which command

On Mon, Sep 10, 2012 at 7:38 AM, Eric Williams wrote: part line

...
argument(s) could be removed to provide maximum speed-up.

Are you running this on NFS-mounted disks by any chance?

None of these should be especially time-consuming - the only big bottleneck in the program is "--use-model". I just tried running the command above on a reasonably large structure (1zy8) and it was very quick (< 15s). On the off chance that something big changed in the last couple of weeks, can you update to the latest nightly build and see if it's still running slowly? If so, send us some IDs that take a particularly long time and we'll see if we can reproduce the problem.

thanks, Nat _______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb

Reply

Sign in to reply online Use email software

Nathaniel Echols

7:51 p.m.

On Mon, Sep 10, 2012 at 12:31 PM, Eric Williams wrote:

I'm running it on a CIFS-mounted disk. Phenix and the data are located on the same disk, though.

It doesn't matter where Phenix is - the IO overhead from reading the CIF and writing the MTZ could be substantial though. Try running a few jobs with files on a local disk and see if that goes faster. -Nat

Reply

Sign in to reply online Use email software

Eric Williams

18 Sep 18 Sep

11:53 p.m.

It seems that the location of Phenix does matter (at least on my system). I ran cif_as_mtz (from 1.8.1-1155) on 3a0n, using only the "--merge" switch. Running from the CIFS mount, it took ~5 minutes. Running from the local file system, it took ~21 seconds. The CIF file was on the local file system in both experiments. Weird. Eric On Mon, Sep 10, 2012 at 3:51 PM, Nathaniel Echols wrote:

On Mon, Sep 10, 2012 at 12:31 PM, Eric Williams wrote:

...
I'm running it on a CIFS-mounted disk. Phenix and the data are located on the same disk, though.

It doesn't matter where Phenix is - the IO overhead from reading the CIF and writing the MTZ could be substantial though. Try running a few jobs with files on a local disk and see if that goes faster.

-Nat _______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb

Reply

Sign in to reply online Use email software

Pavel Afonine

19 Sep 19 Sep

5:36 p.m.

Hi Eric, I just tried this command: phenix.cif_as_mtz 3a0n-sf.cif --ignore-bad-sigmas --merge --remove-systematic-absences --map-to-asu --symmetry=3a0n.pdb and it tool approximately 5 seconds to run on my computer. Pavel On 9/18/12 4:53 PM, Eric Williams wrote:

It seems that the location of Phenix does matter (at least on my system). I ran cif_as_mtz (from 1.8.1-1155) on 3a0n, using only the "--merge" switch. Running from the CIFS mount, it took ~5 minutes. Running from the local file system, it took ~21 seconds. The CIF file was on the local file system in both experiments.

Weird.

Eric

On Mon, Sep 10, 2012 at 3:51 PM, Nathaniel Echols mailto:[email protected]> wrote:

On Mon, Sep 10, 2012 at 12:31 PM, Eric Williams mailto:[email protected]> wrote: > I'm running it on a CIFS-mounted disk. Phenix and the data are located on > the same disk, though.

It doesn't matter where Phenix is - the IO overhead from reading the CIF and writing the MTZ could be substantial though. Try running a few jobs with files on a local disk and see if that goes faster.

-Nat _______________________________________________ phenixbb mailing list [email protected] mailto:[email protected] http://phenix-online.org/mailman/listinfo/phenixbb

_______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb

Reply

Sign in to reply online Use email software

Pavel Afonine

11 Sep 11 Sep

6:38 p.m.

Hi Eric,

I'm currently running cif_as_mtz and model_vs_data on the whole PDB as part of my dissertation. Time has become a critical concern, and cif_as_mtz (dev-1141) is running very, very slowly (1-4 minutes per entry).

clearly something is wrong, as the command you quoted should take from few to ~10-30 seconds, not minutes.

Here's how I'm calling it:

phenix.cif_as_mtz --ignore-bad-sigmas --merge --remove-systematic-absences --map-to-asu --symmetry= --output-file-name=

Clearly I missed a moment when "--ignore-bad-sigmas" was added, otherwise I would pick up on this earlier. What is the definition of "bad-sigmas"? Thanks, Pavel

Reply

Sign in to reply online Use email software

Nathaniel Echols

6:45 p.m.

On Tue, Sep 11, 2012 at 11:38 AM, Pavel Afonine wrote:

Clearly I missed a moment when "--ignore-bad-sigmas" was added, otherwise I would pick up on this earlier. What is the definition of "bad-sigmas"?

It's right there in the code: # if all sigmas for an array are set to zero either raise an error, or set sigmas to None So it's not a per-reflection filter, just a way of detecting nonsense data items in the CIF file. -Nat

Reply

Sign in to reply online Use email software

4318

Age (days ago)

4327

Last active (days ago)

Download

8 comments

4 participants

tags

participants (4)

Eric Williams
Nathaniel Echols
Pavel Afonine
Phil Jeffrey