Hi Nat, Although having a sysadmin change the location of your home directory is unusual, a much more common problem is buying a new computer and copying your home directory from one to the other. It can be very difficult (e.g. when going from Linux to Mac or the reverse) to get the same absolute path for your home directory. So hopefully your project migration tool will allow that too. Regards, Randy On 2 Nov 2010, at 18:47, Nathaniel Echols wrote:

On Tue, Nov 2, 2010 at 11:31 AM, Ben Eisenbraun wrote:

...

Our local sysadmins moved everyone's home directory from /nfs/home/foo to /nfs/userdocs/home/foo, and it appears to have broken most people's PHENIX projects.

Uh-oh. (Doesn't this break CCP4i too? I took a look at my old grad school data and it looks like everything is using absolute paths there too - although perhaps since they're text files, it's easier to fix...)

...

Is there a straightforward way to fix this? Poking around in the ~/.phenix/project_db.pkl files, I can see the old paths, but I'm not exactly sure how to rewrite. My brute force attempt with sed was unsuccssful.

They're technically binary files (Python pickle format), so I doubt sed will do the job. I can write you a script to rewrite paths, but it's not a trivial problem. Migrating projects has been on the to-do list for a while and I'm close to getting it working, but I hadn't planned on anyone moving entire home directories like this. I don't suppose you can beg the sysadmins to add a symlink named /nfs/home?

-Nat _______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb

------ Randy J. Read Department of Haematology, University of Cambridge Cambridge Institute for Medical Research Tel: + 44 1223 336500 Wellcome Trust/MRC Building Fax: + 44 1223 336827 Hills Road E-mail: [email protected] Cambridge CB2 0XY, U.K. www-structmed.cimr.cam.ac.uk

On Wed, Nov 3, 2010 at 2:00 AM, Randy Read wrote:

Although having a sysadmin change the location of your home directory is unusual, a much more common problem is buying a new computer and copying your home directory from one to the other.  It can be very difficult (e.g. when going from Linux to Mac or the reverse) to get the same absolute path for your home directory.  So hopefully your project migration tool will allow that too.

Working on it. . . a lot of stuff to clean up before this will behave reliably, but it's starting to be functional. Among other things, I'm going back to text files, but with some extra optimizations for speed. I'm also going to make the default file handling more rational along the way. If anyone has strong feelings about how the project management/interfaces in the GUI should (or should not) work, now would be an excellent time to make them known. (Comments can be sent to me directly instead of the list.) -Nat

Hi Nat, My 2c: First off, I'm 100% behind the idea of using text files rather than pickles...so if anything goes wrong it can be fixed with a judicious sed script. Second, I would suggest using relative paths whenever practical... i.e. if object (file/directory) is in the project directory or a subdirectory thereof use a relative path, but if it's somewhere else on the file tree entirely (i.e /data for images) use an absolute path. At this point I think most of the migration issues that have been mentioned would vanish. Third, it would be really great if you could export a collection of jobs (including input files, output files and 'control' (phenix) files) as a zip-file or tar file that could be re-imported into another user's phenix project. The use case is where you give someone else in the lab a hand with building or running some phenix jobs so you create a new project, run those jobs and send them back the result (say the model). For teaching purposes especially it would be much better if you could send back the 'jobs' to re-include in their project (so that they know how to run the job themselves next time rather than bugging you!). On the up side, if job inport/export via a tar/zip file worked you wouldn't necessarily need any other project migration tool as people could just zip up their phenix project then 'unpack' it on the new machine. Fourth, perhaps I'm just used to CCP4i but I'd really like to see the job history on the main phenix window rather than hidden in a dialog box. Sorry, that ended up more like $2. Stephen On 4 November 2010 18:16, Nathaniel Echols wrote:

On Wed, Nov 3, 2010 at 2:00 AM, Randy Read wrote:

...

Although having a sysadmin change the location of your home directory is unusual, a much more common problem is buying a new computer and copying your home directory from one to the other.  It can be very difficult (e.g. when going from Linux to Mac or the reverse) to get the same absolute path for your home directory.  So hopefully your project migration tool will allow that too.

Working on it. . . a lot of stuff to clean up before this will behave reliably, but it's starting to be functional.  Among other things, I'm going back to text files, but with some extra optimizations for speed.  I'm also going to make the default file handling more rational along the way.

If anyone has strong feelings about how the project management/interfaces in the GUI should (or should not) work, now would be an excellent time to make them known.  (Comments can be sent to me directly instead of the list.)

-Nat _______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb

-- Dr Stephen Graham 1851 Research Fellow Cambridge Institute for Medical Research Wellcome Trust/MRC Building Addenbrooke's Hospital, Hills Road Cambridge, CB2 0XY, UK Phone: +44 1223 762 638

Hi Nat, Thanks for the excellent software. Would it be possible to have each project directory as a variable, much like the CCP4i "directories & projectDir" list allows you to set. Once you have /Users/brooks/lysozyme and set the project directory to /home/brooks/lysozyme, that would work straight away on a linux computer, for example. I don't think you should clone CCP4i, but that aspect is quite powerful. Even in the .def file, instead of listing "/Users/brooks/lysozyme/peak.mtz" explicitly, one could have a setting: input { project_directory = "/Users/brooks/lysozyme" } ...then set file names by paths relative to that one. That could be useful for migration of existing .def files, for example. Thanks again for being receptive to ideas. Yours, Mark On 5 November 2010 20:49, Nathaniel Echols wrote:

On Fri, Nov 5, 2010 at 12:26 AM, Stephen Graham wrote:

...

First off, I'm 100% behind the idea of using text files rather than pickles...so if anything goes wrong it can be fixed with a judicious sed script.

Agreed.

...

Second, I would suggest using relative paths whenever practical... i.e. if object (file/directory) is in the project directory or a subdirectory thereof use a relative path, but if it's somewhere else on the file tree entirely (i.e /data for images) use an absolute path. At this point I think most of the migration issues that have been mentioned would vanish.

This is practical in some situations but not others; for instance, the .eff files used for each job still need to have the full path for any input files. The pickled result objects are an even bigger problem - fixing this would be a long-term project. Doesn't CCP4i store some intermediate format between the GUI and the actual run script?

...

Third, it would be really great if you could export a collection of jobs (including input files, output files and 'control' (phenix) files) as a zip-file or tar file that could be re-imported into another user's phenix project. The use case is where you give someone else in the lab a hand with building or running some phenix jobs so you create a new project, run those jobs and send them back the result (say the model). For teaching purposes especially it would be much better if you could send back the 'jobs' to re-include in their project (so that they know how to run the job themselves next time rather than bugging you!). On the up side, if job inport/export via a tar/zip file worked you wouldn't necessarily need any other project migration tool as people could just zip up their phenix project then 'unpack' it on the new machine.

Exporting and importing projects as a compressed archive is relatively easy; I'm not sure how to make transferring individual jobs this way work, however. Keeping the directories and job IDs from overlapping is going to be tough.

...

Fourth, perhaps I'm just used to CCP4i but I'd really like to see the job history on the main phenix window rather than hidden in a dialog box.

It's not an unreasonable request, and not the first time I've heard this. I'm still trying to figure out how to fit everything into the main window cleanly - maybe tabs, but I worry about it getting too cluttered. (On the other hand, as we keep adding programs this will probably happen anyway unless we redesign the layout.) Although it's also possible to make this a setting in preferences, and let individual users decide how they want it to look.

-Nat _______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb

Hi Fengyun,

I have two datasets, one of which the completeness is only about 81%, the other one is even lower, about 45%.

The resulting "missing fobs filled map" is much better than the unfilled map in the loop region. I read one file of Pavel on the fact of maps, the concern of using missing fobs filled map is that the risk of the mdoel bias.

My question is that could the missing fobs filled map be used for the model building?

a "missing fobs filled map" is always computed using 100% set of reflections. For example, if your original dataset is 81% complete, that means the "missing fobs filled map" is computed using existing 81% of data plus 19% of reflections to fill the gap. These 19% are DFc. In extreme case of 0% or close to 0% data completeness this because just Fcalc map. Therefore it's clear that "missing fobs filled map" map should be used with care, and it is a good idea to look at both maps (one computed using original data, and another one using the data where missing reflections filled in with DFc). "filled" maps can produce a clearer picture and may disambiguate hard-to-interpret density, while may be biased. Regular maps may be of poorer quality (due to data incompleteness), but are not model biased due to "filling" (although may be model biased caused by other reasons than "filling"). Given data completeness you have I wouldn't use "filled" maps. I'm wondering what Refmac does in this case (as far as I know it always outputs "filled" map)? Pavel.

2010/11/8 :

I am interesting in that at what completeness of the dataset, will one use the missing fobs filled map for model building confidently without too much bias included?

It partly depends on why the data are incomplete, but I'd certainly avoid it for anything less than 90%, possibly 95% if I was feeling especially paranoid. The effect on map quality when you're at, say, 97% is very subtle, but as the completeness drops, filling in missing reflections will mask major artifacts, which is dangerous. -Nat

Hi Fengyun, The one with 81% completeness is likely to be fine, but do compare results with the unfilled map. The one with 45% completeness is going to be very difficult for model-building...unless your starting model is almost correct to start with. I would recommend trying in both cases: instead of using these maps, use density-modified maps which can replace this model information with information based on solvent flattening or (better yet, if you have NCS), ncs averaging. You can do this with phenix.autobuild data=data.mtz model=coords.pdb \ maps_only=True seq_file=seq.dat resolve_command_list=" 'fill' " which will fill in missing reflections with information from the density modification process. All the best, Tom T

...

Hi all,

I have two datasets, one of which the completeness is only about 81%, the other one is even lower, about 45%.

The resulting "missing fobs filled map" is much better than the unfilled map in the loop region. I read one file of Pavel on the fact of maps, the concern of using missing fobs filled map is that the risk of the mdoel bias.

My question is that could the missing fobs filled map be used for the model building?

Thanks! Fengyun

_______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb