Aside from the problem that the program is looking for the geometry libraries in one place and they are located in another place (that is a CCP4 issue), I could not get Phenix (1.7.1-743) to communicate with Coot. I supplied the path to coot (/Applications/coot.app) and Phenix started coot, but could not load the refinement information. I tested this during a refinement run on a 10.5.x machine, and from the "Results" pane for a finished job under 10.6.8. In the latter case, pressing "Open in Coot" does fire up the program, but the molecule and maps are not loaded. On the other hand, choosing "Open in PyMOL" does indeed display the molecule and maps.
You just described the problem that we had at two recent workshops: even though Coot was installed in /Applications/coot.app, PHENIX could not use it. Even providing the whole path to Coot in PHENIX GUI preferences did not help. Pavel.