Hi Phil, Probably, but in the meantime using MTZ mean we can do things like running Aimless on the output data to see if we have made any kind of useful measurements :o) Has been a game though making full-on unmerged MTZ files! Cheerio, Graeme On 12 Feb 2014, at 12:29, Phil Evans wrote:

I thought you DIALS guys were going to use a new file format Phil

On 12 Feb 2014, at 10:43, [email protected] wrote:

...

In related news….

Should

void object::adjust_column_array_sizes(int new_nref) { CMtz::MTZ* p = ptr(); if (!p->refs_in_memory) return; if (new_nref > p->nref) { reserve(new_nref); for(int i=0;i<p->nxtal;i++) { for(int j=0;j<p->xtal[i]->nset;j++) { for(int k=0;k<p->xtal[i]->set[j]->ncol;k++) { CMtz::MTZCOL* col_k = p->xtal[i]->set[j]->col[k]; int old_size = column_array_size(col_k); if (new_nref > old_size) { ccp4array_resize(col_k->ref, new_nref); for(int iref=old_size;iref( &col_k->ref[iref])) = not_a_number_value_; } } } } } } }

assign nref? otherwise repeated calling of this will keep on re-allocating the arrays

Thanks Graeme

On 12 Feb 2014, at 10:40, Graeme Winter wrote:

...

All,

By adding

m_out.adjust_column_array_sizes(len(mi))

to the bpl wrapper and defining

m_out.set_n_reflections(len(mi))

in the object.h and bpl wrapper I can do what I want…

I will add some tests and some warnings that this may make nonsense of the values… however before committing any comments on the interface? or behaviour?

Thanks Graeme

On 12 Feb 2014, at 09:11, wrote:

...

Hi Folks,

I am trying to create a new MTZ file containing unmerged and it seems to be less simple than I would think - set_reals() on a column requires a set of Miller indices and values and does a lookup internally (which I don’t want for unmerged values) and set_values() requires that the array is already the right size - when it starts off as size 0 and I have no idea how to make enough room in there from the Python API…

Any clues? I have done quite a chunk of grepping around to try and figure this one out and have run out of ideas…

Thanks Graeme_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Pointless can read unmerged Scalepack files, which can go into Aimless, but it can't do any further scaling (probably unnecessary anyway) as the geometrical information is missing Phil On 12 Feb 2014, at 18:14, Nathaniel Echols wrote:

Random dumb question: does Aimless read unmerged Scalepack files? A few months ago I added a module in iotbx to export those, which might be simpler than dealing with the MTZ API. (Not that writing unmerged MTZ files would't be a good idea, but if you're looking for a quick solution that may not be the best route.)

-Nat

On Wed, Feb 12, 2014 at 4:48 AM, wrote: Hi Phil,

Probably, but in the meantime using MTZ mean we can do things like running Aimless on the output data to see if we have made any kind of useful measurements :o)

Has been a game though making full-on unmerged MTZ files!

Cheerio, Graeme

On 12 Feb 2014, at 12:29, Phil Evans wrote:

...

I thought you DIALS guys were going to use a new file format Phil

On 12 Feb 2014, at 10:43, [email protected] wrote:

...

In related news….

Should

void object::adjust_column_array_sizes(int new_nref) { CMtz::MTZ* p = ptr(); if (!p->refs_in_memory) return; if (new_nref > p->nref) { reserve(new_nref); for(int i=0;i<p->nxtal;i++) { for(int j=0;j<p->xtal[i]->nset;j++) { for(int k=0;k<p->xtal[i]->set[j]->ncol;k++) { CMtz::MTZCOL* col_k = p->xtal[i]->set[j]->col[k]; int old_size = column_array_size(col_k); if (new_nref > old_size) { ccp4array_resize(col_k->ref, new_nref); for(int iref=old_size;iref( &col_k->ref[iref])) = not_a_number_value_; } } } } } } }

assign nref? otherwise repeated calling of this will keep on re-allocating the arrays

Thanks Graeme

On 12 Feb 2014, at 10:40, Graeme Winter wrote:

...

All,

By adding

m_out.adjust_column_array_sizes(len(mi))

to the bpl wrapper and defining

m_out.set_n_reflections(len(mi))

in the object.h and bpl wrapper I can do what I want…

I will add some tests and some warnings that this may make nonsense of the values… however before committing any comments on the interface? or behaviour?

Thanks Graeme

On 12 Feb 2014, at 09:11, wrote:

...

Hi Folks,

I am trying to create a new MTZ file containing unmerged and it seems to be less simple than I would think - set_reals() on a column requires a set of Miller indices and values and does a lookup internally (which I don’t want for unmerged values) and set_values() requires that the array is already the right size - when it starts off as size 0 and I have no idea how to make enough room in there from the Python API…

Any clues? I have done quite a chunk of grepping around to try and figure this one out and have run out of ideas…

Thanks Graeme_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

and we know from previous discussions that the current unmerged MTZ format is not the best possible On 13 Feb 2014, at 08:11, wrote:

... and we want to write data out from DIALS which does want to be properly scaled by Aimless :o)

Also it is a good exercise anyway to pass as much information as possible down the wire - this is a good part of the motivation. May just require one or two small tweaks of iotbx...

Cheerio, Graeme

-----Original Message----- From: Phil Evans [mailto:[email protected]] Sent: 12 February 2014 20:04 To: cctbx mailing list Subject: Re: [cctbxbb] Adding columns to new *unmerged* MTZ object

Pointless can read unmerged Scalepack files, which can go into Aimless, but it can't do any further scaling (probably unnecessary anyway) as the geometrical information is missing Phil

On 12 Feb 2014, at 18:14, Nathaniel Echols wrote:

...

Random dumb question: does Aimless read unmerged Scalepack files? A few months ago I added a module in iotbx to export those, which might be simpler than dealing with the MTZ API. (Not that writing unmerged MTZ files would't be a good idea, but if you're looking for a quick solution that may not be the best route.)

-Nat

On Wed, Feb 12, 2014 at 4:48 AM, wrote: Hi Phil,

Probably, but in the meantime using MTZ mean we can do things like running Aimless on the output data to see if we have made any kind of useful measurements :o)

Has been a game though making full-on unmerged MTZ files!

Cheerio, Graeme

On 12 Feb 2014, at 12:29, Phil Evans wrote:

...

I thought you DIALS guys were going to use a new file format Phil

On 12 Feb 2014, at 10:43, [email protected] wrote:

...

In related news....

Should

void object::adjust_column_array_sizes(int new_nref) { CMtz::MTZ* p = ptr(); if (!p->refs_in_memory) return; if (new_nref > p->nref) { reserve(new_nref); for(int i=0;i<p->nxtal;i++) { for(int j=0;j<p->xtal[i]->nset;j++) { for(int k=0;k<p->xtal[i]->set[j]->ncol;k++) { CMtz::MTZCOL* col_k = p->xtal[i]->set[j]->col[k]; int old_size = column_array_size(col_k); if (new_nref > old_size) { ccp4array_resize(col_k->ref, new_nref); for(int iref=old_size;iref( &col_k->ref[iref])) = not_a_number_value_; } } } } } } }

assign nref? otherwise repeated calling of this will keep on re-allocating the arrays

Thanks Graeme

On 12 Feb 2014, at 10:40, Graeme Winter wrote:

...

All,

By adding

m_out.adjust_column_array_sizes(len(mi))

to the bpl wrapper and defining

m_out.set_n_reflections(len(mi))

in the object.h and bpl wrapper I can do what I want...

I will add some tests and some warnings that this may make nonsense of the values... however before committing any comments on the interface? or behaviour?

Thanks Graeme

On 12 Feb 2014, at 09:11, wrote:

...

Hi Folks,

I am trying to create a new MTZ file containing unmerged and it seems to be less simple than I would think - set_reals() on a column requires a set of Miller indices and values and does a lookup internally (which I don't want for unmerged values) and set_values() requires that the array is already the right size - when it starts off as size 0 and I have no idea how to make enough room in there from the Python API...

Any clues? I have done quite a chunk of grepping around to try and figure this one out and have run out of ideas...

Thanks Graeme_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

AFAIK mmCIF is not conveniently endowed with what's needed for unmerged data, i.e. a clean way of recording the provenance of every observation, diffraction geometry, crystal properties, beam properties, detector properties etc, although much of this is defined somewhere. This is one of the things that the DIALS people I thought were looking into, probably storing data in an hdf5/Nexus container (is that right?). I'm not convinced that CIF is a good working format for reflection data, which I think should be binary. If we have well-defined structured data, then converting file formats is in principle straightforward (so e.g. Pointless can read XDS and Saint files which have much of the necessary information, unlike Scalepack files) If that is done, defined and a library available in C/C++, then it would be easy(ish) to make Pointless & Aimless read such a format. Phil On 13 Feb 2014, at 15:39, Nathaniel Echols wrote:

On Thu, Feb 13, 2014 at 12:45 AM, Phil Evans wrote: and we know from previous discussions that the current unmerged MTZ format is not the best possible

Perhaps, but it's the only widely supported format that isn't broken by design, so ideally the CCTBX wrapper would provide access to as many features as possible. (I'm assuming that Aimless doesn't read CIF either - correct?)

Graeme: I have no strong opinions on how the API should work as long as existing code does not break, and right now you're the only group trying to output unmerged data. Unless Nick has specific thoughts, I think you should feel free to make it work the way you think is appropriate; we'll probably find things to change later but an imperfect solution is better than none in this case.

-Nat _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Nat, Turns out I only needed to be able to pre-allocate how many reflections I plan to add and after that everything was fine - I do this with: m_out.adjust_column_array_sizes(m_in.n_reflections()) m_out.set_n_reflections(m_in.n_reflections()) which is kind-of messy but works…. no other API changes were needed to make an MTZ file which Aimless will read (I made one from XDS INTEGRATE.HKL, this was essentially fine…) Should adjust_column_array_sizes redefine nref anyway? Looking at the code i would expect it to… Cheerio, Graeme On 13 Feb 2014, at 15:39, Nathaniel Echols wrote:

On Thu, Feb 13, 2014 at 12:45 AM, Phil Evans wrote: and we know from previous discussions that the current unmerged MTZ format is not the best possible

Perhaps, but it's the only widely supported format that isn't broken by design, so ideally the CCTBX wrapper would provide access to as many features as possible. (I'm assuming that Aimless doesn't read CIF either - correct?)

Graeme: I have no strong opinions on how the API should work as long as existing code does not break, and right now you're the only group trying to output unmerged data. Unless Nick has specific thoughts, I think you should feel free to make it work the way you think is appropriate; we'll probably find things to change later but an imperfect solution is better than none in this case.

-Nat _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb