Dear Jan,
this is an interesting problem and there are no perfect solutions. 狢'm not currently a cctbx user so I can't say whether such an algorithm is implemented but I done some work on this myself and can maybe point you in the direction of some approaches you could take.
* Sorted distances (possibly the simplest approach)
1) Calculate all r_ij distances from all the atoms in your unit cell to all neighbours within a cutoff.
2) Sort the list of distances.
3) To get the 'difference' between the two structures calculate the rms of the differences in distances between the two lists.
Generally this approach works well although it can give false positives (by virtue of the fact that it's throwing away a lot of information)
* Histogram radial distribution comparison (real space)
Basically calculate the radial distribution histogram for each species pair and then use some method to calculate the difference between the histograms (e.g. rms differences, cosine distance, etc.)
* Frequency space comparison
Similar to calculating the r-factor although in this paper they use a more sophisticated approach that I don't fully understand.
* Finding ideal transformation between structures
Hundt's approach (CMPZ):
Lonie's approach (XtalComp):
for details and
to try the algorithm for yourself.
Basically the idea is to find the more ideal transformation to take you from one crystal structure to another. 硓rom the transformation you can then pull out one number to get a difference or use it to map from one system to another.
I think I'll stop there but I'd be curious to hear about what you find and how you get on.
All the best,
-Martin