[cctbxbb] Simple C++11 async example

[Winter, Graeme (DLSLtd,RAL,LSCI)]

As someone who spent a few days recently banging my head against boost.python bindings, I can’t help but feel any alternative must be more straightforward to use / debug…

So, possibly desirable, but I would suspect that cctbx is very closely tied to boost so probably not feasible (at a guess)

Cheers Graeme

On 2 Oct 2020, at 09:47, David Waterman <dgwaterman at gmail.com<mailto:dgwaterman at gmail.com>> wrote:

Thanks Tristan,

This does look useful indeed. I note you use pybind11 in your example. Should this work just as easily with Boost.Python?

On that topic, I've heard many good things about pybind11. Does anyone have a feeling for how much work it would be to convert cctbx to pybind11 bindings rather than Boost.Python. Is this feasible? Is it desirable?

Cheers
-- David


On Thu, 1 Oct 2020 at 16:45, Tristan Croll <tic20 at cam.ac.uk<mailto:tic20 at cam.ac.uk>> wrote:
Hi all,

For anyone who's interested: following on from my quick demo of the use of std::async to simplify threading of long-running tasks called from Python, I've put some example code at https://github.com/tristanic/async-hello-world/tree/master/src (implemented as a ChimeraX plugin for demo purposes, but the C++ code and bindings are entirely generic).

To re-iterate what this is about, and why I think it's important: in brief, for most real-world problems Python-level threading is not particularly useful since (except in specific cases) the Global Interpreter Lock limits the overall performance to no better than - and often much worse than - single-threaded code. While multiprocessing can be an option for some tasks, it's (a) slow to get started, (b) memory-hungry due to the need for each process to have a complete copy of all the data, (c) rather difficult to organise with respect to intercommunication, and (d) not very cross-platform friendly (I'm looking at you, Windows!). The point of this example is to show that with C++11 organising the threading at the C++ level can often end up being both the simplest and fastest approach - it's only a dozen or so extra lines of fairly boilerplate code required to allow the Python programmer to choose between running the exact same function either in the main thread or asynchronously in a new one.

Why I think it's important: I've said in the past, and continue to believe, that we should be aiming for (interactive) model building, validation and refinement to merge together into a single integrated process - as much as anything, ISOLDE is meant to be the proof of principle that this is a possibility. But for that to work, high-performance and cross-platform parallelism is a real necessity. The approach described here is how I'm currently getting ChimeraX to maintain 30+ fps graphics performance while simultaneously running interactive molecular dynamics and structure factor calculations. I suspect there are many functions in CCTBX that could potentially benefit from this approach even when not considering GUI applications.

Best regards,

Tristan

_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
http://phenix-online.org/mailman/listinfo/cctbxbb
_______________________________________________
cctbxbb mailing list
cctbxbb at phenix-online.org<mailto:cctbxbb at phenix-online.org>
http://phenix-online.org/mailman/listinfo/cctbxbb


-- 
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/cctbxbb/attachments/20201002/00e0fbb3/attachment.htm>