Creation Review

Eclipse Scanning allows experiments to be conducted by coordinating the operation of scientific instruments, for example motors or detectors. It sequences the movements of these instruments (or devices) in order to scan different parts of the experimental space. For instance you might scan a temperature controller to conduct an experiment at different temperatures or move a goniometer through a range of optical angles or combine the two in a two dimensional scan. Scanning is useful as an open source project because the algorithms which complete scans during experiments are the same in many areas of research. Hardware is experiment specific so scanning algorithms can be used in many settings, whereever electronically controlled hardware does automated experiments. 

Detailed Description

Introduction

Scientific facilities operating high end hardware, for instance robots, motorized stages and detectors, have well defined layers for integrating these devices. For instance it is possible for a call to be made to move a motor or expose a detector (similar to taking an image with a digital camera). Examples of these control layers are the EPICS framework, and the TANGO framework. These frameworks are open source and in wide use at facilities around the globe. Facilities also require low latency operation, for instance a fast acting motor and a detector may need to coordinate specialized hardware. For people not familiar with this idea, one example of this is the Zebra box which combines an ARM processor with an FPGA and can orchestrate hardware times down to the nanosecond scale. There is also a device called TFG.

Design

Introduction

The project will be divided into a services layer to provide the scanning features, an example user interface layer implemented in RCP/jface (users may choose to use another UI layer) (this UI will be used at Diamond Light Source) and a cpython/jython layer for easily scripting the scans.

Server and Clients

The services layer may be operated on a separate server and messages passed between the client and server with a messaging system provided by the project. The scanning project does not prescribe exactly how services are used however if they are started in an OSGi container, declarative services will work. If they are not some manual ‘wiring together’ might be required. One possible arrangement of the services for scanning is shown below. This arrangement is site specific to Diamond Light Source and represents a possible deployment of the services layer. (DAWN is used for plotitng and analysis here but any package may be used with the scanning project for this purpose.) Different facilities would use the various services in this project different ways.

Services Layer

IPointGeneratorService

The point generator service takes a model and provides a generator, IPointGenerator, which gives each nD point in the scan. So for instance in a mapping scan, each point has a x,y stage value for the two-dimensional scan.

Image showing a spiral scan path over a 2D stage

Generators may be nested (unlimited) to provide complex scans. For instance it is possible to combine a grid with temperature step scan. Generators may be added by extension point so that scan paths not envisaged by the developer of the service can be added. Generators may be added by cpython/jython to allow users to define experimental specific scan procedures, including logic to be executed for each point.

IRunnableDeviceService

This service returns devices, IRunnableDevice, which conform to a state machine for driving scans. It is used to run a scan and is the core interface for scanning. It returns the top level scan which uses a point generator to define each scan point. It returns each device in the scan, for instance 2D detectors or devices driving EPICS Area Detector or a Malcolm Device, conform to this interface and pass through the states defined.

See http://pymalcolm.readthedocs.io/en/latest/arch/statemachine.html

AcquisitionDevice (IRunnableDevice)

When doing a CPU scan the Java-based runnable device uses a thread pool to manage the scan. It combines a move with a detector readout to maximize the scan speed. So for scanning one point we have:

_|....|________   run() Tell detector(s) to collect current position

______|.....|___  write() Tell detector(s) to write data,

______|.............|___ setPosition() move motor(s) to next position

IDeviceConnectorService

Provides a connection to devices which are to be scanned. The devices take part in the scan by setting/getting value so they represent things like motors, temperature controllers and goniometer angles. Each device connected to must conform to a simple interface called IScannable.

IScannable<T> … {

 public T getPosition() throws Exception;

 public void setPosition(T value, IPosition position) throws Exception;

 ...

}

IEventService

An event system is required to receive scan requests, notify the user of scan progress and maintain queues. The event service provides the following models:

• Publish/Subscribe   (events like scan finished)
• Submit/Consume    (queues like those for running scans)
• Request/Response (ask the server a question such as how many detectors)

This event system is also used to manage analysis algorithms which are outside the scope of the scanning project.

IMalcolmService

To make it really easy to talk to any hardware, a middleware layer has been designed called ‘Malcolm’ which bridges between hardware and the runnable device interface/state machine. Malcolm is outside the scope of the scanning project. Malcolm devices which implement IRunnableDevice are made available from it using the IMalcolmService internally to the IRunnableDeviceService. (Malcolm is similar in this respect to TANGO and it may be desirable in future to allow TANGO devices to be scanned and/or allow the scanning device to conform to a TANGO device.)

NeXus Builder Service

The NexusBuilderFactory is used in the scanning to write legal NeXus HDF5 files. NeXus is a self-describing binary file format used in many facilities to record large numerical data efficiently. The DAWN product can read any correctly written NeXus file and provides a large armoury of tools with which to analyse data, for example running fast analysis pipelines on clusters. Devices can be integrated with NeXus by implementing a declarative interface called INexusDevice.

Other Services

There are several other services for connecting to python, running scripts etc. which are included in the project.

User Interface Layer

The project provides user interface parts for visualizing the queues of scans. It also provides a simple scan builder perspective, called ‘Scanning’, which allows a scan to be created and run. This perspective will work without configuration to the scanning project and execute mock devices provided in the project examples. The user interface allows a scan to be defined and configured with available devices and then submitted to the scanning service using the event service. This allows scans to be received on a server if one is implemented. This may be desirable if the client and acquisition are separate, for instance in the case of a remote thick client or a web client.

There is also be an example called ‘X-Ray Centering’ which shows a simple submit/consume using the scanning project. This example requires DAWN libraries to be available because it uses the DAWN plotting system as a service. In the future it may be desirable to remove this dependency as the dawnsci project intends to release plotting separately.

Scripting Layer

The scripting layer is intended to provide a python API which is easy to use and drives the runnable device service. It can submit scans to the scan queue or to be directly run. It uses a ScanRequest object on the java side of the service which is identical to the user interface. The doc from the mscan method is shown here which defines how it works.

Usage:

mscan(scan model(s), detector model(s))

   A simple usage of this function is as follows:

   > mscan(step(my_scannable, 0, 10, 1), det=mandelbrot(0.1))

   The above invokation says "please perform a mapping scan over my scannable

   from 0 to 10 with step size 1, collecting data from the 'Mandelbrot'

   detector with an exposure time of 0.1 seconds at each step".

   You can specify multiple detectors with a list (square brackets):

   > mscan(..., det=[mandelbrot(0.1), another_detector(0.4)])

   You can specify a scannable or list of scannables to monitor:

   > mscan(..., mon=my_scannable, ...)  # or:

   > mscan(..., mon=[my_scannable, another_scannable], ...)

   You can embed one scan path inside another to create a compound scan path:

   > mscan([step(s, 0, 10, 1), step(f, 1, 5, 1)], ...)

   The above invocation says "for each point from 0 to 10 on my slow axis, do

   a scan from 1 to 5 on my fast axis". In fact, for the above case, a grid-

   type scan would be more idiomatic:

   > mscan(grid(axes=(f, s), step=(1, 1), origin=(0, 0), size=(10, 4)), ...)

   By default, this function will submit the scan request to a queue and

   return immediately. You may override this behaviour with the "now" and

   "block" keywords:

   > # Don't return until the scan is complete.

   > mscan(..., ..., block=True)

   > # Skip the queue and run the scan now (but don't wait for completion).

   > mscan(..., ..., now=True)

   > # Skip the queue and return once the scan is complete.

   > # For some beamlines now=True, block=True may need to be defaulted in localstation.

   > mscan(..., ..., now=True, block=True)

NeXus Writing

It is important to ensure that writing of NeXus files is performant. Diamond Light Source have partially funded the new SWMR (“swimmer”) functionality in HDF5 which ensures that one process may write to a file, quickly, while other processes read from the same binary file. This is HDF5 library version 2.10 and has been tested with the scanning project. The project has been designed for devices to write correct NeXus records by implementing a single interface i.e. it should be straightforward to add devices. Other file formats would be out of the scope of the initial phase of the project but would be desirable in the future depending on those willing to be involved with the project.

Current Deployment

The project is deployed at Diamond Light Source and is intended to be the backbone of the next generation of data acquisition at the facility. A screenshot of a user interface developed using the project is included below to show how scanning can be reused. The user interface is for mapping experiments, these experiments per-se are outside the scope of the scanning project and their user interfaces not included however the controller and model from scanning are used to deliver a mapping functionality at Diamond Light Source.

Example of polygonal scanning in a mapping experiment