Creation Review

January is a set of libraries for handling numerical data in Java. It is inspired in part by NumPy and aims to provide similar functionality.

Why use it?

• Familiar. Provide familiar functionality, especially to NumPy users.
• Robust. Has test suite and is used in production heavily at Diamond Light Source.
• No more passing double[]. IDataset provide a consistent object for basing APIs on with significantly improved clarity over using double arrays or similar.
• Optimized. Optimized for speed and getting better all the time.
• Scalable. Allows handling of data sets larger than available memory with "Lazy Datasets".
• Focus on your algorithms. By reusing this library it allows you to focus on your code.

For a basic example, have a look at the example project: BasicExample.java

Browse through the more advanced examples.

• NumPy Examples show how common NumPy constructs map to Eclipse Datasets.
• Slicing Examples demonstrate slicing, including how to slice a small amount of data out of a dataset too large to fit in memory all at once.
• Error Examples demonstrate applying an error to datasets.
• Iteration Examples demonstrate a few ways to iterate through your datasets.
• Lazy Examples demonstrate how to use datasets which are not entirely loaded in memory.

This project provides features to allow scientific software to be inter-operable. Algorithms exist today which could be shared between existing Eclipse-based applications however in practice they describe data and do plotting using specific APIs which are not inter-operable or interchangeable. This project defines a structure to make this a thing of the past. It defines an inter-operable layer upon which to build RCP applications such that your user interface, data or plotting can be reused by others.

The Eclipse Integrated Computational Environment (ICE) will address the usability needs of the scientific and engineering community for the Big Four modeling and simulation activities. The focus of the ICE will be to develop an easily extended and reusable set of tools that can be used by developers to create rich user interfaces for their modeling and simulation products. Custom widgets and data structures with well-defined interfaces and high-coverage unit tests will be provided for plugin developers. Plugins based on the ICE tools will also be developed and released with the ICE for codes that the developers use and the community contributes to the extent that resources allow. The idea is that the tools should be available for developers to do what they do and the deployment mechanism should be ready and waiting when they are finished.

Each of the big four tasks presents unique challenges, but much of the capability required can be handled by extensions to the existing Eclipse tools.

Creating models for scientific and engineering projects is a very difficult task that is, presently, very time consuming and hampered by the lack of an integrated tool chain. For example, consider the screenshot in Figure 1 of an input for a battery simulation configured using the code in the Initial Contribution.

The activity depicted in Figure 1 appears to be straightforward. Some information is required to pick a case and some other information is required in a table. After that information is provided, some buttons can be clicked and an input file will be generated. This seems relatively simple when described as such and presented with an Eclipse Form, but on the backend it requires detailed knowledge about the input format and four simulation codes. It also requires dependency tracking and a little bit of math. Both the input format and the four simulation codes are relatively well known in the battery simulation community, but no tools exist for using them easily.

Figure 1 - Setting up the input for a battery simulation.

A user who was not using the tool would have to learn all of those things themselves, much like a Java programmer not using Eclipse for Java Developers or a similar tool would have to learn the details of using the “javac” and “java” command line tools. However, with the tool the user can perform that task far quicker.

The Eclipse Form in the example takes advantage of several data structures and widgets that are automatically created and rendered based on very little information from the developer. The instructions used to draw the widgets on the screen are separate from the domain-specific information (business logic) provided by the plugin developers. While the developer is responsible for saying that they want information from users about time integration or component configurations, they never have to be responsible for drawing that information. An extension API exists, but they can (and tend to) leave that task up to the computer scientists.

The second of the Big Four activities, launching jobs, is in some ways the defining activity of modeling and simulation. It can also be extremely difficult if the simulation in question requires an exotic machine, a special operating system or really any system with which the user is not intimately familiar. In interviews, users tend to say that they would rather push a button, enter a password if the job is remote and watch the information stream back than “work the shell” themselves. In some cases a user may not even know how to use a shell on the operating system in question and cannot easily run the job.

Similar to the proposed modeling tools, the ICE will provide a simple developer interface for configuring job information - by template - for users. These tools will handle input matching with jobs, the actual execution and streaming information back to the user in an organized and recoverable way that is easily reproducible. This burden on users and developers is simplified to that of saying what needs to be launched, although one does this in the user interface and the other writes, for example, a reusable, templated shell script.

Analyzing and managing data, the last two of the Big Four activities, go hand in hand. There are some cases where data can not be managed or analyzed on a local machine because the data files are too big. The exact ways of doing that with a suite like the proposed ICE are open areas of research. However, for smaller data sets there are many things that can be done to improve usability.

The most obvious thing that can be done is to provide users a way to read their data. It is increasingly common in the science and engineering community to use a binary data format, so providing some way to read that data is key. The second most obvious thing to do is provide simple information on the provenance and present location of the data. (That may seem obvious, but it is commonly skipped in scientific codes.)

It is much more useful to provide visualization capabilities for users that can be pre-programmed by developers that help them extract knowledge from the data, as shown in Figure 2. It is important to make a distinction between different types of data since there are different types of users. Some users will prefer full 3D views of their simulations on the mesh as shown in the left pane of Figure 2. Other users will prefer parameterized views of derived macroscopic quantities as shown in the right pane of Figure 2.

In both cases it is important to make these tools easily accessible to users and plugin developers. The view on the left of Figure 2 uses an extension that enables viewing 3D meshes in Eclipse with VisIt, https://wci.llnl.gov/codes/visit/, which some years ago was successful in rendering images with over one trillion elements of data. The parameterized view in the right pane of Figure 2 is also a good example. It is a Cartesian grid drawn with the Graphical Editing Framework and is fully interactive, resizable and can be exported to an image for users with a few clicks. It has a simple interface for user interface developers and can handle hexagonal and circular grids in addition to Cartesian grids.

Figure 2 - A 3D image of the mesh in a battery simulation on the left and a parameterized view of a nuclear fuel assembly and the power on a few rods.