Summary​

This year, our primary focus has been maintaining the NDIP platform - updating packages, fixing bugs, and adding more automation. We also upgraded Galaxy to the latest version (Galaxy Release Notes).

The bulk of our work centered on the NOVA project, where we are developing advanced visualization tools for analyzing neutron scattering experiments.

Additionally, we launched the NOVA Dashboard, which enables users to run interactive tools without relying on the Galaxy GUI.

Neutron Tools​

Our primary focus was on developing Trame applications and configuring them as interactive tools for NDIP.

Summary​

In this release, nearing the end of the project, we focused on improving platform stability, reorganizing and refactoring our scientific tools, and continuing the development of interactive tools based on the Trame framework. We also updated Galaxy to the latest version (see the Galaxy Release Notes).

Great news — we secured funding for the next year through the follow-up project, NOVA, which will focus on creating advanced visualization tools for analyzing neutron scattering experiments.

Neutron Tools​

Our primary focus was on developing interactive tools and customized interfaces. We finalized the Jana and Amira interactive tools, added the SaSView tool, and enhanced the Live Neutron Data Monitor tool.

In total, we now have 71 tools, most of them either completed or in the final stages of development.

Security​

We opened Calvera for external users via XCAMS. Additionally, we modified Galaxy to enable external Python applications to communicate with Galaxy using OIDC tokens.

Summary​

In this release, we focused on creating interactive tools based on the Trame framework, continued adding tools from scientific projects, and finalized work on submitting jobs to Frontier.

Neutron Tools​

We added several interactive tools based on Trame (still prototypes), including Topaz and SANS data reduction, working with Neutron users in an iterative approach to improve the tools.

In total, we now have 68 tools in different stages of development.

Data Management​

We added an option to ingest whole folders from the SNS analysis cluster to Calvera and implemented and tested an option to automatically ingress data from the WebMon Service into Calvera.

Computational Resources​

Calvera can now send jobs to Frontier (to get Frontier time, one needs an approved OLCF project).

Security​

We moved Calvera to an external IP and working now with the security team on the firewall configuration request.

Summary​

In this release, we continued adding Neutron tools, improving user experience based on feedback, implemented access to the platform for XCAMS users, and enhanced testing and logging capabilities.

Neutron Tools​

We added several interactive tools, including Jana2000 (a Windows application) and Amira, Paraview Visualizer, investigating tools based on the Trame framework.

The new LDRD projects are also adding their tools to Calvera:

• Machine Learning Assisted SANS Data Analysis Platform (MLSANS)
• GPU-based Resolution And Visualization Interface for Triple-Axis Spectrometers (GRAVITAS)
• Experimental Steering for Powder Diffraction (ESPD)
• Smart System for Neutron Crystallography (SGUI)

In total, we now have over 60 tools (+30 in this cycle) in different stages of development.

Computational Resources​

In addition to the CADES cloud VMs, one can now send jobs to a new GPU machine (4 NVIDIA L40 GPUs). We are also able to submit jobs to Frontier, but this is still a work in progress.

Security​

We now support login for XCAMS users, though access still needs to be within the ORNL network. Work is in progress on allowing external access.

Introduction​

Calvera provides the basis for creating modular scientific workflows for the analysis and interpretation of neutron scattering data obtained during experiments. It encourages experimentation and exploration while providing the ability to create reproducible end-to-end workflows, interactive tools, visualizations, ...

Calvera is based on the open-source platform Galaxy, and the main work done in this release was to adopt Galaxy to run on heterogeneous ORNL infrastructure and to create an initial set of neutron scattering data analysis tools.

Neutron Tools​

over 30 pilot tools have been created for various use cases:

• Advanced Neutron Data Analysis for Quantum Materials
• Atomistic Modelling and Machine Learning
• Automated Structural Refinement
• Inelastic Neutron Scattering
• Instrument Data Reduction - USANS, GPSANS, TOPAZ
• Interactive Jupyter Notebook
• Monte Carlo Ray Tracing

Computational Resources​

User jobs can be sent to CADES cloud VMs (with and without GPU) and Summit. This happens transparently for the user via the Calvera web interface or using API, so a user never has to log in to a specific resource.

Data Management​

Data is also managed by Calvera transparently for the users. The following functionality is implemented:

• SNS/HFIR data ingest (no copy involved)
• uploading data from the user local storage
• efficiently manage data between various storage enclaves (cloud storage, GPFS on Summit, SNS analysis cluster)
• export results to SNS/HFIR storage.

Security​

Users log in to Calvera with their UCAMS account using ORNL's single sign-in page. No further authentication is required. User jobs always run "as user" using containers (Docker in CADES and Singularity on Summit) therefore, naturally getting access to their data.

Documentation and Training Material​

• Documentation is available at https://calvera.ornl.gov/docs
• Tutorials are available at https://calvera.ornl.gov/training