ICALEPCS2023 - List of Keywords (data-management)

Paper	Title	Other Keywords	Page
TUPDP070	Open Time Proposal Submission System for the MeerKAT Radio Telescope	operation, instrumentation, software, site	666
	R.L. Schwartz, T.B. Baloyi, S.S. Sithole SARAO, Cape Town, South Africa
	Through periodic Call for Proposals, the South African Radio Astronomy Observatory (SARAO), allocates time on the MeerKAT Radio Telescope to the international community for the purpose of maximizing the scientific impact of the telescope, while contributing to South African scientific leadership and human capital development. Proposals are submitted through the proposal submission system, followed by a stringent review process where they are graded based on certain criteria. Time on the telescope is then allocated based on the grade and rank achieved. This paper outlines the details of the Open Time proposal submission and review process, and the design and implementation of the software used to grade the proposals and allocate the time on the MeerKAT Radio Telescope.
	Poster TUPDP070 [0.490 MB]
DOI •	reference for this paper ※ doi:10.18429/JACoW-ICALEPCS2023-TUPDP070
About •	Received ※ 27 September 2023 — Accepted ※ 13 October 2023 — Issued ※ 19 October 2023
Cite •	reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)

WE1BCO02	Data Management Infrastructure for European XFEL	experiment, FEL, network, hardware	952
	J. Malka, S. Aplin, D. Boukhelef, K. Filippakopoulos, L.G. Maia, T. Piszczek, Mr. Previtali, J. Szuba, K. Wrona EuXFEL, Schenefeld, Germany S. Dietrich, MA. Gasthuber, J. Hannappel, M. Karimi, Y. Kemp, R. Lueken, T. Mkrtchyan, K. Ohrenberg, F. Schlünzen, P. Suchowski, C. Voss DESY, Hamburg, Germany
	Effective data management is crucial to ensure research data is easily accessible and usable. We will present design and implementation of the European XFEL data management infrastructure supporting high level data management services. The system architecture comprises four layers of storage systems, each designed to address specific challenges. The first layer, referred to as online, is designed as a fast cache to accommodate extreme high rates (up to 15GB/s) of data generated during experiment at single scientific instrument. The second layer, called high-performance storage, provides necessary capabilities for data processing both during and after experiments. The layers are incorporated into a single infiniband fabric and connected through a 4km long 1Tb/s link. This allows fast data transfer from the European XFEL experiment hall to the DESY computing center. The third layer, mass-storage, extends the capacity of data storage system to allow mid-term data access for detailed analysis. Finally, the tape archive, provides data safety and long-term archive (5-10years). The high performance and mass storage systems are connected to computing clusters. This allows users to perform near-online and offline data analysis or alternatively export data outside of the European XFEL facility. The data management infrastructure at the European XFEL has the capacity to accept and process up to 2PB of data per day, which demonstrates the remarkable capabilities of all the sub-services involved in this process.
DOI •	reference for this paper ※ doi:10.18429/JACoW-ICALEPCS2023-WE1BCO02
About •	Received ※ 06 October 2023 — Revised ※ 23 October 2023 — Accepted ※ 08 December 2023 — Issued ※ 12 December 2023
Cite •	reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)

WE3BCO06	Assonant: A Beamline-Agnostic Event Processing Engine for Data Collection and Standardization	experiment, controls, software, synchrotron	1025
	P.B. Mausbach, E.X. Miqueles, A. Pinto LNLS, Campinas, Brazil
	Synchrotron radiation facilities comprise beamlines designed to perform a wide range of X-ray experimental techniques which require complex instruments to monitor thermodynamic variables, sample-related variables, among others. Thus, synchrotron beamlines can produce heterogeneous sets of data and metadata, hereafter referred to as data, which impose several challenges to standardizing them. For open science and FAIR principles, such standardization is paramount for research reproducibility, besides accelerating the development of scalable and reusable data-driven solutions. To address this issue, the Assonant was devised to collect and standardize the data produced at beamlines of Sirius, the Brazilian fourth-generation synchrotron light source. This solution enables a NeXus-compliant technique-centric data standard at Sirius transparently for beamline teams by removing the burden of standardization tasks from them and providing a unified standardization solution for several techniques at Sirius. The Assonant implements a software interface to abstract data format-related specificities and to send the produced data to an event-driven infrastructure composed of streaming processing and microservices, able to transform the data flow according to NeXus. This paper presents the development process of Assonant, the strategy adopted to standardize beamlines with different operating stages, and challenges faced during the standardization process for macromolecular crystallography and imaging data at Sirius. M. Könnecke et al., ’The nexus data format’, Journal of applied crystallography, vol. 48, no. 1, pp. 301-305, 2015.
	Slides WE3BCO06 [4.909 MB]
DOI •	reference for this paper ※ doi:10.18429/JACoW-ICALEPCS2023-WE3BCO06
About •	Received ※ 05 October 2023 — Accepted ※ 08 December 2023 — Issued ※ 18 December 2023
Cite •	reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)

THPDP048	SARAO Science Repository: Sustainable Use of MeerKAT Data	software, database, interface, framework	1415
	Z. Kukuma, G. Coetzer, R.S. Kupa, C. Schollar SARAO, Cape Town, South Africa
	Funding: National Research Foundation (South Africa) The South African Radio Astronomy Observatory (SARAO) is excited to announce the forthcoming release of its digital repository for managing and preserving astronomical data. The repository, built using the DSpace platform, will allow researchers to catalogue and discover research data in a standardised way, while Digital Object Identifiers (DOIs) through the Datacite service will ensure the unique identification and persistent citation of data. The data will be hosted on a Ceph archive, which provides reliable storage and efficient retrieval using the s3 protocol. We are looking forward to hosting science data from any scientist who has used SARAO instruments. Researchers will be able to apply to host their data on the SARAO digital repository service, which will be released in the coming month. This repository will serve as a critical resource for the astronomy community, providing easy access to valuable data for research and collaboration. With the increasing demand for digital preservation and data accessibility, we believe that the SARAO digital repository will set a standard for other astronomical institutions to follow. We are committed to ensuring that our data remains available and accessible for the long term, and we invite all interested researchers to participate in this exciting initiative.
DOI •	reference for this paper ※ doi:10.18429/JACoW-ICALEPCS2023-THPDP048
About •	Received ※ 05 October 2023 — Revised ※ 12 October 2023 — Accepted ※ 17 December 2023 — Issued ※ 22 December 2023
Cite •	reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)

Paper

Title

Other Keywords

Page

TUPDP070

Open Time Proposal Submission System for the MeerKAT Radio Telescope

operation, instrumentation, software, site

666

R.L. Schwartz, T.B. Baloyi, S.S. Sithole
SARAO, Cape Town, South Africa

Through periodic Call for Proposals, the South African Radio Astronomy Observatory (SARAO), allocates time on the MeerKAT Radio Telescope to the international community for the purpose of maximizing the scientific impact of the telescope, while contributing to South African scientific leadership and human capital development. Proposals are submitted through the proposal submission system, followed by a stringent review process where they are graded based on certain criteria. Time on the telescope is then allocated based on the grade and rank achieved. This paper outlines the details of the Open Time proposal submission and review process, and the design and implementation of the software used to grade the proposals and allocate the time on the MeerKAT Radio Telescope.

Poster TUPDP070 [0.490 MB]

DOI •

reference for this paper ※ doi:10.18429/JACoW-ICALEPCS2023-TUPDP070

About •

Received ※ 27 September 2023 — Accepted ※ 13 October 2023 — Issued ※ 19 October 2023

Cite •

reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)

WE1BCO02

Data Management Infrastructure for European XFEL

experiment, FEL, network, hardware

952

J. Malka, S. Aplin, D. Boukhelef, K. Filippakopoulos, L.G. Maia, T. Piszczek, Mr. Previtali, J. Szuba, K. Wrona
EuXFEL, Schenefeld, Germany
S. Dietrich, MA. Gasthuber, J. Hannappel, M. Karimi, Y. Kemp, R. Lueken, T. Mkrtchyan, K. Ohrenberg, F. Schlünzen, P. Suchowski, C. Voss
DESY, Hamburg, Germany

Effective data management is crucial to ensure research data is easily accessible and usable. We will present design and implementation of the European XFEL data management infrastructure supporting high level data management services. The system architecture comprises four layers of storage systems, each designed to address specific challenges. The first layer, referred to as online, is designed as a fast cache to accommodate extreme high rates (up to 15GB/s) of data generated during experiment at single scientific instrument. The second layer, called high-performance storage, provides necessary capabilities for data processing both during and after experiments. The layers are incorporated into a single infiniband fabric and connected through a 4km long 1Tb/s link. This allows fast data transfer from the European XFEL experiment hall to the DESY computing center. The third layer, mass-storage, extends the capacity of data storage system to allow mid-term data access for detailed analysis. Finally, the tape archive, provides data safety and long-term archive (5-10years). The high performance and mass storage systems are connected to computing clusters. This allows users to perform near-online and offline data analysis or alternatively export data outside of the European XFEL facility. The data management infrastructure at the European XFEL has the capacity to accept and process up to 2PB of data per day, which demonstrates the remarkable capabilities of all the sub-services involved in this process.

DOI •

reference for this paper ※ doi:10.18429/JACoW-ICALEPCS2023-WE1BCO02

About •

Received ※ 06 October 2023 — Revised ※ 23 October 2023 — Accepted ※ 08 December 2023 — Issued ※ 12 December 2023

Cite •

reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)

WE3BCO06

Assonant: A Beamline-Agnostic Event Processing Engine for Data Collection and Standardization

experiment, controls, software, synchrotron

1025

P.B. Mausbach, E.X. Miqueles, A. Pinto
LNLS, Campinas, Brazil

Synchrotron radiation facilities comprise beamlines designed to perform a wide range of X-ray experimental techniques which require complex instruments to monitor thermodynamic variables, sample-related variables, among others. Thus, synchrotron beamlines can produce heterogeneous sets of data and metadata, hereafter referred to as data, which impose several challenges to standardizing them. For open science and FAIR principles, such standardization is paramount for research reproducibility, besides accelerating the development of scalable and reusable data-driven solutions. To address this issue, the Assonant was devised to collect and standardize the data produced at beamlines of Sirius, the Brazilian fourth-generation synchrotron light source. This solution enables a NeXus-compliant technique-centric data standard at Sirius transparently for beamline teams by removing the burden of standardization tasks from them and providing a unified standardization solution for several techniques at Sirius. The Assonant implements a software interface to abstract data format-related specificities and to send the produced data to an event-driven infrastructure composed of streaming processing and microservices, able to transform the data flow according to NeXus*. This paper presents the development process of Assonant, the strategy adopted to standardize beamlines with different operating stages, and challenges faced during the standardization process for macromolecular crystallography and imaging data at Sirius.
* M. Könnecke et al., ’The nexus data format’, Journal of applied crystallography, vol. 48, no. 1, pp. 301-305, 2015.

Slides WE3BCO06 [4.909 MB]

DOI •

reference for this paper ※ doi:10.18429/JACoW-ICALEPCS2023-WE3BCO06

About •

Received ※ 05 October 2023 — Accepted ※ 08 December 2023 — Issued ※ 18 December 2023

Cite •

reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)

THPDP048

SARAO Science Repository: Sustainable Use of MeerKAT Data

software, database, interface, framework

1415

Z. Kukuma, G. Coetzer, R.S. Kupa, C. Schollar
SARAO, Cape Town, South Africa

Funding: National Research Foundation (South Africa)
The South African Radio Astronomy Observatory (SARAO) is excited to announce the forthcoming release of its digital repository for managing and preserving astronomical data. The repository, built using the DSpace platform, will allow researchers to catalogue and discover research data in a standardised way, while Digital Object Identifiers (DOIs) through the Datacite service will ensure the unique identification and persistent citation of data. The data will be hosted on a Ceph archive, which provides reliable storage and efficient retrieval using the s3 protocol. We are looking forward to hosting science data from any scientist who has used SARAO instruments. Researchers will be able to apply to host their data on the SARAO digital repository service, which will be released in the coming month. This repository will serve as a critical resource for the astronomy community, providing easy access to valuable data for research and collaboration. With the increasing demand for digital preservation and data accessibility, we believe that the SARAO digital repository will set a standard for other astronomical institutions to follow. We are committed to ensuring that our data remains available and accessible for the long term, and we invite all interested researchers to participate in this exciting initiative.

DOI •

reference for this paper ※ doi:10.18429/JACoW-ICALEPCS2023-THPDP048

About •

Received ※ 05 October 2023 — Revised ※ 12 October 2023 — Accepted ※ 17 December 2023 — Issued ※ 22 December 2023

Cite •

reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)