Keyword: data-management
Paper Title Other Keywords Page
TUPDP070 Open Time Proposal Submission System for the MeerKAT Radio Telescope operation, instrumentation, software, site 666
 
  • R.L. Schwartz, T.B. Baloyi, S.S. Sithole
    SARAO, Cape Town, South Africa
 
  Through periodic Call for Proposals, the South African Radio Astronomy Observatory (SARAO), allocates time on the MeerKAT Radio Telescope to the international community for the purpose of maximizing the scientific impact of the telescope, while contributing to South African scientific leadership and human capital development. Proposals are submitted through the proposal submission system, followed by a stringent review process where they are graded based on certain criteria. Time on the telescope is then allocated based on the grade and rank achieved. This paper outlines the details of the Open Time proposal submission and review process, and the design and implementation of the software used to grade the proposals and allocate the time on the MeerKAT Radio Telescope.  
poster icon Poster TUPDP070 [0.490 MB]  
DOI • reference for this paper ※ doi:10.18429/JACoW-ICALEPCS2023-TUPDP070  
About • Received ※ 27 September 2023 — Accepted ※ 13 October 2023 — Issued ※ 19 October 2023  
Cite • reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)  
 
WE1BCO02 Data Management Infrastructure for European XFEL experiment, FEL, network, hardware 952
 
  • J. Malka, S. Aplin, D. Boukhelef, K. Filippakopoulos, L.G. Maia, T. Piszczek, Mr. Previtali, J. Szuba, K. Wrona
    EuXFEL, Schenefeld, Germany
  • S. Dietrich, MA. Gasthuber, J. Hannappel, M. Karimi, Y. Kemp, R. Lueken, T. Mkrtchyan, K. Ohrenberg, F. Schlünzen, P. Suchowski, C. Voss
    DESY, Hamburg, Germany
 
  Effective data management is crucial to ensure research data is easily accessible and usable. We will present design and implementation of the European XFEL data management infrastructure supporting high level data management services. The system architecture comprises four layers of storage systems, each designed to address specific challenges. The first layer, referred to as online, is designed as a fast cache to accommodate extreme high rates (up to 15GB/s) of data generated during experiment at single scientific instrument. The second layer, called high-performance storage, provides necessary capabilities for data processing both during and after experiments. The layers are incorporated into a single infiniband fabric and connected through a 4km long 1Tb/s link. This allows fast data transfer from the European XFEL experiment hall to the DESY computing center. The third layer, mass-storage, extends the capacity of data storage system to allow mid-term data access for detailed analysis. Finally, the tape archive, provides data safety and long-term archive (5-10years). The high performance and mass storage systems are connected to computing clusters. This allows users to perform near-online and offline data analysis or alternatively export data outside of the European XFEL facility. The data management infrastructure at the European XFEL has the capacity to accept and process up to 2PB of data per day, which demonstrates the remarkable capabilities of all the sub-services involved in this process.  
DOI • reference for this paper ※ doi:10.18429/JACoW-ICALEPCS2023-WE1BCO02  
About • Received ※ 06 October 2023 — Revised ※ 23 October 2023 — Accepted ※ 08 December 2023 — Issued ※ 12 December 2023
Cite • reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)  
 
WE3BCO06 Assonant: A Beamline-Agnostic Event Processing Engine for Data Collection and Standardization experiment, controls, software, synchrotron 1025
 
  • P.B. Mausbach, E.X. Miqueles, A. Pinto
    LNLS, Campinas, Brazil
 
  Synchrotron radiation facilities comprise beamlines designed to perform a wide range of X-ray experimental techniques which require complex instruments to monitor thermodynamic variables, sample-related variables, among others. Thus, synchrotron beamlines can produce heterogeneous sets of data and metadata, hereafter referred to as data, which impose several challenges to standardizing them. For open science and FAIR principles, such standardization is paramount for research reproducibility, besides accelerating the development of scalable and reusable data-driven solutions. To address this issue, the Assonant was devised to collect and standardize the data produced at beamlines of Sirius, the Brazilian fourth-generation synchrotron light source. This solution enables a NeXus-compliant technique-centric data standard at Sirius transparently for beamline teams by removing the burden of standardization tasks from them and providing a unified standardization solution for several techniques at Sirius. The Assonant implements a software interface to abstract data format-related specificities and to send the produced data to an event-driven infrastructure composed of streaming processing and microservices, able to transform the data flow according to NeXus*. This paper presents the development process of Assonant, the strategy adopted to standardize beamlines with different operating stages, and challenges faced during the standardization process for macromolecular crystallography and imaging data at Sirius.
* M. Könnecke et al., ’The nexus data format’, Journal of applied crystallography, vol. 48, no. 1, pp. 301-305, 2015.
 
slides icon Slides WE3BCO06 [4.909 MB]  
DOI • reference for this paper ※ doi:10.18429/JACoW-ICALEPCS2023-WE3BCO06  
About • Received ※ 05 October 2023 — Accepted ※ 08 December 2023 — Issued ※ 18 December 2023  
Cite • reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)  
 
THPDP048 SARAO Science Repository: Sustainable Use of MeerKAT Data software, database, interface, framework 1415
 
  • Z. Kukuma, G. Coetzer, R.S. Kupa, C. Schollar
    SARAO, Cape Town, South Africa
 
  Funding: National Research Foundation (South Africa)
The South African Radio Astronomy Observatory (SARAO) is excited to announce the forthcoming release of its digital repository for managing and preserving astronomical data. The repository, built using the DSpace platform, will allow researchers to catalogue and discover research data in a standardised way, while Digital Object Identifiers (DOIs) through the Datacite service will ensure the unique identification and persistent citation of data. The data will be hosted on a Ceph archive, which provides reliable storage and efficient retrieval using the s3 protocol. We are looking forward to hosting science data from any scientist who has used SARAO instruments. Researchers will be able to apply to host their data on the SARAO digital repository service, which will be released in the coming month. This repository will serve as a critical resource for the astronomy community, providing easy access to valuable data for research and collaboration. With the increasing demand for digital preservation and data accessibility, we believe that the SARAO digital repository will set a standard for other astronomical institutions to follow. We are committed to ensuring that our data remains available and accessible for the long term, and we invite all interested researchers to participate in this exciting initiative.
 
DOI • reference for this paper ※ doi:10.18429/JACoW-ICALEPCS2023-THPDP048  
About • Received ※ 05 October 2023 — Revised ※ 12 October 2023 — Accepted ※ 17 December 2023 — Issued ※ 22 December 2023
Cite • reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)