Author: Filippakopoulos, K.
Paper Title Page
WE1BCO02 Data Management Infrastructure for European XFEL 952
 
  • J. Malka, S. Aplin, D. Boukhelef, K. Filippakopoulos, L.G. Maia, T. Piszczek, Mr. Previtali, J. Szuba, K. Wrona
    EuXFEL, Schenefeld, Germany
  • S. Dietrich, MA. Gasthuber, J. Hannappel, M. Karimi, Y. Kemp, R. Lueken, T. Mkrtchyan, K. Ohrenberg, F. Schlünzen, P. Suchowski, C. Voss
    DESY, Hamburg, Germany
 
  Ef­fec­tive data man­age­ment is cru­cial to en­sure re­search data is eas­ily ac­ces­si­ble and us­able. We will pre­sent de­sign and im­ple­men­ta­tion of the Eu­ro­pean XFEL data man­age­ment in­fra­struc­ture sup­port­ing high level data man­age­ment ser­vices. The sys­tem ar­chi­tec­ture com­prises four lay­ers of stor­age sys­tems, each de­signed to ad­dress spe­cific chal­lenges. The first layer, re­ferred to as on­line, is de­signed as a fast cache to ac­com­mo­date ex­treme high rates (up to 15GB/s) of data gen­er­ated dur­ing ex­per­i­ment at sin­gle sci­en­tific in­stru­ment. The sec­ond layer, called high-per­for­mance stor­age, pro­vides nec­es­sary ca­pa­bil­i­ties for data pro­cess­ing both dur­ing and after ex­per­i­ments. The lay­ers are in­cor­po­rated into a sin­gle in­fini­band fab­ric and con­nected through a 4km long 1Tb/s link. This al­lows fast data trans­fer from the Eu­ro­pean XFEL ex­per­i­ment hall to the DESY com­put­ing cen­ter. The third layer, mass-stor­age, ex­tends the ca­pac­ity of data stor­age sys­tem to allow mid-term data ac­cess for de­tailed analy­sis. Fi­nally, the tape archive, pro­vides data safety and long-term archive (5-10years). The high per­for­mance and mass stor­age sys­tems are con­nected to com­put­ing clus­ters. This al­lows users to per­form near-on­line and of­fline data analy­sis or al­ter­na­tively ex­port data out­side of the Eu­ro­pean XFEL fa­cil­ity. The data man­age­ment in­fra­struc­ture at the Eu­ro­pean XFEL has the ca­pac­ity to ac­cept and process up to 2PB of data per day, which demon­strates the re­mark­able ca­pa­bil­i­ties of all the sub-ser­vices in­volved in this process.  
DOI • reference for this paper ※ doi:10.18429/JACoW-ICALEPCS2023-WE1BCO02  
About • Received ※ 06 October 2023 — Revised ※ 23 October 2023 — Accepted ※ 08 December 2023 — Issued ※ 12 December 2023
Cite • reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)