JACoW is a publisher in Geneva, Switzerland that publishes the proceedings of accelerator conferences held around the world by an international collaboration of editors.
TY - CONF AU - Malka, J. AU - Aplin, S. AU - Boukhelef, D. AU - Dietrich, S. AU - Filippakopoulos, K. AU - Gasthuber, MA. AU - Hannappel, J. AU - Karimi, M. AU - Kemp, Y. AU - Lueken, R. AU - Maia, L.G. AU - Mkrtchyan, T. AU - Ohrenberg, K. AU - Piszczek, T. AU - Previtali, Mr. AU - Schlünzen, F. AU - Suchowski, P. AU - Szuba, J. AU - Voss, C. AU - Wrona, K. ED - Schaa, Volker RW ED - Götz, Andy ED - Venter, Johan ED - White, Karen ED - Robichon, Marie ED - Rowland, Vivienne TI - Data Management Infrastructure for European XFEL J2 - Proc. of ICALEPCS2023, Cape Town, South Africa, 09-13 October 2023 CY - Cape Town, South Africa T2 - International Conference on Accelerator and Large Experimental Physics Control Systems T3 - 19 LA - english AB - Effective data management is crucial to ensure research data is easily accessible and usable. We will present design and implementation of the European XFEL data management infrastructure supporting high level data management services. The system architecture comprises four layers of storage systems, each designed to address specific challenges. The first layer, referred to as online, is designed as a fast cache to accommodate extreme high rates (up to 15GB/s) of data generated during experiment at single scientific instrument. The second layer, called high-performance storage, provides necessary capabilities for data processing both during and after experiments. The layers are incorporated into a single infiniband fabric and connected through a 4km long 1Tb/s link. This allows fast data transfer from the European XFEL experiment hall to the DESY computing center. The third layer, mass-storage, extends the capacity of data storage system to allow mid-term data access for detailed analysis. Finally, the tape archive, provides data safety and long-term archive (5-10years). The high performance and mass storage systems are connected to computing clusters. This allows users to perform near-online and offline data analysis or alternatively export data outside of the European XFEL facility. The data management infrastructure at the European XFEL has the capacity to accept and process up to 2PB of data per day, which demonstrates the remarkable capabilities of all the sub-services involved in this process. PB - JACoW Publishing CP - Geneva, Switzerland SP - 952 EP - 957 KW - experiment KW - FEL KW - data-management KW - network KW - hardware DA - 2024/02 PY - 2024 SN - 2226-0358 SN - 978-3-95450-238-7 DO - doi:10.18429/JACoW-ICALEPCS2023-WE1BCO02 UR - https://jacow.org/icalepcs2023/papers/we1bco02.pdf ER -