1.
|
|
2.
|
|
A scheme to implement local server computation on EOS system based on Xrootd plug-in
/ zhang, minxing (speaker) (The Institute of High Energy Physics of the Chinese Academy of Sciences)
Particle physics computing model has a kind of high statistical calculation, such applications need to access a large amount of data for analysis, the data I/O capability is very high requirements.For example, the LHAASO experiment generates trillions of events each year, and the large raw data needs to be decode to encode and mark before it can be analyzed. In this process, very high I/O bandwidth is required, otherwise an I/O bottleneck will form.When using the EOS file system, the user cannot know the physical storage location of the file, and when the user needs to access the file, it needs to search the MGM, transfer the file from the FST to the client, and the client provides the target file to the user.In this process, if the user needs to perform such IO intensive operations as mentioned above, there are two limitations on I/O bandwidth, one is the storage node's hard disk read and write efficiency, the other is the network bandwidth between the FST and the client.In this case, if the data storage unit and the computing unit can be integrated into one, the data handling can be significantly reduced, and the parallelism and energy efficiency of computing can be greatly improved.Currently, the potential of this kind of integrated memory and computing storage is attracting the attention of many companies and standards bodies.SNIA has formed a working group to establish standards for interoperability between computable storage devices, and the OpenFog Consortium is also working on standards for computable storage.
Therefore, we propose a scheme to implement local server computation on EOS system based on Xrootd plug-in.Flags can be added after a file is accessed when a user needs to use computable storage.After receiving the access request, the client will forward the request to the FST where the file is located and perform the default decode calculation in the background on the FST.After testing, we found that using this method to simultaneously decode 10 1G raw files stored on the same FST can save about 45.9% of the time compared to the traditional method.The next work plan is to sink the computable module onto the hard disk to reduce the CPU consumption of the FST, and to customize the acceleration module on the hardware to increase the speed of the computation..
2021 - 1135.
HEP Computing; EOS workshop
External links: Talk details; Event details
In : EOS workshop
|
|
3.
|
|
CloudStor Minio: Improving S3 performance in CloudStor
/ D'Silva, Michael (speaker) (AARNet)
We at AARNet as well as the research community in Australia need bulk data access to our synch servers because one-off ingest of seriously large datasets performs subpar across the webdav/synch pathway. This presentation will discuss AARNet’s experiences, journey and many iterations to achieve high-speed data transfers via S3 protocol (the de facto standard) and the challenges and improvements made along the way.
Minio helps some users interact with CloudStor using the S3 protocol. [...]
2019 - 753.
HEP Computing; EOS workshop
External links: Talk details; Event details
In : EOS workshop
|
|
4.
|
|
5.
|
|
Scalable Metadata Management Using Onedata and OpenFaaS
/ Dutka, Lukasz (speaker)
Onedata [1] is a global high-performance, transparent data management system, that unifies data access across globally distributed infrastructures and multiple types of underlying storages, such as NFS, Amazon S3, Ceph, OpenStack Swift, WebDAV, XRootD and HTTP and HTTPS servers, as well as other POSIX-compliant file systems.
Onedata allows users to collaborate, share, and perform computations on data using applications relying on POSIX compliant data access. Thanks to a fully distributed architecture, Onedata allows for the creation of complex hybrid-cloud infrastructure deployments, including private and commercial cloud resources. [...]
2021 - 709.
HEP Computing; CS3 2021- Cloud Storage Synchronization and Sharing
External links: Talk details; Event details
In : CS3 2021- Cloud Storage Synchronization and Sharing
|
|
6.
|
Scaling the EOS namespace
/ Peters, Andreas J (CERN) ; Sindrilaru, Elvin A (CERN) ; Bitzes, Georgios (CERN)
EOS is the distributed storage system being developed at CERN with the aim of fulfilling a wide range of data storage needs, ranging from physics data to user home directories. Being in production since 2011, EOS currently manages around 224 petabytes of disk space and 1.4 billion files across several instances. [...]
2017
In : ISC High Performance 2017 International Workshops, DRBSD, ExaComm, HCPM, HPC-IODC, IWOPH, IXPUG, P^3MA, VHPC, Visualization at Scale, WOPSSS, Frankfurt, Germany, 18 - 22 Jun 2017, pp.731-740
|
|
7.
|
File Access Optimization with the Lustre Filesystem at Florida CMS T2
/ Avery, P (Florida U.) ; Bourilkov, D (Florida U.) ; Fu, Y (Florida U.) ; Kim, B (Florida U.)
The Florida CMS Tier2 center, one of the CMS Tier2 centers, has been using the Lustre filesystem for its data storage backend system since 2004. Recently, the data access pattern at our site has changed greatly due to various new access methods that include file transfers through the GridFTP servers, read access from the worker nodes, and the remote read access through the xrootd servers. [...]
2015 - 7 p.
- Published in : J. Phys.: Conf. Ser. 664 (2015) 042028
IOP Open Access article: PDF;
In : 21st International Conference on Computing in High Energy and Nuclear Physics, Okinawa, Japan, 13 - 17 Apr 2015, pp.042028
|
|
8.
|
|
|
Next Generation PanDA Pilot for ATLAS and Other Experiments
/ Nilsson, P (Texas U., Arlington) ; Barreiro Megino, F (CERN) ; Caballero Bejar, J (Brookhaven Natl. Lab.) ; De, K (Texas U., Arlington) ; Hover, J (Brookhaven Natl. Lab.) ; Love, P (Lancaster U.) ; Maeno, T (Brookhaven Nat) ; Medrano Llamas, R (CERN) ; Walker, R (Munich U.) ; Wenaus, T (Brookhaven Natl. Lab.)
The Production and Distributed Analysis system (PanDA) has been in use in the ATLAS Experiment since 2005. [...]
ATL-SOFT-PROC-2013-025.
-
2014. - 5 p.
Original Communication (restricted to ATLAS) - Full text - Full text
|
|
9.
|
Next Generation PanDA Pilot for ATLAS and Other Experiments
/ Nilsson, P ; Barreiro Megino, F ; Caballero Bejar, J ; De, K ; Hover, J ; Love, P ; Maeno, T ; Medrano Llamas, R ; Walker, R ; Wenaus, T
The Production and Distributed Analysis system (PanDA) has been in use in the ATLAS Experiment since 2005. It uses a sophisticated pilot system to execute submitted jobs on the worker nodes. [...]
ATL-SOFT-SLIDE-2013-812.-
Geneva : CERN, 2013 - 1 p.
Fulltext: PDF; External link: Original Communication (restricted to ATLAS)
In : 20th International Conference on Computing in High Energy and Nuclear Physics 2013, Amsterdam, Netherlands, 14 - 18 Oct 2013
|
|
10.
|
|