Izvorni znanstveni članak

https://doi.org/10.17559/TV-20211213044600

A Novel Completely Local Repairable Code Algorithm Based on Erasure Code

Ying Fang ; Shangqiu Normal University, China
Hai Tan ; 1) Nanjing Audit University, China 2) East China University of Technology, China
Shuaifang Wang ; Shangqiu Normal University, China
Xin Zhang ; East China University of Technology, China
Gejian Liao ; East China University of Technology, China
Jun Zhang ; East China University of Technology, China

Sažetak

Hadoop Distributed File System (HDFS) is widely used in massive data storage. Because of the disadvantage of the multi-copy strategy, the hardware expansion of HDFS cannot keep up with the continuous volume of big data. Now, the traditional data replication strategy has been gradually replaced by Erasure Code due to its smaller redundancy rate and storage overhead. However, compared with replicas, Erasure Code needs to read a certain amount of data blocks during the process of data recovery, resulting in a large amount of overhead for I/O and network. Based on the Reed-Solomon (RS) algorithm, we propose a novel Completely Local Repairable Code (CLRC) algorithm. By grouping RS coded blocks and generating local check blocks, CLRC algorithm can optimize the locality of the RS algorithm, which can reduce the cost of data recovery. Evaluations show that the CLRC algorithm can reduce the bandwidth and I/O consumption during the process of data recovery when a single block is damaged. What's more, the cost of decoding time is only 59% of the RS algorithm.

Ključne riječi

data recovery; erasure code; HDFS; RS algorithm, storage overhead

Hrčak ID:

281681

URI

https://hrcak.srce.hr/281681

Datum izdavanja:

15.10.2022.

Posjeta: 904 *