登录 | 注册 | 退出 | 公司首页 | 繁体中文 | 满意度调查
综合馆
一种面向海量存储系统的高效元数据集群管理方案
  • 摘要

    高效的、去中心化的元数据管理方案对大型分布式存储系统的可靠性、可扩展性起至关重要的作用.针对基于Hash划分和基于子树划分的元数据管理方案扩展代价巨大、对集群变动敏感等问题,提出一种基于一致性 Hash结构的元数据服务器(metadata server ,MDS)集群化方案———CH‐MMS (consistent Hash based metadata management schema).CH‐MMS 在一致性 MDS 集群上引入虚拟MDS(Virtual MDS),有效平衡MDS集群负载;将Standby机制与延迟更新策略融合并应用于MDS集群,实现MDS快速失效恢复以及集群变动时零数据迁移量.阐述了CH‐MMS的体系结构,介绍了核心数据结构layout‐table、虚拟MDS结构、延迟更新机制及相关算法,并对CH‐MMS扩展性、容错性作了定性分析.最后通过原型系统和模拟实验说明,CH‐MMS具有元数据平衡分布、快速失效恢复、灵活的扩展性以及零结点变动数据迁移量等特点,能满足数据量不断增加的大规模存储集群元数据灵活、高效管理的需求.

  • 作者

    肖中正  陈宁江  魏峻  张文博  Xiao Zhongzheng  Chen Ningjiang  Wei Jun  Zhang Wenbo 

  • 作者单位

    广西大学计算机与电子信息学院 南宁 530004/中国科学院软件研究所软件工程技术研究开发中心 北京 100190

  • 刊期

    2015年4期 ISTIC EI PKU

  • 关键词

    元数据管理  一致性Hash  大数据存储  元数据服务器  分布式文件系统  metadata management  consistent Hash  large-scale data storage  metadata server (MDS)  distributed file system 

参考文献
  • [1] Traeger A;Zadok E;Joukov N. A nine year study of file system and storage benchmarking. ACM Trans on Storage, 2008,02
  • [2] Borthakur D;Gray J;Sarma J S. Apache hadoop goes realtime at Facebook. New York:ACM, 2011
  • [3] Pawlowski B;Juszczak C;Staubach P. NFS version 3:Design and implementation. Berkeley,CA:USENIX Association, 1994
  • [4] Satyanarayanan M;Kistler J J;Kumar P. Coda:A highly available file system for a distributed workstation environment. {H}IEEE Transactions on Computers, 1990,04
  • [5] Weil S A;Brandt S A;Miller E L. Ceph:A scalable,high-performance distributed file system. Berkeley,CA:USENIX Association, 2006
  • [6] Liu Jiangchuan;Xu Jianliang. Proxy caching for media streaming over the Internet. {H}IEEE Communications Magazine, 2004,08
  • [7] Rodeh O;Teperman A. zFS-a scalable distributed file system using object disks. Los Alamitos:IEEE Computer Society, 2003
  • [8] Brandt S A;Miller E L;Long D D E. Efficient metadata management in large distributed storage systems. Los Alamitos:IEEE Computer Society, 2003
  • [9] Mateljan V;Cisic D;Ogrizovic D. Cloud database-as-a-service(DaaS)-ROI. Los Alamitos:IEEE Computer Society, 2010
  • [10] The Apache Software Foundation. The apache cassandra project. http://cassandra.apache.org, 2013-07-15
  • [11] Ou L;Engelmann C;He X. Symmetric active/active metadata service for highly available cluster storage systems. Calgary,AB,Canada:A C T A, 2007
  • [12] Roselli D S;Lorch J R;Anderson T E. A comparison of file system workloads. Berkeley,CA:USENIX Association, 2000
  • [13] Ts'o T Y;Tweedie S. Planned extensions to the Linux EXT2/EXT3 filesystem. Berkeley,CA:USENIX Association, 2002
  • [14] Mathur A;Cao M;Bhattacharya S. The new ext4 filesystem:Current status and future plans. Ottawa:Linux Symposium Inc, 2007
  • [15] Hua Yu;Jiang Hong;Zhu Yifeng. SmartStore:A new metadata organization paradigm with semantic-awareness for next-generation file systems. Los Alamitos:IEEE Computer Society, 2009
  • [16] Hua Yu;Jiang Hong;Zhu Yifeng. Semantic-aware metadata organization paradigm in next-generation file systems. {H}IEEE Transactions on Parallel and Distributed Systems, 2012,02
  • [17] Hua Yu;Jiang Hong;Zhu Yifeng. Rapport:Semantic-sensitive namespace management in large-scale file systems. Lincoln,NE:Department of Computer Science and Engineering,University of Nebraska-Lincoln, 2010
  • [18] Leung A W;Shao M;Bisson T. Spyglass:Fast,scalable metadata search for large-scale storage systems. Berkeley,CA:USENIX Association, 2009
  • [19] Weil S A;Pollack K T;Brandt S A. Dynamic metadata management for petabyte-scale file systems. Los Alamitos:IEEE Computer Society, 2004
  • [20] Hua Yuhua;Zhu Yifeng;Jiang Hong. Scalable and adaptive metadata management in ultra large-scale file systems. Los Alamitos:IEEE Computer Society, 2008
  • [21] Karger D R;Ruhl M. Simple efficient load balancing algorithms for peer-to-peer systems. New York:ACM, 2004
  • [22] Stoica I;Morris R;Karger D. Chord:A scalable peer-to-peer lookup service for Internet applications. ACM SIGCOMM Computer Communication Review, 2001,04
  • [23] Karger D;Lehman E;Leighton T. Consistent hashing and random trees:Distributed caching protocols for relieving hot spots on the world wide Web. New York:ACM, 1997
  • [24] Ghemawat S;Gobioff H;Leung S T. The google file system. New York:ACM, 2003
  • [25] Shvachko K;Kuang H;Radia S. The hadoop distributed file system. Los Alamitos:IEEE Computer Society, 2010
查看更多︾
相似文献 查看更多>>
18.232.188.251