The Construction of Large Graph Data Structures in a Scalable Distributed Message System

A large-scale distributed graph data substrate is the foundation of graph databases and knowledge base systems. Existing systems either scale up the capacity of a single compute node or build on distributed relational models (table based systems). The former limits the scalability of the graph system, and the latter can constrain the performance. We have designed a new graph data system based on a scalable message system. The new data model enables most of existing graph algorithms designed for a monolithic computer to run smoothly on a plurality of distributed nodes, and achieves high performance on complex graph operations. The implementation supports 100 million vertices and one billion arcs on a commodity computer cluster, and provides scalability to potentially operate on tens of billions of vertices.