Panopticon is an end-to-end system for monitoring
very large computer networks

Overview

Monitoring Systems are required for the management of anything beyond the smallest networks of computers. While specialised monitoring systems can be deployed to detect specific problems, more general systems are required to detect unexpected issues, and track performance trends.

Current monitoring systems are generally designed for small networks. While large fleets of computers are becoming more common, there are no existing general monitoring systems that can scale to monitor such fleets and provide effective visualisation. Scale is an issue in both the design and presentation of large-scale monitoring systems.

Architecture

Our solution divided the problem up into three components:

Visualisation, Storage & Retrieval, Collection Visualisation Storage & Retrieval Collection

Results

Our system should be able to scale to monitor tens of thousands of nodes, on commodity hardware. Our prototype system was tested to scale to approximately 25,000 monitored nodes under artificial test conditions.

With a few improvements, we believe that it can scale far further.