SARAO, Cape Town, South Africa
The 64 receptor (with 20 more being built) radio telescope in the Karoo, South Africa, comprises a large number of devices and components connected to the Control-and-Monitoring (CAM) system via the Karoo Array Telescope Communication Protocol (KATCP). KATCP is used extensively for internal communications between CAM components and other subsystems. A KATCP interface exposes requests and sensors; sampling strategies are set on sensors, ranging from several updates per second to infrequent on-change updates. The sensor samples are of different types, from small integers to text fields. The samples and associated timestamps are permanently stored and made available for scientists, engineers and operators to query and analyze. This is a presentation on how to apply Machine Learning tools which utilize data-driven algorithms and statistical models to analyze sensor data sets and then draw inferences from identified patterns or make predictions based on them. The algorithms learn from the sensor data as they run against it, unlike traditional rules-based analytics systems that follow explicit instructions. Since this involves data preprocessing, we will go through how the MeerKAT telescope data storage infrastructure (called Katstore) manages the voluminous variety, velocity and volume of this data.
