Monitoring System Components - Data Collection

This area covers the different ways that the system collects the metrics, status and other information about what it's monitoring. There are 3 basic types of data the monitoring system cares about:
(a) Metrics
(b) Status
(c) Configuration

I cover Data for Configuration in the Configuration Management section.
I cover Data for Status in the Status Tracking section.
Data for metrics can find its way into the monitoring system by the monitoring system either polling (pulling) the system/component being monitored, or by the target system pushing the data asynchronously into the monitoring system. A good monitoring system has the flexibility to provide both options as each fits a particular system behavior characteristics and requirements.


In the above diagram, the target system sends metrics to the monitoring system asynchronously (1). The monitoring system didn't initiate the request for metrics. This is typically done by either leveraging a scheduler in the target system, or some other trigger (e.g. metrics generated by activity on the target system). When the monitoring system initiates the metrics request (2), the scheduling of that request is performed on the monitoring system, and the request is sent to the target system. After the request is sent, the monitoring system (typically) waits for the response. This would ensure that the metrics that are returned in the payload are associated with that particular request (and more importantly with the timestamp of the request). In rare occasions, the monitoring system can send the request, and can accept the result asynchronously via method (1). This option is typically difficult to accommodate since the slight clock difference between the monitoring server and the target system can throw off the logic of associating the metrics to the real timestamp.

There are pros and cons of each option.