Hyperic Monitoring Availability Grey Circles

I setup Hyperic Monitoring on my company’s server cluster so that I could get all the awesome data about the tomcat servers and jboss servers because most of our servers are running something in java. It seemed like the better way to go than say Cacti or Nagios. I have monitoring and graphs of data. Perfect right? Wrong! This piece of software has been a nightmare from the beginning. From problems getting the server started properly to getting the agents to report proper data. The Alerts are unreliable in my opinion too. With all the initial setup problems out of the way, here is my newest problem that I fixed after asking questions on their support forum. I Googled this and couldn’t find a single thing so I thought I would share.

Here is the problem: I have 3 nodes that are identical except for hostname and ip address. All three were working fine until all of a sudden node2 quit showing metric data. Now, it reported as available as a “platform” but all services showed that damn gray circle for availability. So it seems that the agent is not sending the data. For those of you who don’t know, there are 4 states in hyperic - Green=all metrics reporting, Yellow=Some metrics are reporting, Red=appears down, and Gray=not receiving a thing for this service. Usually gray comes up when something needs to be configured. Well, out of the blue, I get all gray circles for node2 and the other 2 nodes report fine. I tried to restart the agent. I even tried re-setting the agent’s connection using the setup command. Nothing worked. Well after a day of Googling and searching their forums I found nothing about the problem. So I decided to ask the question on the forum.

The reponse I got was that I should check the system times. If the metric reports things that are timestamped ahead of the HQ server time, the HQ says “I can’t get things from the future…that makes no sense” and considers it bad data. Hence the gray circles.

So, if you have this problem in your Hyperic HQ Dashboard, try syncing all the servers’ time (I suggest setting up an NTP server locally so that all the servers are syncing to each other and not an unreliable source). Then Restart the agents and watch the magic.

Helpful links:
My Forum Thread
Hyperic’s Site

Technorati Tags: Hyperic, Server, Monitoring, HQ, SysAdmin