GATHERING HADOOP METRICS TO GANGLIA

Standard

These days I was working with Ganglia. In UCD Lab, We have 10 slave + 1 master Hadoop Cluster. We need to monitor our cluster during Hadoop jobs. For this problem we used ganglia which is fancy monitoring tool. According to Ganglia People: “Ganglia is a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids.” Also It provides Web UI so we can check our cluster status. It’s bit oldy but still is goldy. But it still receives updates.

I was looking for way to connect my hadoop cluster to ganglia. Fortunately, Hadoop community provide solution connecting Hadoop to ganglia.

Let’s configure…

For Master:

Change directory to Hadoop/etc/hadoop/

Edit hadoop-metrics2.properties(PS: period defines the number of seconds between each update cycle.)

namenode.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
namenode.sink.ganglia.period=10
namenode.sink.ganglia.servers=PUT_HERE_GMETAD_IP:8649

resourcemanager.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
resourcemanager.sink.ganglia.period=10
resourcemanager.sink.ganglia.servers=PUT_HERE_GMETAD_IP:8649

For each slave:

Change directory to Hadoop/etc/hadoop/

Edit hadoop-metrics2.properties

datanode.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
datanode.sink.ganglia.period=10
datanode.sink.ganglia.servers=PUT_HERE_GMETAD_IP:8649

nodemanager.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
nodemanager.sink.ganglia.period=10
nodemanager.sink.ganglia.servers=PUT_HERE_GMETAD_IP:8649

After editing, now restart Hadoop Cluster:

mr-jobhistory-deamon.sh stop historyserver
stop-yarn.sh
stop-dfs.sh

mr-jobhistory-deamon.sh start historyserver
start-yarn.sh
start-dfs.sh

Now restart gmetad

/etc/init.d/gmetad restart

As a last, restart each ganglia-monitor:

service ganglia-monitor restart

Perfect now you can monitor your cluster from heartbeat to jvm status.

ganglia-heartbeat

Leave a Reply

Your email address will not be published. Required fields are marked *