Open Stack

Hi,

  I'd like to validate an observation:

  In coding the data locality feature in HDP, I've notice that if I set the property 'enable_hypervisor_awareness' to 'False', the rack mapping generated by the topology_helper is incompatible with the specified net.toplogy.impl (org.apache.hadoop.net.NetworkTopologyWithNodeGroup).  Such a mismatch causes datanode failures with the following exceptions:

2013-12-05 14:05:39,423 INFO org.apache.hadoop.ipc.Server: IPC Server handler 39 on 50300, call heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus at 421a5817, true, true, true, -1) from 10.1.0.6:55722: error: java.io.IOException: java.lang.IllegalArgumentException: Unexpected data node /rack7/test2-slave-002.novalocal at an illegal network location
java.io.IOException: java.lang.IllegalArgumentException: Unexpected data node /rack7/test2-slave-002.novalocal at an illegal network location
	at org.apache.hadoop.net.NetworkTopologyWithNodeGroup.add(NetworkTopologyWithNodeGroup.java:199)
	at org.apache.hadoop.mapred.JobTracker.addHostToNodeMapping(JobTracker.java:2831)
	at org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2823)
	at org.apache.hadoop.mapred.JobTracker.addNewTracker(JobTracker.java:2801)
	at org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3301)
	at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3001)
	at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1440)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1438)

  Changing the setting's value to 'org.apache.hadoop.net.NetworkTopology' rectifies the issue.

  I'd like to know if setting the value to 'false' has been tested with the vanilla plugin and worked?  If that is the case then I'll make the appropriate change to the HDP plugin code (manipulate the value of net.toplogy.impl based on hypervisor awareness setting).  However, if that setting has always been left as 'True' (the default) during testing, perhaps this is simply a testing gap and I should actually make the change in the topology_helper?

-- Jon


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.