Ranger cannot record Knox audit log to hdfs

[#1]

Hello, I would like to ask a question.

I integrated Ranger + Knox + LDAP on my test cluster.

Ranger could records the audit logs into HDFS except the audit log of Knox.

It doesn't have any user or group synchronize problem with LDAP on my cluster.

Let me list up those component version of my cluster:

Ambari 2.2.1

HDP 2.4.2.0

Ranger: 0.5.0.2.4

Knox: 0.6.0.2.4

The following error message is from /var/log/knox/gateway.log

I cannot figure out why it couldn't recognize my cluster nameservice.

My HDFS directory is hdfs://testcluster/ranger/audit

2016-07-06 19:24:18,591 ERROR queue.AuditFileSpool (AuditFileSpool.java:logError(710)) - Error sending logs to consumer. provider=knox.async.batch, consumer=knox.async.batch.hdfs
2016-07-06 19:25:18,669 ERROR provider.BaseAuditHandler (BaseAuditHandler.java:logError(329)) - Error writing to log file.
java.lang.IllegalArgumentException: java.net.UnknownHostException: testcluster
	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:411)
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:311)
	at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:150)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
	at org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:221)
	at org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:123)
	at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:890)
	at org.apache.ranger.audit.queue.AuditFileSpool.runDoAs(AuditFileSpool.java:838)
	at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:759)
	at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:757)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:360)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1689)
	at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:765)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: testcluster
	... 22 more

Please help me figure this out.

Thanks

[#2]

is this HA cluster?

can you please provide the knox audit confguration

[#3 · Accepted solution]

can you please refer to this document :http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Ranger_Install_Guide/content/save_audits_.... and please see the steps to create the following symbolic links:

Link /etc/hadoop/conf/hdfs-site.xml file to /etc/knox/conf/hdfs-site.xml

Link /etc/hadoop/conf/core-site.xml file to /etc/knox/conf/core-site.xml

More

[#4]

After configured by following the document.

It works now.

Thank you very much.

[#5]

Soft linking of hdfs and core site xmls on KNOX Gateway server fixed the UnknownHostException issue.

The versions I was working with are

HDP - 2.5

Ranger - 0.6.0.2.5

Knox - 0.9.0.2.5

Thanks

[#6]

did it solve the problem ?

[#7]

Is 'testcluster' the name of the topology file that you've configured via Knox?

You should have a file named 'testcluster.xml' under conf/topologies