Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Ranger cannot record Knox audit log to hdfs

avatar
Explorer

Hello, I would like to ask a question.

I integrated Ranger + Knox + LDAP on my test cluster.

Ranger could records the audit logs into HDFS except the audit log of Knox.

It doesn't have any user or group synchronize problem with LDAP on my cluster.

Let me list up those component version of my cluster:

Ambari 2.2.1

HDP 2.4.2.0

Ranger: 0.5.0.2.4

Knox: 0.6.0.2.4

The following error message is from /var/log/knox/gateway.log

I cannot figure out why it couldn't recognize my cluster nameservice.

My HDFS directory is hdfs://testcluster/ranger/audit

2016-07-06 19:24:18,591 ERROR queue.AuditFileSpool (AuditFileSpool.java:logError(710)) - Error sending logs to consumer. provider=knox.async.batch, consumer=knox.async.batch.hdfs
2016-07-06 19:25:18,669 ERROR provider.BaseAuditHandler (BaseAuditHandler.java:logError(329)) - Error writing to log file.
java.lang.IllegalArgumentException: java.net.UnknownHostException: testcluster
	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:411)
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:311)
	at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:150)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
	at org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:221)
	at org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:123)
	at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:890)
	at org.apache.ranger.audit.queue.AuditFileSpool.runDoAs(AuditFileSpool.java:838)
	at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:759)
	at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:757)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:360)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1689)
	at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:765)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: testcluster
	... 22 more

Please help me figure this out.

Thanks

1 ACCEPTED SOLUTION

avatar

can you please refer to this document :http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Ranger_Install_Guide/content/save_audits_.... and please see the steps to create the following symbolic links:

Link /etc/hadoop/conf/hdfs-site.xml file to /etc/knox/conf/hdfs-site.xml

Link /etc/hadoop/conf/core-site.xml file to /etc/knox/conf/core-site.xml

More

View solution in original post

6 REPLIES 6

avatar

is this HA cluster?

can you please provide the knox audit confguration

avatar

can you please refer to this document :http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Ranger_Install_Guide/content/save_audits_.... and please see the steps to create the following symbolic links:

Link /etc/hadoop/conf/hdfs-site.xml file to /etc/knox/conf/hdfs-site.xml

Link /etc/hadoop/conf/core-site.xml file to /etc/knox/conf/core-site.xml

More

avatar
Explorer

After configured by following the document.

It works now.

Thank you very much.

avatar
Contributor

Soft linking of hdfs and core site xmls on KNOX Gateway server fixed the UnknownHostException issue.

The versions I was working with are

HDP - 2.5

Ranger - 0.6.0.2.5

Knox - 0.9.0.2.5

Thanks

avatar

did it solve the problem ?

avatar
Expert Contributor

Is 'testcluster' the name of the topology file that you've configured via Knox?

You should have a file named 'testcluster.xml' under conf/topologies