question Re: Ranger cannot record Knox audit log to hdfs in Archives of Support Questions (Read Only)

Ranger cannot record Knox audit log to hdfs

sula678 — Wed, 06 Jul 2016 18:12:43 GMT

Hello, I would like to ask a question.

I integrated Ranger + Knox + LDAP on my test cluster.

Ranger could records the audit logs into HDFS except the audit log of Knox.

It doesn't have any user or group synchronize problem with LDAP on my cluster.

Let me list up those component version of my cluster:

Ambari 2.2.1

HDP 2.4.2.0

Ranger: 0.5.0.2.4

Knox: 0.6.0.2.4

The following error message is from /var/log/knox/gateway.log

I cannot figure out why it couldn't recognize my cluster nameservice.

My HDFS directory is hdfs://testcluster/ranger/audit

2016-07-06 19:24:18,591 ERROR queue.AuditFileSpool (AuditFileSpool.java:logError(710)) - Error sending logs to consumer. provider=knox.async.batch, consumer=knox.async.batch.hdfs
2016-07-06 19:25:18,669 ERROR provider.BaseAuditHandler (BaseAuditHandler.java:logError(329)) - Error writing to log file.
java.lang.IllegalArgumentException: java.net.UnknownHostException: testcluster
	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:411)
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:311)
	at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:150)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
	at org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:221)
	at org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:123)
	at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:890)
	at org.apache.ranger.audit.queue.AuditFileSpool.runDoAs(AuditFileSpool.java:838)
	at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:759)
	at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:757)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:360)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1689)
	at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:765)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: testcluster
	... 22 more

Please help me figure this out.

Thanks

Re: Ranger cannot record Knox audit log to hdfs

dsharma — Wed, 06 Jul 2016 18:15:04 GMT

is this HA cluster?

can you please provide the knox audit confguration

Re: Ranger cannot record Knox audit log to hdfs

dsharma — Wed, 06 Jul 2016 18:28:36 GMT

can you please refer to this document :http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Ranger_Install_Guide/content/save_audits_to_hdfs.html. and please see the steps to create the following symbolic links:

Link /etc/hadoop/conf/hdfs-site.xml file to /etc/knox/conf/hdfs-site.xml

Link /etc/hadoop/conf/core-site.xml file to /etc/knox/conf/core-site.xml

Re: Ranger cannot record Knox audit log to hdfs

dsharma — Wed, 06 Jul 2016 19:26:15 GMT

did it solve the problem ?

Re: Ranger cannot record Knox audit log to hdfs

zblanco — Thu, 07 Jul 2016 01:23:29 GMT

Is 'testcluster' the name of the topology file that you've configured via Knox?

You should have a file named 'testcluster.xml' under conf/topologies

Re: Ranger cannot record Knox audit log to hdfs

sula678 — Thu, 07 Jul 2016 08:45:10 GMT

After configured by following the document.

It works now.

Thank you very much.

Re: Ranger cannot record Knox audit log to hdfs

jay1ram2 — Fri, 28 Oct 2016 04:14:25 GMT

Soft linking of hdfs and core site xmls on KNOX Gateway server fixed the UnknownHostException issue.

The versions I was working with are

HDP - 2.5

Ranger - 0.6.0.2.5

Knox - 0.9.0.2.5

Thanks