Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ranger cannot record Knox audit log to hdfs

Solved Go to solution
Highlighted

Ranger cannot record Knox audit log to hdfs

New Contributor

Hello, I would like to ask a question.

I integrated Ranger + Knox + LDAP on my test cluster.

Ranger could records the audit logs into HDFS except the audit log of Knox.

It doesn't have any user or group synchronize problem with LDAP on my cluster.

Let me list up those component version of my cluster:

Ambari 2.2.1

HDP 2.4.2.0

Ranger: 0.5.0.2.4

Knox: 0.6.0.2.4

The following error message is from /var/log/knox/gateway.log

I cannot figure out why it couldn't recognize my cluster nameservice.

My HDFS directory is hdfs://testcluster/ranger/audit

2016-07-06 19:24:18,591 ERROR queue.AuditFileSpool (AuditFileSpool.java:logError(710)) - Error sending logs to consumer. provider=knox.async.batch, consumer=knox.async.batch.hdfs
2016-07-06 19:25:18,669 ERROR provider.BaseAuditHandler (BaseAuditHandler.java:logError(329)) - Error writing to log file.
java.lang.IllegalArgumentException: java.net.UnknownHostException: testcluster
	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:411)
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:311)
	at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:150)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
	at org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:221)
	at org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:123)
	at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:890)
	at org.apache.ranger.audit.queue.AuditFileSpool.runDoAs(AuditFileSpool.java:838)
	at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:759)
	at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:757)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:360)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1689)
	at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:765)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: testcluster
	... 22 more

Please help me figure this out.

Thanks

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Ranger cannot record Knox audit log to hdfs

can you please refer to this document :http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Ranger_Install_Guide/content/save_audits_.... and please see the steps to create the following symbolic links:

Link /etc/hadoop/conf/hdfs-site.xml file to /etc/knox/conf/hdfs-site.xml

Link /etc/hadoop/conf/core-site.xml file to /etc/knox/conf/core-site.xml

More

View solution in original post

6 REPLIES 6
Highlighted

Re: Ranger cannot record Knox audit log to hdfs

is this HA cluster?

can you please provide the knox audit confguration

Highlighted

Re: Ranger cannot record Knox audit log to hdfs

can you please refer to this document :http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Ranger_Install_Guide/content/save_audits_.... and please see the steps to create the following symbolic links:

Link /etc/hadoop/conf/hdfs-site.xml file to /etc/knox/conf/hdfs-site.xml

Link /etc/hadoop/conf/core-site.xml file to /etc/knox/conf/core-site.xml

More

View solution in original post

Highlighted

Re: Ranger cannot record Knox audit log to hdfs

New Contributor

After configured by following the document.

It works now.

Thank you very much.

Re: Ranger cannot record Knox audit log to hdfs

Explorer

Soft linking of hdfs and core site xmls on KNOX Gateway server fixed the UnknownHostException issue.

The versions I was working with are

HDP - 2.5

Ranger - 0.6.0.2.5

Knox - 0.9.0.2.5

Thanks

Highlighted

Re: Ranger cannot record Knox audit log to hdfs

did it solve the problem ?

Highlighted

Re: Ranger cannot record Knox audit log to hdfs

Rising Star

Is 'testcluster' the name of the topology file that you've configured via Knox?

You should have a file named 'testcluster.xml' under conf/topologies

Don't have an account?
Coming from Hortonworks? Activate your account here