Nifi HDFS audit error with HA NameNode

couron_mathieu · ‎09-02-2019

Hello, I've got an error when nifi and nifi-registry try to write audit ranger logs to HDFS. 2019-09-02 16:00:13,452 ERROR [XXXXXXX_nifi_registry.async.multi_dest.batch_XXXXXXX_nifi_registry.async.multi_dest.batch.hdfs_destWriter] o.a.r.audit.provider.BaseAuditHandler Error writing to log file. java.lang.IllegalArgumentException: java.net.UnknownHostException: XXXXXXX at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:444) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:132) ~[hadoop-hdfs-client-3.0.0.jar:na] at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:355) ~[hadoop-hdfs-client-3.0.0.jar:na] at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:289) ~[hadoop-hdfs-client-3.0.0.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:163) ~[hadoop-hdfs-client-3.0.0.jar:na] at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3288) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3337) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3305) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476) ~[hadoop-common-3.0.0.jar:na] at org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:284) ~[ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.destination.HDFSAuditDestination.access$000(HDFSAuditDestination.java:43) ~[ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:156) ~[ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:153) ~[ranger-plugins-audit-1.1.0.jar:1.1.0] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_112] at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_112] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) ~[hadoop-common-3.0.0.jar:na] at org.apache.ranger.audit.provider.MiscUtil.executePrivilegedAction(MiscUtil.java:523) ~[ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:153) ~[ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:879) [ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:827) [ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:757) [ranger-plugins-audit-1.1.0.jar:1.1.0] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] Caused by: java.net.UnknownHostException: XXXXXXX ... 23 common frames omitted My namenode is in HA so I set for the configuration xasecure.audit.destination.hdfs.dir the value hdfs://XXXXX/ranger/audit Nifi seams not to accept the name of the cluster as HDFS node ? Any idea ? I use HDP 3.1 and HDF 3.4. Thank's a lot Mathieu

couron_mathieu · ‎08-30-2019

Hi @IvanLeung May be the same error as http://apache-nifi-developer-list.39713.n7.nabble.com/failed-to-build-nifi-1-8-0-SNAPSHOT-on-OSX-high-sierra-td19426.html You can try unistall node if you previously install it on your machine and re-run maven clean install.

couron_mathieu · ‎08-20-2019

Hi @kerwin1217 These are the steps to install DAS on HDP 3.1 (os centos) from open source code : First clone https://github.com/hortonworks/data_analytics_studio and run maven install (you must not be under a proxy, that does not work) Then you can install the mpack you obtain (mpack/target/hdp3-data-analytics-studio-mpack-1.2.0.tar.gz) as describe here (do not start add service at this time). Create a rpm (my cluster is on centos) hdp3_data_analytics_studio-1.2.0-0.0.x86_64.rpm containing data_analytics_studio-event-processor-1.2.0.jar and data_analytics_studio-webapp-1.2.0.jar (jars obtained from step 1). The rpm should copy the jars on /usr/das/1.2.0.0.0/data_analytics_studio/lib and create /etc/das/conf and /var/log/das directories. Put the rpm on your local yum repo. I use Postgres database, so I created das database with das user. On ambari, use Manage Ambari to modify HDP 3.1 version to indicate the yum repo for DAS. On ambari, add service DAS (uncheck create database). Hope this can help. Mathieu

couron_mathieu · ‎07-09-2019

Hi @kerwin chen I finally gave up on installing Hue and chose to install Data Analytics Studio. I compiled the product from open source code under github and installed the mpack and the product on my edge node. Mathieu

couron_mathieu · ‎04-19-2019

Hi @AKRAM JEBALI We will try next week this installation and configuration (authentification, grafana integration, ...). To be continued

couron_mathieu · ‎04-18-2019

Hello @luan ha Did you get hue working on hdp 3.x ?

couron_mathieu · ‎04-15-2019

Hi @Naveenraj Devadoss I've just done it on HDP 3.1 cluster. Create a new database on postgres (we use postgres for managment database) for registry : create database registry; CREATE USER registry WITH PASSWORD 'registry'; GRANT ALL PRIVILEGES ON DATABASE "registry" to registry; Download hdf-ambari-mpack-<version>.tar.gz https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.0/release-notes/content/hdf_repository_locations.html on your admin node in /tmp dir Install mpack : ambari-server install-mpack --mpack=/tmp/hdf-ambari-mpack-3.4.0.0-155.tar.gz --verbose Then go to ambari, indicate the HDF base URL on Stack & Version and just add HDF services (NiFi, NiFi registry, Schema Registry). @see https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/installing-hdf-and-hdp/content/hdf-install-ambari.html

couron_mathieu · ‎04-15-2019

Thanks @Lester Martin I keep in mind the balancer admin command. I solve the issue simply by removing a very huge file created by a data scientist executing a very huge request on hive. The temporary files located at /tmp/hive/[user] seems to be not replicated (i'am not sure of that).

couron_mathieu · ‎04-12-2019

Hi all, I have 6 datanodes on my hortonworks cluster (HDP 2.6.3) and one of them is 91% full. The others are "only" 65% full. Il don't understand why the replication is not homogeneous and how i can fix it ? I check the file system and the same difference is observed : On a safe node : # pwd /grid1/hadoop/hdfs/data/current/BP-332877091-10.136.82.11-1500650625087/current/finalized # du -h . --summarize 1.9T . On the unsafe node : # pwd /grid1/hadoop/hdfs/data/current/BP-332877091-10.136.82.11-1500650625087/current/finalized # du -h . --summarize 2.7T . Same things on each DataNode directories. Thanks for your help. Mathieu

couron_mathieu · ‎04-10-2019

Hi @Ruslan Fialkovsky Is your /tmp/ranger_hdfs_audit.log file got write permission for hdfs (linux) user ?

couron_mathieu · ‎04-08-2019

Hi @Ti Ba I'am facing the same issue. Did you find out the solution ? Thanks, Mathieu

couron_mathieu · ‎11-08-2017

Thank's a lot for your response @kgautam. That's what I thought but now I have arguments to convince.

couron_mathieu · ‎10-18-2017

Hello, My company and I are new users of hortonworks. We plan to use oozie as scheduler but for most of us, we found strange to put scripts and librairies into the HDFS. The main role of the HDFS isn't to store data and only data ? As oozie's clients can be installed on several nodes of the cluster, I think it's pertinent to get these librairies somewhere accessible by all nodes. So the HDFS is in fact the best place (and files are replicated). Can someone tell me why oozie made this choice ? Greathfully Mathieu

Online	Offline
Last Visited	‎01-29-2020 03:32 AM

Date Registered	‎10-18-2017 08:28 AM
Last Visited	‎01-29-2020 03:32 AM
Total Messages Posted	13
Total Kudos Received	2

Nifi HDFS audit error with HA NameNode

Re: Can I install Hue in HDP 3.1.0

Re: Can I install Hue in HDP 3.1.0

Re: Can I install Hue in HDP 3.1.0

Re: Can I install Hue in HDP 3.1.0

Re: Can I install Hue in HDP 3.1.0

Re: How to install Nifi on existing HDP - Ambari

Re: One datanode nearly full but not the others

One datanode nearly full but not the others

Re: ranger HDFS Audit to local file log4j error

Re: Ambari Infra Update - HDP(3.0.1) - Could not f...

Re: Why does Oozie pick up scripts and librairies ...

Why does Oozie pick up scripts and librairies from...