Member since
10-18-2017
13
Posts
2
Kudos Received
0
Solutions
09-02-2019
07:38 AM
1 Kudo
Hello, I've got an error when nifi and nifi-registry try to write audit ranger logs to HDFS.
2019-09-02 16:00:13,452 ERROR [XXXXXXX_nifi_registry.async.multi_dest.batch_XXXXXXX_nifi_registry.async.multi_dest.batch.hdfs_destWriter] o.a.r.audit.provider.BaseAuditHandler Error writing to log file.
java.lang.IllegalArgumentException: java.net.UnknownHostException: XXXXXXX
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:444) ~[hadoop-common-3.0.0.jar:na]
at org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:132) ~[hadoop-hdfs-client-3.0.0.jar:na]
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:355) ~[hadoop-hdfs-client-3.0.0.jar:na]
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:289) ~[hadoop-hdfs-client-3.0.0.jar:na]
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:163) ~[hadoop-hdfs-client-3.0.0.jar:na]
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3288) ~[hadoop-common-3.0.0.jar:na]
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123) ~[hadoop-common-3.0.0.jar:na]
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3337) ~[hadoop-common-3.0.0.jar:na]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3305) ~[hadoop-common-3.0.0.jar:na]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476) ~[hadoop-common-3.0.0.jar:na]
at org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:284) ~[ranger-plugins-audit-1.1.0.jar:1.1.0]
at org.apache.ranger.audit.destination.HDFSAuditDestination.access$000(HDFSAuditDestination.java:43) ~[ranger-plugins-audit-1.1.0.jar:1.1.0]
at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:156) ~[ranger-plugins-audit-1.1.0.jar:1.1.0]
at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:153) ~[ranger-plugins-audit-1.1.0.jar:1.1.0]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_112]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_112]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) ~[hadoop-common-3.0.0.jar:na]
at org.apache.ranger.audit.provider.MiscUtil.executePrivilegedAction(MiscUtil.java:523) ~[ranger-plugins-audit-1.1.0.jar:1.1.0]
at org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:153) ~[ranger-plugins-audit-1.1.0.jar:1.1.0]
at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:879) [ranger-plugins-audit-1.1.0.jar:1.1.0]
at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:827) [ranger-plugins-audit-1.1.0.jar:1.1.0]
at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:757) [ranger-plugins-audit-1.1.0.jar:1.1.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
Caused by: java.net.UnknownHostException: XXXXXXX
... 23 common frames omitted
My namenode is in HA so I set for the configuration xasecure.audit.destination.hdfs.dir the value hdfs://XXXXX/ranger/audit
Nifi seams not to accept the name of the cluster as HDFS node ?
Any idea ?
I use HDP 3.1 and HDF 3.4.
Thank's a lot
Mathieu
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache NiFi
-
Apache Ranger
08-30-2019
01:06 AM
Hi @IvanLeung May be the same error as http://apache-nifi-developer-list.39713.n7.nabble.com/failed-to-build-nifi-1-8-0-SNAPSHOT-on-OSX-high-sierra-td19426.html You can try unistall node if you previously install it on your machine and re-run maven clean install.
... View more
08-20-2019
12:58 AM
Hi @kerwin1217 These are the steps to install DAS on HDP 3.1 (os centos) from open source code : First clone https://github.com/hortonworks/data_analytics_studio and run maven install (you must not be under a proxy, that does not work) Then you can install the mpack you obtain (mpack/target/hdp3-data-analytics-studio-mpack-1.2.0.tar.gz) as describe here (do not start add service at this time). Create a rpm (my cluster is on centos) hdp3_data_analytics_studio-1.2.0-0.0.x86_64.rpm containing data_analytics_studio-event-processor-1.2.0.jar and data_analytics_studio-webapp-1.2.0.jar (jars obtained from step 1). The rpm should copy the jars on /usr/das/1.2.0.0.0/data_analytics_studio/lib and create /etc/das/conf and /var/log/das directories. Put the rpm on your local yum repo. I use Postgres database, so I created das database with das user. On ambari, use Manage Ambari to modify HDP 3.1 version to indicate the yum repo for DAS. On ambari, add service DAS (uncheck create database). Hope this can help. Mathieu
... View more
07-09-2019
09:43 AM
Hi @kerwin chen I finally gave up on installing Hue and chose to install Data Analytics Studio. I compiled the product from open source code under github and installed the mpack and the product on my edge node. Mathieu
... View more
04-19-2019
08:17 AM
Hi @AKRAM JEBALI We will try next week this installation and configuration (authentification, grafana integration, ...). To be continued
... View more
04-15-2019
08:46 AM
1 Kudo
Hi @Naveenraj Devadoss I've just done it on HDP 3.1 cluster. Create a new database on postgres (we use postgres for managment database) for registry : create database registry;
CREATE USER registry WITH PASSWORD 'registry';
GRANT ALL PRIVILEGES ON DATABASE "registry" to registry; Download hdf-ambari-mpack-<version>.tar.gz https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.0/release-notes/content/hdf_repository_locations.html on your admin node in /tmp dir Install mpack : ambari-server install-mpack --mpack=/tmp/hdf-ambari-mpack-3.4.0.0-155.tar.gz --verbose Then go to ambari, indicate the HDF base URL on Stack & Version and just add HDF services (NiFi, NiFi registry, Schema Registry). @see https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/installing-hdf-and-hdp/content/hdf-install-ambari.html
... View more
04-15-2019
08:32 AM
Thanks @Lester Martin I keep in mind the balancer admin command. I solve the issue simply by removing a very huge file created by a data scientist executing a very huge request on hive. The temporary files located at /tmp/hive/[user] seems to be not replicated (i'am not sure of that).
... View more
04-12-2019
08:34 AM
Hi all, I have 6 datanodes on my hortonworks cluster (HDP 2.6.3) and one of them is 91% full. The others are "only" 65% full. Il don't understand why the replication is not homogeneous and how i can fix it ? I check the file system and the same difference is observed : On a safe node : # pwd
/grid1/hadoop/hdfs/data/current/BP-332877091-10.136.82.11-1500650625087/current/finalized
# du -h . --summarize
1.9T . On the unsafe node : # pwd
/grid1/hadoop/hdfs/data/current/BP-332877091-10.136.82.11-1500650625087/current/finalized
# du -h . --summarize
2.7T . Same things on each DataNode directories. Thanks for your help. Mathieu
... View more
Labels:
- Labels:
-
Apache Hadoop
04-10-2019
07:32 AM
Hi @Ruslan Fialkovsky Is your /tmp/ranger_hdfs_audit.log file got write permission for hdfs (linux) user ?
... View more
04-08-2019
12:51 PM
Hi @Ti Ba I'am facing the same issue. Did you find out the solution ? Thanks, Mathieu
... View more
11-08-2017
07:44 AM
Thank's a lot for your response @kgautam. That's what I thought but now I have arguments to convince.
... View more
10-18-2017
08:57 AM
Hello, My company and I are new users of hortonworks. We plan to use oozie as scheduler but for most of us, we found strange to put scripts and librairies into the HDFS. The main role of the HDFS isn't to store data and only data ? As oozie's clients can be installed on several nodes of the cluster, I think it's pertinent to get these librairies somewhere accessible by all nodes. So the HDFS is in fact the best place (and files are replicated). Can someone tell me why oozie made this choice ? Greathfully Mathieu
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Oozie