Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
couron_mathieu
couron_mathieu
New Contributor

Nifi HDFS audit error with HA NameNode

by New Contributor couron_mathieu in Support Questions
‎09-02-2019 07:38 AM
‎09-02-2019 07:38 AM
Hello, I've got an error when nifi and nifi-registry try to write audit ranger logs to HDFS.     2019-09-02 16:00:13,452 ERROR [XXXXXXX_nifi_registry.async.multi_dest.batch_XXXXXXX_nifi_registry.async.multi_dest.batch.hdfs_destWriter] o.a.r.audit.provider.BaseAuditHandler Error writing to log file. java.lang.IllegalArgumentException: java.net.UnknownHostException: XXXXXXX at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:444) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:132) ~[hadoop-hdfs-client-3.0.0.jar:na] at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:355) ~[hadoop-hdfs-client-3.0.0.jar:na] at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:289) ~[hadoop-hdfs-client-3.0.0.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:163) ~[hadoop-hdfs-client-3.0.0.jar:na] at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3288) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3337) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3305) ~[hadoop-common-3.0.0.jar:na] at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476) ~[hadoop-common-3.0.0.jar:na] at org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:284) ~[ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.destination.HDFSAuditDestination.access$000(HDFSAuditDestination.java:43) ~[ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:156) ~[ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:153) ~[ranger-plugins-audit-1.1.0.jar:1.1.0] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_112] at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_112] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) ~[hadoop-common-3.0.0.jar:na] at org.apache.ranger.audit.provider.MiscUtil.executePrivilegedAction(MiscUtil.java:523) ~[ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:153) ~[ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:879) [ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:827) [ranger-plugins-audit-1.1.0.jar:1.1.0] at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:757) [ranger-plugins-audit-1.1.0.jar:1.1.0] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] Caused by: java.net.UnknownHostException: XXXXXXX ... 23 common frames omitted        My namenode is in HA so I set for the configuration xasecure.audit.destination.hdfs.dir the value hdfs://XXXXX/ranger/audit   Nifi seams not to accept the name of the cluster as HDFS node ? Any idea ?   I use HDP 3.1 and HDF 3.4.   Thank's a lot   Mathieu ... View more

Re: Can I install Hue in HDP 3.1.0

by New Contributor couron_mathieu in Support Questions
‎08-30-2019 01:06 AM
‎08-30-2019 01:06 AM
Hi @IvanLeung    May be the same error as http://apache-nifi-developer-list.39713.n7.nabble.com/failed-to-build-nifi-1-8-0-SNAPSHOT-on-OSX-high-sierra-td19426.html   You can try unistall node if you previously install it on your machine and re-run maven clean install. ... View more

Re: Can I install Hue in HDP 3.1.0

by New Contributor couron_mathieu in Support Questions
‎08-20-2019 12:58 AM
‎08-20-2019 12:58 AM
Hi @kerwin1217  These are the steps to install DAS on HDP 3.1 (os centos) from open source code : First clone https://github.com/hortonworks/data_analytics_studio and run maven install (you must not be under a proxy, that does not work) Then you can install the mpack you obtain (mpack/target/hdp3-data-analytics-studio-mpack-1.2.0.tar.gz) as describe here (do not start add service at this time). Create a rpm (my cluster is on centos) hdp3_data_analytics_studio-1.2.0-0.0.x86_64.rpm containing data_analytics_studio-event-processor-1.2.0.jar and data_analytics_studio-webapp-1.2.0.jar (jars obtained from step 1). The rpm should copy the jars on /usr/das/1.2.0.0.0/data_analytics_studio/lib and create /etc/das/conf and /var/log/das directories. Put the rpm on your local yum repo. I use Postgres database, so I created das database with das user. On ambari, use Manage Ambari to modify HDP 3.1 version to indicate the yum repo for DAS. On ambari, add service DAS (uncheck create database). Hope this can help.   Mathieu ... View more

Re: Can I install Hue in HDP 3.1.0

by New Contributor couron_mathieu in Support Questions
‎07-09-2019 09:43 AM
‎07-09-2019 09:43 AM
Hi @kerwin chen I finally gave up on installing Hue and chose to install Data Analytics Studio. I compiled the product from open source code under github and installed the mpack and the product on my edge node. Mathieu ... View more

Re: Can I install Hue in HDP 3.1.0

by New Contributor couron_mathieu in Support Questions
‎04-19-2019 08:17 AM
‎04-19-2019 08:17 AM
Hi @AKRAM JEBALI We will try next week this installation and configuration (authentification, grafana integration, ...). To be continued ... View more

Re: Can I install Hue in HDP 3.1.0

by New Contributor couron_mathieu in Support Questions
‎04-18-2019 08:55 AM
‎04-18-2019 08:55 AM
Hello @luan ha Did you get hue working on hdp 3.x ? ... View more

Re: How to install Nifi on existing HDP - Ambari

by New Contributor couron_mathieu in Support Questions
‎04-15-2019 08:46 AM
1 Kudo
‎04-15-2019 08:46 AM
1 Kudo
Hi @Naveenraj Devadoss I've just done it on HDP 3.1 cluster. Create a new database on postgres (we use postgres for managment database) for registry : create database registry; CREATE USER registry WITH PASSWORD 'registry'; GRANT ALL PRIVILEGES ON DATABASE "registry" to registry; Download hdf-ambari-mpack-<version>.tar.gz https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.0/release-notes/content/hdf_repository_locations.html on your admin node in /tmp dir Install mpack : ambari-server install-mpack --mpack=/tmp/hdf-ambari-mpack-3.4.0.0-155.tar.gz --verbose Then go to ambari, indicate the HDF base URL on Stack & Version and just add HDF services (NiFi, NiFi registry, Schema Registry). @see https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/installing-hdf-and-hdp/content/hdf-install-ambari.html ... View more

Re: One datanode nearly full but not the others

by New Contributor couron_mathieu in Support Questions
‎04-15-2019 08:32 AM
‎04-15-2019 08:32 AM
Thanks @Lester Martin I keep in mind the balancer admin command. I solve the issue simply by removing a very huge file created by a data scientist executing a very huge request on hive. The temporary files located at /tmp/hive/[user] seems to be not replicated (i'am not sure of that). ... View more

One datanode nearly full but not the others

by New Contributor couron_mathieu in Support Questions
‎04-12-2019 08:34 AM
‎04-12-2019 08:34 AM
Hi all, I have 6 datanodes on my hortonworks cluster (HDP 2.6.3) and one of them is 91% full. The others are "only" 65% full. Il don't understand why the replication is not homogeneous and how i can fix it ? I check the file system and the same difference is observed : On a safe node : # pwd /grid1/hadoop/hdfs/data/current/BP-332877091-10.136.82.11-1500650625087/current/finalized # du -h . --summarize 1.9T    . On the unsafe node : # pwd /grid1/hadoop/hdfs/data/current/BP-332877091-10.136.82.11-1500650625087/current/finalized # du -h . --summarize 2.7T    . Same things on each DataNode directories. Thanks for your help. Mathieu ... View more
Labels:

Re: ranger HDFS Audit to local file log4j error

by New Contributor couron_mathieu in Support Questions
‎04-10-2019 07:32 AM
‎04-10-2019 07:32 AM
Hi @Ruslan Fialkovsky Is your /tmp/ranger_hdfs_audit.log file got write permission for hdfs (linux) user ? ... View more

Re: Ambari Infra Update - HDP(3.0.1) - Could not f...

by New Contributor couron_mathieu in Support Questions
‎04-08-2019 12:51 PM
‎04-08-2019 12:51 PM
Hi @Ti Ba I'am facing the same issue. Did you find out the solution ? Thanks, Mathieu ... View more

Re: Why does Oozie pick up scripts and librairies ...

by New Contributor couron_mathieu in Support Questions
‎11-08-2017 07:44 AM
‎11-08-2017 07:44 AM
Thank's a lot for your response @kgautam. That's what I thought but now I have arguments to convince. ... View more

Why does Oozie pick up scripts and librairies from...

by New Contributor couron_mathieu in Support Questions
‎10-18-2017 08:57 AM
‎10-18-2017 08:57 AM
Hello, My company and I are new users of hortonworks. We plan to use oozie as scheduler but for most of us, we found strange to put scripts and librairies into the HDFS. The main role of the HDFS isn't to store data and only data ? As oozie's clients can be installed on several nodes of the cluster, I think it's pertinent to get these librairies somewhere accessible by all nodes. So the HDFS is in fact the best place (and files are replicated). Can someone tell me why oozie made this choice ? Greathfully Mathieu ... View more
Contact Me
Online
Offline
Last Visited
‎09-03-2019 06:13 AM
Public Statistics
Date Registered ‎10-18-2017 08:28 AM
Last Visited ‎09-03-2019 06:13 AM
Total Messages Posted 13
Total Kudos Received 1