Unfortunately there is no way to downgrade. Best will be to uninstall HDP 2.6.5 https://community.hortonworks.com/articles/97489/completely-uninstall-hdp-and-ambari.html And install HDP 2.6.0.3 Take backup of critical files on HDFS and configuration databases. ... View more

All processors run under the context of user that is running Nifi. Check authentication for the user running Nifi. ... View more

here is a thread that can help you: https://community.hortonworks.com/questions/106987/hive-multiple-small-files.html ... View more

OR .collect()[0][1] ... View more

You will need machine names (DN) for both Nifi and Nifi Registry like "CN=machinename,...dc=example,dc=com" where in CN you will use server name the remaining portion will come under whatever wildcard (sub)domain you have. Then you will enter the full DN, manually as a user in both Nifi and Nifi Registry. This method is similar to when setting up site-to-site policies here: https://community.hortonworks.com/articles/88473/site-to-site-communication-between-secured-https-a.html Finally, you can check logs files for errors when pulling buckets in both Nifi and Nifi registry in files *-app.log and *-user.log, this may as well give you a full DN Nifi is looking for. ... View more

Do you have security enabled? Verify if user that is running Nifi, has read permissions on all .xml configuration files. ... View more

In client mode your jar is running on edge node or local machine which will have smtp connectivity. In cluster mode, any of the data nodes could run your jar, so you will need to check connectivity to smtp from all nodes. ... View more

Yes, you can and should use company signed certs. Ask your company/Admins/infoSec to give you truststore and keystore jks files with passwords. Add these files in nifi.properties along with password. Restart Nifi. ... View more

Have a look at this if you need to by pass Avro for Hive DDL: https://github.com/tspannhw/nifi-convertjsontoddl-processor If you need to convert JSON to ORC (for Hive) Avro will be required. You will need to write/manage Avro schemas (recommended will be to use Schema Registry for that). Alternatively you can use Infer Avro schema to detect incoming schema from JSON but it may not be 100% correct all the time. ... View more

If I remember correctly, you will need to add Nifi Registry Server DN (SSL machine name) to Nifi > Access Policies and Nifi Server DN to Nifi Registry to be able to read and query buckets. Once they both know each other, buckets will load. ... View more

For Hive: https://cwiki.apache.org/confluence/display/Hive/WebHCat+Reference For HBase: https://hbase.apache.org/book.html#_rest ... View more

Hive ACID transactions are totally advisable. Please provide what exact performance issue you are facing along with volume of inserts and updates. Have you tried tuning Hive for ACID transactions like compaction etc.: https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration ... View more

If users and groups are deleted in openLDAP server you should use 'existing' mode with ambari ldap sync: ambari-server sync-ldap --existing https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-security/content/existing_users_and_groups.html ... View more

Check if you have proxy enabled echo $http_proxy try wget or curl on your local repository to get a file and verify connectivity. ... View more

There is no configuration in Zeppelin to disable user login on excessive attempts. However, Zeppelin uses Apache Shiro for authentication which has un-handled exception catch for ExcessiveAttemptsException. You will need to implement and deploy on your own. https://shiro.apache.org/authentication.html#Authentication-Step3%3AHandlingSuccessorFailure http://shiro-user.582556.n2.nabble.com/ExcessiveAttemptsException-How-to-configure-td4534742.html ... View more

Perhaps: "Reading/writing to an ACID table from a non-ACID session is not allowed. In other words, the Hive transaction manager must be set to org.apache.hadoop.hive.ql.lockmgr.DbTxnManager in order to work with ACID tables." SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Limitations ... View more

Following call we do, if kafka does not exist you will get status 404. curl --user admin -sS -G "http://ambari_server_here/api/v1/clusters/CLUSTERNAME_HERE/services/KAFKA" API resource: https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/index.md ... View more

Query Record Processor in this scenario seems a bit of overkill for this problem. And will require more work if you don't have auto incrementing fields. You can use just SplitText processor to do everything. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.SplitText/index.html ... View more

In case you ran user sync multiple times and move/removed users from Ranger, disable incremental sync in Ranger and run again. I find the Ranger LDAP connection tool to be very useful in these scenarios: https://cwiki.apache.org/confluence/display/RANGER/LDAP+Connection+Check+Tool ... View more

This works in Spark 2.X. Where tablenamedirectory is a HDFS directory containing all the orc files. spark.read.format("orc").load("/datalake/tablenamedirectory/") ... View more

Great suggestions up there. We handle this case by dumping small files as ORC in daily partitions and then running Hive ALTER TABLE/PARTITION CONCATENATE very week or so. https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionConcatenate ... View more

Check firewall on all servers. ... View more

How are you trying to connect to Hive Metastore? If Ambari is showing it running it may be running, try using Hive and create a table. /usr/hadoop is not a valid directory. It is either /usr/hdp for binaries and /etc/hadoop for configs. ... View more

What user are you seeing logged in? Is logged in user same as one defined in file authorizers.xml > "Initial Admin Identiry" ? ... View more

If you are using a SQL statement in sqoop, try bring the columns as new alias like select notes-id notesid, ... ... View more

You can use RounteOnAttribute if you have some way to identifying which files goes to which destination like in filename, path etc. Check attributes of the flow file to see if any can be used. Then use those attributes to create 3 paths, something like: https://community.hortonworks.com/questions/54811/redirecting-flow-based-on-certain-condition-nifi.html ... View more

ExecuteStreamCommand with password as command argument should work. Caution: The zipped files will need to be loaded into memory so make sure you have enough heap allocated. ... View more

Yes, this does happen if statistics etc. are out of sync. It is generally best practice to use a column identifier to get correct results like count(id) ... View more

Very comprehensive answer here: https://community.hortonworks.com/questions/73302/hdinsight-vs-hdp-service-on-azure-vs-hdp-on-azure.html ... View more