@hadoop hdfs First, make sure that the unlimited key JCE policy is installed and make sure spnego.service.keytab oermission is 444 as it requires access on all Journal node. ... View more

@Sudheer Nulu 1. when I give hostname command in my aws instance it is showing private DNS name .but, when I am giving configuration files to nifi in windows machine there I am replacing the all the private DNS name withn public DNS name in both core-site.xml and hdfs-site.xml -----> is this the right approach? YES ....You need to copy the core-site.xml and hdfs-site.xml from one of your hadoop nodes to the machine where NiFi is running. Then configure PutHDFS so that the configuration resources are "/path/to/core-site.xml,/path/to/hdfs-site.xml". That is all that is required from the NiFi perspective, those files contain all of the information it needs to connect to the Hadoop cluster. Ensure that the machine where NiFi is running has network access to all of the machines in your Hadoop cluster. You can look through those config files and find any hostnames and IP addresses and make sure they can be accessed from the machine where NiFi is running. 2. I changed the hostname in aws instance to my public DNS and also changed it in both core-site.xml and hdfs-site.xml,but my hostname is not reflecting ambari UI and in both the files the hostnmae is changing to Private DNS dynamically--->Could you please let me know,from where this hostname is being referred? Have a look at this AWS documentation https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-hostname.html https://aws.amazon.com/premiumsupport/knowledge-center/linux-static-hostname-rhel7-centos7/ ... View more

@S. Omar Sounds good can you try starting the components manually one by one like MR below? I can see HDFS,YARN, Zookeeper are up and running hope the alerts for these service has gone by now You could start the rest first maniually # su - mapreduce -c "/usr/hdp/current/hadoop-mapreduce-historyserver/sbin/mr-jobhistory-daemon.sh start historyserver" The check if the process is up $ ps -ef | grep -i jobhistoryserver Are you still getting alerts for the datanodes? Let me know ... View more

@Alexandre GRIFFAUT If possible can you paste the blueprint in here and only scramble sensitive info this can help community members analyze it ... View more

@Dale Preston Hortonworks doesn't practice any licencing but for sure the support level for a Dev can be cheaper. Especially response time from Hortonworks engineers contrary to the prod where you can have a dedicated engineer and o. Site or phone support. Look at the links above for the contact numbers and email, they should give you a clearer policy ... View more

@S. Omar Can you make a screenshot of your Ambari UI.Make sure the zookeepers are running .... ... View more

@S. Omar Please see the attached screenshot, yours could have multiple mount points, so whatever mount point you remove from Namenode should also be removed on the Datanode. ... View more

@S. Omar The Datanodes can't start because of too many failed volumes - current valid volumes: 1, volumes configured: 2, volumes failed: 1, volume failures tolerated: 0 Can you change this mount point Failed to add storage directory [DISK]file:/mnt/data/hadoop/hdfs/data/ Check it in the Ambari UI and remove it if you have more than one . Then restart the datanodes ... View more

@Jeff Watson For a kerberized environment when an authorized user/application grabs a ticket the default lifetime is configurable in /etc/krb5.conf usually ticket_lifetime = 24h, but note should be taken that this ticket is renewed automatically as long as there is a job running previously. Kerberos won't expire the ticket which would lead to the job to fail. I don't know id AD has a mechanism to timeout idle connections but most databases have a configurable session timeout. Can you be more specific on the databases. ... View more

@S. Omar I want to ask the obvious first did you perform the recommended step passwordless,ntp,firewall steps? How did you deploy your cluster Ambari blueprints or cloudbreak? Can you upload in /var/log/hadoop/hdfs ... View more

@Bill Nolf When you are setting up a HDP cluster is recommended you de-activate the firewall for inter-cluster communication. Check the official document Preparation of the environment look at Disable IPTables part Hope that helps ... View more

@jiji big data Update Cassandra to listen for connections on the local machine’s IP address instead of on “localhost” Something like this listen_address: 192.168.3.133 rpc_address: 192.168.3.133 where 192.168.3.133 is the address of the machine running Cassandra in my case. Use your machine’s IP address in your setup. Please revert ... View more

@Sushant Pandey HDInsights is just HDP on windows offering the big data analytics platform on the Azure cloud service. Its managed cloud Hadoop offering using Hortonworks. I haven't come across an implementation on HDInsights. You can try a solution like Dataguise already available on HDInsights. But I still think Protegrity Avatar should work on HDInsights apart from HDP it's certified and works on Cloudera,Pivotal,MapR and IBM. ... View more

@Sushant Pandey Yes these are some of the features Protect and unprotect field-level data using policy-based tokenization with MapReduce, Hive, Pig, and HBase Restrict data access from unauthorized users How Protegrity Vaultless Tokenization can enable seamless, usable and secure data in Hortonworks Data Platform. See documentation ... View more

@jiji big data Can you share your cassandra.yaml in the conf and check the variables listen_address and rpc_address ... View more

@jiji big data Do you have any other logs like in /var/logs/cassandra or /var/lib/cassandra these have more details! ... View more

@jiji big data You are getting the error because Cassandra is not started? Can you share the complete error stack ... View more

@gdeleon Matt posted an answer in HCC, You have encountered the following: https://issues.apache.org/jira/browse/NIFI-4363 You can get around this issue by editing the following script on every NiFi node in your cluster: /var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/files/nifi-toolkit-1.2.0.3.0.1.0-43/bin/encrypt-config.sh Near the bottom of the script you will find this line: "${JAVA}"-cp "${CLASSPATH}"-Xms128m-Xmx256m org.apache.nifi.properties.ConfigEncryptionTool"$@" Edit it so that it now says: "${JAVA}"-cp "${CLASSPATH}"-Xms512m-Xmx1024m org.apache.nifi.properties.ConfigEncryptionTool"$@" Then go back to Ambari and start your NiFi service. Hope that helps ... View more

@Sriram V HDP support on Windows I think is deprecated.HDP 2.4 was the last windows edition supported,its strongly advised to run HDP only on supported Linux platforms. You should look at HDP on Linux or HDInsights Azure I am not sure you will get on-premise support with a custom installation. Have a look HDInsights which provides open-source frameworks such as Hadoop, Spark, Hive, LLAP, Kafka, Storm, R & more. Azure HDInsight enables a broad range of scenarios such as ETL, Data Warehousing, Machine Learning, IoT and more. Better still call Hortonworks support 🙂 Hope that helps ... View more