Q&A

Dear @Jay Kumar SenSharma As per your suggested video link https://www.youtube.com/watch?v=wHaVBoLwzwU i have done all network setup ...and installation of ambari server and ambari-agent...successfully by local repo ....-When i adding only master , there is error to register .....see in ambari-agent log file .....INFO 2019-01-10 08:35:21,750 main.py:145 - loglevel=logging.INFOINFO 2019-01-10 08:35:21,751 main.py:145 - loglevel=logging.INFOINFO 2019-01-10 08:35:21,751 main.py:145 - loglevel=logging.INFOINFO 2019-01-10 08:35:21,754 DataCleaner.py:39 - Data cleanup thread startedINFO 2019-01-10 08:35:21,756 DataCleaner.py:120 - Data cleanup startedINFO 2019-01-10 08:35:21,758 DataCleaner.py:122 - Data cleanup finishedINFO 2019-01-10 08:35:21,820 PingPortListener.py:50 - Ping port listener started on port: 8670INFO 2019-01-10 08:35:21,823 main.py:437 - Connecting to Ambari server at https://master.alethe.com:8440 (192.168.1.191)INFO 2019-01-10 08:35:21,824 NetUtil.py:70 - Connectin

Hi,I've created a parameterized SQL query in the ExecuteSQL processor, this processor is predecessed by an UpdateAttribute with those parameters:here is the part of the where clause that uses this parameter : AND (( ?sql.args.4.value IS NULL AND mm.docno IS NULL ) OR ( mm.docno = ?sql.args.4.value))The issue is that I keep having this error message :"java.sql.SQLDataException: Value of the sql.args.4.type attribute is 'CHAR', which is not a valid JDBC numeral type"Any clues ?

Hello, A simple question: How can I know which queries generate Scratch files?I'm inspecting the Impalad logs and I couldn't find any information about the scratch file generation. Regards,Silva

hi ,i am currently using hdp 3.0 and ambari 2.7,i need to install the nifi service on my cluster ,just followed the hortonworks document they said following query ambari-server install-mpack --mpack=http://.../hdf-mpack-name.tar.gz --purge --verboseadded my centos7 package in the mpack.what we need to setup after?

Installed single node cluster on Centos 6 and enabled kerberos on cluster. ranger not starting..stderr: /var/lib/ambari-agent/data/errors-311.txtTraceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 231, in <module> RangerAdmin().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 93, in start self.configure(env, upgrade_type=upgrade_type, setup_db=params.stack_supports_ranger_setup_db_on_start) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 120, in locking_configure original_configure(obj, *args, **kw) File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 135, in configure setup_ranger_db() File "/var/lib/ambari-agent/c

We have a client who wants to setup a small separate special purpose cluster.  It will consist of 5 nodes and they want to run Kudu on it.  If we configure with 3 Master nodes and 3 Tablet Servers, at least one of the nodes will have both Master roles as well as Tablet Server role.  In any case, in this scenario, if hypothetically the node with both roles fail, and the replication is set to 3, neither the master tablet nor any of the table tablets will be able to satisfy the replication factor since there are now only 2 master roles and 2 table servers available.   Since this is still within the (n-1)/2 failures, will the service still run?  After 5 minutes, and when the nodes are determined to be permanent failures, a new follower needs to be created but since there are only 2 tablet servers still functioning, what will happen?  Will the system limp along, still functioning or will it fail? Thanks in advance for any help.

Team, I have 1000 tables in my source RDBMS and I would like to get them migrated to hive using pyspark I read through documentation and found that below two commands would help. Is there a way I can loop these two commands 1000 times if I have all the list of tables in a python array? arr = ("table1","table2")for x in arr:            df = spark.read.format("jdbc").blah.blah              data.write.saveAsTable.blah.blah If someone has a working solution for this could you please share. I tried but it is not throwing any error but at same time not writing anything. ThanksMeher 

I have a HDP 2.6 cluster that I want to upgrade to HDP 3.1. I currently use postgres 9.2 for Ambari, and a separate postgres 9.2 instance for Ranger and Hive. Looking at the compatibility matrix, postgres 9.2 is no longer supported for HDP 3.1.1) Does this apply just to the postgres instance used by ambari, or also to the postgres instance used by Ranger and Hive too?2) I'm assuming upgrading postgres isn't handled by ambari, so should I just upgrade and migrate the relevant data, keeping the connection parameters the same?Thanks in advance

I have setup kylin in HDP. Now want to build the cube using spark instead of map reduce. Cube is getting build with map reduce but when trying to use spark it is getting hanged. Tried to set the hive execution engine as spark instead of Tez in hive. But it is not working and getting error java.lang.NoClassDefFoundError: org/apache/spark/SparkConf. Is there any way we can run hive on spark in HDP or any suggestion how to run kylin using spark? Help is much appreciated.

I am currently working with Apache Hadoop MapReduce and would like to know: what exactly is speculative execution and how does it work?

Hi Everyone,I am gonna adding a new hosts in existing cluster ....All Network Process are done successfully ...after then did't register new hosts by AMBARI....hdp version details : hdp 2.6 with ambari 2.6Os - centso7Please Suggest How to do , Your Help is Highly appreciated .screenshot-from-2018-12-28-18-17-06.pngscreenshot-from-2018-12-28-18-23-15.pngscreenshot-from-2018-12-28-18-40-11.png

Hi, I'm investigating the approaches for row level security in Impala, and I have just found multiple references for RecordService. However, it's homepage is removed, the source code is gone from GitHub, and it didn't get a new release for 2+ years on Maven, so I assume that it's dead.Does anybody knows what happened to it, why it was cancelled? These are the ones which I found as possible workarounds:Store the data on HDFS in separate files splitted by the filter expression, and use HDFS ACLs for the filesUse HBase as a storage engine and query the data directly from it, as it supports row-level securityDisable access to the original Impala table, create views from it with the appropriate filters and use regular Impala ACLs for the viewsIs there any alternative for row level security in Impala? RecordService sounded like the solution what I needed. Thanks,Peter

Hello,    We currently have CDH 5.15.0 running in our environments and have a necessity to support OpenJDK. Does upgrading to CDH 6  help in supporting OpenJDK? If yes, can we directly upgrade from CDH 5.15.0 to CDH 6 or do we have to downgrade first to CDH 5.14 and then upgrade to CDH 6 ( documentation does not show direct upgrade from CDH 5.15 to CDH 6 https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_install_upgrade_notes.html#topic_1_2). Thank you for your time in advance. 

Query:select 2gusage,count(2gusage) from demo group by 2gusage;Output:MID 765153 HIGH 18095461 LOW 119069472 NULL 25997075I tried the below query to find the count of NULL valuesQueryselect count(*) from demo where 2gusage is 'NULL';Output0Kindly help me out with the query to find the count of NULL values

I have enabled kerberos authentication on my cluster. After that when namenode is not starting due Checksum error. I am using cloudera 6.0.1 on Suse 12 OS hosted on GCP Failed to start namenode.org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: hdfs/nn.asia-south1-c.c.cdh-cluster2.internal@ASIA-SOUTH1-C.C.CDH-CLUSTER2.INTERNAL from keytab hdfs.keytab javax.security.auth.login.LoginException: Checksum failedat org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1843)at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1211)at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1007)at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:313)at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:694)at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:713)

Hi,I'm making a sandbox on AWS. I have 3 instances of t2.medium (2 vCPUs, 4GB RAM), with 32 GB disks. Everything was set up to pass the installation. Instances are running RHEL 7.6 (AWS image). I didn't touch much - just enabled passwordless login for root, added mysql to "master" node for hive. I'm getting no warnings or issues during installation.After installation, during 'startup phase' I get the following log. The "only" error I see is related to port 50070 being in use, but this is not the case - at least not before or after installation (on all three hosts actually).Obviously, almost nothing else starts afterwards.Here's the full log:stderr: <script id="metamorph-23723-start" type="text/x-placeholder"></script>Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 361, in <module> NameNode().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/s

I am not able to launch the Ambari home page because an Oracle agreement page keeps coming up as a pop-up. What is the solution to this?

I have Nifi cluster(two instances). I generate urls using "GenerateFlowFile", then split it using "SplitText". For example;-> "GenerateFlowFile" is generate this;url1url2url3url4-> In "SplitText"(split count 2);url1url2---------url3url4-> I want to send these two flowfiles to other nifi instances in order. For this, i am using round robin. But i always want to go in the same order. What I want;url1, url2 ---> node1url3, url4 ---> node2url1, url2 ---> node1url3, url4 ---> node2url1, url2 ---> node1url3, url4 ---> node2-> But data is being sent this wayurl1, url2 ---> node1url3, url4 ---> node2url3, url4 ---> node1url1, url2 ---> node2How can I send same data to the same Nifi node?

I was able to open ambari and see it is installed but I couldn't find.

I am not able to see new users from linux into ranger ui.below is the error log of usersync.logplease help to resolve this.03 Jan 2019 12:02:15 DEBUG PolicyMgrUserGroupBuilder [UnixUserSyncThread] - GROUP: hadoop03 Jan 2019 12:02:15 DEBUG PolicyMgrUserGroupBuilder [UnixUserSyncThread] - GROUP: users03 Jan 2019 12:02:15 DEBUG PolicyMgrUserGroupBuilder [UnixUserSyncThread] - USER: aapuser03 Jan 2019 12:02:15 DEBUG PolicyMgrUserGroupBuilder [UnixUserSyncThread] - GROUP: aapuser03 Jan 2019 12:02:15 DEBUG PolicyMgrUserGroupBuilder [UnixUserSyncThread] - USER: hdfs03 Jan 2019 12:02:15 DEBUG PolicyMgrUserGroupBuilder [UnixUserSyncThread] - GROUP: hadoop03 Jan 2019 12:02:15 DEBUG PolicyMgrUserGroupBuilder [UnixUserSyncThread] - GROUP: hdfs03 Jan 2019 12:02:15 DEBUG PolicyMgrUserGroupBuilder [UnixUserSyncThread] - GROUP: hue03 Jan 2019 12:02:15 DEBUG PolicyMgrUserGroupBuilder [UnixUserSyncThread] - USER: yarn03 Jan 2019 12:02:15 DEBUG PolicyMgrUserGroupBuilder [UnixUserS

Sqoop import fails on empty table We have a batch job that looks into Teradata table and offloads delta records to HIVE based on a timestamp column. We havent used SQOOP incremental mode, Instead we have setup views on top of teradata tables and a joined to a control table to get the delta records. Once the offload is completed, We update the control table with MAX of timestamp record.   However, We are stuck with below issue: Import failed: com.teradata.connector.common.exception.ConnectorException: Input source table is empty   Could someone assist on the same? Regards, SV

We have a Hive External table where a couple of columns have XML data in them. The XML structure is valid but some of the records contain "?<" or "?>" in the data of an element. As you can see, the XML itself is valid as the value is quoted but Hive xpath is unhappy. Trying to figure out if there's a better way to escape out the data elements or have xpath ignore the extra "<". Example:<Root><data1><data2 id="4" value="622" name="CoID" /><data2 id="9999" value="Company XYZ" name="Company Name" /><data2 id="9999" value="2222345=0000?<" name="OfferId" /></data1></Root>Query:selectxpath_string(xml_column, '//Root [@name="OfferId"]/@value')from table;Here is the Error that we receive:Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 284; The value of attribute "value" associated with an element type "Root" must not contain the '<' character.  at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)  at org

I am facing connection issue for ranger usersync. because of that my users and groups are not able to sync in ranger.below is the error log of usersync.log02 Jan 2019 17:54:38 DEBUG PolicyMgrUserGroupBuilder [UnixUserSyncThread] - ==> PolicyMgrUserGroupBuilder.buildGroupList02 Jan 2019 17:56:45 ERROR UserGroupSync [UnixUserSyncThread] - Failed to initialize UserGroup source/sink. Will retry after 120000 milliseconds. Error details:com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: Connection timed out (Connection timed out) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:151) at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:104) at com.sun.jersey.api.client.Client.handle(Client.java:648) at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)

HelloI've been setting up my secured nifi cluster with the help of the nifi-tls-toolkit in standalone mode. which outputs the nifi.properties file along with keystores and the certificate. I tried to use the tls-toolkit in client/server mode, which outputs the same keystores and certificate. but instead of outputting the nifi.properties file it outputs a config.json containing the keystore passwords,types etc. in a json file. In the nifi docs (https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#client-server) nothing is really mentioned about the config.json file, accept configuration of the tls-toolkit where to output it (client) /read it from (server). and if i choose to use this method (client/server) the only valid solution for configuration of the nifi security properties is to copy the json fields into the nifi.properties of the node the certificate was created for (manually or via script).The main question i'm interested in is what is this config.json file?, is

Hi there!I set up standalone NIFI 1.8.0 and now try to communicate with it using node through his api.I am trying to deploy template which already exist on nifi (made it through GUI) : POST to "www.localhost:8080/nifi-api/process-groups/{THE_TEMPLATE_ID}/template-instance" but recieving error 415 status : media type not supported.I cant figure out what is worng with the request.I did not set up any authentication, but still when i tried to make any change like creating processor i am getting error 405 : not allowed. How is that possible? love for some help,john