Q&A

Hi,I am trying to run spark application which will need access to Hive databases. But Hive databases like FOODMART are not visible in spark session.I did spark.sql("show databases").show(); it is not showing Foodmart database, though spark session is having enableHiveSupport.Below i've tried:1)cp /etc/hive/conf/hive-site.xml /etc/spark2/conf2)Changed spark.sql.warehouse.dir in spark UI from /apps/spark/warehouse to /warehouse/tablespace/managed/hiveEven though it is not working.Please let me know what configuration changes would be required to have this.Please note - Above is working in HDP2.6.5.

Hi Community,   I am we have configured auto TLS on our dev enviroment and trying to switch to custom and not using self signed cert anymore. When i try and upload the.   I have ran    psql (9.2.24)Type "help" for help. postgres=# \c scmYou are now connected to database "scm" as user "postgres".scm=# update CONFIGS set VALUE='false' where ATTR='agent_tls';UPDATE 1scm=# delete from CONFIGS where ATTR='web_tls';DELETE 1scm=# \q   but still cant add cert add custom certs

Hi, I have this situation: my team are using Cloudera Manager, and we want to develop Hue ,one of CDH components, for our need, to replace the original one. I log on the server, replace /mnt/disk1/cloudera/parcels/CDH/lib/hue/ with unzipped tarball documents. What should I do before "make apps"?

The hdfs installation fails with the following error while installing cdh 6.2. Can't open /var/run/cloudera-scm-agent/process/63-hdfs-NAMENODE-format/supervisor_status:Permission denied what would cause this error?      

Community,   Any best practices for the maintenance for production clusters. When we should restart the services generally? - weekly once or monthly once.   Cleaning up log files.  And, also found that yarr logs in HDFS not get deleted according to the retention period. Ref: YARN logs under /tmp/logs/{user.name}/logs not cleared properly   Note: Mainly looking for a cluster in Bank, with regular ingestion using Sqoop and analytics using spark and impala.

Hi All,Today I  have given my CCA175 certification so can anyone please tell me when can I expect the result for the same. RegardsVishal Chhabra

I have a process-group which has to be started via rest-api.Principial it works to start the process-group but processors inside which are marked as "restricted" won't get started, f.e. ExecuteScript, PutFile. These processors remain stopped. No error message. In nifi-app.log are only the started processors listed. No word of the others.In the UI of course I can start these processors.How to solve this via rest-api? Thanks for any help.Using NiFi 1.9.2SUPPLEMENTIt is not a REST-API thing.Tried with NiFi Toolkit (/bin/cli.sh nifi pg-start -pgid 708358fc-016c-1000-0000-00000bf....). All processors get started excepted the restricted ones.So where is the cause and how to fix this?

Hi Team, We are in process on enabling TLS on HS2, I wanted to know/clarify if that happens will Impala be affected?Impala is dependent on HMS and TLS is only limited to hive server 2, given that having TLS enabled on Hive and running Impala without it should be ok or do you see a scenario where this combination may not work? Regards

ERROR transport.SentryTransportPool: Failed to obtain transport for xxx.us.xxx.com:8038: null19/08/08 07:12:27 WARN security.UserGroupInformation: PriviledgedActionException as:xxxx@xxx.xxx.COM (auth:KERBEROS) cause:org.apache.thrift.transport.TTransportException: Peer indicated failure: Problem with callback handler19/08/08 07:12:27 ERROR transport.SentryTransportPool: Failed to obtain transport for xxxx.us.xxx.com:8038: null19/08/08 07:12:30 ERROR tools.SentryShellKafka: Peer indicated failure: Problem with callback handlerorg.apache.sentry.SentryUserException: Peer indicated failure: Problem with callback handlerat org.apache.sentry.core.common.transport.RetryClientInvocationHandler.connect(RetryClientInvocationHandler.java:166)at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:90)at org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41)at com.sun.proxy.$Pro

I am trying to export data from Postgres to Hive through Sqoop in Oozie Coordinator Jobs. I set the frequency to 5 MINS.The job is started after the start time and the data is loaded into hive for the first time and after 5 mins frequency the job is started and it is killed.What is the problem here ?Below are my properties and xml file:   Coordinator.xml file: <coordinator-app name="coordinator1" frequency="${frequency}" start="${startTime}" end="${endTime}"timezone="${timezone}" xmlns="uri:oozie:coordinator:0.4"><action><workflow><app-path>${workflowPath}</app-path></workflow></action></coordinator-app>   Job.properties: frequency=5startTime=2019-08-07T10:22ZendTime=2019-08-07T10:32Ztimezone=UTCnameNode=hdfs://HadoopNode3:8020jobTracker=HadoopNode3:8032queueName=defaultoozie.action.sharelib.for.sqoop=sqoop,hiveoozie.use.system.libpath=trueworkflowPath=/user/oozieschedules/sqoop/coordinatoroozie.coord.application.path=/user/oozi

Hi all, I added some new nodes to our impala cluster.On the new nodes, Impala continues to crash with the following error: F0521 12:34:46.838728 5699 llvm-codegen.cc:112] LLVM hit fatal error: Cannot select: 0x7f62888c84c0: ch = store<ST1[%loc.i]> 0x7f622ebc5850, 0x1068b980, 0x1068b850, undef:i64 0x1068b980: i1,ch = load<LD1[%val_ptr.i.i27]> 0x7f626d3d1830, 0x7f622ebc54c0, undef:i64 0x7f622ebc54c0: i64 = add 0x1068bd10, Constant:i64<8> 0x1068bd10: i64,ch = CopyFromReg 0x7f626d3d1830, Register:i64 %vreg21 0x7f622ebc5ab0: i64 = Register %vreg21 0x7f62888c8d10: i64 = Constant<8> 0x7f61b3668260: i64 = undef 0x1068b850: i64,ch = CopyFromReg 0x7f626d3d1830, Register:i64 %vreg20 0x7f62888c8850: i64 = Register %vreg20 0x7f61b3668260: i64 = undefIn function: _ZN6impala26PartitionedAggregationNode12ProcessBatchILb0EEENS_6StatusEPNS_8RowBatchENS_13TPrefetchMode4typeEPNS_12HashTableCtxE.1The issue seems to be related to an hardware in

Scenario:I recently integrated Altus Director with Active Directory for role based authentication & authorisation. After implementation I noticed that the default admin credential (admin/admin) was not working anymore, which was expected. My question is:Is it possible/recommended to create another 'admin' user in Altus director (or in Active Directory) as a master credential just for back-up?Do Altus Director have a Authentication Backend Order (eg. Database then External) like we have in Cloudera Manager?Suppose, if a user is present in admin group as well as readonly group, then what role does Altus Director assumes for that user?Thank you.

I have reading NiFi documentation but I do not found the specifics metrics. There is nothing that explain me what metrics are available here: https://nifi.apache.org/docs.html.

Hello,My ambari agent has lost its heartbeat. I tried restarting the agent/server starting stopping etc but nothing worked.Im really desperate and dont know what to do. Any help would be greatly appreciated. I included the ambari-agent logs below.ERROR 2019-08-02 10:51:04,424 script_alert.py:123 - [Alert][hive_webhcat_server_status] Failed with result CRITICAL: ['Connection failed to http:///templeton/v1/status?user.name=ambari-qa + \nTraceback (most recent call last):\n File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py", line 190, in execute\n url_response = urllib2.urlopen(query_url, timeout=connection_timeout)\n File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen\n return opener.open(url, data, timeout)\n File "/usr/lib64/python2.7/urllib2.py", line 431, in open\n response = self._open(req, data)\n File "/usr/lib64/python2.7/urllib2.py", line 449, in _open\n \'_open\', req)\n File "/usr/lib64/python2.7/urllib2.py", line

I've read https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_cloud-data-access/content/intro.html where it is not recommended to use a cloud storage connector as the filesystem for HDFS. Can someone point out the reasoning for why these object stores can't be set as the defaultFS, which services wouldn't work/have issues, etc.?

Hello,   How zookeeper will react if we loose one disk assuming we have 3 zookeeper in 3 nodes. And what happen when we replace the broken disk.   Regards

Hello all,I'm trying to kerberize the Ambari 2.7.3 cluster. However, during the setup, I get the following error:Caused by: org.apache.ambari.server.serveraction.kerberos.KerberosOperationException: Unexpected error condition executing the kadmin command. STDERR: kadmin: Matching credential not found (filename: /tmp/ambari_krb_142308985016794830cc) while initializing kadmin interface at org.apache.ambari.server.serveraction.kerberos.MITKerberosOperationHandler.invokeKAdmin(MITKerberosOperationHandler.java:323) at org.apache.ambari.server.serveraction.kerberos.MITKerberosOperationHandler.principalExists(MITKerberosOperationHandler.java:123) at org.apache.ambari.server.serveraction.kerberos.KerberosOperationHandler.testAdministratorCredentials(KerberosOperationHandler.java:314) at org.apache.ambari.server.controller.KerberosHelperImpl.validateKDCCredentials(KerberosHelperImpl.java:2133)All of the authentication settings are okay, because I am able to kinit and

Hi All   I am trying to achieve similar functionality to below Unix command in HDFS level.    find /temp -name '*.avro' -cnewer sample.avro    or to retrieve the files greater than a specific timestamp from HDFS level.    From the Hadoop documentation I came to know that  we have limited functionality of  https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/FileSystemShell.html#find   Let me know how this can be achieved in Hadoop level. Any workarounds.    Thanks - Muthu   

I have to read data from two different hive views(probably two different databases) , extract data from those views and write it into files and then, join those files and perform data formatting and finally , write it into final file. Could you please suggest me some tools/technologies and design for hadoop for this requirement.

I use hive on spark.I made UDF.  Jar file name is 'hive-udf-20190701.jar' I set hive configuration(Hive Service Advanced Cofniguration snipped(Safety Value) for hive-site.xml.hive.reloadable.aux.jars.path/usr/local/bigdata/hive-udf I upload jar file to HiveServer2 filesystem directory./usr/local/bigdata/hive-udf-20190701.jarI create function on Hue.reload;drop temporary function udf_map_tojson;create temporary function udf_map_tojson as 'bigdata.hive.udf.MapToJsonString';I test UDF.select udf_map_tojson(str_to_map("k1:v1,k2:v2")); But, Exception raise.Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed due to: Job aborted due to stage failure: Aborting TaskSet 3.0 because task 0 (partition 0) cannot run anywhere due to node and executor blacklist. Most recent failure: Lost task 0.0 in stage 3.0 (TID 3, worker09.example.com, executor 1): UnknownReason Blacklisting beh

I have a CSV file in which nulls are represented by "/N". How can i go about replacing this with an empty string ''. SampleYear Age City Date2013 \N London \N 2019 56 \N \N 2015 20 \N 12/12/2019I would like to replace this, so that the file looks like the below. Year Age City Date2013 London 2019 56 2015 20 12/12/2019Which processors can i use to achieve this and what is the syntax to be used.

Hello,My ambari agent has lost its heartbeat. I tried restarting the agent/server starting stopping etc but nothing worked.Im really desperate and dont know what to do. Any help would be greatly appreciated. I included the ambari-agent logs below.ERROR 2019-08-02 10:51:04,424 script_alert.py:123 - [Alert][hive_webhcat_server_status] Failed with result CRITICAL: ['Connection failed to http://<My server id was here>/templeton/v1/status?user.name=ambari-qa + \nTraceback (most recent call last):\n File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py", line 190, in execute\n url_response = urllib2.urlopen(query_url, timeout=connection_timeout)\n File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen\n return opener.open(url, data, timeout)\n File "/usr/lib64/python2.7/urllib2.py", line 431, in open\n response = self._open(req, data)\n File "/usr/lib64/python2.7/urllib2.py", line 449, in _open\n \'_open\', req)\n File "/usr/lib6

Hi all,I have an issue to connect to the SQL Server with DBCPConnectionPool 1.9.2 but it is worked with DBCPConnectionPool 1.2Do you know if there are some modifications between 1.2 and 1.9.2 concerning DBCPConnectionPool ?Connection parameter :jdbc:jtds:sqlserver://${host}:${port};databaseName=xxxxxx;user=xxxxxx;password=xxxxxxnet.sourceforge.jtds.jdbc.Driver/var/dlk/squad/squad-jars/jtds-1.3.1.jarThanks for helps.Here error on NIFI 1.9.22019-07-31 12:53:42,483 ERROR [Timer-Driven Process Thread-12] o.a.n.p.standard.ListDatabaseTables ListDatabaseTables[id=db1439dc-aa6f-385b-be8f-66ee7faf84f4] ListDatabaseTables[id=db1439dc-aa6f-385b-be8f-66ee7faf84f4] failed to process session due to java.lang.AbstractMethodError; Processor Administratively Yielded for 1 sec: java.lang.AbstractMethodErrorjava.lang.AbstractMethodError: null        at net.sourceforge.jtds.jdbc.JtdsConnection.isValid(JtdsConnection.java:2833)        at org.apache.commons.dbcp2.Del

Trying to import data from oracle DB and getting error.... 19/07/31 13:07:10 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-myuser/compile/375d3de163797c05cd7b480fddcfe58c/QueryResult.jar 19/07/31 13:07:10 ERROR tool.ImportTool: Import failed: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "null"     at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281)     at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301) ....My sqoop command looks like...sqoop import \    -Dmapreduce.map.memory.mb=3144 -Dmapreduce.map.java.opts=-Xmx1048m \    -Dyarn.app.mapreduce.am.log.level=DEBUG \    -Dmapreduce.map.log.level=DEBUG \    -Dmapreduce.reduce.log.level=DEBUG \    -Dmapred.job.name="Ora import table $tablename" \    -Djava.security.egd=file:///dev/urandom \    -Djava.security.egd=file:///dev/urandom \    -Doraoop.time

I am following the instruction at links below to add users to Hive admin role.https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_dataintegration/content/hive-013-feature-sql-standard-based-grant-revoke.html and https://community.hortonworks.com/articles/4568/sql-based-authorization-in-hive.html However, I could not find this configuration "hive.users.in.admin.role" via Ambari UI (Hive - Configs - Advanced). I downloaded the hive_site.xml (via Ambria's Download Client Configs button) and found no such configuration in the xml file either. What did I miss?Below is my current Hive security settings.Authorization: SQL-Standard Based (SQLStdAuth)Authentication: NoneRun as end user instead of Hive user: False