Solved

I've read https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_cloud-data-access/content/intro.html where it is not recommended to use a cloud storage connector as the filesystem for HDFS. Can someone point out the reasoning for why these object stores can't be set as the defaultFS, which services wouldn't work/have issues, etc.?

Hello,   How zookeeper will react if we loose one disk assuming we have 3 zookeeper in 3 nodes. And what happen when we replace the broken disk.   Regards

Hello all,I'm trying to kerberize the Ambari 2.7.3 cluster. However, during the setup, I get the following error:Caused by: org.apache.ambari.server.serveraction.kerberos.KerberosOperationException: Unexpected error condition executing the kadmin command. STDERR: kadmin: Matching credential not found (filename: /tmp/ambari_krb_142308985016794830cc) while initializing kadmin interface at org.apache.ambari.server.serveraction.kerberos.MITKerberosOperationHandler.invokeKAdmin(MITKerberosOperationHandler.java:323) at org.apache.ambari.server.serveraction.kerberos.MITKerberosOperationHandler.principalExists(MITKerberosOperationHandler.java:123) at org.apache.ambari.server.serveraction.kerberos.KerberosOperationHandler.testAdministratorCredentials(KerberosOperationHandler.java:314) at org.apache.ambari.server.controller.KerberosHelperImpl.validateKDCCredentials(KerberosHelperImpl.java:2133)All of the authentication settings are okay, because I am able to kinit and

Hi All   I am trying to achieve similar functionality to below Unix command in HDFS level.    find /temp -name '*.avro' -cnewer sample.avro    or to retrieve the files greater than a specific timestamp from HDFS level.    From the Hadoop documentation I came to know that  we have limited functionality of  https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/FileSystemShell.html#find   Let me know how this can be achieved in Hadoop level. Any workarounds.    Thanks - Muthu   

I have to read data from two different hive views(probably two different databases) , extract data from those views and write it into files and then, join those files and perform data formatting and finally , write it into final file. Could you please suggest me some tools/technologies and design for hadoop for this requirement.

I use hive on spark.I made UDF.  Jar file name is 'hive-udf-20190701.jar' I set hive configuration(Hive Service Advanced Cofniguration snipped(Safety Value) for hive-site.xml.hive.reloadable.aux.jars.path/usr/local/bigdata/hive-udf I upload jar file to HiveServer2 filesystem directory./usr/local/bigdata/hive-udf-20190701.jarI create function on Hue.reload;drop temporary function udf_map_tojson;create temporary function udf_map_tojson as 'bigdata.hive.udf.MapToJsonString';I test UDF.select udf_map_tojson(str_to_map("k1:v1,k2:v2")); But, Exception raise.Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed due to: Job aborted due to stage failure: Aborting TaskSet 3.0 because task 0 (partition 0) cannot run anywhere due to node and executor blacklist. Most recent failure: Lost task 0.0 in stage 3.0 (TID 3, worker09.example.com, executor 1): UnknownReason Blacklisting beh

I have a CSV file in which nulls are represented by "/N". How can i go about replacing this with an empty string ''. SampleYear Age City Date2013 \N London \N 2019 56 \N \N 2015 20 \N 12/12/2019I would like to replace this, so that the file looks like the below. Year Age City Date2013 London 2019 56 2015 20 12/12/2019Which processors can i use to achieve this and what is the syntax to be used.

Hello,My ambari agent has lost its heartbeat. I tried restarting the agent/server starting stopping etc but nothing worked.Im really desperate and dont know what to do. Any help would be greatly appreciated. I included the ambari-agent logs below.ERROR 2019-08-02 10:51:04,424 script_alert.py:123 - [Alert][hive_webhcat_server_status] Failed with result CRITICAL: ['Connection failed to http://<My server id was here>/templeton/v1/status?user.name=ambari-qa + \nTraceback (most recent call last):\n File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_webhcat_server.py", line 190, in execute\n url_response = urllib2.urlopen(query_url, timeout=connection_timeout)\n File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen\n return opener.open(url, data, timeout)\n File "/usr/lib64/python2.7/urllib2.py", line 431, in open\n response = self._open(req, data)\n File "/usr/lib64/python2.7/urllib2.py", line 449, in _open\n \'_open\', req)\n File "/usr/lib6

Hi all,I have an issue to connect to the SQL Server with DBCPConnectionPool 1.9.2 but it is worked with DBCPConnectionPool 1.2Do you know if there are some modifications between 1.2 and 1.9.2 concerning DBCPConnectionPool ?Connection parameter :jdbc:jtds:sqlserver://${host}:${port};databaseName=xxxxxx;user=xxxxxx;password=xxxxxxnet.sourceforge.jtds.jdbc.Driver/var/dlk/squad/squad-jars/jtds-1.3.1.jarThanks for helps.Here error on NIFI 1.9.22019-07-31 12:53:42,483 ERROR [Timer-Driven Process Thread-12] o.a.n.p.standard.ListDatabaseTables ListDatabaseTables[id=db1439dc-aa6f-385b-be8f-66ee7faf84f4] ListDatabaseTables[id=db1439dc-aa6f-385b-be8f-66ee7faf84f4] failed to process session due to java.lang.AbstractMethodError; Processor Administratively Yielded for 1 sec: java.lang.AbstractMethodErrorjava.lang.AbstractMethodError: null        at net.sourceforge.jtds.jdbc.JtdsConnection.isValid(JtdsConnection.java:2833)        at org.apache.commons.dbcp2.Del

Trying to import data from oracle DB and getting error.... 19/07/31 13:07:10 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-myuser/compile/375d3de163797c05cd7b480fddcfe58c/QueryResult.jar 19/07/31 13:07:10 ERROR tool.ImportTool: Import failed: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "null"     at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3281)     at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301) ....My sqoop command looks like...sqoop import \    -Dmapreduce.map.memory.mb=3144 -Dmapreduce.map.java.opts=-Xmx1048m \    -Dyarn.app.mapreduce.am.log.level=DEBUG \    -Dmapreduce.map.log.level=DEBUG \    -Dmapreduce.reduce.log.level=DEBUG \    -Dmapred.job.name="Ora import table $tablename" \    -Djava.security.egd=file:///dev/urandom \    -Djava.security.egd=file:///dev/urandom \    -Doraoop.time

I am following the instruction at links below to add users to Hive admin role.https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_dataintegration/content/hive-013-feature-sql-standard-based-grant-revoke.html and https://community.hortonworks.com/articles/4568/sql-based-authorization-in-hive.html However, I could not find this configuration "hive.users.in.admin.role" via Ambari UI (Hive - Configs - Advanced). I downloaded the hive_site.xml (via Ambria's Download Client Configs button) and found no such configuration in the xml file either. What did I miss?Below is my current Hive security settings.Authorization: SQL-Standard Based (SQLStdAuth)Authentication: NoneRun as end user instead of Hive user: False

I use IntelliJ IDE installed on my Windows 10 laptop and try to run spark job in yarn mode on my 5 node HDP 3.1.1 cluster.My codes:package p1import org.apache.spark.sql.{SparkSession, functions => F}import org.apache.log4j.{Logger, Level}object SparkDeneme extends App { Logger.getLogger("org").setLevel(Level.INFO) val spark = SparkSession.builder() .appName("SparkDeneme") .master("yarn") .config("spark.hadoop.fs.defaultFS","hdfs://node1.impektra.com:8020") .config("spark.hadoop.yarn.resoursemanager.address","node1.impektra.com:8030") .getOrCreate()} import spark.implicits._ val sc = spark.sparkContext val dfFromList = sc.parallelize(List(1,2,3,4,5,6)).toDF("rakamlar") // dfFromList.printSchema() dfFromList.show()When I run get following error:19/07/26 20:00:32 ERROR SparkContext: Error initializing SparkContext.org.apache.spark.SparkException: Could not parse Master URL: 'yarn'    at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$c

Hi all, We have currently a Hadoop cluster in test in a private cloud with private ip and hostname:- <private_ip_1> mh1.example.com (namenode 1 in private cloud)- <private_ip_2> mh2.example.com (namenode 2 in private cloud)- <private_ip_3> w1.example.com (worker 1 in private cloud)...- <private_ip_n> wn.example.com (worker n private cloud)- <private_ip_m> eg1.example.com (edge node 1 + cloudera manager service in public cloud)- <private_ip_o> eg2.example.com (edge node 2 + hue etc. in public cloud) We can access on the private cloud only through the edge nodes and the cluster is kerberized with sentry.For the moment we use only self-certificate, but now we want to put the cluster in production. My question is that: Could we use official authority certificate in our case ? Because if I undestand well, authority certificate is only for public domain and public IP. Or what I was thinking for production is:- Self-Certificate for Cloud

Hi everyone, I'm using command in subject to fetch all yarn applications and use grep on the output to filter out a specific application . I wanted to find out what is the limit of this command, as in how far in the history does it go to get these states. Is this limited by the job history server limit or does it return something below that like last 1000 jobs etc? Regards

Confused about what Ambari users and groups are.When looking at the docs and the Ambari UI (Admin/Users and Admin/Groups), I get the impression that users / groups created in this interface should appear across all nodes in the cluster, but this does not seem to be the case, eg...[root@HW01 ~]# id <some user created in Ambari UI>id: <some user created in Ambari UI>: no such userSame situation for groups created in Ambari UI admin section.Not sure I understand to use of the Ambari users and groups if they do not somehow have a link back to user and groups locally on the hosts. Can someone please explain what is going on here?

We have a slow query like: select max(partition_col_1) from some_table where partition_col_2 = 'x' and it's super slow, scanning all records (hundreds of billions) in the filtered partitions, even though it's actually not getting anything out of them... select only includes partitioning column.  Absolutely no need to read any files I don't think. Any way or hint to get around this?

I'am struggling with Expression Language concerning nested and-/or-conditions.In my UpdateAttribute the variable KONTO_IST_AN should be getting the result in boolean which is provided by this expression. Sample content of RESPONSE (always exactly one element at one time):[{"antauschbis":"20191231","antauschvon":"20190101","id":"(422)"}]  => expected result: true[{"antauschbis":null,"antauschvon":"20190101","id":"(422)"}]  => expected result: true[{"antauschbis":"20190531","antauschvon":"20190101","id":"(422)"}]  => expected result: false[{"antauschbis":null,"antauschvon":"20200101","id":"(422)"}]  => expected result: falseExpression:${RESPONSE:jsonPath('$.[0].id'):equals(${KONTO_BV}):   and(${RESPONSE:jsonPath('$.[0].antauschvon'):le(${now():format("yyyyMMdd")}):        and(${RESPONSE:jsonPath('$.[0].antauschbis'):ge(${now():format("yyyyMMdd")}):              or(   

Not sure if this is the right subsection of this community forums but I am curious if there are fundamental changes that would prevent a custom CSD that was written for (and works for) a 5.x CM from working on a CM 6.x build. I will more than likely end up testing this myself soon but I guess I am wondering if there any caveats that I should be aware of or watch out for. Cheers!

hi allis it necessary to restart the ambari server after we delete ambari service by REST API ?I as asking because after we delete some service from the ambari by APIwe saw that service really Disappeared from ambari dartboard so I not sure if we need also to restart the ambari server to take affect?

Attempting to use / startup HDFS NFS following the docs (ignoring the instructions to stop the rpcbind service and did not start the hadoop portmap service given that the OS is not SLES 11 and RHEL 6.2), but running into error when trying to set up the NFS service starting the hdfs nfs3service:[root@HW02 ~]# [root@HW02 ~]# [root@HW02 ~]# cat /etc/os-release NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7" [root@HW02 ~]# [root@HW02 ~]# [root@HW02 ~]# service nfs status Redirecting to /bin/systemctl status nfs.service Unit nfs.service could not be found. [root@HW02 ~]# [root@HW02 ~]# [root@HW02 ~]# service nfs stop Redirecting to /bin/systemctl stop nf

Trying to upload a simple csv file into HDFS via the Ambari files view, getting the error below java.net.NoRouteToHostException: hw04.ucera.local:50075: No route to host (Host unreachable)     at java.net.PlainSocketImpl.socketConnect(Native Method)     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)     at java.net.Socket.connect(Socket....For reference, here the hosts on the clusterand the services on the HW04 nodeAccumulo ClientDataNodeHBase ClientHDFS ClientHive ClientHST AgentInfra Solr ClientLog FeederMapReduce2 ClientMetrics MonitorOozie ClientPig ClientSpark2 ClientSqoop ClientTez ClientYARN ClientZooKeeper ClientNot sure what the error m

When I run: hive --service metastore, it raised errors as below /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 24: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-hadoop2-compat.jar: Permission denied/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 25: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/lib/htrace-core.jar: Permission denied/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 26: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-common.jar: Permission denied/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 27: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-server.jar: Permission denied/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 28: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.

Are the ODBC Drivers for Hive the same for Cloudera and Hortonworks? Flexera indicates they are the same product due to the merger. However, there are downloads for both products, and they are not the same version: Cloudera 2.6.4: https://www.cloudera.com/downloads/connectors/hive/odbc/2-6-4.html Hortonworks 2.6.1.1: https://www.cloudera.com/downloads/hdp.html

Hello,We have upgraded our cluster and are getting error " The Oozie Server build version(4.1.0-cdh5.16.2) and the Oozie Server shared library version(4.1.0-cdh5.14.2) do not match". We have done the following:Upgrade the Oozie SharedLibGo to the Oozie serviceIf the Oozie service is stopped, start it:Select Actions > Start and click Start to confirm.Select Actions > Install Oozie SharedLib and click Install Oozie SharedLib to confirm. The service became Green from some time and then again its Red, kindly suggest / guide path to fix this  

Indent to call sqoop on a client node and need to be able to connect to an oracle DB. Currently seeing error...19/07/25 10:48:55 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriverWhen looking into this, appears need to install the proper jdbc driver for sqoop to use. However, the only docs I can find on installing sqoop drivers is for Cloudera CDP and following these docs runs into problems where certain expected directories do not exist on my cluster (particularly /var/lib/sqoop) (and I am assuming that they should already be there and not need to be manually created). Are there any docs on doing this for HDP? Docs for doing this just through the Ambari UI?