Solved

When I reinstall the Ambari Metrics，I find some widget didn't work。After carefully analyzation，I find out that in the all error widget the expression have subtraction. Anybody knows how to solve it ?

Hello All,I have installed Nifi(HDF) on server A. I connect to Ambari and Launch NiFi UI. Everything looks good till here.FIrst i tried to copy a file from 1 directory to another using GetFile. It did not recognize the directory and got the following error"input directory invalidated because input directory does not exist while creating getfile in nifi". I am on the same server where Nifi is Launched. I checked all permissions and everything looks good but not sure why it complains directory does not exist.I tried to generate a file and write to a directory, files are getting generated but getting below error."failure due to java.nio.file.AccessDeniedException: /home/xxx/receive: java.nio.file.AccessDeniedException"can someone please help me in fixing this issue.

I am trying to install HDF 3.3.1 (NIFI) to my existing HDP 3.1 cluster with Ambari 2.7.3. (https://supportmatrix.hortonworks.com/)I have followed all the steps in the documentation (Installing HDF Services on an Existing HDP Cluster), installed HDF management pack, ambari server restart = yes but I still cannot add HDF services through Add Service Wizard. Nothing to choose in 'Choose Serivce' .. Also cannot find space to update HDF Base URLCan someone help me with this issue.Thanks.

We have defined several YARN queues. Say that you have queue Q1, where users A and B run Spark processes.If A submits a job that demands all of the queue resources, they are allocated by YARN. Subsequently, when B submits his job, he is affected by resource scarcity.We need to prevent this situation, by assigning resources more evenly between A and B (and all other incoming users), within Q1. We have already set Scheduler to Fair. Can this eager resource allocation behaviour be prevented?

When I try to acces to ambari throw http://sandbox-hdp.hortonworks.com:8080/ it bring me to Apache Tomcat/9.0.8.I try every thing i could and even that soulution, no even in Advanced HDP Quick Links worked to me Ambari , that issue is similar but the solusion doesnt work to me.Any help would be apprecciated.Thanks in Advance

I config  prostro on hue ：Hue Service → Configuration → Service-Wide → Advanced → Hue ServiceAdvanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini， when I add the following:   [notebook] [[interpreters]] [[[presto]]] name=Presto JDBC interface=jdbc options='{"url": "jdbc:presto://172.31.17.252:9080/", "driver": "com.facebook.presto.jdbc.PrestoDriver"}' thus the hue-->query-->editor only show  "prosto JDBC ", how to config presto without losting the hive and impala button on edior ????   I try to config all these  [notebook] ## Show the notebook menu or not # show_notebooks=true ## Flag to enable the selection of queries from files, saved queries into the editor or as snippet. # enable_external_statements=true ## Flag to enable the bulk submission of queries as a background task through Oozie. # enable_batch_execute=true ## Flag to turn on the SQL indexer. # enable_sql_indexer=false ## Flag to turn on the Present

Spark is getting data with a header showing col's A,C,D,B and the data under it. Next day we get the same data like this Col's B,D,A,C and then the next day we get data like col's A,C,B,D and so on randomly. Now we have to put this data in a hive table with col's as A,B,C,D. Can any one suggest me an idea how to write this script in spark?

How can I add a node to a cluster using Ambari?

@Timothy SpannIn Apache NIFI http://localhost:8080/nifiI configured NIFI to Ingest Minifi logs to Nifi as below: a) Reading Minifi Logs using "Tail File" process and then the flow file is forwarded to NIFI http://localhost:8080/nifi using Remote Process Group. b) After receiving flowfile data at Nifi content of log file is read and written in CSV format in output file using put file. Now using SITE TO SITE TO PROVENANCE REPORT I can send NIFI's Provenance data to another instance of NIFI. It is working fine. Now need to know how to send MINIFI's Provenance data to another instance of NIFI. Is there a way to read Provenance data at MINIFI end. Note: There is Provenance Repository folder in both NIFI and MINIFI installation directory. Please help Thanks

I have a mapreduce program that communicates with a process on localhost machine. When running this job using yarn docker containers, the job fails because the container fails to communicate with the process. I tried running the same job without launching docker container using yarn but using docker run and specifying --ipc="host" and it worked. Is it possible to configure yarn to run docker containers with "--ipc" option?

Hello all, I'm trying to config Hiveserver2 use Spark and it's working perfect with small file. But with large file ( ~ 1.5GB ) , it will be crash by "GC overhead limit exceeded" . My flow is simple like this :1. Load data from text file into table_text ( text file ~ 1.5G )Sql: load data local path 'home/abc.txt' into table table_text;2. select data from table_text to insert to table_orc ( crash in this flow )SQL : Insert into table table_orc select id,time,data,path,size from table_text;I guess spark have to load all data from table_text and save it in memory before insert to table_orc . I researched and know that spark can config if data does not fit in memory, store the partitions that don't fit on disk, and read them from there when they're needed ( RDD Persistence ).My environment:Ubuntu 16.04Hive version : 2.3.0Free memory when launch sql : 4GMy config in hive-site.xml: <property> <name>hive.execution.engine</name> <value>spark</value></propert

I am attempting to configure Zeppelin behind Knox gateway running within an HDP 3.1.0 cluster with HDF 3.3.1 installed, all administered by Ambari 2.7.3. I am following the docs here: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/configuring-proxy-knox/sec_configuring_proxy_knox.pdf. As per the docs, I have set gateway.websocket.feature.enabled=true in Ambari's Knox configs, and when I access Zeppelin at the proxied URL I get the UI, saying the websocket is disconnected, and in the knox logs I see this:2019-01-08 20:26:46,217 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /zeppelin/ws2019-01-08 20:26:46,217 DEBUG knox.gateway (UrlRewriteProcessor.java:rewrite(163)) - Rewrote URL: https://KNOXHOST:8443/gateway/CLUSTERNAME/zeppelin/ws, direction: IN via explicit rule: ZEPPELINWS/zeppelin/ws/inbound to URL: ws://ZEPPELINHOST:9995/ws/ws2019-01-08 20:26:46,218 DEBUG knox.gateway (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request:

Dear @Jay Kumar SenSharma As per your suggested video link https://www.youtube.com/watch?v=wHaVBoLwzwU i have done all network setup ...and installation of ambari server and ambari-agent...successfully by local repo ....-When i adding only master , there is error to register .....see in ambari-agent log file .....INFO 2019-01-10 08:35:21,750 main.py:145 - loglevel=logging.INFOINFO 2019-01-10 08:35:21,751 main.py:145 - loglevel=logging.INFOINFO 2019-01-10 08:35:21,751 main.py:145 - loglevel=logging.INFOINFO 2019-01-10 08:35:21,754 DataCleaner.py:39 - Data cleanup thread startedINFO 2019-01-10 08:35:21,756 DataCleaner.py:120 - Data cleanup startedINFO 2019-01-10 08:35:21,758 DataCleaner.py:122 - Data cleanup finishedINFO 2019-01-10 08:35:21,820 PingPortListener.py:50 - Ping port listener started on port: 8670INFO 2019-01-10 08:35:21,823 main.py:437 - Connecting to Ambari server at https://master.alethe.com:8440 (192.168.1.191)INFO 2019-01-10 08:35:21,824 NetUtil.py:70 - Connectin

Hi,I've created a parameterized SQL query in the ExecuteSQL processor, this processor is predecessed by an UpdateAttribute with those parameters:here is the part of the where clause that uses this parameter : AND (( ?sql.args.4.value IS NULL AND mm.docno IS NULL ) OR ( mm.docno = ?sql.args.4.value))The issue is that I keep having this error message :"java.sql.SQLDataException: Value of the sql.args.4.type attribute is 'CHAR', which is not a valid JDBC numeral type"Any clues ?

Hello, A simple question: How can I know which queries generate Scratch files?I'm inspecting the Impalad logs and I couldn't find any information about the scratch file generation. Regards,Silva

hi ,i am currently using hdp 3.0 and ambari 2.7,i need to install the nifi service on my cluster ,just followed the hortonworks document they said following query ambari-server install-mpack --mpack=http://.../hdf-mpack-name.tar.gz --purge --verboseadded my centos7 package in the mpack.what we need to setup after?

Installed single node cluster on Centos 6 and enabled kerberos on cluster. ranger not starting..stderr: /var/lib/ambari-agent/data/errors-311.txtTraceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 231, in <module> RangerAdmin().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 93, in start self.configure(env, upgrade_type=upgrade_type, setup_db=params.stack_supports_ranger_setup_db_on_start) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 120, in locking_configure original_configure(obj, *args, **kw) File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 135, in configure setup_ranger_db() File "/var/lib/ambari-agent/c

We have a client who wants to setup a small separate special purpose cluster.  It will consist of 5 nodes and they want to run Kudu on it.  If we configure with 3 Master nodes and 3 Tablet Servers, at least one of the nodes will have both Master roles as well as Tablet Server role.  In any case, in this scenario, if hypothetically the node with both roles fail, and the replication is set to 3, neither the master tablet nor any of the table tablets will be able to satisfy the replication factor since there are now only 2 master roles and 2 table servers available.   Since this is still within the (n-1)/2 failures, will the service still run?  After 5 minutes, and when the nodes are determined to be permanent failures, a new follower needs to be created but since there are only 2 tablet servers still functioning, what will happen?  Will the system limp along, still functioning or will it fail? Thanks in advance for any help.

Team, I have 1000 tables in my source RDBMS and I would like to get them migrated to hive using pyspark I read through documentation and found that below two commands would help. Is there a way I can loop these two commands 1000 times if I have all the list of tables in a python array? arr = ("table1","table2")for x in arr:            df = spark.read.format("jdbc").blah.blah              data.write.saveAsTable.blah.blah If someone has a working solution for this could you please share. I tried but it is not throwing any error but at same time not writing anything. ThanksMeher 

I have a HDP 2.6 cluster that I want to upgrade to HDP 3.1. I currently use postgres 9.2 for Ambari, and a separate postgres 9.2 instance for Ranger and Hive. Looking at the compatibility matrix, postgres 9.2 is no longer supported for HDP 3.1.1) Does this apply just to the postgres instance used by ambari, or also to the postgres instance used by Ranger and Hive too?2) I'm assuming upgrading postgres isn't handled by ambari, so should I just upgrade and migrate the relevant data, keeping the connection parameters the same?Thanks in advance

I have setup kylin in HDP. Now want to build the cube using spark instead of map reduce. Cube is getting build with map reduce but when trying to use spark it is getting hanged. Tried to set the hive execution engine as spark instead of Tez in hive. But it is not working and getting error java.lang.NoClassDefFoundError: org/apache/spark/SparkConf. Is there any way we can run hive on spark in HDP or any suggestion how to run kylin using spark? Help is much appreciated.

I am currently working with Apache Hadoop MapReduce and would like to know: what exactly is speculative execution and how does it work?

Hi Everyone,I am gonna adding a new hosts in existing cluster ....All Network Process are done successfully ...after then did't register new hosts by AMBARI....hdp version details : hdp 2.6 with ambari 2.6Os - centso7Please Suggest How to do , Your Help is Highly appreciated .screenshot-from-2018-12-28-18-17-06.pngscreenshot-from-2018-12-28-18-23-15.pngscreenshot-from-2018-12-28-18-40-11.png

Hi, I'm investigating the approaches for row level security in Impala, and I have just found multiple references for RecordService. However, it's homepage is removed, the source code is gone from GitHub, and it didn't get a new release for 2+ years on Maven, so I assume that it's dead.Does anybody knows what happened to it, why it was cancelled? These are the ones which I found as possible workarounds:Store the data on HDFS in separate files splitted by the filter expression, and use HDFS ACLs for the filesUse HBase as a storage engine and query the data directly from it, as it supports row-level securityDisable access to the original Impala table, create views from it with the appropriate filters and use regular Impala ACLs for the viewsIs there any alternative for row level security in Impala? RecordService sounded like the solution what I needed. Thanks,Peter

Hello,    We currently have CDH 5.15.0 running in our environments and have a necessity to support OpenJDK. Does upgrading to CDH 6  help in supporting OpenJDK? If yes, can we directly upgrade from CDH 5.15.0 to CDH 6 or do we have to downgrade first to CDH 5.14 and then upgrade to CDH 6 ( documentation does not show direct upgrade from CDH 5.15 to CDH 6 https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_install_upgrade_notes.html#topic_1_2). Thank you for your time in advance.