Q&A

Hi,   Could you please help me out by giving answer for the below questions   If  HDFS services goes down in kafka cluster, what will happen to kafka service and data   Regards, R.

Hi, Could you please help me out by giving answer for the below questions? 1. What is command to set queue name for hive job 2. Which of the files in the input directory will be skipped by hive job?A._SUCCESSB.  IGNOREC. Input.metadataD.   .skip

How do I remove a currency symbol from a string and convert it into an int in Hive? For example, how do I change '124$' into 124?

Issue: Loading data into Hive ORC table is infinite, I should manually kill the load process.I am trying to load data into ORC Hive table from another Hive TEXTFILE table. Since the source files are TXT/Json, loading data first into TEXT table and then trying to load into ORC table.Cluster: HDP 2.6.5-292Hive version: 1.2.1000.2.6.5.0-292Here is the Hive TEXTFILE table schema:Create external table if not exists TEXTTable(ID bigint, DOCUMENT_ID bigint, NUM varchar(20), SUBMITTER_ID bigint, FILING string, CODE varchar(10), RECEIPTNUM varchar(20))row format delimited fields terminated by '|' Location '/data/3rdPartyData/Hive/ TEXTTable 'TBLPROPERTIES ('skip.header.line.count'='1');Load Data into TEXTFILE table:load data local inpath '/data/TextFile.txt' overwrite into table TEXTTable;Here is the Hive ORC table schema:Create external table if not exists ORCTable(ID bigint, DOCUMENT_ID bigint, NUM varchar(20), SUBMITTER_ID bigint, FILING TIMESTAMP, CODE varchar(10), RECEIPTNUM varchar(20))

I have nifi cluster summarry json data like this;[{ "nodeId": "29bed24c-6d73-4652-930b-0065cad4ef66", "address": "b1518aec7e38", "apiPort": 8080, "status": "CONNECTED", "heartbeat": "01/16/2019 16:24:32 UTC", "roles": ["Primary Node", "Cluster Coordinator"], "activeThreadCount": 0, "queued": "0 / 0 bytes", "events": [{ "timestamp": "01/16/2019 15:21:50 UTC", "category": "INFO", "message": "Received first heartbeat from connecting node. Node connected." }, { "timestamp": "01/16/2019 15:21:48 UTC", "category": "INFO", "message": "Connection requested from existing node. Setting status to connecting." }], "nodeStartTime": "01/16/2019 15:20:44 UTC" }, { "nodeId": "22aofpp4-87rf-asf4-930b-0065cad4ef66", "address": "b67uf98tkl5", "apiPort": 8080, "status": "CONNECTED", "heartbeat": "01/16/2019 16:24:32 UTC", "roles": [], "activeThreadCount": 0, "queued": "0 / 0 bytes", "events": [{ "timestamp": "01/16/2019 15:21:50 UTC", "category": "INFO", "message": "Received first heartbeat f

If any node is down, can nifi create new node automatically? The unused node will be ready to use. When a node is down(fail, disconnect or etc.), I just want to use this new node instead of failed node. Can nifi do this automatically? Is this possible?Thank you.

Hello, i am new to atlas and i am wondering if there is a solution making atlas expose, in addition to a dataset metadata, a sample data of this dataset ( 10 first rows for example).for a purpose of data governance, the problem i am solving is that the metadata could not be so comprehensive functionally, and reading a sample data will make more sens to explain a dataset content.if , actually, there is not a solution please help me find were should i invest my effort ( inner development within atlas, or treat the problem using a third party accessing hive,scoop or hbase directly to get the sample data)thanks in advance.

Does anyone know what are the changes we need to use the https://issues.cloudera.org/browse/HUE-7407 changed code effectively. We have in our user_filter:|(memberof=CN=admin_group,OU=OUname,DC=DC,DC=DC,DC=com)(memberof=CN=nonadmin_group,OU=OUname,DC=DC,DC=DC,DC=com) Ideally we want to someway set all the poeple in the admin group as super_user's for hue.It would be of great help because currently when we rehydrate our AWS EMR which has HUE enabled all the user's gets removed and when someone logins to the new AWS EMR they will become super_user.I have circumvented this issue temporarly  by login a user in the bootstrap action.   

I'm currently upgrading from HDP 2.6.3 to HDP 3.1.0, following the docs.On my test cluster, upgrading ambari-infra-solr went fine, but on our main cluster running the backup stage failed. I restarted the ambari-infra-solr server, but it doesn't seem to have come back properly - loading the web UI shows a few GUI elements and a big "loading" spinning. Logging seems broken too - while the server is running, nothing appears in solr.log and there are log4j configuration warnings in stdout.I added a second ambari-infra-solr server, and it sees the collections but can't access the shards.Long story short, I don't particularly mind losing the content of the collections - is there an easy way to drop them and start from scratch?Thanks

Hello,When installing new HDP version in Ambari, the following error is displayed: "Cluster <cluster_name> upgrade is in progress. Cannot install packages."I had tried to use instruction like this https://community.hortonworks.com/content/supportkb/177629/errorcluster-xyz-upgrade-is-in-progress-cannot-ins.htmlBut not successfully. How can i work around this problem?

We would like to take advantage of Oracle JDK 11.1.0. So, Will it be possible to use Oracle JDK 11.1.0 with HDP 3.1.0 ?.Anyone have any success stories on this ?.Please share your experiences?

Please tell me the requirements of Impala.In the HW requirements of Catalog Server and Impalad, the CPUs are listed as well, are they correct?Does Catalog Server also require CPU, Min: 4 - Recommended: 16?I referred to the following page. * Hardware Requirements  * impala   * https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_hardware_requirements.html#concept_usf_qln_3bb * Impala  * Impala Requirements   * https://www.cloudera.com/documentation/enterprise/6/6.0/topics/impala_prereqs1.html#prereqs_hardware

we have ambari cluster with 176 datanodes machines ( workers )each datanode have the following component:capture.pngDataNode / HDFSMetrics Monitor / Ambari MetricsNodeManager / YARNhow to restart all these component on all datanode machines by API* because it will take time to restart all datanode component manual as ( access each datanode machine from ambari and perform --> host actions --> restart all )

When I reinstall the Ambari Metrics，I find some widget didn't work。After carefully analyzation，I find out that in the all error widget the expression have subtraction. Anybody knows how to solve it ?

Hello All,I have installed Nifi(HDF) on server A. I connect to Ambari and Launch NiFi UI. Everything looks good till here.FIrst i tried to copy a file from 1 directory to another using GetFile. It did not recognize the directory and got the following error"input directory invalidated because input directory does not exist while creating getfile in nifi". I am on the same server where Nifi is Launched. I checked all permissions and everything looks good but not sure why it complains directory does not exist.I tried to generate a file and write to a directory, files are getting generated but getting below error."failure due to java.nio.file.AccessDeniedException: /home/xxx/receive: java.nio.file.AccessDeniedException"can someone please help me in fixing this issue.

I am trying to install HDF 3.3.1 (NIFI) to my existing HDP 3.1 cluster with Ambari 2.7.3. (https://supportmatrix.hortonworks.com/)I have followed all the steps in the documentation (Installing HDF Services on an Existing HDP Cluster), installed HDF management pack, ambari server restart = yes but I still cannot add HDF services through Add Service Wizard. Nothing to choose in 'Choose Serivce' .. Also cannot find space to update HDF Base URLCan someone help me with this issue.Thanks.

We have defined several YARN queues. Say that you have queue Q1, where users A and B run Spark processes.If A submits a job that demands all of the queue resources, they are allocated by YARN. Subsequently, when B submits his job, he is affected by resource scarcity.We need to prevent this situation, by assigning resources more evenly between A and B (and all other incoming users), within Q1. We have already set Scheduler to Fair. Can this eager resource allocation behaviour be prevented?

When I try to acces to ambari throw http://sandbox-hdp.hortonworks.com:8080/ it bring me to Apache Tomcat/9.0.8.I try every thing i could and even that soulution, no even in Advanced HDP Quick Links worked to me Ambari , that issue is similar but the solusion doesnt work to me.Any help would be apprecciated.Thanks in Advance

I config  prostro on hue ：Hue Service → Configuration → Service-Wide → Advanced → Hue ServiceAdvanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini， when I add the following:   [notebook] [[interpreters]] [[[presto]]] name=Presto JDBC interface=jdbc options='{"url": "jdbc:presto://172.31.17.252:9080/", "driver": "com.facebook.presto.jdbc.PrestoDriver"}' thus the hue-->query-->editor only show  "prosto JDBC ", how to config presto without losting the hive and impala button on edior ????   I try to config all these  [notebook] ## Show the notebook menu or not # show_notebooks=true ## Flag to enable the selection of queries from files, saved queries into the editor or as snippet. # enable_external_statements=true ## Flag to enable the bulk submission of queries as a background task through Oozie. # enable_batch_execute=true ## Flag to turn on the SQL indexer. # enable_sql_indexer=false ## Flag to turn on the Present

Spark is getting data with a header showing col's A,C,D,B and the data under it. Next day we get the same data like this Col's B,D,A,C and then the next day we get data like col's A,C,B,D and so on randomly. Now we have to put this data in a hive table with col's as A,B,C,D. Can any one suggest me an idea how to write this script in spark?

How can I add a node to a cluster using Ambari?

@Timothy SpannIn Apache NIFI http://localhost:8080/nifiI configured NIFI to Ingest Minifi logs to Nifi as below: a) Reading Minifi Logs using "Tail File" process and then the flow file is forwarded to NIFI http://localhost:8080/nifi using Remote Process Group. b) After receiving flowfile data at Nifi content of log file is read and written in CSV format in output file using put file. Now using SITE TO SITE TO PROVENANCE REPORT I can send NIFI's Provenance data to another instance of NIFI. It is working fine. Now need to know how to send MINIFI's Provenance data to another instance of NIFI. Is there a way to read Provenance data at MINIFI end. Note: There is Provenance Repository folder in both NIFI and MINIFI installation directory. Please help Thanks

I have a mapreduce program that communicates with a process on localhost machine. When running this job using yarn docker containers, the job fails because the container fails to communicate with the process. I tried running the same job without launching docker container using yarn but using docker run and specifying --ipc="host" and it worked. Is it possible to configure yarn to run docker containers with "--ipc" option?

Hello all, I'm trying to config Hiveserver2 use Spark and it's working perfect with small file. But with large file ( ~ 1.5GB ) , it will be crash by "GC overhead limit exceeded" . My flow is simple like this :1. Load data from text file into table_text ( text file ~ 1.5G )Sql: load data local path 'home/abc.txt' into table table_text;2. select data from table_text to insert to table_orc ( crash in this flow )SQL : Insert into table table_orc select id,time,data,path,size from table_text;I guess spark have to load all data from table_text and save it in memory before insert to table_orc . I researched and know that spark can config if data does not fit in memory, store the partitions that don't fit on disk, and read them from there when they're needed ( RDD Persistence ).My environment:Ubuntu 16.04Hive version : 2.3.0Free memory when launch sql : 4GMy config in hive-site.xml: <property> <name>hive.execution.engine</name> <value>spark</value></propert

I am attempting to configure Zeppelin behind Knox gateway running within an HDP 3.1.0 cluster with HDF 3.3.1 installed, all administered by Ambari 2.7.3. I am following the docs here: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/configuring-proxy-knox/sec_configuring_proxy_knox.pdf. As per the docs, I have set gateway.websocket.feature.enabled=true in Ambari's Knox configs, and when I access Zeppelin at the proxied URL I get the UI, saying the websocket is disconnected, and in the knox logs I see this:2019-01-08 20:26:46,217 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /zeppelin/ws2019-01-08 20:26:46,217 DEBUG knox.gateway (UrlRewriteProcessor.java:rewrite(163)) - Rewrote URL: https://KNOXHOST:8443/gateway/CLUSTERNAME/zeppelin/ws, direction: IN via explicit rule: ZEPPELINWS/zeppelin/ws/inbound to URL: ws://ZEPPELINHOST:9995/ws/ws2019-01-08 20:26:46,218 DEBUG knox.gateway (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: