Q&A

I am new to Ambari ... It looks like it is not allowed to manually edit configuration files like /etc/hive/conf/hive-site.xml. I tried to make some changes but after I re-started Hive via Ambari - all my changes are gone. Can somebody comment on this and point me to some Amabari "best practices" documentation?

Hello,I have used Ambari 2.7.3 to setup an HDP 3.1.0 + HDF 3.3.0 cluster, and am trying to configure Knox proxy functionality but am having problems...the Knox admin UI doesn't seem to respect the setting of 30 minute sessions with the ShiroProvider. I have verified that the configs in Ambari all specify sessionTimeout of 30, which is supposed to be 30 minutes, but I get logged-out almost immediately when trying to use the admin UI. If I click down into the default read-only topologies, and then try to navigate back into the providers, I have to re-login, even though it's on;y been like 30 seconds.Any idea what the problem might be?Thanks in advance,Doug

I am using 3.0.0.0-1634. The annoying part is that the issue is random.The same job runs fine and suddenly it fails with this issue and next time some other job might fail after couple of days which had successfully ran earlier and which will run properly in future. There is no other jobs running during that time on the cluster. 18/12/29 20:25:21 INFO mapreduce.Job: Running job: job_1546013184089_004618/12/29 20:49:47 INFO mapreduce.Job: Job job_1546013184089_0046 running in uber mode : false 18/12/29 20:49:47 INFO mapreduce.Job: map 0% reduce 0%18/12/29 20:49:47 INFO mapreduce.Job: Job job_1546013184089_0046 failed with state FAILED due to: Application application_1546013184089_0046 failed 2 times due to ApplicationMaster for attempt appattempt_1546013184089_0046_000002 timed out. Failing the application.18/12/29 20:49:47 INFO mapreduce.Job: Counters: 018/12/29 20:49:47 ERROR crawl.DeduplicationJob: DeduplicationJob: java.io.IOException: Job failed! In Resource manager in Di

hi allwe have ambari cluster with HDP version 2.6.0when we try to start the metrics collector from ambari , we get the following logs:2019-01-02 06:35:16,406 ERROR org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: hconnection-0x4bdeaabb-0x1680d44c1810001, quorum=master02.sys32.com:61181, baseZNode=/ams-hbase-unsecure Received unexpected KeeperException, re-throwing exceptionorg.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ams-hbase-unsecure/table/SYSTEM.CATALOG at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:622)2019-01-02 06:35:18,256 INFO org.apache.zookeeper.ClientCnxn: Opening sock

hi allwe try to start the ambari metrics and we get the following logs:2019-01-02 06:35:16,406 ERROR org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: hconnection-0x4bdeaabb-0x1680d44c1810001, quorum=master02.sys32.com:61181, baseZNode=/ams-hbase-unsecure Received unexpected KeeperException, re-throwing exceptionorg.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ams-hbase-unsecure/table/SYSTEM.CATALOG at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:622)and also2019-01-02 06:35:18,256 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server master02.sys32.com/101.23.98.60

unable to login with root user using hadoop password, I never changed the password

I tried to add another host via cloudera manager . I followed the below steps 1. Hosts - > Add hosts -> provided the ip address -> CM asked me to create a host template but i gave none and finised . 2. Found the host in good health . 3. Cluster -> HDFS -> Instances -> Add -> I added a Datanode with the new hosts.   (Now the Datanode i added was not starting , When i search for logs via UI i dont find any . I searched for logs in logs/datanode folder , but there was no Datanode folder in Var .)   4. Cluster -> Deploy client Configuration -> Completed -> Still the Datanode was not starting .   Please help .   Thanks Bala

i am trying to deploy client configuration on hbase , and it always fails , i am having 1 master , and 2 workers  , and here are the logs:     Generate client configuration.Successfully generated client configuration.   Run 3 steps in parallelCompleted 2/3 steps successfully       Cleanup client configs that failed to deploy.Successfully cleaned up client configs that failed to deploy.   Can't open /var/run/cloudera-scm-agent/process/740-deploy-client-config/hbase-conf/hive-env.sh: No such file or directory.+ '[' '!' -e /etc/hbase ']'+ cd /etc/hbase+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg,topology.py,topology.map}'+ '[' -e /etc/hbase/conf.cloudera.hbase/taskcontroller.cfg ']'+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executor.cfg,topology.py,topology.map}'+ '[' -e /etc/hbase/conf.cloudera.hbase/container-executor.cfg ']'+ for SPECIAL_FILE in '$DEST_PATH/{taskcontroller.cfg,container-executo

In stanalone and secured NiFi 1.7.0 using oauth 2.0 as authentication mechanism. I try to use the SiteToSiteProvenanceReportingTask The property 'Transport Protocol' of Remote Process Group is 'HTTP'. I get the error message: Attached is the snapshot of the Reporting Task configurations: I get the below error in bulletin and logs: Unable to refresh Remote Group's peers due to response code 403:Forbidden with explanation: null 2018-11-12 06:20:27,984 INFO [Timer-Driven Process Thread-4] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling SiteToSiteProvenanceReportingTask[id=<id>] to run The Site to Site properties nifi.properties are as below: nifi.remote.input.host=<hostname>, nifi.remote.input.secure=true, nifi.remote.input.socket.port=10443, nifi.remote.input.http.enabled=true, nifi.remote.input.http.transaction.ttl=30 sec nifi.remote.contents.cache.expiration=30 secs How to resolve it? Thanks for your help. Poonam

So I have 4 740xd Servers with 80 Threads on each and 384GB of ram each. How should I best handle heap for each nifi VM on the server? Option 1: 1, VM 80 cpus, NiFi set to 300GB of Ram for heap? And have the 4 nifis clustered. Option 2: 10, VMs 8 cpus each, NiFi set to 32GB of Ram for heap for each VM, These NiFis would also be clustered, across the Virtualized 740 and across the 4 hardware 740s Note: all these files going through these nifis are less than 4GB, and 90% of them are under 10MB ALSO what generation should I use for Java Garbage Collection? Attached are two pictures explaining my options. John

We're looking to automate the shutdown/restart the processor instead of manually stopping through Nifi UI. We don't want to bring down everything through nifi.sh stop, instead looking to bring down a specific processor.

Hi All, I have problem with kerberbos util command (ktutil), on client machine i created a new principal with add_entry and it's work fine but when i try to init ticket with kinit it's failed with message : "principal don't existe on kerberos database". when i create principal with kadmin.local on krb-server and try to init ticket on client it's work. any suggestion pleasee ?? Thanks for your help. Best Regards suggestion

Hi All, I have a scenario where i have a 3 node HDF cluster and i am making an API call to get the validation token(valid for 1 hour only) using InvokeHTTP every one hour on primary node only, since its not preferred to make multiple API calls. i am putting this token in Distributed Cache, but since i want to fetch the token from the distributedCache on all nodes, is there a way i can duplicate the flowfile from primary node to all nodes to the processors Downstreams. is there a way i can achieve this ?

HBase Master cannot start up!! Here is the error (trace): <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> 2015-05-08 09:31:02,675 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/lib/native2015-05-08 09:31:02,675 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp2015-05-08 09:31:02,675 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>2015-05-08 09:31:02,675 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux2015-05-08 09:31:02,675 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd642015-05-08 09:31:02,675 INFO org.apache.zookeeper.ZooKeeper: Client environment:

I have a processor group that fetches data from ElasticSearch and stores it in mysql. Every time I run this processor group, NiFi will restart automatically after a while, which will cause my 'Scroll Duration=2m' to be disabled in ScrollElasticSearchHttp,and then I will never get the new data from the ES. I checked the log file and found no error message. this is the nifi-bootstrap_2018-10-09.log 2018-10-09 09:43:15,167 WARN [main] org.apache.nifi.bootstrap.RunNiFi Apache NiFi appears to have died. Restarting...2018-10-09 09:43:15,343 INFO [main] org.apache.nifi.bootstrap.Command Launched Apache NiFi with Process ID 158612018-10-09 09:43:15,373 INFO [main] org.apache.nifi.bootstrap.RunNiFi Successfully started Apache NiFi with PID 158612018-10-09 09:43:16,401 INFO [NiFi logging handler] org.apache.nifi.StdOut Listening for transport dt_socket at address: 81112018-10-09 09:43:21,932 INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and list

Hi, We are planning to use NiFi's Data Provenance on long term to use it for audits. What is the best way to configure NiFi ? If I "simply" backup the Data Provenance disk content, will I be able to use it later? By reinjecting it in a working NiFi? Should I use ReportingTaskProcessor? But again how do you query backuped data later? By keeping it in a dedicated NiFi used only for backup? I also did not understand the management of the FlowFile contents, is it supposed to be stored in the DataProvenance Disk (which would greatly increase its size...)? Or is the "replay button" from Data Provenance UI working only if the content is still fresh and present in the "Content repository"? Or is Data Provenance just not meant to used for long term purpose? Sorry if mess up multiple concepts. Thanks.

Hi, I am using RouteONAttribute Processor and i defined a property and its value contains multiple AND conditions like the following: Property Name: ROUTE1 Property Value: ${text.tag.type:equals('XXX'):and(${txt:contains('AAA')}):and(${txt:contains('BBB')})} The Issue i am facing is that after generating a flow file with attributes text.tag.type = XXX and txt = AAA,BBB it always route to unmatched while it should route to ROUTE1 Thanks ...

Hi All, Thanks a lot everyone on this awesome community. I have data flows in PROD now, and I am adding new data flows and testing them, however accidentally I stopped one the flows which was good and running in PROD, reuslting in data loss. Is there any way I can freeze the data flow or dataflow confirms twice before I stop a data flow? Thanks Dheeru

Hi Team, I am using PUTSQL processor to execute copy command in redshift database.This processor usually processes more than 20 flowfiles which are merged files containing json objects.I have configured it this way:(please see attached image). I want to know have i configured run-duration and run schedule correctly? My nifi canvas has more than 2000 processor and i am facing ui slowness issues.Please suggest how to decide these configurations in order to get high throughput and less resource consumption. Thanks in advance!! Sri

I have seen a situation wher some of the jobs suddenly failed with error saying "Cannot create a file. Namenode in safe mode." When I check the cloudera manager, HDFS_CANARY_HEALTH check failed. I also observed that the Active namenode become standaby and standby become active. So, if the HDFS_CANARY_HEALTH test fails then Namenode restarts by itself? Can anyone respond to this.

Hi everyone,   My system have 2 datanode, 2  namenode, 3 journalnode, 3 zookeeper service    I had config cluster namenode ok , when browsing the admin page namenode:50070 , I had see 1 name node status (active) and one namenode status (standby). => OK    When I stop active namenode the other with standby become active .  => OK   But the problem is how start the namenode which I had stop again ?   I do the following :   sudo -u hdfs hdfs namenode -bootstrapStandby -force/etc/init.d/hadoop-hdfs-namenode start With above process sometime namenode start ok with standby mode , but sometime it start with active mode and then I have 2 active node (split brain !!)   So what I have wrong , what is the right process to start a namenode had stop again   Thanks you          

Hi I am trying to send data from one nifi to another nifi (which are working in different nodes) using remote process group. So i created one remote process group and mention the url of other nifi host. Now whenever i am trying to connect a getfile or generate flowfile with this RPG its giving me error that it does not have input ports for RPG. I have configured the nifi.properties file as mentioned in here https://nifi.apache.org/minifi/getting-started.html and mention the site to site port number but it still not working. I am attaching some screenshots regarding my problem. Hoping for a solution. nifi1.png nifi2.png nifi3.png Thanks in Advance. Pratik

Hi, I am performing a basic check to see if a file exists in HDFS or not. I am using hdfs dos -test command for the same. But it doesn't seem to work correctly. In the documentation its mentioned as it would return 0 if the file exists. But I am not getting any output when the command is run. Let me know what need to be done to get this working. Please see the screenshot attached Thanks

We have setup Nifi site-to-site to transfer Snort logs (alert.csv) to a remote Nifi server, using Apache Metron (Hortonworks Cybersecurity). Once Snort is running, it produces a vast amount of logs in CSV format, more than 500 records only in 5 seconds for instance. The problem is that Nifi Tailfile cannot detect and separate all 500 records, it may detect around 20 changes (totally include 500 records). Current configurations are attached to this message.How can we configure the Tailfile processor to detect every single line as a new packet (event)? Any help would be appreciated.

HI, I would like to create data flow,which put file to one directory, and the if it has succeded, put the same file to another directory I use for that Notify/Wait, but I wonder if uuid is a good solution for the signal, Are Wait/Notify properties adequate ? or if there is another way ?