I was looking into the Open ID way of user authentication for Nifi Registry. I see that we can implement OpenID for Nifi but it is not available for Nifi Registry? Is there a way we can implement Open ID authentication for NiFi registry also? ... View more

We had a data node disk failure recently and we got it replaced. However the new disk is not recognized and NameNode UI still shows the disk is failed? What should be next steps so that new disk is recognized and data gets replicated? ... View more

In one of our data nodes the root partition is almost 98% full and I see that there is nothing much I can clean up. I saw below tar.gz files in the root partition: rw-r--r-- 1 root root 196M /usr/hdp/2.5.0.0-1245/hadoop/mapreduce.tar.gz r--r--r-- 1 root root 378M /usr/hdp/2.5.0.0-1245/oozie/oozie-sharelib.tar.gz -rw-r--r-- 1 root root 117M /usr/hdp/2.5.0.0-1245/pig/pig.tar.gz -rwxr-xr-x 1 root root 121M /usr/hdp/share/hst/smartsense-activity-explorer-1.3.0.0-22.tar.gz Will there be any impact if I move these tar.gz files to a different lodcation so that we can have more space in the root partition? ... View more

I am trying to deploy the application I built in SAM, but while I deploy I am getting error: Topology submission failed due to: java.io.IOException: Cannot run program "/usr/java/jre1.8.0_131/bin/jar": error=2, No such file or directory When I see the log, below are the lines printed. Below is printed in : INFO [14:39:26.872] [dw-241414 - GET /api/v1/catalog/streams/componentbundles/SOURCE/1/hints/namespaces/1-EventThread] o.a.z.ClientCnxn - EventThread shut down WARN [14:50:08.543] [ForkJoinPool-4-worker-7] c.h.s.s.c.t.c.TopologyComponentFactory - Type JOIN not found in provider map, returning an instance of com.hortonworks.streamline.streams.layout.component.StreamlineProcessor ERROR [14:50:08.846] [ForkJoinPool-4-worker-7] c.h.s.s.a.t.s.TopologyStates - Error while trying to deploy the topology in the streaming engine java.io.IOException: Cannot run program "/usr/java/jre1.8.0_131/bin/jar": error=2, No such file or directory at java.lang.ProcessBuilder.start(Unknown Source) at com.hortonworks.streamline.streams.actions.storm.topology.StormTopologyActionsImpl.executeShellProcess(StormTopologyActionsImpl.java:733) Also I see below error in : Exception in thread "main" com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Connection to abc.xyz.net:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. Request for assistance ... View more

Hi All, I am facing a wierd problem today. We noticed that the space in the root partition was full. The /var/log/nifi is configured in the same partition. I cleared almost all the nifi logs, however within a minute they are appearing back and making root partition full. What might be the cause for this ? How do I get it resolved? I have a standalone nifi instance ... View more

We have a query which has about 4 UNIONS on the same table and an order by clause. I wanted to know if it is ever possible to make a Phoenix query with 4 unions and orderby clause a sub second query? Although I have created required indexes but query is taking about 3 sec to execute. Is there a way of achieveing sub second query with 4 UNION operations? ... View more

Hi All, I have a requirement where in I have to setup a new HDP cluster which will be replica of our one another cluster. WHat I am planning is to export the Ambari Blueprint and use the same blue print to setup another server. What are the right steps for me to do and what else I need to take care to get this task done? ... View more

We have a requirement where in we fetch the log information real time from linux servers via Nifi.(streaming data) We need to do some business calculation where in we have to substract 2 subsequent time stamps. So it becomes very important to maintain the order. We put the data in Kafka (topic has one partition to maintain ordering) and fetch it from there. Once data is in Kafka, I would like to know what should I use (Spark Streaming or Storm) to process the data by maintaining the order. ... View more

I have a requirement where in I have to maintain a cache of about 200,000 records. I want to use H2 database as a cache. If any body have used H2 as a cache please let me know the steps. ... View more

I have a requirement where in I have to join 2 table from Oracle DB and HBase (populated using NiFi) to do some calculation and insert data into HBase. I would like to know what would be the better appoach to do it. One of the approach that I was thinking is: Create the solr index on Oracle Table and HBase table and then join them to perform the calculation. My concern is we have around 2 million incoming streaming records perday. Will doing Solr Index joins will be a viable option? If there is any other option to do it please suggest. Can I use GETSOLR processor to simplify my task? ... View more

Hi All, I have taken a Snapshot in my Dev Environment and Exported to PreProd environment so that I can clone a new table from snapshot or else I can restore it. I exported the snapshot from Dev to Pre Prod using below command: /usr/bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapshot_table_name -copy-to hdfs://NAME_NODE:8020/hbase -mappers 5 My problem is when I try to clone the table from this exported snapshot I get error Unknown Snapshot and also this exported snapshot is not showing up when I hit list_snapshot commad in Hbase shell. I can see the exported snapshot in hdfs using below command as I gave the hdfs path as hdfs://NAME_NODE:8020/hbase: hadoop fs -ls /hbase However all the other snapshots which I created in Pre Prod are present in the below location: hadoop fs -ls /apps/hbase/data How can I resolve this issue? Is there a problem in the way I exported the Snapshot? Do i have to export the snapshot and give the path as hdfs://NAME_NODE:8020/apps/hbase/data ? Request for assistance. ... View more

Hi All, I have a requirement in which we have a table which has about 100 Million records. On this table we want to enable Salting. Inorder to enable the salting I would have to recreate the table which has 100 Million records. Can any one suggest what would be the best way to migrate these 100 million records to a different place, recreate the table with SALTING and put the data back into the SALTED table. Also I am reading in the official documentation that "There are some cautions and difference in behavior you should be aware about when using a salted table." Can anyone guide what could be the possible difference in behavior we can see? ... View more

I have created an INDEX on my tables using below DDL: CREATE INDEX my_index ON my_table (col1, col2, col3 , EVENT DESC) INCLUDE (col4 , col5); However the EVENT column is not sorted in descending order. Is there anything I am missing? Have I created the Index correctly? One thing I wanted to highlight was my EVENT column datatype is VARCHAR....is that making EVENT column not to sort? ... View more

Hi, I have a requirement in which I want the event column to be indexed. However I wanted to know if it is possible to ORDER the event column in Descending order something like below: CREATE INDEX my_index ON my_table (EVENT ORDER BY DESC) INCLUDE (v2); My Base table has EVENT column but it is not ordered, I am thinking if indexing EVENT column in Descending order will have me get away with the ORDERBY clause and reduce query time. Request for comments. ... View more

Hi , I am trying to configure nifi Execute Process to connect to MSSQL and execute the store procedure . I followed this link https://community.hortonworks.com/questions/26170/does-executesql-processor-allow-to-execute-stored.html where in @M. Mashayekhi has provided the steps how he connected to MSSQL and executed the Stored Procedure. @Mashayekhi , I wanted to know how your execute process configuration screen looks like and also please let me know if any additional client tools are required to be installed so that I can execute the stored procedure. Thank you. ... View more

Hi All , I went through the comments in this page https://community.hortonworks.com/questions/26170/does-executesql-processor-allow-to-execute-stored.html and figured out how to execute a store procedure by connecting to MSSQL server. I wanted some pointers as to how I can pull only those records from MSSQL database from the point the last execution had fetched me the data using Nifi. For example, If I execute stored procedure every minute then in the first min I will get 1000 records, by the time procedure got executed 100 records got inserted in the DB ,and when the stored procedure gets executed second time how can I make sure I get only those new 100 records in Nifi? ... View more

Hi All, we have a HBase/Phoenix Table, wherein we have created a covered index in improve read performance. There is a requirement that we create one more covered index on the same HBase/Phoenix Table. Will that cause any performance issues on the table due to two indexes on the same table? ... View more

We have an Phoenix table in which records are inserted by Nifi workflow. Our table and primary key looks something like below: PRIMARY KEY COL1 COL2 COL3 COL4 COL5 COL1 | COL2 | COL3 We faced some issues in Nifi and |COL2|COL3 was inserted as primary keys for all rows without COL1 value. Now our primary key looks like this -> ( |COL2 | COL3) . Is there any upsert statement to update existing primary key with COL1 value so that out primary key become like this --> COL1|COL2|COL3 ? @Sergey Soldatov @Josh Elser ... View more