I am using confluent HDFS sink connector and would like to know how to get consumer properties to expose through either JMS or rest api. I checked the following two properties, however, I don't know how to expose matrix to jmx port 1. connect-standalone.properties 2. consumer.properties ... View more

Repo Description I have created a NiFi Utlities for doing following things. Find stopped processor groups Following script finds the processor group in which no processor is running. find_stopped_processor_groups.py Find invalid processor group Following script finds the processor group in which has at least one invalid processor. As even if with single invalid processor you can't run that processor group. find_invalid_processor_groups.py Find duplicate processor group Following script finds the duplicate processor group based in it's name. find_duplicate_processor_groups.py Find controller services Following script finds all the database controller services used by NiFi find_duplicate_processor_groups.py Instrunction about how to install and run is on repo page. Repo Info Github Repo URL https://github.com/Gaurang033/nifapi Github account name Gaurang033 Repo name nifapi ... View more

Hi Guys, I am trying to check the service status with Ambari rest API, However I am not able to find any documention which explain thing in great details. For example, If I hit following REST URL http://localhost:8080/api/v1/clusters/Sandbox/services/HDFS I get following output "maintenance_state": "OFF", "repository_state": "CURRENT", "service_name": "HDFS", "state": "STARTED" }, "alerts_summary": { "CRITICAL": 0, "MAINTENANCE": 8, "OK": 293, "UNKNOWN": 4, "WARNING": 0 }, However, I am not sure how to interpret this things. Should I care about MAINTENANCE, UNKNOWN and WARNING and just checking that nothing CRITICAL is good enough. This is mainly for developers to understand and track the time how much any service is down. ... View more

hi Guys, I am building a small app to see the status of each service under Ambari. And I am using following API to get that information. http://localhost:8080/api/v1/clusters/Sandbox/services/FLUME Would someone please let me know the possible values for each service. For now I know it's either (STARTED, or INSTALLED) ... View more

Hi Guys. Is there anyway to get all the connection ids from Nifi? I know how to get the connection ids from a particular processor group, however that't not what I want. We have a multiple processor group and each group of hierarchical processor group and they are connected using output and input port. I want to clean all the flowfile from all these connection and so want to get the ids. ... View more

I am trying to understand the hive query plan for a simple distinct query and I have small confusion regrading output of one of the stage. I have simple table with just two column, id and value. and just 4 rows as mentioned below. Data: <code>hive> select * from temp.test_distinct; OK 1 100 2 100 3 100 4 150 Plan <code>hive> explain select distinct value from temp.test_distinct; OK Plan not optimized by CBO. Vertex dependency in root stage Reducer 2 <- Map 1 (SIMPLE_EDGE) Stage-0 Fetch Operator limit:-1 Stage-1 Reducer 2 File Output Operator [FS_6] compressed:false Statistics:Num rows: 2 Data size: 10 Basic stats: COMPLETE Column stats: NONE table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"} Group By Operator [GBY_4] | keys:KEY._col0 (type: string) | outputColumnNames:["_col0"] | Statistics:Num rows: 2 Data size: 10 Basic stats: COMPLETE Column stats: NONE |<-Map 1 [SIMPLE_EDGE] Reduce Output Operator [RS_3] key expressions:_col0 (type: string) Map-reduce partition columns:_col0 (type: string) sort order:+ Statistics:Num rows: 4 Data size: 20 Basic stats: COMPLETE Column stats: NONE Group By Operator [GBY_2] keys:value (type: string) outputColumnNames:["_col0"] Statistics:Num rows: 4 Data size: 20 Basic stats: COMPLETE Column stats: NONE Select Operator [SEL_1] outputColumnNames:["value"] Statistics:Num rows: 4 Data size: 20 Basic stats: COMPLETE Column stats: NONE TableScan [TS_0] alias:test_distinct Statistics:Num rows: 4 Data size: 20 Basic stats: COMPLETE Column stats: NONE Time taken: 0.181 seconds, Fetched: 35 row(s) Confusion: TableScan, Select Operator and Group By Operator shows that they processed 4 rows which make sense to me. But shouldn't the next stage after Group By Operator get only 2 rows the process. As group by will remove other rows. In my DAG I can see the output of mapper is just two rows and not four, however that doesn't see to match with Plan. May i looking it wrong? ... View more

hi Guys, I have a hive job which is running without any issue and I am trying to convert that to spark job to see if I am able to get any performance improvement or not. And I would like to know how to do apple to apple comparison, I mean how do I compare other parameters apart from time? (i.e CPU, Memeory and others) ... View more

on the multiple different blog posts I have found that one mapper task is created per block. However when I tried to to check that, I don't see it happening. I have 100 files each of 385 Bytes only however occupies the whole block. How do I know that? [root@sandbox-hdp ~]# hdfs fsck /apps/hive/warehouse/temp.db/emp_orc_small_files/000000_0_copy_11 -files -blocks Connecting to namenode via http://sandbox-hdp.hortonworks.com:50070/fsck?ugi=root&files=1&blocks=1&path=%2Fapps%2Fhive%2Fwarehouse%2Ftemp.db%2Femp_orc_small_files%2F000000_0_copy_11 FSCK started by root (auth:SIMPLE) from /172.17.0.2 for path /apps/hive/warehouse/temp.db/emp_orc_small_files/000000_0_copy_11 at Thu Sep 13 02:58:47 UTC 2018 /apps/hive/warehouse/temp.db/emp_orc_small_files/000000_0_copy_11 385 bytes, 1 block(s): OK 0. BP-32082187-172.17.0.2-1517480669419:blk_1073745283_4489 len=385 repl=1 However when I run the query to calculate max value of a particular column, I see only 1 mapper and 1 reducer task is being created? Shouldn't there be 100 mapper tasks being created? ... View more

I am trying to prepare and give exam for HDPCD, and I have few question as mentioned below, would be nice if someone can answer that. Approximately how many questions ares being asked? Is there distribution of question among sqoop, flume, pig and hive. it says I will have limited internet connection, what does that mean? how many question I need to answer right in order to pass the exam. Would I be able to run(execute) the queries I form for any given task. How difficult the question are compared to what they have provided in practice exam. What is the purpose of the exam, is it more about the technology or the logic? what I mean to ask is, are the question for pig and hive need to implement complicated logic? ... View more

I am trying to connect to hive metastore to my HDP sandbox. However it's throwing following error. Hive Catalogonnector.name=hive-hadoop2 hive.metastore.uri=thrift://sandbox-hdp.hortonworks.com:9083 hive.metastore.authentication.type=NONE Error: Query 20180810_005352_00000_umgac failed: Failed connecting to Hive metastore: [sandbox-hdp.hortonworks.com:9083] I tried using following values for hive.metastore.uri however I am gettting the same error. thrift://localhost:9083 thrift://127.0.0.1:9083 thrift://<IP of docker container>:9083 thrift://<IP of local machine>:9083 ... View more

am trying to connect hbase using JAVA HBase Rest Client. However it's giving following error. The Hbase Rest uses kerberos authentication, and so I have created a kerberos ticket and trying to authenticate using this ticket. code: public class RestExample { public static void main(String[] args) throws IOException { Configuration conf = HBaseConfiguration.create(); UserGroupInformation.setConfiguration(conf); String projectDir = System.getProperty("user.dir"); System.out.println(projectDir); UserGroupInformation.loginUserFromKeytab("gaurang.shah@mydomain.com", projectDir+"/gaurang.shah.keytab"); // vv RestExample Cluster cluster = new Cluster(); cluster.add("hbase_host.mydomain.com", 17000); Client client = new Client(cluster); TableName tableName = TableName.valueOf("bda:aaa"); RemoteAdmin remoteAdmin = new RemoteAdmin(client, conf); HTableDescriptor tableDesc = new HTableDescriptor(tableName); remoteAdmin.createTable(tableDesc); } } StackTrace: Exception in thread "main" java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: http://hbase_host.mydomain.com:17000/bda:aaa/schema?user.name=gaurang.shah, status: 403, message: Forbidden at org.apache.hadoop.hbase.rest.client.Client.negotiate(Client.java:285) at org.apache.hadoop.hbase.rest.client.Client.executeURI(Client.java:239) at org.apache.hadoop.hbase.rest.client.Client.executePathOnly(Client.java:204) at org.apache.hadoop.hbase.rest.client.Client.execute(Client.java:265) at org.apache.hadoop.hbase.rest.client.Client.put(Client.java:557) at org.apache.hadoop.hbase.rest.client.Client.put(Client.java:504) at org.apache.hadoop.hbase.rest.client.Client.put(Client.java:474) at org.apache.hadoop.hbase.rest.client.RemoteAdmin.createTable(RemoteAdmin.java:294) at ca.cantire.RestExample.main(RestExample.java:42) Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: http://hbase_host.mydomain.com:17000/bda:aaa/schema?user.name=gaurang.shah, status: 403, message: Forbidden at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:281) at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:212) at org.apache.hadoop.hbase.rest.client.Client.negotiate(Client.java:280) ... 8 more ... View more

hi Guys. I am trying to load data files into my hive table and facing issue. if files are located on local it doesn't work. if I move the file to hdfs then it works without any issue. Following command is not working in beeline, however it works perfectly in hive load data local inpath '/home/gaurang.shah/test.json' into table temp.test; Data is located on the node where one of the instance of hiverserver2 is running. I have given it all the permission as well. [gaurang.shah@aa ~] pwd /home/gaurang.shah [gaurang.shah@aa ~]$ ll test.json -rwxrwxrwx 1 gaurang.shah domain users 56 Jul 11 13:54 test.json ... View more

hi Guys, I am thinking to use hbase reset api to interact with hbase reset server. would someone please let me know if there is a rest client available for hbase rest server. I found following two for python, if someone has used any of this would you please share the exeperince. https://github.com/barseghyanartur/starbase https://github.com/wbolster/happybase ... View more

hi Guys, Could someone let me know how they decide which hive-exec jar version would be compatible with their Environment. Here are the two approches I have take, the first one is failing and second one passes. However second approach is really messy and I would like to have first approach work somehow. First Approach - Not Working use the maven project to compile and build single (fat) jar. check which version of hive-exec.jar hadoop is using from following directory /usr/hdp/current/hive-client/lib/hive-exec.jar -> hive-exec-1.2.1000.2.6.4.0-91.jar use the matching version as maven dependancy <dependency> <groupId>org.apache.hive</groupId> <artifactId>hive-exec</artifactId> <version>1.2.1</version> </dependency> Second Approach - Working Create the simple JAVA project (not the maven) copy the JAR from hadoop and add in classpath compile and create the class files Create a thin Jar (without any dependancies), Provide class path in mainfest.mf file Class-path: /usr/hdp/current/hive-client/lib/hive-exec.jar ... View more

I am checking how to create a custom UDF in Hive. I have created a custom UDF which does nothing, just returns the given text as it is. I am able to load this JAR in hive without any issue I am also able to create function from this jar and I am able to execute this function. Problem if this Jar is added I can't execute load table/hdfs from another table. following simple query fails. insert into demo1 select * from demo; Stacktrace: Vertex failed, vertexName=Map 1, vertexId=vertex_1523501275422_0010_3_00, diagnostics=[Task failed, taskId=task_1523501275422_0010_3_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Container container_1523501275422_0010_01_000007 finished with diagnostics set to [Container completed. ]], TaskAttempt 1 failed, info=[Container container_1523501275422_0010_01_000008 finished with diagnostics set to [Container completed. ]], TaskAttempt 2 failed, info=[Container container_1523501275422_0010_01_000009 finished with diagnostics set to [Container completed. ]], TaskAttempt 3 failed, info=[Container container_1523501275422_0010_01_000010 finished with diagnostics set to [Container completed. ]]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1523501275422_0010_3_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE] DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0 <br> Please note that both the table has same structure. If I remove the jar then I can execute above query without any issue. Code: ca.abc.demo public class demo extends UDF { public String evaluate(String s) { if (s == null) { return null; }else{ return s; } } }<br> pom.xml <?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>ca.cantire</groupId> <artifactId>hive-normalize</artifactId> <version>1.0</version> <dependencies><!-- https://mvnrepository.com/artifact/org.apache.hive/hive-exec --><dependency> <groupId>org.apache.hive</groupId> <artifactId>hive-exec</artifactId> <version>2.3.3</version> </dependency> </dependencies> <build> <pluginManagement> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.8</version> </plugin> <plugin> <artifactId>maven-assembly-plugin</artifactId> <configuration> <archive> <manifest> <mainClass>ca.cantire.demo</mainClass> </manifest> </archive> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> </configuration> </plugin> </plugins> </pluginManagement> </build></project> ... View more

hi Guys, Is there any way to Normalize the Hive UTF-8 data. I am talking about NFC format. Currently I have written custom UDF that does that, however wanted to know if there is any better easier way that does for whole table/hdfs file. ... View more

Hi Guys, I have written a custom UDF function which works fine if I run that in select query. However, if I try to export data to HDFS using select query as custom UDF is part of that query, it fails. Following query gets executed successfully. CREATE temporary FUNCTION normalize as 'ca.test.Normalize' USING JAR 'hdfs://hadoopdev/tmp/udf/normalizer-1.0-jar-with-dependencies.jar'; select normalize(test_desc, 'NFC') from temp.test_special_char; Following query Fails CREATE temporary FUNCTION normalize as 'ca.test.Normalize' USING JAR 'hdfs://hadoopdev/tmp/udf/normalizer-1.0-jar-with-dependencies.jar'; INSERT OVERWRITE DIRECTORY '/tmp/test_special_char/' ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' NULL DEFINED AS '' select normalize(test_desc, 'NFC') from temp.test_special_char; StackTrace INFO : Tez session hasn't been created yet. Opening session ERROR : Failed to execute tez graph. org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1519224124029_102976 failed 2 times due to AM Container for appattempt_1519224124029_102976_000002 exited with exitCode: 255 For more detailed output, check the application tracking page: http://abc:8088/cluster/app/application_1519224124029_102976 Then click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_e71_1519224124029_102976_02_000001 Exit code: 255 Stack trace: org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException: Launch container failed at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DefaultLinuxContainerRuntime.launchContainer(DefaultLinuxContainerRuntime.java:109) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:89) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:392) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:317) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:83) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Shell output: main : command provided 1 main : run as user is gaurang.shah main : requested yarn user is gaurang.shah Getting exit code file... Creating script paths... Writing pid file... Writing to tmp file /data/10/hadoop/yarn/local/nmPrivate/application_1519224124029_102976/container_e71_1519224124029_102976_02_000001/container_e71_1519224124029_102976_02_000001.pid.tmp Writing to cgroup task files... Creating local dirs... Launching container... Getting exit code file... Creating script paths... Container exited with a non-zero exit code 255 Failing this attempt. Failing the application. at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:779) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:217) at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:272) at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:152) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1151) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197) at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76) at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:253) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:264) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1) ... View more

I am trying to use Sqoop Export to Export Data from HDFS to Netezza table. And I am facing issue with few of french characters. I have table with just only one column and one row as mentioned below. En fil métallique résistant mâchouillag I am able to see this text without any issue in Hive Table and corresponding HDFS directory as well. I am also able to insert this same line in Netezza table without any issue, so I know my Netezza database handles these character. Problem: When I use sqoop to export these data to Netezza table, I get following row as compared to mentioned above. En fil meÌtallique résistant maÌchouillag It seems like there is any issue with the Netezza JDBC driver I am using to export, I should specify somewhere the encoding or something like that. However, I am not sure. Could someone please let me know how to fix this issue. ... View more

I am trying to export the data from HDFS to Netezza and few french characters are giving me trouble. The only related post I found on internet is following. http://grokbase.com/t/sqoop/user/137gtanzx8/sqoop-utf-8-data-load-issue However the problem is I am not sure which configuration file he is talking about, would someone please let me know in which configuration file I need to provide, Connection Encoding? ... View more