Hi all, I want to load only new files from the Apache Pig  LOAD statement. I have 25 files in HDFS directory and on other days I upload another few files in the same directory of HDFS. But now I just want to load only those files which are still unloaded (New files) in the Pig. How could I achieve this?   Thanks, Jay. ... View more

HI all, I want to pull data from the SFTP server and load into HDFS. Scenario 1: I configured Agent as in, Source as `SFTP server`, SInk as `HDFS` and channel as `File-Channel`. But in this scenario, HDFS sink creates too many small files. I have searched for that. But I didn't get any specific solution for that. so I thought the second scenario. Scenario 2: In this scenario, I have configured two flume agent. Agent1: Source >>> SFTP Server, Sink >>> file_roll , Channel >>> file-channel Agent2: Source >>> SpoolDir, Sink >>> HDFS, Channel >>> file-channel So first of all, I've configured one flume agent named Agent1 which will load data into Hadoop local from SFTP server. Now here the second step, I have configured the second agent name Agent2 which will load data from Hadoop local and store into HDFS. But whenever I started the second agent the main file renamed as .COMPLETE extension. And I am running twos agent simultaneously. Agent1 will load all data into one file (here rollcountInterval = 0) but Agent2 cannot load data from that particular file. Can anybody help me out for this block? Thanks. ... View more

Hi All, I want to install scala in Linux as in sandbox centOS. I have already downloaded scala-2.10.5 and also create a soft link for it in /us/hdp/current folder. After that, I executed below commands to set PATH for SCALA_HOME. echo "export SCALA_HOME = /usr/hdp/current/scala" or echo "export SCALA_HOME = /usr/hdp/2.5.0.0-1245/scala" echo "export PATH=$PATH:$SCALA_HOME/bin" After that, I jumped at root directory in local and executed below command to execute .bashrc and .bash_profile files. source ~/.bashrc source ~/.bash_profile I thought that is all. But I couldn't able to go into scala shell from anywhere. Can anyone help me with this? Regards, ... View more

Hi, I want to implement flume for trial. I have seen many blogs that Flume implemented on Twitter data only. As Twitter is not accessible in my area. Is there any real-time or streaming source that can be implemented on Flume? Thanks, Jay. ... View more

Hi all, I am new to Druid, I have too many running tasks in Coordinator. So now unable to submit more tasks in Druid. I tried to shutdown task by executing below command; curl -X POST http://xxx-xx-xxx-xxx-x.compute-1.amazonaws.com:8081/druid/indexer/v1/task/<taskID>/shutdown But somewhere I don't know, there is too many tasks are running, and I don't know many are remain to stop. So I want to know, is there any way to delete/shutdown all task at once? Regards, Jay. ... View more

Hi All, I want to perform incremental import using the Sqoop command. I have following data into MySQL table. mysql> select * from data; +------+------+------------+ | id | name | join_date | +------+------+------------+ | 1 | aaa | 2017-11-26 | | 2 | bbb | 2018-03-06 | | 3 | ccc | 2018-04-25 | +------+------+------------+ 3 rows in set (0.02 sec) I have imported these MySQL data into hive table. hive> select * from datahive; OK 1 aaa 2017-11-26 2 bbb 2017-11-26 3 ccc 2018-04-25 Time taken: 3.876 seconds, Fetched: 3 row(s) hive> Now, I am updating one record and inserting a new record in MySQL table for incremental Sqoop import. mysql> update data set name="new_update" where id = 1; mysql> insert into data values (5,"new_insert","2018-06-01"); Sqoop command; sqoop import \ --connect "jdbc:mysql://sandbox.hortonworks.com:3306/sqltempdb" \ --username root \ --password hadoop \ --table data \ --hive-import \ --hive-table datahive \ --fields-terminated-by "," \ -m 1 \ --incremental lastmodified \ --check-column join_date \ --last-value "2017-01-01" By executing the above Sqoop command I am getting the following records; hive> select * from datahive; OK 1 aaa 2017-11-26 2 bbb 2017-11-26 3 ccc 2018-04-25 1 new_update 2017-11-26 2 bbb 2017-11-26 3 ccc 2018-04-25 5 new_insert 2018-06-01 Time taken: 30.438 seconds, Fetched: 7 row(s) hive> But I only want the updated record and newly inserted record into Hive table, I don't want to append all data with existing data. Required Data into Hive: 1 aaa 2017-11-26 2 bbb 2018-03-06 3 ccc 2018-04-25 1 new_update 2017-11-26 5 new_insert 2018-06-01 Kindly help me to reach out this solution. Regards, Jay. ... View more

Hi All, I want to load data into druid from Kafka topic. I have created the specification file in JSON format. I have already fetched the data as CSV format into Kafka broker. Sample data: timestamp,open,high,low,close,volume 2018-07-20 05:08:00,1990.8000,1991.5500,1990.8000,1991.0000,1321 2018-07-20 05:07:00,1991.1000,1991.1500,1990.6000,1991.0500,2387 2018-07-20 05:06:00,1991.0000,1991.3000,1991.0000,1991.1000,1776 2018-07-20 05:05:00,1991.7500,1991.8000,1990.5000,1991.0500,5988 2018-07-20 05:04:00,1991.9500,1992.0000,1991.7500,1991.7500,1646 2018-07-20 05:03:00,1992.0000,1992.0500,1991.8500,1991.9500,3272 Now, I want to push this data into Druid datastore named "stockexchange". supervisor.json [ { "dataSource" : [ { "spec" : { "dataSchema" : { "granularitySpec" : { "queryGranularity" : "none", "type" : "uniform", "segmentGranularity" : "hour" }, "dataSource" : "stockexchange", "parser" : { "type" : "string", "parseSpec" : { "format" : "csv", "timestampSpec" : { "format" : "auto", "column" : "timestamp" }, "columns" : ["timestamp","open","high","low","close","volume"], "dimensionsSpec" : { "dimensions" : ["open","high","low","close","volume"] } } }, }, "ioConfig" : { "type" : "realtime" }, "tuningConfig" : { "type" : "realtime", "intermediatePersistPeriod" : "PT10M", "windowPeriod" : "PT10M", "maxRowsInMemory" : 75000 } }, "properties" : { "task.partitions" : "2", "task.replicants" : "2", "topicPattern" : "stockexchange.*", "topicPattern.priority" : "1" } } ], "properties" : { "zookeeper.connect" : "ip-xxx-xx-xxxx-xx.ec2.internal:2181", "zookeeper.timeout" : "PT20S", "druid.selectors.indexing.serviceName" : "druid/overlord", "druid.discovery.curator.path" : "/druid/discovery", "kafka.zookeeper.connect" : "ip-xxx-xx-xxxx-xx.ec2.internal:2181", "kafka.group.id" : "xxxx-xxxxx-xxxx", "consumer.numThreads" : "2", "commit.periodMillis" : "15000", "reportDropsAsExceptions" : "false" } } ] But, when I post data into Druid datastore using following curl command, it throws following error: Command: curl -X POST -H 'Content-Type: application/json' -d @supervisor.json http://ec2-xxx-xx-xxxx-xxxx.compute-1.amazonaws.com:8090/druid/indexer/v1/supervisor Error: {"error":"Unexpected token (START_OBJECT), expected VALUE_STRING: need JSON String that contains type id (for subtype of io.druid.indexing.overlord.supervisor.SupervisorSpec)\n at [Source: HttpInputOverHTTP@1f0c750b; line: 1, column: 2]"} I have searched for this error, but I didn't find any particular solution. Please kindly help me to solve this error. Regards, Jay. ... View more

Hi All, I want to export CSV data into MsSQL using Sqoop. I have created a table in MsSQL which have one auto-increment column named 'ID'. I have one CSV file in HDFS directory. I have executed below Sqoop export command; Sqoop Command: sqoop export --connect 'jdbc:sqlserver://xxx.xxx.xx.xx:xxxx;databasename=<mssql_database_name>'--username xxxx-passwordxxxx--export-dir /user/root/input/data.csv--table <mssql_table_name> I am facing the following error; Error: 18/07/11 10:30:48 INFO mapreduce.Job: map 0% reduce 0% 18/07/11 10:31:12 INFO mapreduce.Job: map 75% reduce 0% 18/07/11 10:31:13 INFO mapreduce.Job: map 100% reduce 0% 18/07/11 10:31:17 INFO mapreduce.Job: Job job_1531283775339_0005 failed with state FAILED due to: Task failed task_1531283775339_0005_m_000003 Job failed as tasks failed. failedMaps:1 failedReduces:0 18/07/11 10:31:18 INFO mapreduce.Job: Counters: 31 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=163061 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=261 HDFS: Number of bytes written=0 HDFS: Number of read operations=7 HDFS: Number of large read operations=0 HDFS: Number of write operations=0 Job Counters Failed map tasks=3 Launched map tasks=4 Data-local map tasks=4 Total time spent by all maps in occupied slots (ms)=87423 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=87423 Total vcore-milliseconds taken by all map tasks=87423 Total megabyte-milliseconds taken by all map tasks=21855750 Map-Reduce Framework Map input records=0 Map output records=0 Input split bytes=240 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=346 CPU time spent (ms)=470 Physical memory (bytes) snapshot=106921984 Virtual memory (bytes) snapshot=1924980736 Total committed heap usage (bytes)=39321600 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=0 18/07/11 10:31:18 INFO mapreduce.ExportJobBase: Transferred 261 bytes in 71.3537 seconds (3.6578 bytes/sec) 18/07/11 10:31:18 INFO mapreduce.ExportJobBase: Exported 0 records. 18/07/11 10:31:18 ERROR mapreduce.ExportJobBase: Export job failed! 18/07/11 10:31:18 ERROR tool.ExportTool: Error during export: Export job failed! Sample Data: abc,1223 abck,1332 abckp,2113 Regards, Jay. ... View more

Hi, I am new to learning HBase and I am using Hortonworks sandbox on virtual box. I just have open HBase Shell and ran the first simple command to show status but it is giving me following error. hbase(main):010:0> status 'simple' ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2402) at org.apache.hadoop.hbase.master.MasterRpcServices.getClusterStatus(MasterRpcServices.java:778) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:57174) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2127) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at java.lang.Thread.run(Thread.java:745) Here is some help for this command: Show cluster status. Can be 'summary', 'simple', 'detailed', or 'replication'. The default is 'summary'. Examples: hbase> status hbase> status 'simple' hbase> status 'summary' hbase> status 'detailed' hbase> status 'replication' hbase> status 'replication', 'source' hbase> status 'replication', 'sink' hbase(main):011:0> I have checked that HBase Master is running well on Ambari. I have restarted Sandbox as well. But I am facing same error. Regards, Jay. ... View more

Hi All, I have created HDP cluster on AWS. Now I want to execute a spark-submit command using shell action. Spark-submit command is simple, that take input from HDFS and store output in HDFS and .jar file taken from Hadoop local. My spark-submit command is running well on a command line. It can read data and store output on HDFS in a specific directory. And I could also create a script and run on command line, it also worked well. But the problem is while executing oozie workflow for this. script.sh #!/bin/bash /usr/hdp/current/spark2-client/bin/spark-submit --class org.apache.<main> --master local[2] <jar_file_path> <HDFS_input_path> <HDFS_output_path> job.properties nameNode=hdfs://<HOST>:8020 jobTracker=<HOST>:8050 queueName=default oozie.wf.application.path=${nameNode}/user/oozie/shelloozie workflow.xml <workflow-app name="ShellAction" xmlns="uri:oozie:workflow:0.3"> <start to='shell-node' /> <action name="shell-node"> <shell xmlns="uri:oozie:shell-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <exec>script.sh</exec> <file>${nameNode}/user/oozie/shelloozie/script.sh#script.sh</file> </shell> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message>Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name='end' /> </workflow-app> Anyways, I have checked my yarn log, it is giving me following, I didn't get it what it is explaining. LogType:stderr Log Upload Time:Sat Jun 16 07:00:47 +0000 2018 LogLength:1721 Log Contents: Jun 16, 2018 7:00:24 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a provider class Jun 16, 2018 7:00:24 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class Jun 16, 2018 7:00:24 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices as a root resource class Jun 16, 2018 7:00:24 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM' Jun 16, 2018 7:00:24 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton" Jun 16, 2018 7:00:25 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton" Jun 16, 2018 7:00:26 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to GuiceManagedComponentProvider with the scope "PerRequest" log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. End of LogType:stderr Kindly help me to solve this. Thank You, Jay. ... View more

Hi all, I want to create Oozie workflow for Spark Action. Anyways, I have created workflow but I am getting the following error: Main class [org.apache.oozie.action.hadoop.SparkMain], exit code [1] I searched up on the error, I saw that most common cause of this error is oozie sharelib. So I installed all new jars and update sharelib by running following command: su oozie oozie admin -sharelibupdate Ensure that sharelib is installed properly and none none of this has stopped the error occurring. My workflow files are following below; job.properties nameNode=hdfs://sandbox.hortonworks.com:8020 jobTracker=sandbox.hortonworks.com:8050 queueName=default projectRoot=user/root/oozie/sparkoozie master=local[2] mode=cluster class=org.apache.TransformationOper hiveSite=hive-site.xml workflowAppUri=${nameNode}/${projectRoot}/lib/TransformationOper.jar oozie.use.system.libpath=true oozie.action.sharelib.for.spark=spark,hive oozie.wf.application.path=${nameNode}/${projectRoot}/ workflow.xml <workflow-app name="SparkAction" xmlns="uri:oozie:workflow:0.4"> <start to="spark-node"/> <action name="spark-node"> <spark xmlns="uri:oozie:spark-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <prepare> <delete path="${projectRoot}/output"/> </prepare> <job-xml>${nameNode}/${projectRoot}/hive-site.xml</job-xml> <configuration> <property> <name>mapred.compress.map.output</name> <value>true</value> </property> </configuration> <master>${master}</master> <mode>${mode}</mode> <name>Testing Spark Action</name> <class>${class}</class> <jar>${nameNode}/${projectRoot}/lib/TransformationOper.jar</jar> <arg>INPUT=${nameNode}/${projectRoot}/input/error.log</arg> <arg>OUTPUT=${projectRoot}/output</arg> </spark> <ok to="end"/> <error to="error"/> </action> <kill name="error"> <message>Spark Test WF failed. [${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name ="end"/> </workflow-app> command: oozie job -oozie http://127.0.0.1:11000/oozie -config job.properties -run I also checked yarn log yarn logs -applicationId application_1528173243110_0007 following is logerror; LogType:stderr Log Upload Time:Tue Jun 05 16:31:27 +0000 2018 LogLength:2329 Log Contents: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/hadoop/yarn/local/filecache/685/spark-assembly-1.6.2.2.5.0.0-1245-hadoop2.7.3.2.5.0.0-1245.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/hadoop/yarn/local/filecache/25/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Using properties file: null Parsed arguments: master local[2] deployMode cluster executorMemory null executorCores null totalExecutorCores null propertiesFile null driverMemory null driverCores null driverExtraClassPath null driverExtraLibraryPath null driverExtraJavaOptions -Dlog4j.configuration=spark-log4j.properties supervise false queue null numExecutors null files null pyFiles null archives null mainClass org.apache.TransformationOper primaryResource hdfs://sandbox.hortonworks.com:8020/user/root/oozie/sparkoozie/lib/TransformationOper.jar name Testing Spark Action childArgs [INPUT=hdfs://sandbox.hortonworks.com:8020/user/root/oozie/sparkoozie/input/error.log OUTPUT=user/root/oozie/sparkoozie/output] jars null packages null packagesExclusions null repositories null verbose true Spark properties used, including those specified through --conf and those from the properties file null: spark.yarn.security.tokens.hive.enabled -> false spark.executor.extraJavaOptions -> -Dlog4j.configuration=spark-log4j.properties spark.yarn.security.tokens.hbase.enabled -> false spark.driver.extraJavaOptions -> -Dlog4j.configuration=spark-log4j.properties Error: Cluster deploy mode is not compatible with master "local" Run with --help for usage help or --verbose for debug output Intercepting System.exit(1) Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], exit code [1] End of LogType:stderr I went thru it, but I m not getting this what is the exact error. It is very grateful If you could help me to solve this issue. Regards, Jay. ... View more

Hi all , I have install Hortownworks Sandbox in Oracle virtual machine. Now I want to upload files from Windows to HDFS. I have tried following command. C:\InputFileWindows>scp -p 22 datafile.txt root@localhost: But I am not able to do that. It is giving me permission denied error; ... View more