We are using HDP 3.0, while we are loading .xlsx file to spark data frame, strangely string type of column was taken as number in data frame. giving the exception as java.text.ParseException: Unparseable number: "TMF Study 331-102-00088 Contributor" . the code we used is df_load_temp = spark.read.format("com.crealytics.spark.excel").option("treatEmptyValuesAsNulls", "true") \ .option("location",file_name) \ .option("useHeader", "true") \ .option("inferSchema", "true") \ .option("addColorColumns", "False").load() Hope any body help as early as possible. thanks in advance.... ... View more

currently we are using hdinsight, in spark streaming we are unable to create the checkpoint for the directory hdfs://mycluster/*/* , but instead of hdfs prefix we are able to create the checkpoint directory in default hdi container. kindky help us how to create checkpoint directory as hdfs://mycluster/*/* . ... View more

currently we are using Ambari, we are unable to open the logs of yarn from oozie workflows. kindly help us how to resolve this issue ... View more

i am unable to understand the difference between drop and replace in alter table of hive, can any body give the info what it mean exactly. ( even though drop exists why people are using replace to drop the column) ... View more

how to rename the parquet format table columns i have set the flag parquet.column.index.access=false but when i want to retrieve data using "select * from tablename" , i am getting the error, Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.UnsupportedOperationException: Cannot inspect org.apache.hadoop.io.IntWritable ... View more

how to create the external parquet partitioned table from the existing partitioned table and how to load the data from existing table to newly created table, log is syaing Container exited with a non-zero exit code 1 ], containerId=container_1502450195556_0170_01_000024, nodeId=sandbox.hortonworks.com:45454, nodeHttpAddress=sandbox.hortonworks.com:8042, counters=Counters: 0 2017-08-14 10:29:59,675 [INFO] [Dispatcher thread {Central}] |impl.TaskImpl|: Scheduling new attempt for task: task_1502450195556_0170_6_00_000000, currentFailedAttempts: 3, maxFailedAttempts: 4 2017-08-14 10:29:59,676 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Source task attempt completed for vertex: vertex_1502450195556_0170_6_01 [Reducer 2] attempt: attempt_1502450195556_0170_6_00_000000_2 with state: FAILED vertexState: RUNNING 2017-08-14 10:29:59,676 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved sandbox.hortonworks.com to /default-rack 2017-08-14 10:29:59,676 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Ignoring removal of unknown task: attempt_1502450195556_0170_6_00_000000_2 2017-08-14 10:29:59,678 [INFO] [TaskSchedulerEventHandlerThread] |rm.TaskSchedulerEventHandler|: Task: attempt_1502450195556_0170_6_00_000000_2 has no container assignment in the scheduler 2017-08-14 10:29:59,678 [ERROR] [TaskSchedulerEventHandlerThread] |rm.TaskSchedulerEventHandler|: No container allocated to task: attempt_1502450195556_0170_6_00_000000_2 according to scheduler. Task reported container id: container_1502450195556_0170_01_000024 2017-08-14 10:29:59,678 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Allocation request for task: attempt_1502450195556_0170_6_00_000000_3 with request: Capability[]Priority[1] host: null rack: null 2017-08-14 10:30:00,692 [INFO] [AMRM Callback Handler Thread] |util.RackResolver|: Resolved sandbox.hortonworks.com to /default-rack 2017-08-14 10:30:00,692 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: Assigning container to task: containerId=container_1502450195556_0170_01_000025, task=attempt_1502450195556_0170_6_00_000000_3, containerHost=sandbox.hortonworks.com:45454, containerPriority= 1, containerResources=, localityMatchType=NodeLocal, matchedLocation=sandbox.hortonworks.com, honorLocalityFlags=true, reusedContainer=false, delayedContainers=0 2017-08-14 10:30:00,692 [INFO] [DelayedContainerManager] |util.RackResolver|: Resolved sandbox.hortonworks.com to /default-rack 2017-08-14 10:30:00,696 [INFO] [ContainerLauncher #3] |launcher.ContainerLauncherImpl|: Launching container_1502450195556_0170_01_000025 2017-08-14 10:30:00,696 [INFO] [ContainerLauncher #3] |impl.ContainerManagementProtocolProxy|: Opening proxy : sandbox.hortonworks.com:45454 2017-08-14 10:30:00,717 [INFO] [ContainerLauncher #3] |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1502450195556_0170_01_000025, launchTime=1502706600717 2017-08-14 10:30:00,944 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated container completed:container_1502450195556_0170_01_000025 last allocated to task: attempt_1502450195556_0170_6_00_000000_3 2017-08-14 10:30:00,946 [INFO] [Dispatcher thread {Central}] |container.AMContainerImpl|: Container container_1502450195556_0170_01_000025 exited with diagnostics set to Container failed, exitCode=1. Exception from container-launch. Container id: container_1502450195556_0170_01_000025 ... View more