Member since
08-14-2017
9
Posts
0
Kudos Received
0
Solutions
03-29-2019
06:26 PM
We are using HDP 3.0, while we are loading .xlsx file to spark data frame, strangely string type of column was taken as number in data frame. giving the exception as java.text.ParseException: Unparseable number: "TMF Study 331-102-00088 Contributor" . the code we used is df_load_temp = spark.read.format("com.crealytics.spark.excel").option("treatEmptyValuesAsNulls", "true") \ .option("location",file_name) \ .option("useHeader", "true") \ .option("inferSchema", "true") \ .option("addColorColumns", "False").load() Hope any body help as early as possible. thanks in advance....
... View more
Labels:
12-04-2017
09:11 AM
currently we are using hdinsight, in spark streaming we are unable to create the checkpoint for the directory hdfs://mycluster/*/* , but instead of hdfs prefix we are able to create the checkpoint directory in default hdi container. kindky help us how to create checkpoint directory as hdfs://mycluster/*/* .
... View more
Labels:
11-28-2017
05:51 AM
Hello All, we need to bring down the hive log level to ERROR follow are the files related to hive log properties, . in which file we need to change INFO to ERROR find / -name "hive-*.properties" /etc/hive/2.5.0.0-1245/0/hive-log4j.properties
/etc/hive/2.5.0.0-1245/0/hive-exec-log4j.properties
/etc/hive/2.5.0.0-1245/0/conf.server/hive-log4j.properties
/etc/hive/2.5.0.0-1245/0/conf.server/hive-exec-log4j.properties
/etc/hive/conf.backup/hive-log4j.properties /etc/hive/conf.backup/hive-exec-log4j.properties
/usr/hdp/2.5.0.0-1245/etc/hive/conf.dist/hive-log4j.properties /usr/hdp/2.5.0.0-1245/etc/hive/conf.dist/hive-exec-log4j.properties this we want to automate through power shell, kindly guide us how to do this.
... View more
Labels:
09-06-2017
10:03 AM
thanks we will try to open the logs by adding FQDN to hosts file
... View more
09-04-2017
10:52 AM
currently we are using Ambari, we are unable to open the logs of yarn from oozie workflows. kindly help us how to resolve this issue
... View more
Labels:
08-22-2017
07:33 AM
i am unable to understand the difference between drop and replace in alter table of hive, can any body give the info what it mean exactly. ( even though drop exists why people are using replace to drop the column)
... View more
Labels:
08-17-2017
05:09 PM
yes, after altering the column and after set the flag(parquet.column.index.access) in hive shell i am getting the error, if the flag set to false, all values in the changed column are nulls.
... View more
08-17-2017
10:32 AM
how to rename the parquet format table columns i have set the flag parquet.column.index.access=false but when i want to retrieve data using "select * from tablename" , i am getting the error, Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.UnsupportedOperationException: Cannot inspect org.apache.hadoop.io.IntWritable
... View more
Labels:
08-14-2017
12:38 PM
how to create the external parquet partitioned table from the existing partitioned table and how to load the data from existing table to newly created table, log is syaing Container exited with a non-zero exit code 1 ], containerId=container_1502450195556_0170_01_000024, nodeId=sandbox.hortonworks.com:45454, nodeHttpAddress=sandbox.hortonworks.com:8042, counters=Counters: 0 2017-08-14 10:29:59,675 [INFO] [Dispatcher thread {Central}] |impl.TaskImpl|: Scheduling new attempt for task: task_1502450195556_0170_6_00_000000, currentFailedAttempts: 3, maxFailedAttempts: 4 2017-08-14 10:29:59,676 [INFO] [Dispatcher thread {Central}] |impl.VertexImpl|: Source task attempt completed for vertex: vertex_1502450195556_0170_6_01 [Reducer 2] attempt: attempt_1502450195556_0170_6_00_000000_2 with state: FAILED vertexState: RUNNING 2017-08-14 10:29:59,676 [INFO] [Dispatcher thread {Central}] |util.RackResolver|: Resolved sandbox.hortonworks.com to /default-rack 2017-08-14 10:29:59,676 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Ignoring removal of unknown task: attempt_1502450195556_0170_6_00_000000_2 2017-08-14 10:29:59,678 [INFO] [TaskSchedulerEventHandlerThread] |rm.TaskSchedulerEventHandler|: Task: attempt_1502450195556_0170_6_00_000000_2 has no container assignment in the scheduler 2017-08-14 10:29:59,678 [ERROR] [TaskSchedulerEventHandlerThread] |rm.TaskSchedulerEventHandler|: No container allocated to task: attempt_1502450195556_0170_6_00_000000_2 according to scheduler. Task reported container id: container_1502450195556_0170_01_000024 2017-08-14 10:29:59,678 [INFO] [TaskSchedulerEventHandlerThread] |rm.YarnTaskSchedulerService|: Allocation request for task: attempt_1502450195556_0170_6_00_000000_3 with request: Capability[]Priority[1] host: null rack: null 2017-08-14 10:30:00,692 [INFO] [AMRM Callback Handler Thread] |util.RackResolver|: Resolved sandbox.hortonworks.com to /default-rack 2017-08-14 10:30:00,692 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: Assigning container to task: containerId=container_1502450195556_0170_01_000025, task=attempt_1502450195556_0170_6_00_000000_3, containerHost=sandbox.hortonworks.com:45454, containerPriority= 1, containerResources=, localityMatchType=NodeLocal, matchedLocation=sandbox.hortonworks.com, honorLocalityFlags=true, reusedContainer=false, delayedContainers=0 2017-08-14 10:30:00,692 [INFO] [DelayedContainerManager] |util.RackResolver|: Resolved sandbox.hortonworks.com to /default-rack 2017-08-14 10:30:00,696 [INFO] [ContainerLauncher #3] |launcher.ContainerLauncherImpl|: Launching container_1502450195556_0170_01_000025 2017-08-14 10:30:00,696 [INFO] [ContainerLauncher #3] |impl.ContainerManagementProtocolProxy|: Opening proxy : sandbox.hortonworks.com:45454 2017-08-14 10:30:00,717 [INFO] [ContainerLauncher #3] |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1502450195556_0170_01_000025, launchTime=1502706600717 2017-08-14 10:30:00,944 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated container completed:container_1502450195556_0170_01_000025 last allocated to task: attempt_1502450195556_0170_6_00_000000_3 2017-08-14 10:30:00,946 [INFO] [Dispatcher thread {Central}] |container.AMContainerImpl|: Container container_1502450195556_0170_01_000025 exited with diagnostics set to Container failed, exitCode=1. Exception from container-launch. Container id: container_1502450195556_0170_01_000025
... View more
- Tags:
- Data Processing
- Hive
Labels: