As per the OP's response if the data isn't well distributed along the partitioned column you will end up having some very large partitions while others will be very small. Writing into a single large partition can lead Kudu to fail.   If your partitioned column is skewed aim for redesigning your table partitioning. Final note: As per Kudu's documentation (Apache Kudu - Apache Kudu Schema Design) Typically the primary key columns are used as the columns to hash, but as with range partitioning, any subset of the primary key columns can be used.   ... View more

Sorry for late response, I use oozie to submit a spark job ... View more

Hi @tomasz  Can you please confirm which graph it is for heap utilization ? any custom query you have used to build this?   ... View more

ls: cannot access '/var/lib/ranger/yarn/': No such file or directory ... View more

thanks i have disabled the dynamic allocation and it was working now. ... View more

Hi @JiteshHirani    You can make a rest API call to provenance to fetch the flowfile attributes if we have provenance data available as per our defined provenence repository storage time and size [1] [1]  https://nifi.apache.org/docs/nifi-docs/rest-api/index.html   ... View more

to resolve problem you need to create folder    sudo mkdir /var/lib/hadoop-yarn/ sudo chmod +077 /var/lib/hadoop-yarn/ sudo chown yarn:hadoop /var/lib/hadoop-yarn/     ... View more

@Ganeshk Yes, we do not have any flag equivalent to -v in beeline. However, if we are looking for the command itself, you could find it in the line stating "Compiling command" or "Executing command" ... View more

 I am getting errors while running a simple MR job    2023-01-02 23:29:42,462 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Downloading public resource: { hdfs://hp8300one:8020/user/yarn/mapreduce/mr-framework/3.0.0-cdh6.3.4-mr-framework.tar.gz, 1672446065301, ARCHIVE, null } 2023-01-02 23:29:42,462 ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Local path for public localization is not found. May be disks failed. org.apache.hadoop.util.DiskChecker$DiskErrorException: No space available in any of the local directories. at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:400) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:152) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:589) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$PublicLocalizer.addResource(ResourceLocalizationService.java:883) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:781) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:723) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126) at java.lang.Thread.run(Thread.java:750) 2023-01-02 23:29:42,462 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_1672729727095_0002_01_000001 2023-01-02 23:29:42,463 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed for container_1672729727095_0002_01_000001 org.apache.hadoop.util.DiskChecker$DiskErrorException: No space available in any of the local directories. at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:400) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:152) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:133) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:117) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:584) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1205) 2023-01-02 23:29:42,463 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1672729727095_0002_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2023-01-02 23:29:42,463 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Container container_1672729727095_0002_01_000001 sent RELEASE event on a resource request { hdfs://hp8300one:8020/user/yarn/mapreduce/mr-framework/3.0.0-cdh6.3.4-mr-framework.tar.gz, 1672446065301, ARCHIVE, null } not present in cache. 2023-01-02 23:29:42,463 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Execution exception when running task in DeletionService #0 2023-01-02 23:29:42,464 WARN org.apache.hadoop.util.concurrent.ExecutorHelper: Caught exception in thread DeletionService #0: java.lang.NullPointerException: path cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:270) at org.apache.hadoop.fs.FileContext.delete(FileContext.java:768) at org.apache.hadoop.yarn.server.nodemanager.containermanager.deletion.task.FileDeletionTask.run(FileDeletionTask.java:109) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-01-02 23:29:42,464 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=sanjay OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1672729727095_0002 CONTAINERID=container_1672729727095_0002_01_000001 ... View more