Member since
06-16-2016
43
Posts
21
Kudos Received
0
Solutions
01-15-2019
08:20 PM
Do you mean what is the use/need ? If so, for anyone who is a platform admin, if they would like to know how many hive tables are there in all the hive DBs put together, this query can help them answer. This was asked by a customer in one of the calls we had and thought it would help others with similar request. There is also a similar thread as this, feel free to refer this.
... View more
01-12-2019
12:40 AM
The number can be found in the database which is used to store hive meta data. If MySql is the backend db, select count(*) from TBLS;
... View more
- Find more articles tagged with:
- Hadoop Core
- Hive
- How-ToTutorial
Labels:
11-09-2018
03:20 AM
2 Kudos
Hadoop archives is one of the methodology which is followed to reduce the load on the Namenode by archiving the files and referring all the archives as a single file via har reader. Testing:
To understand the behavior of the HAR, we try following example.
1. Create test folders
harSourceFolder2 : Where the initial set of small files are stored. Ex. (In HDFS ) /tmp/harSourceFolder2
harDestinationFolder2 : Where the final archived files are stored. Ex. (In HDFS) /tmp/harDestinationFolder2
2. Ingest small files in the source folder. sudo -u hdfs hadoop fs -copyFromLocal /tmp/SampleTest1.txt /tmp/harSourceFolder2 NOTE: thid command shows one file "SampleTest1", however in our example we used five files with index extending till 5 (SampleTest5.txt) 3. Capture fsck report across the "/" and NN report after small files are ingested. sudo -u hdfs hdfs fsck / -files > ./fsckWhenFilesCreated.txt 143 files and directories, 48 blocks = 191 total filesystem object(s). 4. Execute hadoop archive commands sudo -u hdfs hadoop archive -archiveName hartest2.har -p /tmp harSourceFolder2 /tmp/harDestinationFolder2 5. Capture fsck report across the "/" and NN report after after hadoop archives are created. sudo -u hdfs hdfs fsck / -files > ./fsckAfterHARCreated.txt 156 files and directories, 55 blocks = 211 total filesystem object(s). 6. Compare the Namenode report and fsck report. 143 files and directories, 48 blocks = 191 total filesystem object(s).
156 files and directories, 55 blocks = 211 total filesystem object(s).
Analysis: Upon analyzing the fsck reports that were captured (fsckWhenFilesCreated amd fsckAfterHARCreated) we see that there are multiple files and blocks that are created. In this case, 13 files and folders and 7 blocks. Which can be explained with following output. /app-logs/hdfs/logs-ifile/application_1541612686625_0001 <dir>
/app-logs/hdfs/logs-ifile/application_1541612686625_0001/c3187-node3.squadron-labs.com_45454 17656 bytes, 1 block(s): OK
/app-logs/hdfs/logs-ifile/application_1541612686625_0001/c3187-node4.squadron-labs.com_45454 6895 bytes, 1 block(s): OK
/mr-history/done/2018/11 <dir>
/mr-history/done/2018/11/07 <dir>
/mr-history/done/2018/11/07/000000 <dir>
/mr-history/done/2018/11/07/000000/job_1541612686625_0001-1541618133969-hdfs-hadoop%2Darchives%2D2.7.3.2.6.5.0%2D292.jar-1541618159397-1-1-SUCCEEDED-default-1541618141722.jhist 33597 bytes, 1 block(s): OK
/mr-history/done/2018/11/07/000000/job_1541612686625_0001_conf.xml 149808 bytes, 1 block(s): OK
/tmp/harDestinationFolder2/hartest2.har <dir>
/tmp/harDestinationFolder2/hartest2.har/_SUCCESS 0 bytes, 0 block(s): OK
/tmp/harDestinationFolder2/hartest2.har/_index 619 bytes, 1 block(s): OK
/tmp/harDestinationFolder2/hartest2.har/_masterindex 23 bytes, 1 block(s): OK
/tmp/harDestinationFolder2/hartest2.har/part-0 120 bytes, 1 block(s): OK
Above list comprises of the the new 13 files/folders that are added. Except for the "harDestinationFolder2/hartest2.har" and its content, rest of the data are temporary which are triggered as a result of the MapReduce job that is triggered as a result of hadoop archive command shown above. Also, we see that there are seven occurrences of "1 block(s):" in the above output which contributes to the total block increase. Out of these, three are permanent and rest are temporary. Also, at this point of time, the source small files can be deleted as there is a new archive for these files. Since, there are constant number of blocks (_index, _masterindex, part-0) that are created for each archives, it would be worthy to consider archiving large number of small files instead for small datasets, which can have negative effect. It can also be noted that in the fsck report executed after creating the archive file, we do not see the source files(SampleTest[1-5].txt) inside the directory "hartest2.har" which could be seen when we list it via a regular "hadoop fs -lsr har:" command. This shows that HDFS does not consider the initial source files once it is archived via HAR. This helps to answer that even though source text files could be seen, they do not add to the load on the Namenode. hadoop fs -lsr har:///tmp/harDestinationFolder2/hartest2.har
lsr: DEPRECATED: Please use 'ls -R' instead.
drwxr-xr-x - hdfs hdfs 0 2018-11-07 18:49 har:///tmp/harDestinationFolder2/hartest2.har/harSourceFolder2
-rw-r--r-- 3 hdfs hdfs 24 2018-11-07 18:48 har:///tmp/harDestinationFolder2/hartest2.har/harSourceFolder2/SampleTest1.txt
-rw-r--r-- 3 hdfs hdfs 24 2018-11-07 18:48 har:///tmp/harDestinationFolder2/hartest2.har/harSourceFolder2/SampleTest2.txt
-rw-r--r-- 3 hdfs hdfs 24 2018-11-07 18:48 har:///tmp/harDestinationFolder2/hartest2.har/harSourceFolder2/SampleTest3.txt
-rw-r--r-- 3 hdfs hdfs 24 2018-11-07 18:48 har:///tmp/harDestinationFolder2/hartest2.har/harSourceFolder2/SampleTest4.txt
-rw-r--r-- 3 hdfs hdfs 24 2018-11-07 18:49 har:///tmp/harDestinationFolder2/hartest2.har/harSourceFolder2/SampleTest5.txt
... View more
- Find more articles tagged with:
- archive
- FAQ
- hadoop
- Hadoop Core
- har
Labels:
10-29-2018
06:13 PM
1 Kudo
The idea of this article is to help admins detect artifacts(files/folders) in the cluster that are older than certain days. Also, in certain cases, there may be empty directories that are lying in the cluster which are not used and hence contribute to the small file issue. Hence, we have the attached script which performs,
1. Identifies files older than X days. 2. Identifies folders older than X days.
3. Deletes empty folders. Script Execution Script name is "findAll.sh" which expects 2 parameters which is 1. Age of the artifact (file/folder) in terms of days.
2. Actual location of the artifact (file/folder) in HDFS.
Based upon the type of artifact and kind of operation you would have to choose one of the three operations. NOTE:
1. Please make sure the user running the script has permissions to execute the command on the artifacts that is passed as parameter to script.
2. Also, running this script once may take some time based upon the size/hierarchy of the folders. But once the list is procured, you can act upon it as per need. Hence, I would recommend you to test the script in lower ENV and run it in PROD when the load on HDFS is less. 3. Please exercise caution on the folders on which you run the scripts. Example executions: Execution 1: To list the old folders:
[hive@c2187-node2 tmp]$ ./findAll.sh 9 /tmp/hive/hive
Please select your option
1. Identify folders/directories that are older than 9 days
2. Identify files that are older than 9 days
3. Delete empty folders
1
Please check the output in ./OldFolders-202054.txt ;
Execution 2: To list the old files:
[hive@c2187-node2 tmp]$ ./findAll.sh 9 /tmp/hive/hive
Please select your option
1. Identify folders/directories that are older than 9 days
2. Identify files that are older than 9 days
3. Delete empty folders
2
Please check the output in file ./Oldfiles.txt-202148
Execution 3 : To delete empty folders
[hive@c2187-node2 tmp]$ ./findAll.sh 9 /tmp/hive/hive
Please select your option
1. Identify folders/directories that are older than 9 days
2. Identify files that are older than 9 days
3. Delete empty folders
3
rmdir: `/tmp/hive/hive/_tez_session_dir': Directory is not empty
Please feel free to tweak and extend the functionalities of the script.findall.tar.gz
... View more
- Find more articles tagged with:
- delete
- help
- How-ToTutorial
- oldata
- older
- small-files
- solutions
09-24-2018
06:07 PM
Great article @Dinesh Chitlangia Just to add to the details mentioned in the blog here, the property ("upgrade.parameter.nn-restart.timeout") change during the upgrade may not take effect even if Ambari servers are restarted. So have the property in place before the upgrade starts.
... View more
08-30-2018
12:27 AM
2 Kudos
JSTACK and JMAP Collection Jstack Collection Step 1: Switch as the service user that started the process. #su - <service-user-who-started-the-process> Step 2: Capture the process ID #ps -ef | grep <process-name> #ps -ef | grep hive
hive 21887 1 0 Aug01 ? 00:58:04 /usr/jdk64/jdk1.8.0_112/bin/java -Xmx1024m -Dhdp.version=2.6.5.0-292 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.5.0-292 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.5.0-292/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.5.0-292/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.5.0-292/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.5.0-292/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx2048m -Djava.util.logging.config.file=/usr/hdp/current/hive-server2/conf/conf.server/parquet-logging.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.6.5.0-292/hive/lib/hive-service-1.2.1000.2.6.5.0-292.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris= -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive From above output, Parent service account is hive Process ID is 21887 Java version used is /usr/jdk64/jdk1.8.0_112/bin/java Step 3: Capture the java used by the process to start the service. From the above output it is /usr/jdk64/jdk1.8.0_112/bin/java Step 4: (In the order of priority) NOTE: We would need to consider running the command multiple times (min 5 times ) separated by 20-30 seconds. 4.1: Simple jstack for a responding process #<jstack-used-by-process>/jstack -l <pid> > <location-to-redirect-the-output>/jstack.out 4.2: Use kill for a hung process #kill -3 <pid> Corresponding output is captured in .out file of the process. 4.3: Use -F for a hung process #<jstack-used-by-process>/jstack -F <pid> > <location-to-redirect-the-output>/jstack.out JMap Collection Step 1: #su - <service-user-who-started-the-process> Step 2: Capture the process ID Step 3: Capture the java used by the process to start the service. Step 4: Determining the appropriate flag to be used, We use "-heap" option to determine if it is needed to use "-dump" option. #<jmap-used-by-process>/jmap -heap <pid> > jmapHEAP.out Upon multiple executions of the above command, if the percentage used is above 90% then we use the -dump flag as below, Sample output of above command is, Attaching to process ID 21887, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.112-b15
using thread-local object allocation.
Parallel GC with 8 thread(s)
Heap Configuration:
MinHeapFreeRatio = 0
MaxHeapFreeRatio = 100
MaxHeapSize = 2147483648 (2048.0MB)
NewSize = 87031808 (83.0MB)
MaxNewSize = 715653120 (682.5MB)
OldSize = 175112192 (167.0MB)
NewRatio = 2
SurvivorRatio = 8
MetaspaceSize = 21807104 (20.796875MB)
CompressedClassSpaceSize = 1073741824 (1024.0MB)
MaxMetaspaceSize = 17592186044415 MB
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 141557760 (135.0MB)
used = 36859416 (35.151878356933594MB)
free = 104698344 (99.8481216430664MB)
26.038428412543404% used
From Space:
capacity = 5767168 (5.5MB)
used = 4211840 (4.0167236328125MB)
free = 1555328 (1.4832763671875MB)
73.0313387784091% used
To Space:
capacity = 5767168 (5.5MB)
used = 0 (0.0MB)
free = 5767168 (5.5MB)
0.0% used
PS Old Generation
capacity = 277872640 (265.0MB)
used = 161075720 (153.61377716064453MB)
free = 116796920 (111.38622283935547MB)
57.9674630794885% used
From the above output, 57% of heap is being used. The two general flags that are used while collecting the heapdumps are “-dump” and “-histo”. While former gives the heapdump in the form of binary file with the collection of objects at a particular time, latter provides the details of live objects in a text format. #<jmap-used-by-process>/jmap -dump:file=<location-to-redirect-the-output>/heapdump.hprof,format=b <PID> If histo label needs to be used, #<jmap-used-by-process>/jmap -histo <pid> > jmap.out NOTE: 1. Jmap/Jstack is high CPU intensive process, so please use it with caution. 2. Please try not to use -F as much as possible as critical data are missed with this option. If -F option needs to be used by any of the commands, Example: #/usr/jdk64/jdk1.8.0_112/bin/jmap -dump:file=/tmp/jmap21887.hprof,format=b -F 21887 #/usr/jdk64/jdk1.8.0_112/bin/jmap -histo -F 21887 > /tmp/jmaphistoF.out Thanks @Rajkumar Singh @Vinod Bonthu and @Kevin Wong for reviewing.
... View more
- Find more articles tagged with:
- heap
- heapdump
- How-ToTutorial
- jmap
- jstack
- Sandbox & Learning
- thread
- threaddump
05-17-2018
08:22 PM
@slachterman I am trying to set the execution mode in Ambari but even with LLAP enabled and the daemons running, I am not able to set the engine to LLAP mode. Currently, would we have to set the execution engine on a beeline session level ? and not in Ambari. I am using 2.6.1.0
... View more
05-08-2018
02:16 AM
5 Kudos
As an extension to the article mentioned here we are using custom Ambari alerts to monitor the current state of the Journal Node edits health.
With the default monitoring that is present in the Ambari, we would not be alerted about the failure of edits that may happen in the one of the JN quorum. In typical HDFS HA env, there are three Journal node daemons that are deployed. If any one of the daemons fails to maintain the edits, then we are at risk of failovers and eventual cluster outage if another journal node hits similar issue as other journal node (Because, if quorum of edits are not maintained, then NN fails to be up). Hence, we need to have necessary alerting mechanism put in place for such failures. Journal Nodes may not get updated due to various reasons such as, 1. Disk getting full.
2. Corrupt Permissions.
3. Exhausted HDFS handlers in JN host, etc..
Attached are the artifacts, which contains, 1. alerts-test.json
2. jn_edits_tracker.py
jn_edits_tracker.py have preconfigured values, OK_CEIL = 9
WARN_FLOOR = 10
WARN_CEIL = 19
CRITICAL_FLOOR = 20
Which defines the corresponding time ranges in seconds for alerts to be triggered. This would alert in Ambari, if the "edits_inprogress" file is not updated for above configured time interval. Steps to configure the alert 1. Copy the jn_edits_tracker.py to /var/lib/ambari-server/resources/host_scripts 2. Now restart the Ambari-Server. 3. Run the following command to list all the existing alerts: curl -u admin:admin -i -H 'X-Requested-By:ambari' -X GET http://node1.example.com:8080/api/v1/clusters/ClusterDemo/alert_definitions 4. Install the custom alert using Curl command as following: curl -u admin:admin -i -H 'X-Requested-By:ambari' -X POST -d @alerts-test.json http://node1.example.com:8080/api/v1/clusters/ClusterDemo/alert_definitions Attachments : jneditsarchive.zip
... View more
- Find more articles tagged with:
- Ambari
- HDFS
- How-ToTutorial
- journalnode
- Sandbox & Learning
Labels:
04-12-2018
06:54 AM
@Neil Tu By disabling it, znode creation and cleanup will not be performed and for the fact that registry being disabled, the load in parsing the ZK hierarchy is relieved.
... View more
04-03-2018
07:45 PM
@Saikiran Parepally Please accept the answer if it has helped you.
... View more
03-08-2018
09:05 PM
1 Kudo
A Hive Database can contain both transactional and non transactional table. Hence, if we are doing some quick checks to determine if the table is ACID enabled, please run the following command. # hive -e "describe extended <Database>.<tablename>;" | grep "transactional=true" If you get an output with the string that you grep for, then the table is transactional. Example: #hive -e "describe extended default.hello_acid;" | grep "transactional=true"
... View more
- Find more articles tagged with:
- acid
- Hive
- How-ToTutorial
- Metastore
- Sandbox & Learning
- transactional
Labels:
01-10-2018
08:14 PM
1 Kudo
In large clusters, where there are multiple services which makes use of single Zookeeper quorum, the state store is maintained as znodes. Hence the count of such znodes are directly proportional to the services that are deployed and also the activity on the cluster.
If LLAP apps are deployed in such clusters, it is imperative that slider is enabled (by setting the property, "hadoop.registry.rm.enabled"), this will introduce an overhead in the Znode scans for all the application containers that are created and destroyed on timely basis. The behavior of the scans are as described below, If the property is set in core-site.xml or yarn-site.xml, the YARN Resource Manager will behave as follows: 1. On startup: create the initial root paths of /, /services and /users. On a secure cluster, access will be restricted to the system accounts (see below). 2. When a user submits a job: create the user path under /users. 3. When a container is completed: delete from the registry all service records with a yarn:persistence field of value container, and a yarn:id field whose value matches the ID of the completed container. 4. When an application attempt is completed: remove all service records with yarn:persistence set to application-attempt and yarn:id set to the pplication attempt ID. 5. When an application finishes: remove all service records with yarn:persistence set to application and yarn:id set to the application ID. Ref: Registry scan Hence, this leads to registry scan across all the znodes irrespective of rmservice znode. Meaning, even if there are few thousand (<10K) of applications in /rmstore (/rmstore-secure), the scan would be from root level (/). If the count of znodes under root exceeds 10k limit, this leads to registry scan and hence the connectivity issues between ZK and RM which leads to timeout and hence RM failover and hence its stability. This is addressed in this Apache JIRA. ROOT CAUSE: https://issues.apache.org/jira/browse/YARN-6136 RESOLUTION: To implement change in the ZK scan behavior. WORKAROUND: 1. If LLAP (slider) is not used: Disable, hadoop.registry.rm.enabled 2. If LLAP (slider) is used: i) Assume only LLAP uses slider, if nobody else is using the same ZK cluster, the only way to reduce ZK load is lower yarn.resourcemanager.state-store.max-completed-applications to 3k ii) If other services use ZK quorum, please reach out to HWX support.
... View more
Labels:
12-18-2017
09:21 PM
Labels:
- Labels:
-
Apache Ambari
-
Apache Ranger
09-14-2017
09:28 PM
It is expected in large clusters where node count ranges to few hundreds, the master services tend to be busy. One such master service is Namenode. Some of the critical activities that NN does includes, 1. Addressing client requests which includes verifying proper permissions, auth checks for HDFS resources. 2. Constant block report monitoring from all the Datanodes. 3. Updating the service and audit logs. are to name a few. In certain situations when there are rogue applications which tries to access multiple resources in HDFS or a data ingestion that is trying to load high data volumes, NN tends to be very busy. In such situations and cluster like these NN FSImage tends to be in $$GB. Hence, operations such as checkpointing would consume considerable bandwidth across the two Namenodes. Hence, high volume of edits sync along with loggings would cause high disk utilization which can lead to NameNode instability. Hence, it is recommended to have dedicated disks for service logs and edit logs. We can monitor the IO on the disks using `iostat` output.
... View more
- Find more articles tagged with:
- Design & Architecture
- disk
- disk-usage
- FAQ
- HDFS
Labels:
08-08-2017
12:49 AM
3 Kudos
This article tries to compare the data recovery period of accidentally deleted data in HDFS. We would compare two scenarios,
1. When trash is enabled.
2. When snapshot is enabled.
Data Recovery from trash:
When a data from HDFS is deleted, metadata in HDFS is updated to delete the file from the source folder. However, the blocks from the datanode is not immediately deleted. The trash folder in HDFS is updated with the file along with the directory from where it is deleted in the user's .trash folder. The deleted data could be recovered from the trash folder.
Example: 1. Existing data in HDFS. #hadoop fs -ls /tmp/test1.txt
-rw-r--r-- 3 hdfs hdfs 4 2017-08-07 23:47 /tmp/test1.txt
2. Deleted data in HDFS. #hadoop fs -rm /tmp/test1.txt
17/08/07 23:52:13 INFO fs.TrashPolicyDefault: Moved: 'hdfs://vnn/tmp/test1.txt' to trash at: hdfs://vnn/user/hdfs/.Trash/Current/tmp/test1.txt
3. Recovering a deleted data #hadoop fs -cp /user/hdfs/.Trash/Current/tmp/test1.txt /tmp/
#hadoop fs -ls /tmp/test1.txt
-rw-r--r-- 3 hdfs hdfs 4 2017-08-07 23:57 /tmp/test1.txt
Data recovery from snapshots: Snapshots are read-only point in time copies of HDFS file system. Enable a directory to be snapshot-able to recovery any accidental data loss. 1. Enabling snapshot. #hdfs dfsadmin -allowSnapshot /tmp/snapshotdir
Allowing snaphot on /tmp/snapshotdir succeeded
2. Create snapshot for a directory. #hdfs dfs -createSnapshot /tmp/snapshotdir
Created snapshot /tmp/snapshotdir/.snapshot/s20170807-180139.568 3. Contents of HDFS snapshot based folder. #hdfs dfs -ls /tmp/snapshotdir/
Found 3 items
hadoop fs -rm $1
-rw-r--r-- 3 hdfs hdfs 1083492818 2017-07-31 19:01 /tmp/snapshotdir/oneGB.csv
-rw-r--r-- 3 hdfs hdfs 10722068505 2017-08-02 17:19 /tmp/snapshotdir/tenGB.csv
#hdfs dfs -ls /tmp/snapshotdir/.snapshot/s20170807-180139.568
Found 3 items
-rw-r--r-- 3 hdfs hdfs 1083492818 2017-07-31 19:01 /tmp/snapshotdir/.snapshot/s20170807-180139.568/oneGB.csv
-rw-r--r-- 3 hdfs hdfs 10722068505 2017-08-02 17:19 /tmp/snapshotdir/.snapshot/s20170807-180139.568/tenGB.csv
4. Delete and recovering lost data. #hadoop fs -rm /tmp/snapshotdir/oneGB.csv
17/08/07 19:37:46 INFO fs.TrashPolicyDefault: Moved: 'hdfs://vinodnn/tmp/snapshotdir/oneGB.csv' to trash at: hdfs://vinodnn/user/hdfs/.Trash/Current/tmp/snapshotdir/oneGB.csv1502134666492
#hadoop fs -cp /tmp/snapshotdir/.snapshot/s20170807-180139.568/oneGB.csv /tmp/snapshotdir/
It is seen in the above methods that hadoop copy "hadoop fs -cp <source> <dest>" is used to recover the data. However, the time taken by "cp" operation would increase as the size of the lost data increases. One of the optimizations would be to use the move command, "hadoop fs -mv <source> <destination>" in place of copy operation, as former operation fairs better over latter. Since, snapshot folders are read-only, the only supported operation is "copy" ( but not move ). Following are the metrics that are used to compare the performance of "copy" operation over "move" for one GB and ten GB data file. Time to recover a file using copy (cp) operations: screen-shot-2017-08-07-at-60552-pm.png Time to recover a file using move (mv) operations: screen-shot-2017-08-07-at-60602-pm.png Hence, we observe that recovery of data using trash along with move operation is efficient in certain cases to tackle accidental data loss and recovery. NOTE: Recovering the data from trash would be possible if trash interval (fs.trash.interval) are properly configured to give Hadoop admins enough time to detect the data loss and recover it. If not, snapshot would be recommended for eventual recovery.
... View more
- Find more articles tagged with:
- Cloud & Operations
- delete
- HDFS
- help
- How-ToTutorial
- recovery
- snapshot
Labels:
07-18-2017
06:54 PM
I had to remove following properties as well, oozie.service.ELService.ext.functions.workflow
oozie.service.ELService.ext.functions.coord-sla-create On the whole, had to make sure there are no oozie.extensions and ELService Functions that were used in the oozie-site.xml. Thank you @Kuldeep Kulkarni and @Orlando Teixeira
... View more
05-26-2017
10:50 PM
1 Kudo
I am trying to 1. Enable HDFS ranger plugin. 2. Add an additional HDFS service in Ranger I am following the documentation, https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+0.5+-+User+Guide which suggests to add the service ("Add Service" under "Service Manager" using "+" next to HDFS). I am trying to add the basic entries that are needed to bring up the service and have a successful "Test Connection". Following are the values for the fields that I am entering, Service Name : ranger1_hadoop
Username : admin
Password : admin
Namenode URL : hdfs://<hostname -f>:8020
Authentication Type : Simple
Test connection was failing with the below error, Connection Failed.
Unable to retrieve any files using given parameters, You can still save the repository and start creating policies, but you would not be able to use autocomplete for resource names. Check ranger_admin.log for more info. Observation: 1. There is no file by the name "ranger_admin.log" in my Ranger hosts as specified by the above logs. Is this expected? 2. In xa_portal.log, I see the following stack trace, 2017-05-26 22:38:29,578 [timed-executor-pool-0] INFO apache.ranger.services.hdfs.client.HdfsClient (HdfsClient.java:208) - ===> HdfsClient.testConnection()
2017-05-26 22:38:29,579 [timed-executor-pool-0] ERROR org.apache.ranger.plugin.util.PasswordUtils (PasswordUtils.java:127) - Unable to decrypt password due to error
javax.crypto.IllegalBlockSizeException: Input length must be multiple of 8 when decrypting with padded cipher
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:913)
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:824)
at com.sun.crypto.provider.PBES1Core.doFinal(PBES1Core.java:416)
at com.sun.crypto.provider.PBEWithMD5AndDESCipher.engineDoFinal(PBEWithMD5AndDESCipher.java:316)
at javax.crypto.Cipher.doFinal(Cipher.java:2165)
at org.apache.ranger.plugin.util.PasswordUtils.decryptPassword(PasswordUtils.java:112)
at org.apache.ranger.plugin.client.BaseClient.login(BaseClient.java:113)
at org.apache.ranger.plugin.client.BaseClient.<init>(BaseClient.java:59)
at org.apache.ranger.services.hdfs.client.HdfsClient.<init>(HdfsClient.java:52)
at org.apache.ranger.services.hdfs.client.HdfsClient.connectionTest(HdfsClient.java:221)
at org.apache.ranger.services.hdfs.client.HdfsResourceMgr.connectionTest(HdfsResourceMgr.java:47)
at org.apache.ranger.services.hdfs.RangerServiceHdfs.validateConfig(RangerServiceHdfs.java:58)
at org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:560)
at org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:547)
at org.apache.ranger.biz.ServiceMgr$TimedCallable.call(ServiceMgr.java:508)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-05-26 22:38:29,580 [timed-executor-pool-0] ERROR apache.ranger.services.hdfs.client.HdfsResourceMgr (HdfsResourceMgr.java:49) - <== HdfsResourceMgr.testConnection Error: Unable to login to Hadoop environment [ranger1_hadoop]
org.apache.ranger.plugin.client.HadoopException: Unable to login to Hadoop environment [ranger1_hadoop]
at org.apache.ranger.plugin.client.BaseClient.login(BaseClient.java:136)
at org.apache.ranger.plugin.client.BaseClient.<init>(BaseClient.java:59)
at org.apache.ranger.services.hdfs.client.HdfsClient.<init>(HdfsClient.java:52)
at org.apache.ranger.services.hdfs.client.HdfsClient.connectionTest(HdfsClient.java:221)
at org.apache.ranger.services.hdfs.client.HdfsResourceMgr.connectionTest(HdfsResourceMgr.java:47)
at org.apache.ranger.services.hdfs.RangerServiceHdfs.validateConfig(RangerServiceHdfs.java:58)
at org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:560)
at org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:547)
at org.apache.ranger.biz.ServiceMgr$TimedCallable.call(ServiceMgr.java:508)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Unable to decrypt password due to error
at org.apache.ranger.plugin.util.PasswordUtils.decryptPassword(PasswordUtils.java:128)
at org.apache.ranger.plugin.client.BaseClient.login(BaseClient.java:113)
... 12 more
Caused by: javax.crypto.IllegalBlockSizeException: Input length must be multiple of 8 when decrypting with padded cipher
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:913)
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:824)
at com.sun.crypto.provider.PBES1Core.doFinal(PBES1Core.java:416)
at com.sun.crypto.provider.PBEWithMD5AndDESCipher.engineDoFinal(PBEWithMD5AndDESCipher.java:316)
at javax.crypto.Cipher.doFinal(Cipher.java:2165)
at org.apache.ranger.plugin.util.PasswordUtils.decryptPassword(PasswordUtils.java:112)
... 13 more
2017-05-26 22:38:29,580 [timed-executor-pool-0] ERROR org.apache.ranger.services.hdfs.RangerServiceHdfs (RangerServiceHdfs.java:60) - <== RangerServiceHdfs.validateConfig Error: Unable to login to Hadoop environment [ranger1_hadoop]
org.apache.ranger.plugin.client.HadoopException: Unable to login to Hadoop environment [ranger1_hadoop]
at org.apache.ranger.plugin.client.BaseClient.login(BaseClient.java:136)
at org.apache.ranger.plugin.client.BaseClient.<init>(BaseClient.java:59)
at org.apache.ranger.services.hdfs.client.HdfsClient.<init>(HdfsClient.java:52)
at org.apache.ranger.services.hdfs.client.HdfsClient.connectionTest(HdfsClient.java:221)
at org.apache.ranger.services.hdfs.client.HdfsResourceMgr.connectionTest(HdfsResourceMgr.java:47)
at org.apache.ranger.services.hdfs.RangerServiceHdfs.validateConfig(RangerServiceHdfs.java:58)
at org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:560)
at org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:547)
at org.apache.ranger.biz.ServiceMgr$TimedCallable.call(ServiceMgr.java:508)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Unable to decrypt password due to error
at org.apache.ranger.plugin.util.PasswordUtils.decryptPassword(PasswordUtils.java:128)
at org.apache.ranger.plugin.client.BaseClient.login(BaseClient.java:113)
... 12 more
Caused by: javax.crypto.IllegalBlockSizeException: Input length must be multiple of 8 when decrypting with padded cipher
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:913)
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:824)
at com.sun.crypto.provider.PBES1Core.doFinal(PBES1Core.java:416)
at com.sun.crypto.provider.PBEWithMD5AndDESCipher.engineDoFinal(PBEWithMD5AndDESCipher.java:316)
at javax.crypto.Cipher.doFinal(Cipher.java:2165)
at org.apache.ranger.plugin.util.PasswordUtils.decryptPassword(PasswordUtils.java:112)
... 13 more
2017-05-26 22:38:29,580 [timed-executor-pool-0] ERROR org.apache.ranger.biz.ServiceMgr$TimedCallable (ServiceMgr.java:510) - TimedCallable.call: Error:org.apache.ranger.plugin.client.HadoopException: Unable to login to Hadoop environment [ranger1_hadoop]
2017-05-26 22:38:29,580 [http-bio-6080-exec-7] ERROR org.apache.ranger.biz.ServiceMgr (ServiceMgr.java:188) - ==> ServiceMgr.validateConfig Error:org.apache.ranger.plugin.client.HadoopException: org.apache.ranger.plugin.client.HadoopException: Unable to login to Hadoop environment [ranger1_hadoop]
After enabling the HDFS plugin in HDFS service section of Ambari, there is a service that is created by the name "Ranger_hadoop" in Ranger UI. However, I am not able to add another HDFS service.
... View more
Labels:
- Labels:
-
Apache Ranger
05-26-2017
12:58 AM
1 Kudo
ISSUE: While configuring NFS mounts to access HDFS as a part of local FS, we do tend to control the access using nfs proxies as shown below, <property>
<name>hadoop.proxyuser.nfsserver.groups</name>
<value>nfs-users1,nfs-users2</value>
<description>
The 'nfsserver' user is allowed to proxy all members of the
'nfs-users1' and 'nfs-users2' groups. Set this to '*' to allow
nfsserver user to proxy any group.
</description>
</property>
<property>
<name>hadoop.proxyuser.nfsserver.hosts</name>
<value>nfs-client-host1.com</value>
<description>
This is the host where the nfs gateway is running. Set this to
'*' to allow requests from any hosts to be proxied.
</description>
</property> However, a user who has access to NFS server would be able to access (view) the HDFS file system even if they are not part of "hadoop.proxyuser.nfsserver.groups" and "hadoop.proxyuser.nfsserver.hosts" . This may be a security flaw in certain scenarios, ROOT CAUSE: This is due to a property, "nfs.exports.allowed.hosts" which is used to allow the access to the HDFS from the hosts. RESOLUTION: Make sure the desired hosts and permissions are assigned to HDFS. Permissions for the property can be defined as below, <property>
<name>nfs.exports.allowed.hosts</name>
<value>* rw</value>
</property> NOTE: NFS gateway restart may be needed if the property is altered Links: https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html#Allow_mounts_from_unprivileged_clients
... View more
- Find more articles tagged with:
- authorization
- FAQ
- HDFS
- nfs
- nfsgateway
- Security
Labels:
05-25-2017
09:06 PM
2 Kudos
What is the recommended Livy heap size settings for optimal performance. Is it dependent on any particular factors? Do we have any formula to calculate the heap settings?
... View more
Labels:
- Labels:
-
Apache Zeppelin
05-24-2017
04:36 PM
@fschneider There was no specific reason. In fact, I searched in for CBD AMI in AWS and I picked the latest date that was mentioned in the image name. Yes, I missed the latest images that were mentioned in the documentation
... View more
05-24-2017
06:37 AM
@rdoktorics I do have the property UAA_DEFAULT_SECRET mentioned in the Profile. Even after multiple restarts, it did not mitigate the issue. However, since it was a trial cluster that I was playing around with, I tried to follow the second way that you mentioned. Incidentally, that (second method) was mentioned by @fschneider this morning which helped me get pass the issue. Also, it was told that this process can lead to loss of configuration data which has to be followed with caution.
... View more
05-23-2017
04:15 PM
Hi @rdoktorics export PUBLIC_IP=<IP.add.res.ss> export UAA_DEFAULT_SECRET='<password>' export UAA_DEFAULT_USER_PW='<password>' Where '<password>' is something like 'badbad' So, while entering the credentials, it is like, username: admin@example.com password: badbad I also checked uaa.yml and '/home/cloudbreak/.hdc/config' which has same credentials. -Krishna
... View more
05-22-2017
10:48 PM
I am trying to deploy a cluster using Cloudbreak. While, bringing up the CBD and trying to login to the cloudbreak UI, I am getting the error "Incorrect email/password or account is disabled." The logs from cbd during the time when I tried to login are as below
identity_1 | [2017-05-22 21:57:42.960] cloudfoundry-identity-server - ???? [http-nio-8080-exec-6] .... DEBUG --- ChainedAuthenticationManager: Attempting chained authentication of org.springframework.security.authentication.UsernamePasswordAuthenticationToken@33b826f9: Principal: admin@example.com; Credentials: [PROTECTED]; Authenticated: false; Details: remoteAddress=172.17.0.1, clientId=sultans; Not granted any authorities with manager:org.cloudfoundry.identity.uaa.authentication.manager.CheckIdpEnabledAuthenticationManager@31c68309 required:null
identity_1 | [2017-05-22 21:57:42.962] cloudfoundry-identity-server - ???? [http-nio-8080-exec-6] .... DEBUG --- AuthzAuthenticationManager: Processing authentication request for admin@example.com
identity_1 | [2017-05-22 21:57:43.105] cloudfoundry-identity-server - ???? [http-nio-8080-exec-6] .... DEBUG --- AuthzAuthenticationManager: Password did not match for user admin@example.com
identity_1 | [2017-05-22 21:57:43.106] cloudfoundry-identity-server - ???? [http-nio-8080-exec-6] .... INFO --- Audit: UserAuthenticationFailure ('admin@example.com'): principal=b8ac52a4-8c11-4bae-8063-421e9e73d380, origin=[remoteAddress=172.17.0.1, clientId=sultans], identityZoneId=[uaa]
identity_1 | [2017-05-22 21:57:43.110] cloudfoundry-identity-server - ???? [http-nio-8080-exec-6] .... INFO --- Audit: PrincipalAuthenticationFailure ('null'): principal=admin@example.com, origin=[172.17.0.1], identityZoneId=[uaa]
identity_1 | [2017-05-22 21:57:43.112] cloudfoundry-identity-server - ???? [http-nio-8080-exec-6] .... DEBUG --- ChainedAuthenticationManager: Chained authentication exception:Bad credentials at:org.cloudfoundry.identity.uaa.authentication.manager.AuthzAuthenticationManager.authenticate(AuthzAuthenticationManager.java:137)
identity_1 | [2017-05-22 21:57:43.112] cloudfoundry-identity-server - ???? [http-nio-8080-exec-6] .... DEBUG --- ChainedAuthenticationManager: Chained Authentication status of org.springframework.security.authentication.UsernamePasswordAuthenticationToken@33b826f9: Principal: admin@example.com; Credentials: [PROTECTED]; Authenticated: false; Details: remoteAddress=172.17.0.1, clientId=sultans; Not granted any authorities with manager:org.cloudfoundry.identity.uaa.authentication.manager.ChainedAuthenticationManager$AuthenticationManagerConfiguration@2e5cfbef; Authenticated:false
I observe that in the logs there credential mismatch issue that is being reported. However, the credentials that I am using are verified from the files, 1. Profile
2. /home/cloudbreak/.hdc/config
3. uaa.yml I used the cloudbreak deployer from Amazon AMI, AMI ID - cloudbreak-deployer-130-2016-06-15 (ami-019b5c6c) and updated to Cloudbreak 1.14.4 using the instructions provided here, http://sequenceiq.com/cloudbreak-docs/latest/update/#update-cloudbreak-deployer Some useful INFO: 1. cbd doctor: cbd doctor
===> Deployer doctor: Checks your environment, and reports a diagnose.
uname: Linux ip-10-144-186-250 4.4.8-20.46.amzn1.x86_64 #1 SMP Wed Apr 27 19:28:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
local version:1.14.4
latest release:1.14.4
docker images:
hortonworks/haveged:1.1.0
hortonworks/socat:1.0.0
hortonworks/cloudbreak-uaa-db:v3.6.0
hortonworks/cloudbreak-uaa:3.6.0
hortonworks/cloudbreak-server-db:1.2.0
hortonworks/cloudbreak:1.14.4
hortonworks/cloudbreak-auth:1.14.4
hortonworks/cloudbreak-web:1.14.4
hortonworks/cloudbreak-autoscale-db:1.2.0
hortonworks/cloudbreak-autoscale:1.14.4
docker command exists: OK
docker client version: 1.12.6
docker client version: 1.12.6
ping 8.8.8.8 on host: OK
ping github.com on host: OK
ping 8.8.8.8 in container: OK
ping github.com in container: OK
2. cbd ps: Name Command State Ports
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
cbreak_cbdb_1 /start postgres Up 172.17.0.1:5432->5432/tcp
cbreak_cloudbreak_1 /start_cloudbreak_app.sh bash Up 0.0.0.0:8080->8080/tcp
cbreak_consul_1 /bin/consul agent -server ... Up 8300/tcp, 8301/tcp, 8301/udp, 8302/tcp,
8302/udp, 0.0.0.0:8400->8400/tcp,
0.0.0.0:8500->8500/tcp, 8600/tcp,
172.17.0.1:53->8600/udp
cbreak_haveged_1 haveged -F Up
cbreak_identity_1 /tmp/run.sh Up 0.0.0.0:8089->8080/tcp
cbreak_logsink_1 socat -u TCP-LISTEN:3333,r ... Up 0.0.0.0:32773->3333/tcp
cbreak_logspout_1 /bin/sh -c sleep 1; ROUTE_ ... Up 0.0.0.0:8000->80/tcp
cbreak_mail_1 /bin/sh -c /opt/install.sh ... Up 172.17.0.1:25->25/tcp
cbreak_pcdb_1 /start postgres Up 172.17.0.1:5433->5432/tcp
cbreak_periscope_1 /start_periscope_app.sh Up 0.0.0.0:8085->8080/tcp
cbreak_registrator_1 /bin/registrator consul:// ... Up
cbreak_sultans_1 /sultans/start-docker.sh Up 0.0.0.0:3001->3000/tcp, 3001/tcp
cbreak_traefik_1 /traefik --debug --web --d ... Up 0.0.0.0:443->443/tcp, 0.0.0.0:80->80/tcp,
0.0.0.0:8081->8080/tcp
cbreak_uaadb_1 /start postgres Up 172.17.0.1:5434->5432/tcp
cbreak_uluwatu_1 /uluwatu/start-docker.sh Up 0.0.0.0:3000->3000/tcp
I also tried to sign up using my personal email id with a password of my own. However, I did not see any confirmation being sent ( to my email account ) as cloudbreak mentioned. Is there a possibility to add new users using "sign up" feature in Cloudbreak UI. Thank you, Krishna
... View more
Labels:
- Labels:
-
Hortonworks Cloudbreak
05-22-2017
06:52 PM
1 Kudo
Apart from checking the topologies from a Storm WebUI, we can also list the active topologies from one of the cluster nodes, We can use the following command to list the same, /usr/hdp/<HDP-version>/storm/bin/storm list If there are no topologies running, we would get an output as follows, No topologies running.
... View more
- Find more articles tagged with:
- Data Ingestion & Streaming
- How-ToTutorial
- Storm
- topologies
- topology
Labels:
04-13-2017
06:18 PM
I see that there are two ways to setup Cloudbreak Deployer, 1. Using AWS cloud image. 2. Installing manually on VM. I would like to know which method is suggested for setting up Production clusters. Are there any known caveats following any specific approach. Thanks for your time.
... View more
Labels:
- Labels:
-
Hortonworks Cloudbreak
04-07-2017
10:54 PM
1 Kudo
ISSUE: Java Heap Space issue in Hive MR engine
While working on a sample data set in hive. Query such as "select count(*)" was seen to fail with below error. Starting Job = job_1491603076412_0001, Tracking URL = http://krishna3.openstacklocal:8088/proxy/application_1491603076412_0001/
Kill Command = /usr/hdp/2.4.2.0-258/hadoop/bin/hadoop job -kill job_1491603076412_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2017-04-07 22:18:09,736 Stage-1 map = 0%, reduce = 0%
2017-04-07 22:18:46,065 Stage-1 map = 100%, reduce = 100%
Ended Job = job_1491603076412_0001 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1491603076412_0001_m_000000 (and more) from job job_1491603076412_0001
Task with the most failures(4):
-----
Task ID:
task_1491603076412_0001_m_000000
URL:
http://krishna3.openstacklocal:8088/taskdetails.jsp?jobid=job_1491603076412_0001&tipid=task_1491603076412_0001_m_000000
-----
Diagnostic Messages for this Task:
Error: Java heap space
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
Checking the corresponding application logs, we observe that 2017-04-07 22:25:40,828 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:986)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:402)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:442)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) ROOT CAUSE: Insufficient HeapSpace in MR Engine for mapreduce.map.memory.mb RESOLUTION: Increasing mapreduce.map.memory.mb from 1.2G to 1.75G and hence increasing the mapreduce.task.io.sort.mb to 1003 mapreduce.map.java.opts to -Xmx1433m and restarting the necessary services did resolve the problem. (NOTE: mapreduce.task.io.sort.mb and mapreduce.map.java.opts value recommendations were made by Ambari )
... View more
- Find more articles tagged with:
- Data Processing
- heap
- Issue Resolution
- Mapreduce
- outofmemory
- Upgrade to HDP 2.5.3 : ConcurrentModificationException When Executing Insert Overwrite : Hive
Labels:
03-30-2017
09:51 PM
@mqureshi thank you for the reply. Yes, "mapreduce.cluster.temp.dir" seems to be legacy property which is no longer available in 2.x.
... View more
03-30-2017
05:13 PM
From, https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml I understand that "mapreduce.cluster.local.dir" is defined to store intermediate data from mapreduce operation. But, the description of "mapreduce.cluster.temp.dir" was minimal to understand what kind of temporary data is written in there. Would it be cache and temp jars sort of data? Thank you for your time.
... View more
Labels:
- Labels:
-
Apache Hadoop
03-30-2017
06:52 AM
ISSUE: FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
kafka.common.KafkaException: Failed to acquire lock on file .lock in /opt/kafka/log. A Kafka instance in another process or thread is using this directory.
at kafka.log.LogManager$anonfun$lockLogDirs$1.apply(LogManager.scala:98)
at kafka.log.LogManager$anonfun$lockLogDirs$1.apply(LogManager.scala:95)
at scala.collection.TraversableLike$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.log.LogManager.lockLogDirs(LogManager.scala:95)
at kafka.log.LogManager.<init>(LogManager.scala:57)
at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:589)
at kafka.server.KafkaServer.startup(KafkaServer.scala:171)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala) [2017-03-30 06:32:06,504] INFO shutting down (kafka.server.KafkaServer) RESOLUTION: move .lock file from /opt/kafka/log to a temporary location and restart the kafka broker on the node.
... View more
- Find more articles tagged with:
- broker
- Data Ingestion & Streaming
- fail
- Issue Resolution
- Kafka
- restart
Labels: