Member since
01-18-2016
169
Posts
32
Kudos Received
21
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1627 | 06-27-2025 06:00 AM | |
| 1339 | 01-14-2025 06:30 PM | |
| 1860 | 04-06-2018 09:24 PM | |
| 2008 | 05-02-2017 10:43 PM | |
| 5200 | 01-24-2017 08:21 PM |
11-29-2016
09:44 PM
@Raf Mohammed - Assuming your index is stored in HDFS rather than the local file system where Solr is running: hdfs dfs -mkdir /solr/DELETEME_core_node1
hdfs dfs -mv /solr/tweets/core_node1/data/tlog/tlog.* /solr/DELETEME_core_node1 When you're ready to delete the files, run this command: hdfs dfs -rm -r /solr/DELETEME_core_node1
... View more
11-29-2016
08:19 PM
@Raf Mohammed - This is a transaction log file: /solr/tweets/core_node1/data/tlog/tlog.0000000000000000338, so just delete them (I always prefer to move files I'm going to delete to another directory called "TODELETE" in the event that I actually need them and once things look good I then delete them). You may need to restart Solr if you delete these files out from under Solr. A manual commit should be like this assuming your collection is named "tweets": http://localhost:8983/solr/tweets/update?commit=true
... View more
11-28-2016
06:26 PM
Note that the above is deletes older files based on file modification time, not based on the timestamp in the filename. I did use the filename with a timestamp, which probably makes the example confusing. So that command could be used with any kind of file such as keeping the last 5 copies of your backup files. Also, if you use logrotate (e.g. where log4j rolling files is not an option), you can use the maxage option, which also uses modified time. This is from the logrotate man page: maxage count
Remove rotated logs older than <count> days. The age is only checked if the logfile is to be rotated. The files are mailed to the configured address if maillast and mail are configured.
... View more
11-28-2016
06:13 PM
1 Kudo
@Avijeet Dash, The suggestion from Sunile is great. But, where you can't do that, here is a solution. If you need to manually delete all but the last X files named with a certain file pattern (*.zip, files*.log, etc), you can run something like this command which finds all but the most recent 5 matching files.
# find MY_LOG_DIR -type f -name "FILE_PATTERN" -printf "%T+\t%p\n" | sort |awk '{print $2}' |head -n -5 |xargs -i CMD_FOR_EACH_FILE {} Replace the bold parts as needed. For example, the following command will find all but the most recent 5 files matching pattern *.log.20##-##-## and deletes them. Note, since this command is a delete command, before running something so drastic, you should test first by replacing the "rm" with "ls -l" or do a "mv" instead. Test, test, test. # find /var/log/hive -type f -name "*.log.20[0-9][0-9]-[0-2][0-9]-[0-9][0-9]" -printf "%T+\t%p\n" | sort |awk '{print $2}' |head -n -5 |xargs -i rm {} There are always many ways to solve a problem and I'm sure there is a more elegant solution.
... View more
11-28-2016
05:41 PM
1 Kudo
@Bilal Arshad All of the files in ATLAS_HOME/conf/solr are probably needed in a directory on the Solr host so that when you run the solr create command it will upload those files into zookeeper for Solr to access for that collection (no matter which host Solr is running on). The files in this directory are as follows (from the link you provided) |- solr
|- currency.xml
|- lang
|- stopwords_en.txt
|- protowords.txt
|- schema.xml
|- solrconfig.xml
|- stopwords.txt
|- synonyms.txt solrconfig.xml is has configuration parameters such as what rest endpoints are available for the collection. schema.xml describes the fields and how they are handled (indexed, stored, and so forth). The other files are used by the schema.xml (assuming they are used) as they should be listed/referenced in the schema.xml. So, for this collection, I assume the following are referenced from the schema: synonyms.txt --- listing words that can be searched and considered equivalent (e.g. car and automobile) stopwords.txt -- listing highly common words that will not be indexed such as "the" and "a" protowords.xml -- listing words that should not be stemmed (broken into equivalent root words) stopwords_en.txt -- same as stopwords above but specific for English. currency.xml -- money exchange rates Even if all of these files are not used, it shouldn't hurt anything including them.
... View more
11-27-2016
07:27 PM
@Yogesh Chaudhari I'm sorry to year about your issue and I am sure someone will get back to you. Hortonworks has been on holiday for a few days for Thanksgiving and I suspect that is why you have not heard back from them. I'm sure the exam support staff will respond early next week.
... View more
11-22-2016
01:49 AM
@Venkat Rangan - I think I found the documentation you need to become the admin user: As the default "cloudbreak" user doesn't have certain permissions (for example, it has no write access to HDFS), you must use the "admin" user to perform certain actions. To use the "admin" user instead of the default "cloudbreak" user, run sudo su - admin . (http://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/using/index.html)
... View more
11-22-2016
01:45 AM
@Venkat Rangan I'm sorry, but at the moment I don't know how to execute commands as hdfs in cloudbreak, but maybe - the cloudbreak user *may* be able to sudo (sudo -u hdfs <COMMAND>). If you are not familiar with sudo, do it something like this: sudo -u hdfs hdfs dfs -mkdir /user/<username> Don't let the "hdfs hdfs" together confuse you. The first one is the username and the second one is the command. Give that a shot and let me know.
... View more
11-21-2016
09:10 PM
From the command line as the hdfs user: # Create the directory $ hdfs dfs -mkdir /user/<username> # Set permissions and ownership $ hdfs dfs -chown <username> /user/<username> $ hdfs dfs -chmod 700 /user/<username> ##Optionally set $ hdfs dfsadmin -setSpaceQuota <bytes_allocated> /user/<username> ## Where bytes_allocated is bytes allowed for this directory (counting replication). This is allocating space for the directory, not by username. So if the user created files in other HDFS directories, this doesn't control that.
... View more