@amateur  There is a Jira out there  ... View more

@Seaport  I would think there is a typo error  the dash  [-] after - put  and before the hdfs path  hdfs dfs -cat /user/testuser/stage1.tar.gz | gzip -d | hdfs dfs -put - /user/testuser/test3   try this after removing the dash -   hdfs dfs -cat /user/testuser/stage1.tar.gz | gzip -d | hdfs dfs -put /user/testuser/test3  Hope that helps  ... View more

@Seaport  It shouldn't be strange to you that Hadoop doesn't perform well with small files, now with that in mind the best solution would be to zip all your small files locally and then copy the zipped file to hdfs using copyFromLocal there is one restriction that is the source of the files can only be on  a local file system. I assume the local Linux box had is the edge node and had the hdfs client installed. If not you will have to copy the myzipped.gz to a node usually the edge node and perform the below steps $ hdfs dfs -copyFromLocal myzipped.gz /hadoop_path". Then unzip the myzipped.gz gzipped file in HDFS using $ hdfs dfs -cat /hadoop_path/myzipped.gz | gzip -d | hdfs dfs -put - /hadoop_path2 Hope that helps ... View more

@Stephbat  Bizarre all the symlinks should point to the newer version 3.1.4.0-315. You should recreate the symlink point to the new version. The re-run the steps   ... View more

@Stephbat  Did you follow the Workaround:1 through 3? I see you have a similar error File "/var/lib/ambari-agent/cache/custom_actions/scripts/remove_previous_stacks.py 1) go to each ambari-agent node and edit the file remove_previous_stacks.py # vi /var/lib/ambari-agent/cache/custom_actions/scripts/remove_previous_stacks.py 2) go to line: 77 edit the line from : all_installed_packages = self.pkg_provider.all_installed_packages() to all_installed_packages = self.pkg_provider.installed_packages() 3)  R etry the operation via curl again, please substitute only the Ambari_host below  ex: curl 'http://Ambari_host:8080/api/v1/clusters/<cluster_name>/requests' -u admin:admin -H "X-Requested-By: ambari" -X POST -d'{"RequestInfo":{"context":"remove_previous_stacks", "action" : "remove_previous_stacks", "parameters" : {"version":"3.1.0.0-78"}}, "Requests/resource_filters": [{"hosts":"Ambari_host"}]}'  And please revert    ... View more

@Stephbat    This thread should help you achieve that the Ambari version could be an issue there is a workaround documented note the DISCLAIMER !! Please let me know ... View more

@Krpyto84  Your permission issue is linked to ZK ACL's my good guess is your Kafka is kerberized. Zookeeper requires you to set up a superuser using the zookeeper.DigestAuthenticationProvider.superDigest property. I don't know how you will integrate that procedure in your Ansible playbook You will then need to append this in y your KAFKA_OPTS env variable to set the JVM parameters export KAFKA_OPTS=-Djava.security.auth.login.config=/path/to/kafka_server_jaas.conf Please let me know whether that is your situation if that's the case then I will try to help you out   ... View more

@SpidyGen  Did you get help with your knox? ... View more

@isanas  I remember writing a long document on the steps either in HWX/Cloudera community. Ranger will automatically sync local users whose USERID is above 500 Please let me know if you what me to do a walkthrough, in that case, I will ask you to give me an accurate scenario and the version of HDP. Happy hadooping   ... View more

@focal_fossa  My guess is you are running out of memory. I would like to know how much memory you have? Copying local files to HDFS is done using the mapreduce job when we use put or copyFromLocal commands it is actually using Streaming by the hadoop client binary client libraries and queues. So my guess is that the Ambari views copy might also be using MR behind the scenes.   Another alternative is to use DistCp [distributed copy] a tool used for large inter/intra-cluster copying. It also uses MapReduce to effect its distribution, error handling and recovery, and reporting. It expands a list of files and directories into input to map tasks, each of which will copy a partition of the files specified in the source list. DistCp at times runs out of memory for big datasets? If the number of individual files/directories being copied from the source path(s) is extremely large, DistCp might run out of memory while determining the list of paths for copy. This is not unique to the new DistCp implementation. To get around this, consider changing the -Xmx JVM heap-size parameters, as follows: $ export HADOOP_CLIENT_OPTS="-Xms64m -Xmx1024m" $ hadoop distcp /source /target Hope that helps ... View more

@saur  Can you explain the latest development ? I compiled a document for you did you go through it step by step? ... View more

@saur  Any updates on this? ... View more

@rwinters  Simplest method is to edit /etc/postgresql/9.2/main/start.conf and replace auto with manual or disabled.   You can also resolve this by adding the following to the end of your shell's initialisation file (e.g. ~/.bashrc if you're using bash): PATH=/usr/psql-10/bin:$PATH reboot the server  now you should have the  postgres 10 as the default Hope that helps ... View more

@Henry2410  MySQL Server is intended for mission-critical, heavy-load production systems as well as for embedding into mass-deployed software. On the other hand, Snowflake is detailed as "The data warehouse built for the cloud".   There's not really an equivalence between MySQL and Snowflake use cases. What you are asking really is whether Snowflake can play the role of an OLTP database. Snowflake is not an OLTP database. It is an OLAP database. So generally speaking I would say no.   Snowflake is a cloud-based warehouse and it would be used most of the times for OLAP purpose back to your questions, Snowflake can be used under the following conditions: If you have only inserts into target table and not much updates to the table we can achieve good performance by using cluster by and other inline views Having said that, to explore your use case a little bit more I would ask yourself or your stakeholders the following questions: Do you need millisecond response times for INSERTs, UPDATEs, and SELECTs? Does your application tool require indexes? Does your application need referential integrity and uniqueness constraints enforced? If you said yes to ANY of 1, 2, 3 then go MySQL. If you said NO to ALL 1, 2, and 3, then Snowflake might be viable. But even then I would not recommend it, as that is not what Snowflake was built for. ... View more

@LeticiaAraujo  Log4j properties set the properties that control logging activities for each service running in your Hadoop cluster. This is something that every site can customize according to its needs like enabling DEBUG will generate detailed by huge logs is you want to analyze the issue with your service. Rotation of the logs, compression format or logfile size etc Logging services is a topic on its own, and here is a good source for reading log4j  the most common task is enabling DEBUG and Log rotation and it's date format. ... View more

@borisgersh  Sorry for your frustration neither I nor  the admin of this platform is responsible for the contents of the CCA175 exams, here we try to help maybe you will need to address your case to certification@cloudera.com I have found a  Hadoop Developer Ashwin Rangarajan who has shared some good content  below  https://ashwin.cloud/blog/cloudera-cca175-spark-developer/   Hope that helps     ... View more

@LeticiaAraujo  That is the warning you should be able to continue, you will need adjust the log4j.properties.Having said that to switch to user hive from a root session you should do the following [root@sandbox-hdp ~]# su - hive Note the dash and space the above switch command change the context to the hive user variables. explanation su - invokes a login shell after switching the user and the login shell resets most environment variables, providing a clean base. su Just switches the user, providing a normal shell with an environment nearly the same as with the old use [root] ... View more

@borisgersh  Sorry for misunderstanding you but you were neither specific nor clear in your earlier posting as being specific to Spark-shell etc, Increasing memory for spark interactively is done by using the --driver-memory option to set the memory for the driver process. Here are simple examples of executions of standalone [1 node] and Cluster executions these are version-specific Run spark-shell on spark installed on standalone mode spark version 1.2.0 ./spark-shell --driver-memory 2g Run spark-shell on spark installed on the cluster ./bin/spark-shell --executor-memory 4g Spark 1.2.0 you can set memory and cores by giving following arguments to spark-shell ./spark-shell --driver-memory 10G --executor-memory 15G --executor-cores 8 Using Spark version 2.4.5 to run the application locally on 8 cores ./bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master local[8] \ /path/to/examples.jar \ 100 Run-on a Spark standalone cluster in client deploy mode ./bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master spark://207.184.161.138:7077 \ --executor-memory 20G \ --total-executor-cores 100 \ /path/to/examples.jar \ 1000 Run-on a Spark standalone cluster in cluster deploy mode with supervise ./bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master spark://207.184.161.138:7077 \ --deploy-mode cluster \ --supervise \ --executor-memory 20G \ --total-executor-cores 100 \ /path/to/examples.jar \ 1000 Run-on a YARN cluster you will need to export HADOOP_CONF_DIR=XXX then ./bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master yarn \ --deploy-mode cluster \ # can be client for client mode --executor-memory 20G \ --num-executors 50 \ /path/to/examples.jar \ 1000 So in the above examples, you have to adjust the below parameters --executor-memory --total-executor-cores Get help commands to spark shell spark-shell --help Hope that answers your question on how to interactive  increase available memory to be able to run rings around spark there is no better source than spark-docs.   Happy hadooping ... View more

@borisgersh    I think what is being said and misunderstood her is the runtime variables like in hive, where you can personalize your environment by executing a file at startup. See this example  and  many more online examples There are total of three available for holding variables. hiveconf   - hive started with this, all the hive configuration is stored as part of this conf. Initially, variable substitution was not part of hive and when it got introduced, all the user-defined variables were stored as part of this as well. Which is definitely not a good idea. So two more namespaces were created. hivevar: To store user variables system: To store system variables. If you do not provide namespace as mentioned below, variable var will be stored in hiveconf namespace. set var="default_namespace"; So, to access this you need to specify hiveconf namespace select ${hiveconf:var};   Hope that helps    ... View more