Hi @Devpriyo Bhattacharya, You cannot install HDP natively on Windows 10. You can, however, run HDP on your Windows 10 laptop via Docker or virtual machines but I expect the behavior to be very unpredictable. You are going to be *very* resource constrained and you will likely experience occasional component failures and/or slow response time. That said, typically I would recommend that you leverage the dockerized sandbox for this situation. However, if you wish to go through the process of a customized installation then you can take the whole thing for a test drive using Docker for Windows. To do this you will need to do the following: Download and install Docker for Windows Launch a CentOS 7 container with appropriate ports opened - most importantly 8080 for Ambari and 10000 for HiveServer2 - you may realize that you need others open later for various UIs and connectivity (e.g., 50070 for the HDFS UI) Connect to the CentOS 7 container and run through the standard Ambari installation process to install your custom single-node HDP installation. I recommend installing the very minimum number of components due to your resource limitation. This will get you a single-node HDP installation running on your laptop that you can use for basic functionality testing. It will be similar to the Sandbox with the exception being that you have hand-selected the components that you wish to install. ... View more

Hi @Dagmawi Mengistu, We do not currently support start/stop of cluster created via HDC. The likely reason that you are seeing the above error is that the local instance storage was chosen to support HDFS at cluster creation. This is short-lived and does not persist through start/stop of instances in EC2. In general, HDC clusters are intended to be for ephemeral workloads. If you want to start and stop the compute resources to control costs then I recommend creating a persistent metastore backed by RDS and providing that when you create new clusters. This way you can spin up and destroy clusters as you see fit and the data can be persisted via the shared metastore and S3. I hope this helps. ... View more

Hi @Saminathan A One thing you can do is drop the SplitLine processor and go straight to the ExtractText processor where you can use a regex to pull out the first 5 lines via a regex. Then you can use the groups within that regex to work on the individual groups (e.g., the first 5 lines) in the UpdateAttribute processor. This regex should work for you: ^(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n.* ... View more

Hi @Houssam Manik, The big benefit that you get by utilizing snapshots with distCP is that you can do incremental backups when distCP'ing the snapshotted directory in the future by leveraging the differential between the snapshots. Jing provides some context around this in the second answer here. The work to complete this is discussed in HDFS-7535 and some more context is provided there. This was first pulled into Hadoop 2.7.0 ... View more

Hi @Santhosh B Gowda, Assuming that this is happening on a single JournalNode then you can try the following: As a precaution, stop HDFS. This will shut down all Journalnodes as well. On the node in question, move the fsimage edits directory (/hadoop/hdfs/journal/stanleyhotel/current) to an alternate location. Copy the fsimage edits directory (/hadoop/hdfs/journal/stanleyhotel/current) from a functioning JournalNode to this node. Start HDFS. This should get this Journalnode back inline with the others and get you back to a properly functioning HA state. ... View more

Hi @Simran Kaur, You can still achieve this using out-of-the-box functions in Hive as you mentioned. You just missed getting the string in the right format. For clarity, the basic steps are: Replace the 'T' in the string with a space so the date is in the format expected by Hive. Convert the string to a unix timestamp using the unix_timestamp function. Convert the timestamp to your preferred date format using the from_unixtime function. Here is a quick example you can run in Hive to see the result for the string you provided: select from_unixtime(unix_timestamp(regexp_replace('2016-09-13T06:03:51Z', 'T',' ')), 'dd-MM-yyyy HH-mm-ss'); Notice that the only additional step is the replace operation. ... View more

Hi @Aengus Rooney and @Tamas Bihari, I just ran into the same issue. I tried to deploy with two different unique stack names and both times I got a failure during cbdinit. For giggles I tried again but this time removed the non-alphanumeric characters ('&' and '*') from my password and it succeeded. Seems like there is an implicit character set limit on the password. Would be nice to either remove this limit [preferable] or at least make a note on the form that only alphanumeric characters are acceptable. ... View more

Hi @Sami Ahmad, You can install NiFi using the steps documented in the installation guide here. ... View more

Hi @Johnny Fugers, What you are talking about sounds like you are trying to build a recommender system. I assume what you are trying to do is understand that users that do X and Y also tend to do Z. From your example, customers that buy diapers and beer also tend to buy milk. Within HDP, you would be looking at Apache Spark and the machine learning capabilities that it offers. There is a clear example of leveraging Collaborative Filtering for this type of problem on the Spark site. You should be able to run this directly on your HDP sandbox or cluster. ... View more

Hi @Smart Solutions, I think this would be sufficient to certify that the libraries are installed and your applications will be able to find them. You can find several examples that are ready to run under /usr/hdp/current/spark-client/examples/src/main/python/mllib. You can substitute python with your preferred language to find examples that correspond to the appropriate API. In terms of optimized configurations, it is hard to tune that upfront as it will be highly dependent upon on your application, dataset, and cluster. ... View more

Hi @Andy Max, This should certainly be doable and should be relatively straight forward. In the end, I would recommend you stand up a small sandbox environment that mimics the current one and test out this process and develop a concrete playbook. The rough steps that I would recommend you try are: Stop all services on the existing cluster. Use Apache Ambari to install a "dummy" HDP 2.4.x cluster on the current cluster: http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_Installing_HDP_AMB/content/ch_Getting_Ready.html Install barebones with only HDFS, ZK, etc. services. Make sure that the HDFS namenode and datanode directories are dummy directories that do not point to your existing data and namenode directories. Stop all services via Ambari. In Ambari, change the data and namenode directories for HDFS to point to your old directories. Start the services back up and verify that the data is available. This should work smoothly with HDP 2.4 because 2.4 also include Apache Hadoop 2.7.1 so the file system version is identical. This all assumes that you are only using HDFS. It could all get a bit hairier if you have Hive tables sitting on top with some metadata that needs to be migrated. Cheers, Brandon ... View more

Hi @Khera What JDBC driver are you using to connect to SQL Server? The one provided by MS does not support WIndows authentication. That said, you can grab another driver that does support it. You have a couple options: Both Simba and Data Direct have drivers to support this authentication method. These have free trials but are ultimately going to require a license for repeated use. There is also jTDS which is free and open source and claims to support Windows Authentication so we can take it for a spin if you would like. You can see the rough JDBC URL that you would need is: <code>jdbc:jtds:sqlserver://123.123.123;instance=server1;databaseName=students;integratedSecurity=true;authenticationScheme=JavaKerberos ... View more