@Tu Nguyen - I'm afraid I don't understand your question. Spark does not use JDBC to communicate with Hive, but it can load Hive with any type of data that's able to be represented as a Spark DataSet. You may want to try a "MSCK REPAIR TABLE <tablename>;" in Hive, though ... View more

See https://community.hortonworks.com/questions/158942/ms-access-odbc-connection-to-hive.html ... View more

You can coalesce using Spark, or MergeContent in NiFi to "compact" processes without needing to go to -getmerge. You should ideally avoid zip files on HDFS. They are not a common format in HDFS since they are not splittable. Reading a large ZIP file will therefore be only processable by a single mapper. Querying multiple part files of uncompressed CSV will be faster. If you need these files compressed in HDFS for archival while also able to query via Hive and other engines, use a different, compressed, binary format like Snappy w/ ORC. If you just want a CSV, use Beeline's output format argument, and write the results to a file, which you can then ZIP. ... View more

According to the Compatibility Matrix, 1.0 should work with older clients. Broker 1.0.0 Basic client compatibility: Java: any version KIP-35 enabled clients: any version Kafka Streams: any version Kafka Connect: any version Doesn't this depend on what you want to do with the data once you consume it, though? HDP itself doesn't provide Kafka 1.0, NiFi doesn't have a 1.0 processor pair... Kafka Streams is a standalone Java process, and therefore is not tied to HDP ... View more

@Geoffrey Shelton Okot I'm not deploying Hue on a server with HDP libraries, so there are no Hadoop client libraries that would be related. I was able to build Hue from source, then built an RPM, and installed and templated the configurations on both Centos6 and Centos7 with Puppet. Both OS Hue installs are pointing at same HDP 2.5.3 cluster, which has been running the same endpoints (Namenode, ResourceManagers, HiveServer, etc) since HDP 2.2, so that first link I provided in the other answer is all that needs to be referenced. The Hue config file itself doesn't change that significantly between releases. ... View more

Not sure what you mean by "hive base" . Can you add you hue.ini to the post? Are you using a kerberized cluster? Are you using the correct ports, and FQDN server names? I have followed these, and it works fine for at least HDFS, YARN, Oozie, Hive, and HBase pages of Hue. http://gethue.com/how-to-configure-hue-in-your-hadoop-cluster/ http://gethue.com/hadoop-hue-3-on-hdp-installation-tutorial/ ... View more

The exception says hdfs://secondnamenode:9000/sample already exists. List the content of the other cluster, and delete the directory if that folder is indeed there. Otherwise, add the overwrite option to your Spark code. ... View more

@Evan Tattam I'm sure it's supported, but the link is just down. ... View more

@Evan Tattam Is there a specific reason you want an old version? The 1.4.0 link is working for me. http://public-repo-1.hortonworks.com/HCP/centos7/1.x/updates/1.4.0.0/tars/metron/hcp-ambari-mpack-1.4.0.0-38.tar.gz ... View more

@Rakesh AN I have not used Flume in a distributed fashion, but whatever agent you choose, it tails the logs from the agent on that server, then ships them to the configured sink destinations. One agent per server makes it collect from different servers. Flume is near real-time, since it is configured with a batch size. It's not clear what doubt you have... Can you please explain how you've configured your Flume agents, and the issues you are experiencing? The Flume documentation is fairly straightforward ... View more

Your open source options include Oozie (included with HDP), Luigi, Airflow, and Azkaban. https://www.bizety.com/2017/06/05/open-source-data-pipeline-luigi-vs-azkaban-vs-oozie-vs-airflow/ NiFi is also capable of scheduling tasks. An enterprise option includes Stonebranch UAC "Best" is relative to your needs and who will support it ... View more

@Markus Wilhelm, That API call tells the Ambari server that it has a service, but yes, now you'll need to add the nifi component. Some combination of these requests should be able to show / add that... https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/components.md https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/create-component.md https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/create-hostcomponent.md I think the Nifi component name is NIFI_MASTER, ... View more

@Markus Wilhelm, Since NiFi is installed on the machine itself, you might be able to post the service to the Ambari database https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/create-service.md Something like this POST ambari-server:8080/clusters/:clusterName/services/NIFI ... View more

@Markus Wilhelm Well, the error does not say the installation of NiFi failed. Maybe the service won't show up in Ambari, but are you able to start NiFi on the server you installed it to? ... View more

The command syntax is hadoop fs -copyFromLocal <localsrc> URI https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html#copyFromLocal If "URI" is not given, it will copy to /home/$(whoami), where in your case `$(whoami) == "hduser"` In other words, running this command as the hduser linux account hadoop fs -copyFromLocal afile in Will copy "afile" to hdfs:///home/hduser/in If you want to copy to a different location on HDFS, give the full path to the destination ... View more

It looks like it created the directory structure for the service. 2017-12-07 16:43:16,148 - Installing package nifi_3_0_* ('/usr/bin/yum -d 0 -e 0 -y install 'nifi_3_0_*'') 2017-12-07 16:44:58,917 - Directory['/usr/hdf/current/nifi'] {'owner': 'nifi', 'group': 'nifi', 'create_parents': True, 'recursive_ownership': True} 2017-12-07 16:44:58,918 - Creating directory Directory['/usr/hdf/current/nifi'] since it doesn't exist. 2017-12-07 16:44:58,918 - Following the link /usr/hdf/current/nifi to /usr/hdf/3.0.0.0-453/nifi to create the directory 2017-12-07 16:44:58,919 - Changing owner for /usr/hdf/3.0.0.0-453/nifi from 0 to nifi 2017-12-07 16:44:58,919 - Changing group for /usr/hdf/3.0.0.0-453/nifi from 0 to nifi 2017-12-07 16:44:58,921 - Directory['/var/run/nifi'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2017-12-07 16:44:58,921 - Directory['/var/log/nifi'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2017-12-07 16:44:58,922 - Directory['/var/lib/nifi'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2017-12-07 16:44:58,922 - Directory['/var/lib/nifi/database_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2017-12-07 16:44:58,923 - Directory['/var/lib/nifi/flowfile_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2017-12-07 16:44:58,923 - Directory['/var/lib/nifi/provenance_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2017-12-07 16:44:58,924 - Directory['/usr/hdf/current/nifi/conf'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2017-12-07 16:44:58,924 - Creating directory Directory['/usr/hdf/current/nifi/conf'] since it doesn't exist. 2017-12-07 16:44:58,924 - Changing owner for /usr/hdf/current/nifi/conf from 0 to nifi 2017-12-07 16:44:58,924 - Changing group for /usr/hdf/current/nifi/conf from 0 to nifi It looks like you are using a custom local repository? Have you tried using the Hortonworks ones? ... View more

@Ravikiran Dasari, You can of course install NiFi as an extra service, just as anything else. You are not locked to only packages HDP provides. You just lose the advantage of using Ambari to monitor and configure it. Feel free to read over the NiFi installation documentation, if you want to use it. Or you can install HDF services (such as NiFi) to your existing HDP cluster. If you want to use Flume, it seems there is an external FTP source, however, I personally don't know to install or configure it Also see https://community.hortonworks.com/questions/150882/ftp-files-to-hdfs.html ... View more

You should use Hive partitioned tables. Make your folders like this /user/hadoop/tableName/day=2017-12-04/ /user/hadoop/tableName/day=2017-12-05/ /user/hadoop/tableName/day=2017-12-06/ And your corresponding Hive query would be create external table test_table(id int, name string) partitioned by (day string) row format delimited fields terminated by ',' location '/user/hadoop/tableName'; ... View more

NiFI is not your only option. You could install a Flume Agent on the SFTP Server to read this folder as a spooling directory. You can use Spark to read from the FTP directory and write to HDFS as it's just a filesystem. Add FTP Java clients to your code, and read from a folder. Whatever route you chose, you either need 1. additional software installed on the SFTP Server itself 2. setup a process "upstream" of the SFTP server that also sends files to HDFS. That could be by WebHDFS, HTTPFS, or a NFS Gateway 3. Some software that HDP does not provide out of the box between that server and HDFS. This includes NiFi, but Streamsets is another option. The official documentation for those softwares are going to tell you more than I would be able to here. If you want to use HDF, I believe you see if this documentation suits your needs. https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.1.1/bk_installing-hdf-and-hdp/content/ch_install-ambari.html ... View more

NiFI has a GetFTP and PutHDFS processor. Are you using an HDF cluster? ... View more

I don't think the message "Unable to lookup the cluster by ID" has anything to do with local repositories, that looks like a problem with how you named your cluster. See "clusterName=clusterID=-1". But yes, you had to first create an "ambari.repo" and "hdp.repo" file in /etc/yum.repos.d/ that point to your local repo before you even can install ambari-server and other related HDP packages on that machine. Try starting over. https://community.hortonworks.com/questions/1110/how-to-completely-remove-uninstall-ambari-and-hdp.html ... View more

The repos are explicitly defined by an ambari stack. For the HDP stack, these are public-repo-1.hortonworks.com. You can reconfigure these yourself by logging into Ambari as an admin, dropping down the username > Manage Ambari > Versions > Version Name. See in the screenshot that I have set the repos to be Artifactory. ... View more

@Michael Bronson Topics are never automatically deleted. The logs are retained for a configured number of bytes (log.retention.bytes) or period of time (log.retention.{hours, minutes, ms}), then the log segments are purged or compacted, which is another Kafka setting (log.cleanup.policy). All the configurations that you seek are defined in the Kafka documentation, and you should really take these tunables into consideration when installing a production Kafka cluster. ... View more

Ambari should manage this already depending on the clients/services installed. And if there is no refresh icon, it will be up-to-date. Feel free to spot-check the configurations for your services in /usr/hdp/current/ or /etc/hadoop/conf ... View more

What documentation did you follow to install Hue? From what I have found, there is no systemd service that it comes with. You should be able to extract the tarball on gethue.com and start the server. ... View more

Connect to which part of Hadoop? HDFS, Hive, HBase? If HDFS, you can use WebHDFS from any programming language with an HTTP client, or you can include the hadoop-common library in your code via Maven, for example. If HBase, there are Java clients you can find. If Hive, you can use JDBC. ... View more