About stevenmatison

stevenmatison · ‎03-27-2020

@Gubbi It sounds like you have the getFile working and just having issues with the matching to target_file route? I would recommend to simply the match in RouteOnAttribute: ${filename:endsWith(${today})} If you are having problems with the GetFile, please share that proc configuration so we can see what you have going on there.

stevenmatison · ‎03-27-2020

@Punjala First, do you have the mysql connector for java enabled? To use MySQL with Ranger, you must download the https://dev.mysql.com/downloads/connector/j/ from MySQL . Once downloaded to the Ambari Server host, run: ambari-server setup --jdbc-db=mysql --jdbc-driver=/path/to/mysql/com.mysql.jdbc.Driver Next, have you confirmed that ranger metastore database is setup, with user, password, and correct permissions from ranger host to metastore host? In Ambari during Ranger Install, you can use the Test Connection to make sure this stuff works.

stevenmatison · ‎03-27-2020

@Savi I think you only need one date format in your nifi expression language. Next, it seems you know what format it is upstream (source), and what it needs to be for downstream. Based on your expression need, you should play the flow up to the csv reader processor and then List Queue on the FlowFiles and inspect the attributes. Adjust the expression, and retest until the attribute format meets the required format. Inspecting the dates upstream won't show you what the date looks like after expression and before the reader. Often times I find myself testing a flow step by step, each step confirming the attributes needed are correct for the next step. Never moving forward to the next step until I am sure the attributes are correct after visual inspection.

stevenmatison · ‎03-27-2020

@murali2425 It sure is possible. I believe anything is possible in NiFi!! The excel processor is: ConvertExcelToCSVProcessor I have also put an Excel to SQL template on my GitHub: https://github.com/steven-dfheinz/NiFi-Templates This template gets an excel file that contains product inventory data (sku, name, price, description, quantity, etc and routes the contents of the excel file to ConvertsExceltoCSV. Then it splits the csv file line by line, parses each line for quantity and sku and uses those attributes to build an insert query, which is finally executed via PutSQL. There are many different ways to deliver the Excel file too ConvertExceltoCsv, different ways to process the CSV, and different ways to insert into SQL. The above is just one way I have used in the past.

stevenmatison · ‎03-26-2020

Update for Hue 4.x: I have finished initial required changes to for branch Hue.4.6.0 and successfully installed Hue 4.6.0 in HDP 3.1.4. There ended up being some big differences in the hue.ini for 4.x. Once these were resolved the only manual work needed for install in the node is creating the hue user and group and running the python to ignore_groupsusers_create. User Commands: useradd -g hue hue usermod -a -G wheel hue chown -R hue:hue /home/hue Reset/Remove Commands: rm -rf /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE rm -rf /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/ rm -rf /usr/local/hue rm -rf /usr/hdp/current/hue-server rm -rf /usr/local/hue-4.6.0/ Once installed it is possible to update/manage changes to hue.ini via ambari to enable HDFS, Hive, RDBMS, enable/disable Hue Plugins, or further configure Hue as necessary.

stevenmatison · ‎03-25-2020

@ParthiCyberPunk In HDP3 the only out of box option for hive is beeline. Additionally many third Party Tools can access hive via jdbc connections. I have been working on Hue with HDP3 and have had some success with Hue 3.x and HDP 3.x in production already. Here is article: https://community.cloudera.com/t5/Community-Articles/How-to-install-Hue-3-11-in-HDP-3-1/ta-p/291280 Here is the repo: https://github.com/steven-dfheinz/HDP3-Hue-Service I am currently workjng on hue 4.x and a management pack for HDP 2.x - 3.x

stevenmatison · ‎03-24-2020

Repo Branch created for Hue 4.6.0. I am working on this right now but should have it operating here shortly. If you want Hue 4.6.0 be sure to download Branch Hue.4.6.0.

stevenmatison · ‎03-24-2020

Are you running hive/beeline as the hive user or root? To get to Hive as the hive user I do: sudo su - hive hive Then execute the queries.

stevenmatison · ‎03-23-2020

There should be a config value for Rollback on Failure. Change accordingly:

stevenmatison · ‎03-23-2020

After you get all the attributes you want, you check them with RouteOnAttribute. In that proc click + and set update / inserts equal to matches you want. You use regex in the value for each. For example insert is ${operation.equals('add')}. After RouteOnAttribute is configured, drag update to the update query proc, and insert to the insert query proc.

stevenmatison · ‎03-18-2020

@manuel_loayza You would need to do a few things to monitor the performance. These include: Watching the nifi logs on all nodes during times of issues. This also includes going back and opening the files and searching for ERRORS. Watching the nodes during times of issues. Looking at "top", looking at ambari metrics for ram/cpu/, garbage collection, etc Check your NiFi min/max ram. Compare these values with system resource and tune higher. Tune nifi UI for min/max threads. Watch garbage collection and tune. Consider upgrading to 1.9 + to make most out of increased functionality with node to node and clustering capabilities. Upgrading version will not address 1-5, so sort them first.

stevenmatison · ‎03-18-2020

@prakashpunj You need to add the admin user to the hdfs user group. Depending on your system, you may need to make some changes to hdfs proxy hosts. Make it *. You can confirm by checking HDFS Files View with the admin user. If admin can view hdfs there, you should be good in Hue too. In Hue, you can also add a user: hdfs with password: hdfs and login as that user to view HDFS. You can do the same for Hive. In one of my recent tests, admin could view HDFS Files, but could not execute queries on hive tables. So I just created a hive user in hue to validate it works. I assume you are working in a test environment so it should be no problem to make minor users/groups changes to satisfy these access issues. Normally you would want LDAP integration and use Ranger to manage permissions. For example, giving a specific user privileges to HDFS and HIVE.

stevenmatison · ‎03-17-2020

@prakashpunj Please accept my first reply as the solution?

stevenmatison · ‎03-16-2020

Working with Hive in HDP 3.x is a challenge for Administrators and Developers because there is no Hive UI. The old Hive View has been removed in HDP 3.x and since then Developers were forced to use external 3rd party tools to access Hive. The only other option for Administrators and Developers was the command line for Hive which certainly works, but is not as friendly as the old Hive View. Both Beeline and the Hive View are limited when compared to the entire capabilities in the WebUI for Hue (HDFS, Hive, Zookeeper, YARN, Oozie, Sqoop, RDBMS, etc.). First a little back story on Hue Ambari Service. First created by Kyle Joe as a Hue Service for Ambari 2.4 the repo did not have any updates in three years. At that time, Hortonworks and Hue removed and changed the location of the 3.11 files, making Kyle's repo not operational. In this article, I am going to outline how I created an HDP 3 test cluster, installed Kyle's 2.4 repo, and began to debug. Next with what I learned debugging, I created an HDP 2.4 cluster, installed Kyle's repo, and finished the installation. With both learning sessions complete, I was able to create my repo version, install a new HDP 3 Single Node Cluster with all the services, install, start, and use Hue for HDFS File Manager, to view an RDMBS Database, view existing Hive Tables, and Execute New HQL Queries. Since creating the original repo, I have installed it in two additional HDP clusters (1 in Production). I will continue to make small updates to the repository as necessary for future projects. I plan to further increase functionality to include adjustments necessary for SSL, High Availability, and clusters without all the required components. I also plan to work towards Hue 4.x and bundling up everything nicely into a Management Pack. Assumptions For the installation: The following are some assumptions for the installation: This is tested on Centos 7 and RHEL 7 Single Node Cluster has the following Components HDFS Yarn Hbase Hive Oozie Sqoop Spark2 Zookeeper No SSL or High Availability (Not Tested Yet) Install includes installing all current Hue Dependencies. See documentation here. Install time lengthy due to compile time during command "make apps". There are conflicts with "Ambari Managing Users" for Custom Components There are conflicts with Spark and Hbase Version Differences The HDP 3 Hue Service Repo https://github.com/steven-dfheinz/HDP3-Hue-Service Creating an HDP 3 Cluster The following commands are used to quickly install a new HDP 3 Cluster [root@hdp3 ~]# history 1 hostnamectl set-hostname hdp3.cloudera.com 2 yum install nano git wget -y 3 nano /etc/hosts 4 nano /etc/selinux/config 5 nano /etc/cloud/cloud.cfg 6 reboot 7 exit 8 ssh-keygen 9 cat ~/.ssh/id_rsa.pub 10 cat ~/.ssh/id_rsa 13 nano ~/.ssh/authorized_keys 14 ssh root@hdp3.cloudera.com 15 wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo && yum --enablerepo=extras install epel-release -y && yum install nano java java-devel mariadb mariadb-server mysql-connector-java ambari-server ambari-agent -y && ambari-server setup -s && ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar && ambari-server start && ambari-agent start && service mariadb start && chkconfig mariadb on 16 mysql Installing Ambari Hue Service Using GIT The following commands will deliver the service to Ambari using GIT yum install git -y sudo git clone https://github.com/steven-dfheinz/HDP3-Hue-Service.git /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE ambari-server restart Installing Ambari Hue Service Using ZIP The following commands will deliver the service to Ambari using Zip Download wget https://github.com/steven-dfheinz/HDP3-Hue-Service/archive/master.zip unzip master.zip mv HDP3-Hue-Service HUE mv HUE /var/lib/ambari-server/resources/stacks/HDP/3.1/services/ Installing Ambari Hue Service via Ambari With everything above you should be able to login to Ambari Server and click on Add Service to choose to install Hue. This article assumes you have some experience Adding Custom Services via Ambari, so I won't go into detail here. If you need more info please reach out. The install should work very well without providing any details to the Installation Wizard. If the service fails to install or start, you should be able to determine the issue and take action to retry install/start. See below for more information about debugging this install process. Lessons Learned From This Project I always love a good technical challenge, but even more, I love to learn new things I can use in the future. Some of those things are below. When a custom Ambari Service install fails, clicking into the Failure Modal, and inspecting the stdout and stderr will provide everything you need to debug. While debugging, you can directly edit python files in the Agent Cache location found in the above modal, then retry the action. When a code change is effective, make the same change in the source repository file set. While debugging python, you can print "some variable", config['clusterHostInfo'] to see the output of the object in the modal. This is useful to fix conflicts in config objects. Be sure to remove these prints when done or just do NOT make those print changes in the source repository file set. If working in the context of Ambari becomes cumbersome, it is possible to work directly in the node as root, making changes to components configs, stopping/starting component, etc to a point where things become more operational. Be sure to be mindful of following: Always make the same effective changes in the source file sets (Ambari-Server, Ambari-Agent, and Parent Repo). Restart the service via Ambari to overwrite local changes. Be careful here. After making any changes directly in the node, make sure all the files are owned to the correct user. This includes the component files and logs. The wrong ownership will break ambari start/stop/restart actions. Work in small forward-moving test cycles. Making too many changes at a time, will confuse you in what did or didn't work to create forward movement. Make a small change, and move forward in successful increments only. If something does not work, back out to the last successful increment and try again. Full History Output I always find it useful to record all of my node history. Here, you can see all the commands I executed while working on my HDP 3 Cluster. The only commands I have removed are typos or misfires. [root@hdp3 ~]# history 1 hostnamectl set-hostname hdp3.cloudera.com 2 yum install nano git wget -y 3 nano /etc/hosts 4 nano /etc/selinux/config 5 nano /etc/cloud/cloud.cfg 6 reboot 7 exit 8 ssh-keygen 9 cat ~/.ssh/id_rsa.pub 10 cat ~/.ssh/id_rsa 11 nano ~/.ssh/authorized_keys 13 nano ~/.ssh/authorized_keys 14 ssh root@hdp3.cloudera.com 15 wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo && yum --enablerepo=extras install epel-release -y && yum install nano java java-devel mariadb mariadb-server mysql-connector-java ambari-server ambari-agent -y && ambari-server setup -s && ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar && ambari-server start && ambari-agent start && service mariadb start && chkconfig mariadb on 16 mysql 17 sudo git clone https://github.com/steven-dfheinz/HDP3-Hue-Service.git /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE 18 ambari-server restart 19 mysql 20 useradd hue 21 useradd -g hue hue 22 python /var/lib/ambari-server/resources/scripts/configs.py -u admin -p admin -n HDP3 -l hdp3.cloudera.com -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true 23 python /var/lib/ambari-server/resources/scripts/configs.py -u admin -p admin -n HDP3 -l hdp3.cloudera.com -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v false 24 top 25 python /var/lib/ambari-server/resources/scripts/configs.py -u admin -p admin -n HDP3 -l hdp3.cloudera.com -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true 26 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py 27 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/hue_server.py 28 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/common.py 29 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py 30 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/common.py 31 cd /usr/local 32 ls 33 ls -al 34 rm -rf hue 35 ls -al 36 du -hs ./* 37 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py 38 ls 0al 39 ls -al 40 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/setup_hue.py 41 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py 42 sudo su - hdfs 44 chown -R hue:hue hue* 45 cd hue 46 ls 47 cd desktop/ 48 ls 49 ls -al 50 cd .. 51 ls 52 ls -al 53 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/hue_server.py 54 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/common.py 55 cd /var/log/hue 56 ls 57 ls -al 58 history 59 rm -r /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE 60 rm -R /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE 61 rm -rf /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE 62 sudo git clone https://github.com/steven-dfheinz/HDP3-Hue-Service.git /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HU 63 sudo git clone https://github.com/steven-dfheinz/HDP3-Hue-Service.git /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE 64 rm -rf /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HU 65 rm -rf /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE 66 rm -rf /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE 67 sudo git clone https://github.com/steven-dfheinz/HDP3-Hue-Service.git /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE 68 nano /var/lib/ambari-server/resources/stacks/HDP/3.1/services/HUE/package/scripts/common.py 69 service ambari-server restart 70 exit 71 cd /usr/local/ 72 ls -al 73 rm -rf /usr/local/hue* 74 ls -al 75 history 76 history | grep params 77 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py 78 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/common.py 79 sudo su - hdfs 80 mysql 81 cd /var/log/hue 82 ls 83 ls -al 84 uptime 85 tail -f error.log 86 cd /etc/hive/conf 87 ls 88 cd /var/log/hue 89 tail -f error.log 90 cd /usr/local/hue 91 ls 92 cd etc 93 ls 94 cd ext 95 ls 96 cd .. 97 cd app 98 cd apps 99 ls 100 cd .. 102 cd /var/log/hue 103 ls 104 ls -al 105 tail -f supervisor.log 106 tail -f runcpserver.log 107 cd /etc/yum.repos.d 108 ls 109 nano epel.repo 110 exit 111 history 112 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/common.py 113 nano /usr/local/hue-3.11.0/desktop/conf 114 cd /usr/local/hue-3.11.0/desktop/conf 115 ls 116 nano hue.ini 117 yum –y install cyrus-sasl-plain cyrus-sasl-gssapi 118 yum install cyrus-sasl-plain cyrus-sasl-gssapi -y 120 ls 121 ls -al 122 nano pseudo-distributed.ini 123 cat hue.ini | grep hiveserver2 124 cat hue.ini | grep hive 125 nano pseudo-distributed.ini 126 ls -al 127 chown -R hue:hue /usr/local/hue* 128 cd /var/log/hue 129 ls 130 ls -al 131 nano audit.log 132 nano error.log 133 history 134 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 135 history | grep params 136 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py 137 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/hue_server.py 138 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py 139 nano /usr/local/hue-3.11.0/desktop/conf/hue.ini 140 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 141 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py 142 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/hue_server.py 143 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/common.py 144 /usr/local/hue/build/env/bin/supervisor 145 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 146 /usr/local/hue/build/env/bin/supervisor 147 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 148 /usr/local/hue/build/env/bin/supervisor 149 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py 150 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 151 cd /usr/local/ 152 ls -al 153 cd hue 154 ls 155 ls -al 156 rm -rf hue 157 cd desktop/ 158 ls 159 cd conf 160 ls 161 ls -al 162 cd /var/log/ 163 ls -al 164 cd hue 165 ls -al 166 chown -R hue:hue /var/log/hue 167 ls -al 168 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py 169 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 170 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py 171 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 172 /usr/local/hue/build/env/bin/supervisor 173 history | grep chown 174 chown -R hue:hue /usr/local/hue* 175 chown -R hue:hue /var/log/hue 176 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 177 /usr/local/hue/build/env/bin/supervisor 178 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 179 chown -R hue:hue /var/log/hue 180 chown -R hue:hue /usr/local/hue* 181 nano /usr/local/hue-3.11.0/desktop/conf/pseudo-distributed.ini 182 nano /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/HUE/package/scripts/params.py 183 history

stevenmatison · ‎03-12-2020

@ajay_mamgain200 Try to go back to the source system, and see if there is something you need to pass on the additional nth calls to indicate you want the "more results" not the initial results. Sometimes this is another flag, or a range to identify a start/stop point. If the additional call is quite a bit different, do not loop back to the original invokeHttp, use a separate invokeHttp and loop to this one until done; unless you can make a single InvokeHttp dynamically handle first and additional nth calls. I would test these calls in postman before attempting in NiFi, so that you know for sure what the first call is, and what 2nd, 3rd, nth is.

stevenmatison · ‎03-12-2020

@ajay_mamgain200 After the first invokeHtml you should do a check for the moreResults attribute to exist. If it exists, route that FlowFile back to invokeHttml to run again. If does not exist or false, go to next step in your flow. Hope this helps..

stevenmatison · ‎03-12-2020

OperationalError: attempt to write a readonly database Need to fix the write perms on the database file... not sure of the path you have now, but mine was in the desktop folder.

stevenmatison · ‎03-12-2020

Your /etc/hosts file is a mapping of a hostname to an IP. What is in /etc/hosts will override any dns settings. That is the whole point of using /etc/hosts/. Also the hostname of each machine should be same as the hostname you use in /etc/hosts to map to the the IP. You should be setting the hostname, and /etc/hosts accordingly before install anything. This should be done on all hosts, and all hosts of cluster should be listed in /etc/hosts on each machine. Assuming you have more than one node in the cluster. You can set hostnames to be whatever FQDN you want, just make sure they persist after reboot. In some OS's the /etc/hosts file and hostname file(s) are managed (manual changes over-written during reboot). So make sure you set them both per node, reboot, and check. You need to get the machines working on a single hostname and a single ip. This is the problem and is a documented requirement for working in CDH/CDP/HDP/etc.

stevenmatison · ‎03-12-2020

looks like you have it installed as /root/hue-3.8.1 try using /root/hue-3.8.1 instead of /usr/local/hue in the startup command? I would recommend to put it in /usr/local and symlink if not installed as just /hue/ folder: ln -s /usr/local/hue-3.8.1 /usr/local/hue also make sure you execute "make apps" in the folder to install everything as the hue user...

stevenmatison · ‎03-12-2020

@DarkStar Looks like I already responded to other post with the Regex Mappings I created for you: timestamp : ([0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[1-2][0-9]|3[0-1]) (2[0-3]|[01][0-9]):([0-5][0-9]):([0-9]{2}).([0-9]{6})) .* source_ip : .* (.*) -> .* dest_ip : .* -> (.*) <.* Mac : ^.*] TS\|[0-9]{10}\|([a-fA-F0-9:]{17}|[a-fA-F0-9]{12})\|.*$ dbm : ^.*\|(.*)$ Here is a template I use to extract text and putSql: https://github.com/steven-dfheinz/NiFi-Templates/blob/master/CSV_to_SQL_Demo.xml The demo is a simple 2 column csv but the concept is the same. Send in the text, split the text, get the values from each line, create an insert query using the values, and execute it.

stevenmatison · ‎03-12-2020

I know some other people with big time Regex Skills can map it differently but the method I use it to get single values using strategic wildcard like searches in ReplaceText like: timestamp : ([0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[1-2][0-9]|3[0-1]) (2[0-3]|[01][0-9]):([0-5][0-9]):([0-9]{2}).([0-9]{6})) .* source_ip : .* (.*) -> .* dest_ip : .* -> (.*) <.* Mac : ^.*] TS\|[0-9]{10}\|([a-fA-F0-9:]{17}|[a-fA-F0-9]{12})\|.*$ dbm : ^.*\|(.*)$ I also test using a regex tester like this one: REGEX

stevenmatison · ‎03-12-2020

Per documentation, the start command is: /usr/local/hue/build/env/bin/supervisor If it doesn't start, make sure you have completed the "make apps" command and installed all dependencies. Also depending on your system you want to chown -R hue:hue /usr/local/hue

stevenmatison · ‎03-10-2020

@TCloud use /etc/hosts to set the fqdn you want to the ip you want. When this host mapping and the hostname is set, then reboot and check again that everything stays as you set it after reboot when it comes back online.

stevenmatison · ‎03-09-2020

@YerFel I think you still need to use the truststore or keystore method. Try to test jdbc connection string non ssl vs ssl with another tool until you get the right connection string, then move to NIFI Execution. At Nifi level make sure the cert file(s) are owned to nifi user. Also watch the nifi log during testing if not already. That log may provide additional hints to the client error.

stevenmatison · ‎03-09-2020

[hiveServer2.async.multi_dest.batch_hiveServer2.async.multi_deso log file. java.net.ConnectException: Call From xxx.xxx.xxx/10.21.16.60 to localhost:8020 ed; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused Check your permissions for Hive User: Ranger? MetaStore? Also something I try to avoid is using or letting components run as "localhost". If you can, always using the fqdn for all services. In clustered environment this fqdn is required. "Localhost" can be the wrong "host" depending on which node you are working in.

stevenmatison · ‎03-09-2020

Have you considered creating a copy of the table as external (in new location you want) and then INSERT into NewTable SELECT * FROM OldTable?

stevenmatison · ‎03-06-2020

@Ivan_M93 Great question!!! I personally prefer the Kafka method. This allows nifi to scale against a separate scale of Kafka (assuming not in same cluster). This also decouples the data from hdfs (which is also usually separate from nifi) at the point of ingestion. With this method you have tons of options to decouple the processing after ingestion and take advantage of the basics of Kafka (pub/sub) to avoid processing duplicate data. That said the other option is arguably good as well as it can be done without Kafka. The decision then becomes what components you have available and comfortable to your team.

stevenmatison · ‎03-05-2020

@Vikas_Samant looks like this might be the issue Exception in thread "main" java.lang.RuntimeException: java.lang.ClassNotFoundException: com.mysql.jdbc.jdbc2.optional.MysqlDataSource Did you install the mysql JDBC? In a demo yesterday (centos/rhel) that looks like: yum install mysql-connector-java ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

stevenmatison · ‎03-05-2020

You need to use the cert to create truststore files and deliver these files to all of the nifi nodes. These files needed to be owned nifi:nifi as well. Then provide the path to these files to the controller service: SSLContextService. Then configure your Processor to use this SSLContextService. An example truststore command looks like: /usr/jdk64/jdk1.8.0_112/bin/keytool -import -file /etc/nifi/ssl/file.crt -alias nifi -keystore /etc/nifi/ssl/truststore.jks

stevenmatison · ‎03-05-2020

Look for additional errors in /var/log/hue/error.log when using the WebUI. The client sider errors within the UI may not be very helpful to trouble shoot.

Online	Offline
Last Visited	‎03-27-2020 06:22 AM

Date Registered	‎07-19-2018 04:45 PM
Last Visited	‎03-27-2020 06:22 AM
Total Messages Posted	314
Total Kudos Received	54

Re: Transfer files based on the YYYYMMDD in their ...

Re: why HDP-3.0 version not showing Hive view..? ...

Re: JSON file with operation values ADD/MODIFY ba...

Re: How do I write the Regular Expression from the...

Re: Hue 3.8.1 on centos 7 - How to start it

Re: Transfer files based on the YYYYMMDD in their ...

Re: Unable to Install Ranger.

Re: Date is not parsed as expected in Csv Reader P...

Re: Is it possible to create a table in mysql usin...

Re: How to install Hue 3.11 in HDP 3.1

Re: why HDP-3.0 version not showing Hive view..? ...

Re: How to install Hue 3.11 in HDP 3.1

Re: Unable to run Hive query through putty

Re: Error SQL statement in PutSQL processor not ro...

Re: JSON file with operation values ADD/MODIFY ba...

Re: NIFI 1.4.0 gets unresponsive after heavy load

Re: Can't access file manager using HUE admin ( 4....

Re: Hue 3.8.1 on centos 7 - How to start it

How to install Hue 3.11 in HDP 3.1

Re: page through results - InvokeHTML process thr...

Re: page through results - InvokeHTML process thr...

Re: Hue 3.8.1 on centos 7 - How to start it

Re: Reverse DNS (PTR) and multiple hostnames.

Re: Hue 3.8.1 on centos 7 - How to start it

Re: Extract attribute from a textfile and push it ...

Re: How do I write the Regular Expression from the...

Re: Hue 3.8.1 on centos 7 - How to start it

Re: Reverse DNS (PTR) and multiple hostnames.

Re: Nifi and Postgresql with certificate: how to c...

Re: Hive select query error in Beeline

Re: Change Hive database location.

Re: Streaming tweets with kafka or not.

Re: Failed to start SAM and schema registry servic...

Re: Nifi and Postgresql with certificate: how to c...

Re: HUE desktop problem - can not open oozie workf...