Member since
01-15-2019
33
Posts
27
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
632 | 07-20-2021 01:05 AM | |
7985 | 11-28-2019 06:59 AM |
01-12-2023
02:16 AM
1 Kudo
To install Apache NiFi on Macbook, do the following:
Download the Zip file from Apache nifi Downloads. Ensure that you download the latest version zip file Apache NiFi Binary 1.19.1 [OpenPGP] [SHA-256] [SHA-512].
Download it to MBP, and then unzip it:
chmod +x bin/nifi.sh
Run the following command for User Generate:
bin/nifi.sh set-single-user-credentials zzeng <my long password>
Run the following command to Start:
bin/nifi.sh start bin/nifi.sh status netstat -an | grep LIST
Check the 8443 port is LISTEN:
tcp4 0 0 127.0.0.1.8443 * . * LISTEN
Login from https://localhost:8443/nifi/
... View more
Labels:
11-13-2022
08:07 AM
https://lists.apache.org/thread/907n11xlvdmckp1045bspzloclthfqsh As NiFi is a pure Java/JVM application, we use Jython rather than
Python for ExecuteScript. This means that you can't import native
(CPython, e.g.) modules into your Jython scripts in ExecuteScript consider using ExecuteStreamCommand with a real Python
interpreter and script. I'm looking at Py4J to try and bridge the gap
... View more
04-10-2022
10:47 PM
The latest document about Airflow in Data Engineering Experience (CDE) within Public Cloud and CDP Private Cloud Data Service is here: CDP Public Cloud https://docs.cloudera.com/data-engineering/cloud/orchestrate-workflows/topics/cde-airflow-dag-pipeline.html CDP Private Cloud Data Service https://docs.cloudera.com/data-engineering/1.3.3/orchestrate-workflows/topics/cde-airflow-dag-pipeline.html
... View more
07-20-2021
01:05 AM
Sorry for my late reply. > what AMI are you using and were there any pre-requisites for OS ? I used the CentOS 7.8, like the matrix showed. https://supportmatrix.cloudera.com/#Cloudera To download the CDP, if you are not paying for CDP, you can try it from here. https://docs.cloudera.com/cdp-private-cloud/latest/release-guide/topics/cdpdc-trial-download-information.html The trial version has the same functions with normal CDP but can only use 60 days. Please have a try and let me know if it works.
... View more
07-09-2021
12:01 AM
Hi Sam, I am also use CDP on AWS for learning. If you don't have enough AWS credit budget, you can use 3 nodes in this way, Use these 3 nodes as both Master and worker nodes. Means put NameNode and DataNode in these nodes. Put DataNode in all the 3 nodes.
... View more
07-08-2021
11:57 PM
I met similar errors, but with different reasons. My reason was, I created a leaf queue behind default queue. Then after Hive restart, it will submit a query to check the health status, however as default queue is not a leaf queue any more, then it went error. My solution was , I removed the child queue behind default queue, then the error resolved.
... View more
04-20-2021
05:06 AM
Here is a URL, written by David W. Streever https://www.streever.com/post/2019/filter-hive-compactions/ It's not official steps. All the ‘extras’ you create against the metastore DB can/may break with the next release. This isn’t a supported method of accessing metadata.
... View more
01-26-2021
08:51 AM
https://nifi.apache.org/docs/nifi-docs/components/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.9.0/org.apache.nifi.processors.gcp.bigquery.PutBigQueryBatch/index.html With this processor, you can do Batch loads flow files content to a Google BigQuery table.
... View more
04-07-2020
07:06 AM
3 Kudos
1) Demarcator, use ${literal(' '):unescapeXml()} 2) Delimiter Strategy = Text
... View more
11-28-2019
06:59 AM
10 Kudos
If you meet this problem in new installing CM+CDP/CDH it can be resolve in this way: in CM, open Solr page, do the 2 actions: Initialize Solr / Create HDFS Home Dir
... View more
10-01-2019
03:00 AM
Or you can also use the option to do this: rs.write.option("compression", "uncompressed").parquet("/user/output01/")
... View more
10-01-2019
02:18 AM
I tried in my CDH 6.3, found no errors. scala> spark.sql("set spark.sql.parquet.compression.codec=uncompressed")
res8: org.apache.spark.sql.DataFrame = [key: string, value: string]
scala>
... View more
07-27-2019
05:16 AM
It worked. Thanks!
... View more
07-16-2019
01:50 PM
https://www.thegeekdiary.com/centos-rhel-7-how-to-disable-ipv6-on-a-specific-interface-only/ # vi /etc/sysctl.conf
net.ipv6.conf.eth0.disable_ipv6 = 1 and then restart you Ambari server: sudo ambari-server restart
... View more
02-17-2019
04:33 PM
Reference URL: https://atlas.apache.org/Hook-Sqoop.html Link <atlas package>/hook/sqoop/*.jar in sqoop lib In HDP, you can find all the hook jar files in atlas hook folder. hook folder: /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl So, after building symbolic link for these jar files in sqoop folder, This problem was resolved. https://gist.github.com/zz22394/6c004731423fb11095aa41ac19de3393 sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/atlas-client-common-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-server/lib/zz-atlas-client-common-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/atlas-client-v1-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-server/lib/zz-atlas-client-v1-1.1.0.3.1.0.0-78.jar
sudo ln -s /usr/hdp/current/atlas-server/hook/sqoop/atlas-sqoop-plugin-impl/atlas-client-v2-1.1.0.3.1.0.0-78.jar /usr/hdp/current/sqoop-server/lib/zz-atlas-client-v2-1.1.0.3.1.0.0-78.jar
... View more
10-10-2018
02:20 PM
Hi @Jeff Storck, Thanks for telling me this. > To create an HDF cluster with Cloudbreak, a KDC must be configured After using a test KDC , the CB 2.7 can provision HDF 3.1 successfully.
... View more
10-08-2018
06:45 AM
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'loginIdentityProvider': FactoryBean threw exception on object creation; nested exception is java.lang.Exception: The specified login identity provider 'kerberos-provider' could not be found. CloudBreak 2.7 built HDF 3.1 has error: 'kerberos-provider' could not be found. The reason is HDF 3.1's blueprint has problem: nifi.security.user.login.identity.provider= is set to kerberos-provider but that provider doesn't exist in login-identity-providers.xml, or is commented out
... View more
- Find more articles tagged with:
- Cloudbreak
- Data Ingestion & Streaming
- hdf-3.1
- Issue Resolution
- login
09-30-2018
03:46 AM
Got error [User home directory not found] when creating WorkFlow Manager view in Ambari Resolve: sudo -u hdfs hdfs dfs -mkdir /user/admin
sudo -u hdfs hdfs dfs -chown admin /user/admin
Resule:
... View more
- Find more articles tagged with:
- ambari-views
- Data Processing
- Issue Resolution
- Oozie
Labels:
05-27-2018
09:02 AM
In Ambari, the default value is /etc/ranger/admin/conf/ranger-admin-keystore.jks
... View more
04-30-2018
12:49 PM
It works! Thanks for your info. yum install -y epel-release
... View more
03-22-2018
09:19 AM
Latest Nifi Monitoring docs: (HDF 3.1.1) https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_getting-started-with-apache-nifi/content/monitoring-nifi.html
... View more
03-21-2018
01:51 PM
In Zeppelin 0.7.0 Document, it is said that library should use 0.13. Reference : How to execute HIVE LLAP queries from Zeppelin in HDP 2.6.1 Hive Interpreter for Apache Zeppelin However it is too old. For LLAP, it need to be newer one. Properties :
hive2.driver = org.apache.hive.jdbc.HiveDriver
hive2.url = < can be obtained from ambari. check HiveServer2 Interactive JDBC URL >
eg.
hive2.url = jdbc:hive2://zzeng-hdp-3.example.com:2181,zzeng-hdp-1.example.com:2181,zzeng-hdp-2.example.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-hive2
Dependencies :
org.apache.hive:hive-jdbc:2.2.0
org.apache.hadoop:hadoop-common:2.6.0
With this setting, I can access Hive LLAP now:
... View more
- Find more articles tagged with:
- Data Science & Advanced Analytics
- FAQ
- llap
- zeppelin
Labels:
03-21-2018
01:31 PM
Before reading this post, I tried official document and failed. https://zeppelin.apache.org/docs/0.7.0/interpreter/hive.html Then I found my problem - I should set the correct properties with interpreter name.
... View more
03-14-2018
03:52 PM
1 Kudo
The article describes how to install and configure Sparkmagic to run in HDP2.5 against Livy Server and Spark 1.6.2 Reference: Using Jupyter with Sparkmagic and Livy Server on HDP 2.5 in HCC
1) Install Jupyter http://jupyter.org/install ###### Init venv
## Just first time
sudo yum install python-pip python-dev python-virtualenv -y
mkdir ~/jupyter_env
## After 2nd
virtualenv --system-site-packages ~/jupyter_env
source ~/jupyter_env/bin/activate
curl -O https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py
sudo easy_install -U pip
python -m pip install --upgrade pip
# for jupyter, it need gcc
sudo yum install gcc -y
sudo pip install jupyter notebook ipython http://zzeng-hdp-ambari:8888/tree?token=17dfdcb7525ff7470a637752450bbd586f607eddccc86a7f
2) Use Jupyter to connect Spark Livy https://community.hortonworks.com/articles/70501/using-jupyter-with-sparkmagic-and-livy-server-on-h.html # Failed building wheel for pykerberos
sudo yum install krb5-devel -y
sudo -H pip install sparkmagic
sudo pip install hdijupyterutils
sudo pip install autovizwidget
sudo pip install sparkmagic
pip show sparkmagic
pip show autovizwidget
cd /usr/lib/python2.7/site-packages
jupyter-kernelspec install --user sparkmagic/kernels/sparkkernel
jupyter-kernelspec install --user sparkmagic/kernels/pysparkkernel
sudo -H jupyter nbextension enable --py --sys-prefix widgetsnbextension
3) Start Notebook cd ~/
jupyter notebook --ip=0.0.0.0 4) Connect from Jupyter to remote Spark cluster Readme: https://github.com/jupyter-incubator/sparkmagic In[ ]: %load_ext sparkmagic.magics In[ ]: %manage_spark
... View more
- Find more articles tagged with:
- Data Science & Advanced Analytics
- How-ToTutorial
- jupyter
- Spark
Labels:
02-24-2018
02:42 PM
2 Kudos
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install mysql-community-release' returned 1. Error: Nothing to do Ambari got the error above. Try install mysql directly: [centos@zzeng-hdp-2 ~]$ sudo yum install mysql-community-release
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirror.web-ster.com
* extras: mirror.hostduplex.com
* updates: mirrors.sonic.net
No package mysql-community-release available.
Error: Nothing to do https://bugs.mysql.com/bug.php?id=81037 Try MySQL 5.7 [centos@zzeng-hdp-2 ~]$ sudo rpm -ivh http://dev.mysql.com/get/mysql57-community-release-el7-8.noarch.rpm
Retrieving http://dev.mysql.com/get/mysql57-community-release-el7-8.noarch.rpm
warning: /var/tmp/rpm-tmp.hGWJu5: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql57-community-release-el7-8 ################################# [100%]
[centos@zzeng-hdp-2 ~]$
This still didn't resolve the problem. At last, I tried to use MySQL 5.6 ( http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm ) [centos@zzeng-hdp-2 ~]$ sudo yum remove mysql57-community-release
Loaded plugins: fastestmirror
Resolving Dependencies
--> Running transaction check
---> Package mysql57-community-release.noarch 0:el7-8 will be erased
--> Finished Dependency Resolution
Dependencies Resolved
============================================================================================================================================================================================================================================
Package Arch Version Repository Size
============================================================================================================================================================================================================================================
Removing:
mysql57-community-release noarch el7-8 installed 8.2 k
Transaction Summary
============================================================================================================================================================================================================================================
Remove 1 Package
Installed size: 8.2 k
Is this ok [y/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Erasing : mysql57-community-release-el7-8.noarch 1/1
warning: /etc/yum.repos.d/mysql-community.repo saved as /etc/yum.repos.d/mysql-community.repo.rpmsave
Verifying : mysql57-community-release-el7-8.noarch 1/1
Removed:
mysql57-community-release.noarch 0:el7-8
Complete!
[centos@zzeng-hdp-2 ~]$ sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-release-el7-5 ################################# [100%]
[centos@zzeng-hdp-2 ~]$
... View more
- Find more articles tagged with:
- ambari-server
- Cloud & Operations
- Issue Resolution
- issue-resolution
- MySQL
Labels:
02-02-2018
09:28 PM
9 Kudos
resource_management.core.exceptions.Fail: Failed to download file from http://ambari1.hdp.hadoop:8080/resources//mysql-connector-java.jar due to HTTP error: HTTP Error 404: Not Found By the following command, the problem is fixed: Ambari Server:(Ref: https://discuss.pivotal.io/hc/en-us/articles/115001611807-Hive-Services-Fail-to-Start-giving-Error-HTTP-Error-404-Not-Found- ) sudo yum install mysql-connector-java*
ls -al /usr/share/java/mysql-connector-java.jar
cd /var/lib/ambari-server/resources/
ln -s /usr/share/java/mysql-connector-java.jar mysql-connector-java.jar
... View more
- Find more articles tagged with:
- ambari-server
- Data Processing
- Issue Resolution
- issue-resolution
- MySQL
Labels:
02-02-2018
12:25 PM
1 Kudo
Thanks. It works for me. sudo yum-config-manager --enable rhui-REGION-rhel-server-optional sudo yum install libtirpc-devel
... View more
01-16-2018
08:42 AM
Yes, you can. https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.1.1/bk_planning-your-deployment/content/ch_deployment-scenarios.html According to the URL above, you can Installing HDF Services on an Existing HDP Cluster. Detail URL: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.1.1/bk_installing-hdf-on-hdp/content/upgrading_ambari.html
... View more
01-16-2018
08:22 AM
You can take a look at this post: https://community.hortonworks.com/articles/10485/introduction-to-presto.html Then let me answer you question: Q1)How we can come to know what is optimum size of memory/ram allocation. A) You can get the value from presto UI. http://<host-name>:8080/ All the queries are listed here. If you click one of them, you can see details , such as peak memory usage. Q3) Is there any way to restrict user from launching such a huge query or any other preventive approach A) Yes. ref url config.properties example: query.max-memory=30GB
query.max-memory-per-node=5GB
... View more