About TimothySpann

TimothySpann · ‎11-02-2016

http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-user-guide/content/using_grafana.html

TimothySpann · ‎11-02-2016

in HDF 2.0.1 it goes to ambari metrics which has grafana http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.1/bk_ambari-user-guide/content/configuring_notifications.html https://cwiki.apache.org/confluence/display/NIFI/Ambari+Metrics see: http://hortonworks.com/blog/hood-ambari-metrics-grafana/

TimothySpann · ‎11-02-2016

can you post the DDL of both tables? Check out my example: https://community.hortonworks.com/articles/64122/incrementally-streaming-rdbms-data-to-your-hadoop.html Are there a lot of nulls? Can you post an example row? What JDK and NIFI versions are you using? What version of Hive and Hadoop are you running? It may be an issue like this when some data is a number and some is an number in quotes https://issues.apache.org/jira/browse/PIG-3407 Maybe this fix is relevant https://community.hortonworks.com/questions/17996/nifi-applying-an-avro-schema-in-convertcsvtoavro.html Do you have the latest HDF 2.01 version? http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.1/index.html

TimothySpann · ‎11-01-2016

Hadoop Installation on RHEL 7.2 Tips Custom Install Tips: Read this article and have it available. Make sure you know your Hadoop Mount Points Follow the networking Best Practices Make sure you follow the Hortonworks HDP minimum requirements. Make sure you have root access and make sure you have yum, rpm, scp, curl, wget, unzip, tar, yum-utils, createrepo, reposync installed, working and in your path. Make sure networking, iptables, proxies and firewalls are all open enough for you to access the Hortonworks repos and the bandwidth is decent. Download the OpenJDK 1.8 64-bit update 51 or higher Make sure you setup passwordless SSH to all machines, including the current machine you are starting from which is the Ambari Server. You may need to SSH to yourself. Make sure you have 20G+ of /var space, 20G+ /usr space and plenty of /tmp space. Many things will go here: /usr/hdp One time I needed to manually install MySQL via sudo yum -y install mysql Any space where you will install your data nodes will need to be owned by hdfs:hadoop and have 755 access. Keep community.hortonworks.com in a browser, you can quickly search and find more answers.

TimothySpann · ‎11-01-2016

you need a different browser

TimothySpann · ‎11-01-2016

If you can use the latest Chrome and make sure there are no issues with Proxies, firewalls or anti-virus software. https://docs.hortonworks.com/HDPDocuments/HDF1/HDF-1.2/bk_AdminGuide/content/how-to-install-and-start-nifi.html

TimothySpann · ‎11-01-2016

what else is running on windows? In your nifi.properties (see the settings in the article) see this for setup: https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html Also the articles give you the details for setting the JVM settings for as much of that 198 GB that you can spare. Any reason you picked Windows? 99% of people run NiFi and other big data servers and services on Linux (including HDInsight) After you change those settings, restart nifi. Watch your logs. Make sure you have the latests JDK 1.8 and plenty of disk space.

TimothySpann · ‎11-01-2016

mergecontent will put all the csv records in one file done

TimothySpann · ‎11-01-2016

https://msdn.microsoft.com/en-us/library/a6cd7c08(v=vs.110).aspx see: https://community.hortonworks.com/questions/64251/can-we-use-hbasecassandra-as-database-for-aspnet-w.html#answer-64375 For C# and .NET https://github.com/hdinsight/hbase-sdk-for-net https://hbasenet.codeplex.com/ For C# Spark https://github.com/Microsoft/Mobius Use HBase for large scale. Do you have to use ASP.NET? Can you move to Java Spring? Scala Play? Scala Spray? NodeJS Express? Ruby on Rails? All of these scale better and run on multiple platforms. Most people store web data to RDBMS like Postgresql and then ingest the data into HDFS and HBase via NiFi, Sqoop, Flume and other tools. Then you run your BI, Zeppelin, Deep Learning, Machine Learning, Spark and analytics workloads on your Hadoop based data lake.

TimothySpann · ‎11-01-2016

For Phoenix, updates and inserts are handled by the Upsert command. If it's the same data it will be updated, if not inserted. 1) For delete, you can call an ExecuteProcess to delete with a command line program. 1b) For delete, you can call ExecuteScript to have Python/Groovy/Javascript delete it. 2) For delete, you can send a message to Spark Streaming to delete it via SparkSQL (using SitetoSite or Kafka) Follows general SQL rules. Apache Phoenix Delete Reference https://phoenix.apache.org/language/index.html#delete

Online	Offline
Last Visited	‎05-20-2024 05:42 PM

Member Since	‎01-07-2019 11:58 AM
Last Visited	‎05-20-2024 05:42 PM
Posts	1,973
Kudos received	1122

Cloudera Community

Re: Has anyone tried NiFi consuming (JMSConsume) f...

Re: NiFi Crash after runing chain of lookups

Re: Recommend approach for listening to RSS Feed i...

Re: NiFi ListenFTP Processor Default Data Port

Re: Nifi: Kafka Producer with Avro format in both ...

Re: Nifi metrics in Grafana/Prometheus

Re: Nifi metrics in Grafana/Prometheus

Re: QueryDatabaseTable - importing from postgresql...

Quick Tips for HDP 2.4+ Installation on RHEL 7.2

Re: NiFi UI Response is Very slow

Re: NiFi UI Response is Very slow

Re: NiFi UI Response is Very slow

Re: Convert JSON to CSV using Nifi

Re: Hadoop can be used as a database for Asp.Net w...

Re: Incrementally Streaming RDBMS Data to Your Had...