Hi, can't guarantee this, but the global index in Phoenix is as per documentation for 'heavy read' usage. So if this should make sense, your query should result in using the secondary index (of course only when using Phoenix). Regards Harald ... View more

Hi @Armanur Rahman, from the error message it seems to be a simple syntax error. From there I think this is the statement with the error: CREATE TABLE IF NOT EXISTS `hr.f_company_xml_tab` ( `RECID` STRING, `XMLTYPE.GETSTRINGVAL(XMLRECORD)` STRING) your second column is tried to be named 'XMLTYPE.GETSTRINGVAL(XMLRECORD)' which includes a '(' just as the error message claims. Can you rename the column to an easier name i.e. 'val', and try again? Regards Harald ... View more

Hi, so the Spark jobs have not finished during the few hours? They have all been hanging? Or did they finish, though there are errors with NoRouteToHost logged? And you shut down a full physical server, which means you 'loose' 4 VMs of 8 VMs at the same time and not just one VM on the server? To make sure that you don't loose data in this event (its a 50% loss), you will need to make your cluster 'rack' aware, so that the replications are ensured to be created in both racks (set the rack correctly in your host overview in Ambari, don't leave it with 'default-rack'). Otherwise the default cluster config is supposed to continue without data loss when max. 2 data nodes are lost as the replication factor is 3. Loosing 4 machines can leave you with data losses that the cluster can't recover, some files/data might be fully located on the machines being shut down. Regards Harald ... View more

Hi Ken, the filesystem /var/lib/docker/tmp is located on the docker host, so dependent on how you install it, this you AWS centos or maybe the Sandbox itself, where a docker is run. So check there on the size of the filesystem (df -h /var/lib/docker/tmp). If you run the sandbox there might even be restriction on the VirtualBox executing the Sandbox. Regards Harald ... View more

Hi @Abhimanyu Dasarwar, where does it show your memory to be 7.7 GB? Regards Harald ... View more

Hi @Pat ODowd, what exactly means the IP address becomes invalid? Will there be another machine getting that IP address? And the machine gets after a restart a new IP address? No route to host is what IP is supposed to respond when the host with that IP disappears from the network. In any case putting the machine down, will have an impact on currently running jobs, which is different to shutting down the service in the Ambari UI, where running jobs will finish and new jobs will be handled by other nodes. Bringing down the machine will interrupt currently active jobs, though it is supposed to be recovered after the TCP timeout. The cluster control needs to become aware of the 'lost' node as well before letting other nodes handle the jobs. UDP communication will simply be lost, while TCP communication will take 3-5 frames before setting the connection to broken. This takes some seconds (dependent on the settings typically between 30 and 90 seconds). Determining that a port isn't listening on an active machine while trying to connect is much faster than determining a TCP connection is broken, due to the specifics of TCP. So I guess the main point is why doesn't your jobs recover without intervention? How long did you wait for it? Regards Harald ... View more

Hi @Shesh Kumar, I don't think you can configure this via Ambari (but not sure anyway). What you could do is either write a replication in Hbase or simply configure both as one cluster making them rack aware to ensure the data is replicated in both racks (with different locations). Regards Harald ... View more

Hi @Praveen Kumar, this is a quite generic question, so a precise answer is difficult. But given the fact that you want to get the data from a RDBMS I think you can go with one cluster. What you need to consider is how much througput you will have to handle, typically the RDBMS will limit this anyway. Just for consideration: Spark and Flink are merely RAM intensive, while HBase uses HDD and RAM, dependent on the load. Hive again uses mainly HDD for M&R. But if you plan to create an external Hive table pointing to HBase, you are again in the Hbase usage pattern. Assuming you have a sufficient RAM available on your nodes (i would go with >= 2 Gb per PCU core) I think one cluster for all would do. If later the load increase, you can scale the cluster, one of the big advantages of Hadoop. Regards Harald ... View more

on your dev server, do you have any hive table defined that you can query? What actually happens when you are querying the table in hive? ... View more

To upload the file from your Windows machine to a Linux machine, you can use a tool like WinSCP. You configure the session for the Linux machine almost identical to the config in Putty. It gives you a GUI to copy files. On the other hand, when you need to access the Windows machine from Linux, you need to configure an FTP or better SFTP server on Windows that allows access to your NTFS path. Or you use the Windows Network to share, and install Samba, a Windows networking implementation, on the Linux machine. ... View more

I am a little guessing here, but I believe its possible that the Hive metastore has statistics (i.e. information on the number of records in the partitions), so that the count might actually not read the complete table. The count on the file must read the file in any case. But still i think 12 min are really long for processing 3.8 GB, even if this is the compressed size. Is the count the very first action on the data frame? So that Spark only executes all previous statements (i guess reading the file, uncompressing it etc) when running the count? ... View more

HI Ajay, here is a sizing guide, which seems to address exactly your questions: https://community.hortonworks.com/articles/135337/nifi-sizing-guide-deployment-best-practices.html Still i personally wouldn't start with 8Gb RAM per node but at least with 16GB (2 GB per core). Anyway you will have to be clear on the throughput needed (Gb/sec.), not only on the overall volume. Regards Harald ... View more

Hi Sindhu, do you get any error message when it fails? Regards Harald ... View more

Can you perhaps also let us know how you try to read the file and the hive table? Also where is the file stored? ... View more

it really depends on how you mapped the primary key during migration to Hbase. If you mapped it into a column pk within the column familiy, you will not see the pk column as from the original table (i.e. oracle). I suggest you try it as below, which would map the Hbase rowkey into the column "rowkey" in your Phoenix table and would also map the column PK from the column family in case it exists. CREATE VIEW "FNL.ADDRESSES" ( rowkey VARCHAR PRIMARY KEY, "cf".HJMPTS DECIMAL(20,0), "cf".CREATEDTS TIME, "cf".MODIFIEDTS TIME, "cf".TYPEPKSTRING DECIMAL(20,0), "cf".OWNERPKSTRING DECIMAL(20,0), "cf".PK DECIMAL(20,0), "cf".P_ORIGINAL DECIMAL(20,0), "cf".P_DUPLICATE DECIMAL(1,0), "cf".P_APPARTMENT VARCHAR(255), "cf".P_BUILDING VARCHAR(255), "cf".P_CELLPHONE VARCHAR(255), "cf".P_COMPANY VARCHAR(255), "cf".P_COUNTRY DECIMAL(20,0), "cf".P_DEPARTMENT VARCHAR(255), "cf".P_DISTRICT VARCHAR(255), "cf".P_EMAIL VARCHAR(255), "cf".P_FAX VARCHAR(255), "cf".P_FIRSTNAME VARCHAR(255), "cf".P_LASTNAME VARCHAR(255), "cf".P_MIDDLENAME VARCHAR(255), "cf".P_MIDDLENAME2 VARCHAR(255), "cf".P_PHONE1 VARCHAR(255), "cf".P_PHONE2 VARCHAR(255), "cf".P_POBOX VARCHAR(255), "cf".P_POSTALCODE VARCHAR(255), "cf".P_REGION DECIMAL(20,0), "cf".P_STREETNAME VARCHAR(255), "cf".P_STREETNUMBER VARCHAR(255), "cf".P_TITLE DECIMAL(20,0), "cf".P_TOWN VARCHAR(255), "cf".P_GENDER DECIMAL(20,0), "cf".P_DATEOFBIRTH TIME, "cf".P_REMARKS VARCHAR(255), "cf".P_URL VARCHAR(255), "cf".P_SHIPPINGADDRESS DECIMAL(1,0), "cf".P_UNLOADINGADDRESS DECIMAL(1,0), "cf".P_BILLINGADDRESS DECIMAL(1,0), "cf".P_CONTACTADDRESS DECIMAL(1,0), "cf".P_VISIBLEINADDRESSBOOK DECIMAL(1,0), "cf".P_STATE VARCHAR(255), "cf".P_LANDMARK VARCHAR(255), "cf".P_CODELIGIBLE DECIMAL(1,0), "cf".ACLTS DECIMAL(20,0), "cf".PROPTS DECIMAL(20,0), "cf".P_ISHOMEADDRESS DECIMAL(1,0) ); ... View more

Have you setup SSL or Kerberos as the security for your Kafka broker? In that case have a look here: https://community.hortonworks.com/content/supportkb/150148/errorwarn-bootstrap-broker-6668-disconnected-orgap.html ... View more

I think you will not be able to do so with SerDe. The SerDe will read record by record, which is line by line, and then the regex is applied on this record that has been read, making it impossible to span the pattern over multiple lines. One way you could solve it, is try to create a table with SerDe with 1 line for each record and do the combination of multiple lines then via query in Hive. Another way would be to process the input file first outside Hive and write out the line combined in one line as needed for your table. ... View more

can you do a klist on the shell with the same user that starts the HBase master (and post the result here). I am a little curios on the point that the TGT expiry timestamp is the exact time of your start. Are you using a ticket or a keytab for Kerberos? And which is the configured principal for HBase? ... View more

but you still have a column name PK in the column family I guess (I remember that table was migrated from Oracle to Hbase originally)? In that case remove the "primary key" from the column definition. The actual primary key must be outside any column family. ... View more

a Hbase table will only have one rowkey, valid for all columnfamilies. It is the only primary key and it can't belong to any column family. So from here we must be clear on what you mean with 'I have a column name "PK" which is a primary key belonging to column family "cf"'. You might have a column PK in the column family cf, but for sure not as the rowkey (primary key) in Hbase. Within a column family any column is optional. Can you provide a table description of the Hbase table you try to map and the Phoenix view statement? ... View more

How do you initialize/load the Azure libs? Maybe you will have to configure the workdir in your processor as well? Do you have any error log? Anyway it might be a good idea to close this issue (by accepting one answer) and create a new post with the next issue your are facing (your script seems to not connect correctly). ... View more

Would be appreciated if you click on 'accept' for the answer. This will let all know the issue is resolved, and is rewarded by the platform. ... View more