Hello,   Hive compaction runs first, then a cleaner thread waits for all readers to finish reading the old base/delta files. When it determines that nobody is reading the old files anymore, only then does the deletion of the old files occur. Please run SHOW COMPACTIONS and look at the state of the compaction in question.   Also would help to know which version are you on for CDH/HDP/CDP? If CDP, public or private cloud?   Hope this helps, Alex ... View more

Hi Nuno,   I've sent you a private message to get some more information. Please respond there and we'll figure out how to help.    Regards, Alex ... View more

@cjervis should be able to help with this. ... View more

Hi Igor,   You can define what users can and cannot do in Atlas by way of defining authorization policies in Ranger. Details on how to do that can be found here: https://docs.cloudera.com/runtime/7.2.6/atlas-securing/topics/atlas-configure-ranger-authorization.html   What you refer to as Bookmarks can potentially be done via Saved Searches (see here), depending on what you want to achieve. As for the popularity score, this could be made a metadata attribute that can be updated by users. There is no automation to derive this score with Atlas out-of-the-box.   Hope this helps, Alex ... View more

That is an odd behaviour. Two things to try: In Cloudera Manager go hdfs -> Configuration -> Enable Log and Query Redaction and switch that to false. Go to HBase -> Configuration -> Enable Audit Collection and turn it off.  Restart the HBase service and run your sqoop job again. See if that helps with performance.  ... View more

Two things to check: 1. Does your nifi service user account have permissions on the table and hdfs location where it's trying to do the insert; 2. Your Hive SQL statement here looks a bit off to me: insert into transformed_db.tbl_airbnb_listing_transformed select a.*, 20210113 partition_date_id from staging_db.etbl_raw_airbnb_listing a Is 20210113 a column name? Are you missing a comma between that and parition_date_id? Is your source staging table partitioned? If you are trying to select only a specific date, than the syntax to do that is different.  ... View more

There could be a problem with running the SQL in your Hive cluster. I would suggest checking the Hive logs for any relevant errors when NiFi flow is triggered. Another thing to check is the FlowFile itself to make sure it has the data that Hive table would be expected (i.e. schema matches). You can do this by forwarding the failed FlowFiles to a rejected flow.     ... View more

Settings look fine. _HOST gets replaced by the actual FQDN of the host at runtime. One thing to check is to make sure reverse DNS lookup works on all hosts.    ... View more

Out-of-the-box Hue can't properly parse this format. There are some potential solutions in this thread: https://stackoverflow.com/questions/13628658/hive-load-csv-with-commas-in-quoted-fields and it depends on what you are comfortable with: pre-processing the file to reformat the input or to use a different SerDe in Hive.    Hope that helps, Alex ... View more

This could be as simple as a typo somewhere in your configs, but it's hard to tell where. Looks like the username was set to "host" somehow and hdfs can't authenticate that as a user. Without more context it's hard to say any more. ... View more

Can you show the first couple lines of your file exactly as they appear in the file. You can open the CSV with a simple text editor of your choice and show the output in a comment here.    When you are in the upload screen in Hue, note that under Extras section there are additional parameters that you might need to adjust to fit your file formatting. ... View more

I was able to reproduce this error and it looks like the problem is the identical column name in your tableA and tableB. Namely, DateColumn is referenced in the subquery. Hive interprets this as a reference to the parent query which is not allowed (per limitation listed here). Essentially it's confused what you mean by this query due to overloaded column name.   To solve this, you can explicitly specify table names when referring to columns: UPDATE tableA SET tableA.ColA = "Value" WHERE year(tableA.DateColumn) >= ( select (max(year(tableB.DateColumn))-1) from tableB )   Let me know if this works. Regards, Alex ... View more

This could be a good start: https://community.cloudera.com/t5/Support-Questions/Using-NiFi-to-load-data-from-localFS-to-HDFS/td-p/212124   ... View more

You also want to check the structure of your raw data. Specifically, look for any instances where there are extra delimiters (e.g. a string field that includes commas as part of the string).  ... View more

Ok, fair enough. There is also a CDH specific connector for Teradata (available here https://www.cloudera.com/downloads/connectors/sqoop/teradata/1-7c6.html). Try that. The installation and usage guide is here: https://docs.cloudera.com/documentation/other/connectors/teradata/1-x/PDF/Cloudera-Connector-for-Teradata.pdf ... View more

Your connection string is "teradat" instead of "teradata" - this can also lead to parsing error.    Otherwise, have you tried exporting the entire table, without giving specific columns? ... View more

The issue is not with --columns parameter, actually. Problem is the sqoop can't parse the command because it expects "-m" instead of "--m". Remember, when using a short-form parameter that is a single letter, use a single dash, otherwise use double dash.      Hope this helps!   ... View more

The error indicates an issue with Kerberos. If things ran fine last week, perhaps your Kerberos ticket expired and needs to be renewed.  ... View more

Can you share a little more about your use case? Are you only appending to the table or also updating records (UPSERT)? Will there be duplicate records? Also, what version of Hive are you working with? ... View more