sc = SparkContext() sqlContext = SQLContext(sc) try: df = sqlContext.createDataFrame(jsonobj) except IOError: logger.exception(jsonobj) schema = df.printSchema() sc.stop() return schema The above code throws, cannot infer schema on empty dataset for some datasets. What does this error mean? How do I fix this? df = sqlContext.createDataFrame(jsonobj) File "/remote/vgrnd104/guntaka/anaconda3/lib/python3.6/site-packages/pyspark/sql/context.py", line 302, in createDataFrame return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema) File "/remote/vgrnd104/guntaka/anaconda3/lib/python3.6/site-packages/pyspark/sql/session.py", line 691, in createDataFrame rdd, schema = self._createFromLocal(map(prepare, data), schema) File "/remote/vgrnd104/guntaka/anaconda3/lib/python3.6/site-packages/pyspark/sql/session.py", line 410, in _createFromLocal struct = self._inferSchemaFromList(data, names=schema) File "/remote/vgrnd104/guntaka/anaconda3/lib/python3.6/site-packages/pyspark/sql/session.py", line 337, in _inferSchemaFromList raise ValueError("can not infer schema from empty dataset") ValueError: can not infer schema from empty dataset ... View more

I was able to GET schema fine. But to POST a new schema, I need some clarification. How do I post a schemaobj with a schema name using the REST? The below command throws error curl -X POST "http://<registry-host>:7788/api/v1/schemaregistry/schemas" -H "accept: application/json" -H "Content-Type: application/json" -d "{ \"type\": \"avro\", \"schemaGroup\": \"myproject\", \"name\": \"testschema\", \"description\": \"testing schema\", \"compatibility\": \"NONE\", \"validationLevel\": \"LATEST\"}" Unmatched ". ... View more

I have registered the schema for my data manually using the Registry UI and then started ingesting data. If I have to validate the schema against the version registered or identify new schemas that may come during ingestion, how do I automate the schema generation part (the schema file)? ... View more

Executing the below command to start streamline gives error. ./streamline-server-start.sh /etc/streamline/conf/streamline.yaml Streamline.yaml # Jar storage configuration fileStorageConfiguration: className: com.hortonworks.registries.common.util.LocalFileSystemStorage properties: directory: /hdf/streamline/jars dashboardConfiguration: url: "http://<streamline_host>:9088" storageProviderConfiguration: properties: db.properties: dataSource.password: "mysql_streamline_password" dataSource.url: "jdbc:mysql://<mysql_host>:3306/streamline" dataSource.user: "streamline" dataSourceClassName: "com.mysql.jdbc.jdbc2.optional.MysqlDataSource" #dataSourceClassName: "com.mysql.jdbc.Driver" db.type: mysql queryTimeoutInSecs: 30 providerClass: "com.hortonworks.registries.storage.impl.jdbc.JdbcStorageManager" Error Exception in thread "main" com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Communication link failure: Bad handshake at com.zaxxer.hikari.pool.HikariPool.throwPoolInitializationException(HikariPool.java:544) at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:536) at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112) at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72) at com.hortonworks.registries.storage.impl.jdbc.connection.HikariCPConnectionBuilder.prepare(HikariCPConnectionBuilder.java:49) at com.hortonworks.registries.storage.impl.jdbc.connection.HikariCPConnectionBuilder.<init>(HikariCPConnectionBuilder.java:38) at com.hortonworks.registries.storage.impl.jdbc.provider.QueryExecutorFactory.getHikariCPConnnectionBuilder(QueryExecutorFactory.java:79) at com.hortonworks.registries.storage.impl.jdbc.provider.QueryExecutorFactory.get(QueryExecutorFactory.java:45) at com.hortonworks.registries.storage.impl.jdbc.JdbcStorageManager.init(JdbcStorageManager.java:240) at com.hortonworks.streamline.webservice.StreamlineApplication.getStorageManager(StreamlineApplication.java:194) at com.hortonworks.streamline.webservice.StreamlineApplication.getDao(StreamlineApplication.java:183) at com.hortonworks.streamline.webservice.StreamlineApplication.registerResources(StreamlineApplication.java:215) at com.hortonworks.streamline.webservice.StreamlineApplication.run(StreamlineApplication.java:101) at com.hortonworks.streamline.webservice.StreamlineApplication.run(StreamlineApplication.java:75) at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:43) at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:85) at io.dropwizard.cli.Cli.run(Cli.java:75) at io.dropwizard.Application.run(Application.java:79) at com.hortonworks.streamline.webservice.StreamlineApplication.main(StreamlineApplication.java:79) Caused by: java.sql.SQLException: Communication link failure: Bad handshake at com.mysql.jdbc.MysqlIO.init(Unknown Source) at com.mysql.jdbc.Connection.connectionInit(Unknown Source) at com.mysql.jdbc.jdbc2.Connection.connectionInit(Unknown Source) at com.mysql.jdbc.Driver.connect(Unknown Source) at com.mysql.jdbc.jdbc2.optional.MysqlDataSource.getConnection(Unknown Source) at com.mysql.jdbc.jdbc2.optional.MysqlDataSource.getConnection(Unknown Source) at com.mysql.jdbc.jdbc2.optional.MysqlDataSource.getConnection(Unknown Source) at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:356) at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:199) at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:444) at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:515) ... 17 more ... View more

When using mysql for streaming Analytics Manager, it fails to start with the below error. Exception in thread "main" java.lang.RuntimeException: java.lang.ClassNotFoundException: com.mysql.jdbc.jdbc2.optional.MysqlDataSource at com.zaxxer.hikari.util.UtilityElf.createInstance(UtilityElf.java:93) at com.zaxxer.hikari.pool.PoolBase.initializeDataSource(PoolBase.java:319) at com.zaxxer.hikari.pool.PoolBase.<init>(PoolBase.java:114) at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:105) at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72) at com.hortonworks.registries.storage.impl.jdbc.connection.HikariCPConnectionBuilder.prepare(HikariCPConnectionBuilder.java:49) at com.hortonworks.registries.storage.impl.jdbc.connection.HikariCPConnectionBuilder.<init>(HikariCPConnectionBuilder.java:38) at com.hortonworks.registries.storage.impl.jdbc.provider.QueryExecutorFactory.getHikariCPConnnectionBuilder(QueryExecutorFactory.java:79) at com.hortonworks.registries.storage.impl.jdbc.provider.QueryExecutorFactory.get(QueryExecutorFactory.java:45) at com.hortonworks.registries.storage.impl.jdbc.JdbcStorageManager.init(JdbcStorageManager.java:240) at com.hortonworks.streamline.webservice.StreamlineApplication.getStorageManager(StreamlineApplication.java:194) at com.hortonworks.streamline.webservice.StreamlineApplication.getDao(StreamlineApplication.java:183) at com.hortonworks.streamline.webservice.StreamlineApplication.registerResources(StreamlineApplication.java:215) at com.hortonworks.streamline.webservice.StreamlineApplication.run(StreamlineApplication.java:101) at com.hortonworks.streamline.webservice.StreamlineApplication.run(StreamlineApplication.java:75) at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:43) at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:85) at io.dropwizard.cli.Cli.run(Cli.java:75) at io.dropwizard.Application.run(Application.java:79) at com.hortonworks.streamline.webservice.StreamlineApplication.main(StreamlineApplication.java:79) Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.jdbc2.optional.MysqlDataSource at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at com.zaxxer.hikari.util.UtilityElf.createInstance(UtilityElf.java:80) ... 19 more ... View more

I installed my HDP cluster as a non-root user. I have my non-root sudoers access setup as specified by ambari,ambari-sudoer-config.txt I installed HDF management pack to install schema registry. But it gives me the below error. Is there anything I'm missing in the sudoer config? /usr/hdf/current/registry/bootstrap/bootstrap-storage.sh migrate' returned 1. sudo: no tty present and no askpass program specified ... View more

I have existing HBase tables for which I build Phoenix Views. CREATE VIEW "MDS" (pk VARCHAR PRIMARY KEY, "auth_group"."id" VARCHAR, "auth_group"."name" CHAR(10)); where MDS is my hbase table, auth_group is one of the column families with columns id, name. I have the below questions: 1. Can I use one of the columns in my column family as a PRIMARY KEY? Or should it something outside of the columns in the column family? How do I build a PRIMARY KEY in such a scenario. 2. When I did the below it gave me the below error. How do I choose the length of CHAR? select * from MDS; Error: ERROR 201 (22000): Illegal data. Expected length of at least 12 bytes, but had 10 (state=22000,code=201) 3. If I picked the min of the CHAR to some minimum number, I get the below result where the characters are cut off(based on the number I gave) and the id column is blank . (It doesn't reflect the same values as in hbase.) PK id name 1 resource_u 2 adminid_id 3 systems_id 4 usersystem 5 uaccess_id 6 numofusers 7 idofusersac ... View more

When data is accidentally removed from HDFS, is there a way to recover it from Trash? Trash is enabled to 360. However I don’t see the files in the user’s /.Trash. ... View more

I had a datanode failure due to the JAVA HEAP SIZE which caused a huge number of under replicated blocks since there were writes that happened when the node failed.I fixed the JAVA HEAP size and got the node alive.When I'm trying to re-replicate the blocks as mentioned here the number doesn't seem to comedown even when the setrep operation running. Also, the other thing I observed was the data looks skewed on the datanodes: 237.95 GB 2567977 222.77 GB (93.62%) 2.7.3.2.6.4.0-91 775.82 GB 2657650 244.16 GB (31.47%) 2.7.3.2.6.4.0-91 776.17 GB 2657711 244.17 GB (31.46%) 2.7.3.2.6.4.0-91 Is the skewed data interfering with the setrep operation? IS there a way I can deal with the skew and the under replicated blocks? ... View more

I'm trying to submit a pyspark job from a spark2 client on HDP-2.6.4-91 like ./bin/spark-submit script.py But this gives me an error: NameError: global name "callable" not defined. ... View more

I'm using an existing MySQL DB for the Hive metastore(I have created the hive user and created the hive DB and setup ambariserver using the jdbc driver). And when trying to install HiveServer2 , I get the below error. Caught an exception while executing custom service command: <type 'exceptions.OSError'>: [Errno 2] No such file or directory: '/var/lib/ambari-agent/cred/conf/hive_server/hive-site.jceks'; [Errno 2] No such file or directory: '/var/lib/ambari-agent/cred/conf/hive_server/hive-site.jceks' ... View more

I'm having an issue starting the metrics monitor on one of my nodes. I get the below error. "ImportError: cannot import name _common" ... View more

Is there any difference between the way High availability is configured between HDP-2.4 and 2.6 using blueprints? I tried setting up HA cluster with exact same JSON files as mentioned in this blog Automate HDP installation using Ambari Blueprints – Part 3 for HDP-2.6 and 2.4 just by changing the stack version. It worked for 2.4 but seems to have issues for 2.6 during zookeeper install. Can anyone point me to the right place for setting up HA cluster on HDP-2.6 using blueprints? ... View more

The install part is failing at the zookeeper client install using ambari-blueprints. It says mkdir: cannot create directory `/usr/hdp/current/zookeeper-client': File exists Here's the error log from the Ambari UI zookeeper-err.txt. ... View more