Member since
07-17-2017
23
Posts
1
Kudos Received
4
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1854 | 04-30-2020 12:06 PM | |
| 2501 | 11-06-2019 12:32 PM | |
| 1892 | 08-22-2019 09:13 AM | |
| 1259 | 08-22-2019 08:39 AM |
04-30-2020
12:06 PM
So after a few days of struggling, I was able to launch a trial CDP-DC cluster on my server and explore myself. I installed the runtime along with most of the CDF parcels. It looks as though the two services mentioned above (Zeppelin and Druid) are not currently available on CDP-DC. If anyone has any information on how we could add these services or any timelines for these services being added, I'd love to know, especially for Druid!
... View more
04-24-2020
08:35 AM
I'm trying to discern what services will be available to us when we upgrade from CDH 5.16.2 to CDP-DC with CDF. From what I can tell, we should have access to any components part of CDF, as well as anything in the Cloudera Runtime. The thing is, I'm seeing some discrepancies.
In the maven artifacts for Cloudera Runtime 7.0.3, it mentions a lot of services, including Druid and Zeppelin: https://docs.cloudera.com/cdpdc/7.0/release-guide/topics/cdpdc-runtime-maven-703.html
But in the component versions list of Cloudera Runtime 7.0.3, Druid and Zeppelin are both not included: https://docs.cloudera.com/runtime/7.0.3/release-notes/topics/rt-runtime-component-versions.html
Additionally, in the documentation for CDF, I don't see Druid listed, even though in the past HDF has had Druid included:
https://www.cloudera.com/content/dam/www/marketing/images/diagrams/cdf-diagram.png
Can someone verify for me whether Druid and/or Zeppelin is available for use by users of CDP-DC with CDF? Additionally, how should I understand the discrepancies between the first two links above?
Thanks!
... View more
Labels:
11-06-2019
12:32 PM
After looking into this some more, we found the error trace below the first time that a paragraph was called after the interpreter was restarted. This didn't show up originally since the above log was only trying to run a paragraph, not necessarily just after the interpreter was restarted. As you can see, in the end there is an exception about a class not being accessible. Once we made sure the wandisco class was accessible to the interpreter in the classpath, then everything started to work properly. 2019-11-06 10:24:48,850 ERROR [pool-2-thread-2] PhoenixInterpreter:108 - Cannot open connection
java.sql.SQLException: ERROR 103 (08004): Unable to establish connection.
at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:386)
at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection(ConnectionQueryServicesImpl.java:288)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.access$300(ConnectionQueryServicesImpl.java:171)
at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:1881)
at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:1860)
at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:77)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:1860)
at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:162)
at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:131)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:133)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.zeppelin.phoenix.PhoenixInterpreter.open(PhoenixInterpreter.java:99)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:493)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:410)
at org.apache.hadoop.hbase.client.ConnectionManager.createConnectionInternal(ConnectionManager.java:319)
at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:144)
at org.apache.phoenix.query.HConnectionFactory$HConnectionFactoryImpl.createConnection(HConnectionFactory.java:47)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection(ConnectionQueryServicesImpl.java:286)
... 22 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
... 27 more
Caused by: java.lang.NoClassDefFoundError: com/wandisco/shadow/com/google/protobuf/InvalidProtocolBufferException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1844)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1809)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1903)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2573)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2586)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2625)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2607)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
... View more
11-05-2019
09:33 AM
I've verified that I can access Phoenix through sqlline.py and psql.py using the configuration in /etc/ams-hbase/conf, and run queries as the activity-explorer user that I'm trying to run through Zeppelin. One thing of note with all this: we've changed the ZNode parent from ams-hbase-secure1 to ams-hbase-secure2. I've verified that the value in /etc/ams-hbase/conf/hbase-site.xml holds the new value, but the value in /etc/ams-metrics-collector/conf/hbase-site.xml is the old value and hasn't been updated recently. activity-env.sh points to /etc/ams-hbase/conf, so I believe this shouldn't be an issue, but it was a bit confusing when I first came across it.
... View more
11-05-2019
08:42 AM
I just noticed that the SmartSense Activity Explorer Zeppelin Notebooks have been failing to run on a production HDP 2.6.1 cluster. I'm not sure how long the issue has been occurring since the dashboards haven't been used much until now. Whenever we try to run the paragraphs, we immediately get an error about unable to make a connection. No other information is given. We are able to connect to Phoenix through psql.py, so we know Phoenix is working properly, just not the dashboard. We've tried restarting the activity explorer, which hasn't fixed the issue. Has someone seen this issue? Any ideas? I'm including the logs we are seeing below. ==> activity-explorer.log <==
2019-11-05 10:34:42,555 INFO [qtp1209702763-1653] NotebookServer:711 - New operation from 10.142.131.4 : 62057 : admin : GET_NOTE : 2BPD7951H
2019-11-05 10:34:42,558 WARN [qtp1209702763-1653] VFSNotebookRepo:292 - Get Note revisions feature isn't supported in class org.apache.zeppelin.notebook.repo.VFSNotebookRepo
2019-11-05 10:34:45,886 INFO [pool-2-thread-31] SchedulerFactory:131 - Job paragraph_1490380022011_880344082 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session255064451
2019-11-05 10:34:45,887 INFO [pool-2-thread-31] Paragraph:366 - run paragraph 20160728-152731_1797959357 using null org.apache.zeppelin.interpreter.LazyOpenInterpreter@4a66e7be
==> zeppelin-interpreter-phoenix-phoenix--<HOSTNAME> <==
2019-11-05 10:34:45,889 INFO [pool-2-thread-4] SchedulerFactory:131 - Job remoteInterpretJob_1572971685889 started by scheduler org.apache.zeppelin.phoenix.PhoenixInterpreter717591913
2019-11-05 10:34:45,889 INFO [pool-2-thread-4] PhoenixInterpreter:192 - Run SQL command 'SELECT file_size_category as "Size category",
total_files as "Total files",
avg_file_size as "Avg file size"
FROM (
SELECT CASE WHEN file_size_range_end <= 10000 THEN 'Tiny (0-10K)'
WHEN file_size_range_end <= 1000000 THEN 'Mini (10K-1M)'
WHEN file_size_range_end <= 30000000 THEN 'Small (1M-30M)'
WHEN file_size_range_end <= 128000000 THEN 'Medium (30M-128M)'
ELSE 'Large (128M+)'
END as file_size_category,
sum(file_count) as total_files,
(sum(total_size) / sum(file_count)) as avg_file_size
FROM ACTIVITY.HDFS_USER_FILE_SUMMARY
WHERE analysis_date in ( SELECT MAX(analysis_date)
FROM ACTIVITY.HDFS_USER_FILE_SUMMARY)
GROUP BY file_size_category
)'
2019-11-05 10:34:45,890 INFO [pool-2-thread-4] SchedulerFactory:137 - Job remoteInterpretJob_1572971685889 finished by scheduler org.apache.zeppelin.phoenix.PhoenixInterpreter717591913
==> activity-explorer.log <==
2019-11-05 10:34:45,891 WARN [pool-2-thread-31] NotebookServer:2067 - Job 20160728-152731_1797959357 is finished, status: ERROR, exception: null, result: %text ERROR 103 (08004): Unable to establish connection.
2019-11-05 10:34:45,909 INFO [pool-2-thread-31] SchedulerFactory:137 - Job paragraph_1490380022011_880344082 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session255064451
... View more
08-22-2019
09:13 AM
So EOL just means that Python 2 will no longer be supported. That doesn't mean anything using Python 2 will not be supported. Any applications that currently use Python 2.7 will still work, but any bugs in Python 2.7 won't be fixed. Additionally, pip will not work with Python 2.7 after version 19.1. As long as the version of pip you have installed is less than this, you'll still be able to use Python 2.7 for a while, until the repository itself is taken down or reconfigured in a way that breaks old pip. See https://stackoverflow.com/questions/54915381/will-pip-work-for-python-2-7-after-its-end-of-life-on-1st-jan-2020 for more details on that. I am not sure on the plans for HDP/CDH/CDP support for Python 2.7 going forward, so that's a question for the dev team directly.
... View more
08-22-2019
08:39 AM
1. I'm not sure I understand what you mean by communicate. When SSSD is first started, it will sync all of the users and groups in AD to the local node, so any existing users will be able to log in, and have the correct groups ready for them (assuming configuration is set up properly). 2. Rolling back SSSD is possible but troublesome. It would consist of stopping the service and uninstalling it from the node. I'm not sure if the users and groups would still be on the node, but you would need to uninstall that as well. There may be some other pieces left around, but none that I would expect to cause any differences, unless you were to try to install SSSD again.
... View more
07-27-2017
08:39 AM
OK, finally got everything working. As for the last error I had been seeing, I had thought for sure my kerberos credentials were still showing up in klist, but this morning when I kinited in, everything worked fine, so that must have been the issue. I then got an error on the consumer side, which I soon realized was because with the new bootstrap-servers parameter, you need to use the same port as the producer (9093 in my case), not the zookeeper port. Once I updated this, everything worked properly.
... View more
07-27-2017
08:29 AM
After researching this a bit, I tried a few more things, none of which changed the error: I moved to using the keytab for the kafka user (https://community.cloudera.com/t5/Data-Ingestion-Integration/Unable-to-connect-to-kerberized-Kafka-2-1-0-10-from-Spark-2-1/m-p/56026) I placed the jaas.conf file in /tmp on every node in the cluster and pointed KAFKA_OPTS to that place (https://stackoverflow.com/questions/43190784/spark-streaming-kafka-kerberos) I exported KAFKA_CLIENT_KERBEROS_PARAMS to be the same as KAFKA_OPTS (https://community.hortonworks.com/content/supportkb/49422/running-kafka-client-bin-scripts-in-secure-envrion.html)
... View more
07-26-2017
08:06 PM
I realized that the client.properties file was using SASL_PLAINTEXT, not SASL_SSL. Updated appropriately. Hitting a new error now, on both producer and consumer. The following error comes up, and then it quits the program. I've verified that jaas.conf is in KAFKA_OPTS properly. org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:718)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:597)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:579)
at kafka.consumer.NewShinyConsumer.<init>(BaseConsumer.scala:53)
at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:69)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:50)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:93)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:109)
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:55)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:84)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:657)
... 6 more
Caused by: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:899)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:55)
at org.apache.kafka.common.security.kerberos.KerberosLogin.login(KerberosLogin.java:100)
at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:52)
at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:81)
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:85) Also, I ran the command you mentioned above, and everything looks right. SSL handshake read 3151 bytes and wrote 499 bytes using TLS v1.2. If you need more information from it, let me know. EDIT: Realized that the properties file actually was wrong. Updating with relevant information because of this.
... View more