Sorry here is the link. https://dzone.com/articles/using-solrj-basic ... View more

You were close. hdfs dfs -ls /lib | awk '{system("hdfs dfs -count " $8) }' | awk '{print $4,$1"\n"$4,$2"\n"$4,$3;}' This throws an usage error for the first run and I haven't looked into why, but prints out all subdirs; three entries for each stat from -count. ... View more

I don't have the answer but the error is when it is trying to verify the SSL certificate of the SOLR instance. This article and code is working to getting around doing the SSL verification. I don't think this is what you want but you could lead you in the correct direction. ... View more

Ah ok. To help I would need to see the script, or at least a snippet of it, and the error it is hitting. I can't think of any issue with doing this in oozie. I could drum something up to test this but it would make it straightforward to know exactly how it is running in parallel outside of oozie. Passing the file with the args, parsing it out, and running whatever command in parallel is all contained withing the script. Let me try to run through this. Say your script takes in a file with just words. You parse it and echo the word to a file. The writing to a file is ran in parallel in the script. Oozie will launch a container on a worker node. The script and the file will both be made available to the job that will run in the container. The script launches and runs. Lets assume 10 words, then 10 processes will be forked and ran. The gist here is that Oozie doesn't do anything to make it run in parallel; it is still dependent on the script for it. What I was thinking previously is that you wanted to run the script/workflow itself in parallel and not a command within the script. And parallel here isn't taking advantage of the cluster for parallelism. It will run just as fast locally on a host and not through Oozie. ... View more

Is that ten different files passes as arguments? And these files contain any number of tables? Can you post the logs or error that oozie hit when trying to run the second version? I think the first issue would be that you need to provide each argument file to the workflow so that it can be passed to the container. I see that you have this for one file but did you change this or add the second arg file? Changing it for each run is not fun. Do you need to pass the arguments from a file or can they be pass to the script directly? Then you can get around the need to provide each argument file to each different workflow. I still think that a coordinate is what you are looking for. I haven't messed with one in a while but from my recollection you can reference an existing workflow multiple times and set its own conditions or schedule. ... View more

I am not 100% on this covering what you are asking but I think it does. Look into Oozie coordinator and bundles. I feel like Coordinators are typically used for trigger workflows based on conditions but you could use one to launching different versions of your workflow (i.e. 10 different tables passed as args for each) and have them run at the same time. https://oozie.apache.org/docs/3.1.3-incubating/CoordinatorFunctionalSpec.html ... View more

As @Fawze mentioned, any role can be moved. From the info provide it looks like you are running 5 worker nodes and 1 master. Depending on the hardware you can make one or two of those workers into Masters; otherwise moving master roles onto worker nodes will potentially cause issues. Some tips for moving, if the role connects to a backend DB, like the Hive Metastore, then you need to update the DB user account to reflect it running from the new hosts. Some other roles are tricky, like the Namenode, as you would need to migrate the metadata. The two candidates to convert to masters from workers would be the two that just have the Datanode role on them. You can just do one if you don't need production level resiliency and redundancy in the master services. First, remove the Nodemanager, HBase Regionserver, and Impala Daemon, Solr server role from the current master (the one with 25 roles). Those are all worker roles. I would move the ResourceMananger, Spark History, Impala Catalog, Impala Statestore, and Secondary Namenode. The last one is the only tricky one and it should be fine as setting up a new one should sync from the Namenode. I also recommend moving the Cloudera Manager roles off to a completely different server, possible a VM. It doesn't need to be on the cluster, is using up valuable resources, and can live in a VM. You will need to update all DB accounts related to them. That is a start, we could go further but that should be enough to give you some breathing room. Good luck. ... View more

I can't say for certain but it is probably because it is running the script on whatever node the container ends up running on. So if /home/mbigelow/logs/20170412... doesn't exist on each and every node then it will fail. I tend to stick with /tmp as it is always there and writable for all users. ... View more

There is probably an issue with the client connecting to the Datanode. It is reporting that you have one live data nodes but it is failing to place any replica on it. I would expect the client to get a different error if it was failing to write out the first replica. Check the NN UI to validate that the DN is live, and check the NN and DN logs to see if there is more information on what the issue is. ... View more