Adding TTL on Solr: cd to this directory Step1: Step2: Step3: vi managed-schema: add these 3 lines <field name="_timestamp_" type="date" indexed="true" stored="true" multiValued="false" /> <field name="_ttl_" type="string" indexed="true" multiValued="false" stored="true" /> <field name="_expire_at_" type="date" multiValued="false" indexed="true" stored="true" /> Step4: vi solrconfig.xml Replace the below 3 lines with the lines after it: <updateRequestProcessorChain name="add-unknown-fields-to-the-schema"> <!-- UUIDUpdateProcessorFactory will generate an id if none is present in the incoming document --> <processor class="solr.UUIDUpdateProcessorFactory"/> as <updateRequestProcessorChain name="add-unknown-fields-to-the-schema"> <processor class="solr.TimestampUpdateProcessorFactory"> <str name="fieldName">_timestamp_</str> </processor> <processor class="solr.DefaultValueUpdateProcessorFactory"> <str name="fieldName">_ttl_</str> <str name="value">+30SECONDS</str> </processor> <processor class="solr.processor.DocExpirationUpdateProcessorFactory"> <str name="ttlFieldName">_ttl_</str> <str name="ttlParamName">_ttl_</str> <int name="autoDeletePeriodSeconds">30</int> <str name="expirationFieldName">_expire_at_</str> </processor> <processor class="solr.FirstFieldValueUpdateProcessorFactory"> <str name="fieldName">_expire_at_</str> </processor> <!-- UUIDUpdateProcessorFactory will generate an id if none is present in the incoming document --> <processor class="solr.UUIDUpdateProcessorFactory" /> Things that might be useful: Make sure to start solr like this so that configs related to solr goes to /solr in zookeeper: 1./opt/lucidworks-hdpsearch/solr/bin/solr start -c –z lake1.field.hortonworks.com:2181, lake2.field.hortonworks.com:2181, lake3.field.hortonworks.com:2181/solr 2.create the collection like this /opt/lucidworks-hdpsearch/solr/bin/solr create -c tweets -d data_driven_schema_configs -s 1 -rf 1 3.to delete the collection: http://testdemo.field.hortonworks.com:8983/solr/admin/collections?action=DELETE&name=tweets 4.also remove it from zkCli.sh as rmr /solr/config/tweets Thanks, Sujitha Sanku please ping me or email me at ssanku@hortonworks.com in case of any issues. ... View more

Special thanks to Michael Young for the help to be my mentor. Step1: cd /opt/lucidworks-hdpsearch/solr/server/solr/configsets/data_driven_schema_configs/conf/ screen-shot-2016-11-21-at-100524-am.png Step2: vi managed-schema: add these 3 lines <field name="_timestamp_" type="date" indexed="true" stored="true" multiValued="false" /> <field name="_ttl_" type="string" indexed="true" multiValued="false" stored="true" /> <field name="_expire_at_" type="date" multiValued="false" indexed="true" stored="true" /> screen-shot-2016-11-21-at-100929-am.png Step3: vi solrconfig.xml on the same directory. Replace the below 3 lines with the lines after it: <updateRequestProcessorChain name="add-unknown-fields-to-the-schema"> <!-- UUIDUpdateProcessorFactory will generate an id if none is present in the incoming document --> <processor /> as <updateRequestProcessorChain name="add-unknown-fields-to-the-schema"> <processor> <str name="fieldName">_timestamp_</str> </processor> <processor> <str name="fieldName">_ttl_</str> <str name="value">+30SECONDS</str> </processor> <processor class="solr.processor.DocExpirationUpdateProcessorFactory"> <str name="ttlFieldName">_ttl_</str> <str name="ttlParamName">_ttl_</str> <int name="autoDeletePeriodSeconds">30</int> <str name="expirationFieldName">_expire_at_</str> </processor> <processor> <str name="fieldName">_expire_at_</str> </processor> <!-- UUIDUpdateProcessorFactory will generate an id if none is present in the incoming document --> <processor class="solr.UUIDUpdateProcessorFactory" /> screen-shot-2016-11-21-at-101045-am.png Hope that helps. Thanks, Sujitha ... View more

Solr indexing the MySQL database table on HDP 2.5 Tech Preview: Solr version used: solr 4.9.0 Step1: Downloaded the solr 4.9.0.zip from https://archive.apache.org/dist/lucene/solr/4.9.0/ Step2: Extract the file: Step3: modify the solrconfig.xml, schema.xml and add the db-data-config.xml at Step4: add the jar at this location a.vi solrconfig.xml: add these lines in between the config tags. <lib dir="../../../contrib/dataimporthandler/lib/" regex=".*\.jar" /> <lib dir="../../../dist/" regex="solr-dataimporthandler-\d.*\.jar" /> <lib dir="../../../lib/" regex="mysql-connector-java-5.0.8-bin.jar" /> <requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler"> <lst name="defaults"> <str name="config">db-data-config.xml</str> </lst> </requestHandler> b.vi schema.xml add the below line: <dynamicField name="*_name" type="text_general" multiValued="false" indexed="true" stored="true" /> c.Create a file called db-data-config.xml at the same path later in this session I would create a database employee in mysql server add these <dataConfig> <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost:3306/employees" user="root" password="hadoop" /> <document> <entity name="id" query="select emp_no as 'id', first_name, last_name from employees limit 1000;" /> </document> </dataConfig> After this is complete run the below command (d) to start solr and check if solr is up and running at url below: 8983 is the default port of solr d.java –jar start.jar http://localhost:8983/solr/#/ e.select the core selector as collection1. f.Click on Data Import, expand configuration and check if its pointing to our db-data-config.xml file we created. g.After the completion of Step5 below click on execute on the page. Step5: Setting up database: Import an already available database into Mysql: Ref: https://dev.mysql.com/doc/employee/en/employees-installation.html shell> tar -xjf employees_db-full-1.0.6.tar.bz2 shell> cd employees_db/ shell> mysql -t < employees.sql With this installation of employees db in mysql is complete. Step6: With this our indexing is complete using Solr. To do: I will try indexing the tables in Mysql using latest version of Solr. Reference: http://blog.comperiosearch.com/blog/2014/08/28/indexing-database-using-solr/ Hope this helps…. Thanks, Sujitha ... View more

Story: From the documentation I was able to add a service to existing stack definition in Ambari. Issue: But I was either not able to stop the service or delete the service just in case. https://cwiki.apache.org/confluence/display/AMBARI/Defining+a+Custom+Stack+and+Services How did I solve the problem? 1.Create and Add Stack: cd /var/lib/ambari-server/resources/stacks/HDP/2.4/services 2.Create a directory that contains the service definition for SAMPLESRV mkdir /var/lib/ambari-server/resources/stacks/HDP/2.4/services/SAMPLESRV cd /var/lib/ambari-server/resources/stacks/HDP/2.4/services/SAMPLESRV 3.Create a metainfo.xml as show in the link above. 4.With this we have a service name as SAMPLESRV and it contains SAMPLESRV_MASTER, SAMPLESRV_SLAVE and SAMPLESRV_CLIENT 5.Next we need to create the command scripts mkdir –p /var/lib/ambari-server/resources/stacks/HDP/2.4/services/SAMPLESRV/package/scriptscd /var/lib/ambari-server/resources/stacks/HDP/2.4/services/SAMPLESRV/package/scripts 6.Browse the scripts directory and create the .py command scripts:master.py, slave.py and sample_client.py under : /var/lib/ambari-server/resources/stacks/HDP/2.4/services/SAMPLESRV/package/scripts Master.py and slave.py here was the issue: in the documentation it doesn’t mention about the dummy.pid that needs to be created. Since we have not installed a real service, there is no PID file created by it. Therefore, we are going to artificially create the PID, remove the PID and check the process status of the dummy pid. 7.Then restart ambari: ambari-server restart and add the service to the stack as shown the document. Just don't want to duplicate the process with steps here. Hope this helps.... ... View more