Created on 06-08-2016 01:47 PM - edited 08-17-2019 12:08 PM
While traditional fraud detection systems have focused on looking for factors such as bad IP addresses or unusual login times based on business rules and events, the Connected Data Platform renovates such an approach by enabling machine learning capabilities at scale. The Credit Fraud Prevention Demo is an excellent example of a Modern Data Application running on the Hortonworks Connected Platform (HDF/HDP).
Part I: Preparing the Demo
Follow the instructions in the README to install the Credit Card Transaction Monitor Application on an HDP 2.4 Sandbox.
Note: There is a known bug with the HAXM emulator used by the Android Studio for the Credit Card Transaction Monitor Mobile App that does not allow it to be launched while a Virtual Box virtual machine is running. To avoid this, I am using a HDP 2.4 Sandbox instance that I created in the Azure Cloud directly from the Microsoft Azure Marketplace.
The install shell script handles the installation and configuration of all the application artifacts necessary for the demo onto the latest version of the Hortonworks Sandbox, including:
The startDemoServices shell script should be run each time the Sandbox VM is (re)started, after all of the default Sandbox services come up successfully. It handles the initialization of all of the application-specific components of the demo, including:
To validate that the Slider Application for the Transaction Monitor UI started, you can take a look at the YARN Resource Manager UI by selecting the YARN service under Ambari and using the Quick Link for Resource Manager UI:
Part II: Demo Code Walk-Through
There are three types of transactions managed by the NiFi data flow:
Incoming Transactions are generated by the TransactionSimulator class and posted as ReST calls over HTTP to port 8082 of the Sandbox.
The Transaction Monitor UI servlet generates Fraud Notifications and posts them to the same Sandbox port.
The NiFi flow creates an HTTP Listener on the correct Sandbox port when it is deployed:
The Credit Card Transaction Monitor mobile application posts Customer Transaction Validation messages to Amazon SQS. The NiFi Flow is configured to receive these messages.
After being received, these transactions are all sent through a simple dataflow that extracts the event source from the payload and determines the event destination accordingly.
Once the event destination is determined, these transactions are routed based on the value of that attribute.
Incoming Transactions are routed to a Kafka topic.
Incoming Transactions posted to that Kafka Topic are then consumed by the CreditCardTransactionMonitor Storm topology.
The Storm topology enriches the incoming transaction with customer details from HBase, passes the enriched transaction into a Spark model to detect fraud, and then either publishes a fraud alert or publishes a legitimate transaction to the TransactionMonitorUI web application using CometD.
While reviewing the code for each of the bolts in the Storm topology above, in addition to reading the execute method, be sure take a close look at the prepare method as well. For example, in the FraudDetector bolt:
The Credit Fraud Detection model uses Spark MLLib Logistic Regression against four primary features of the credit card transaction to detect fraud:
To view the Zeppelin notebook showing how the model is built and trained, using a browser go to sandbox port 9995 and select the Credit Fraud Detection ML link under Notebooks.
Fraud Notifications are posted over HTTP to Google Cloud Messaging where the Credit Card Transaction Mobile Application MessagingClient can pick them up.
Customer Transaction Validation
Customer Transaction Validation events are routed to a Kafka topic.
Customer Transaction Validation event posted to that Kafka Topic are then consumed by the CreditCardTransactionMonitor Storm topology.
The Storm topology processes the Customer Transaction Validation event, updating the HBase table for the customer account, as well as publishes the Account Status Update to the TransactionMonitorUI web application using CometD.
Part III: Running the Demo
Before running the demo, be sure to install and compile the Credit Card Transaction Mobile App by following the instructions in the README. Once you’ve done that, follow these steps:
java -jar CreditCardTransactionSimulator-0.0.1-SNAPSHOT-jar-with-dependencies.jar Customer 1000 Simulation
The Credit Fraud Prevention Demo shows the full power of developing a Modern Data Application for the Connected Data Platform:
The platform identifies and signals fraud in near real time. The result: an improved customer experience, revenue loss prevention due to fraud and reduced cost overall.