This article shows a simple NiFi data flow from the web to HDFS that demonstrates several fundamental capabilities of NiFi, including:
- iterating the flow from an external configuration file
- groovy scripting to transform content
- groovy scripting that imports an external jar library
- NiFi expression language
The article may be particularly useful for newcomers to NiFi.
Overview
The NiFi flow is shown below. At a very high-level, an InvokeHTTP processor will retrieve in sequence the RSS feeds from a list of URLs that are configured locally. RSS feeds return XML content and one feed URL returns multiple XML item elements, each representing one news story and structured by child elements like title, description, etc. A groovy script will transform each XML-formatted feed response to produce a set or records. Each record is one news story (item element) with title, description etc extracted from the XML as tab-separated values. Additionally, HTML formatting is stripped from those values. The tsv result for all configured feed URLs will be merged and put to HDFS as a single file named as <filestamp>.tsv. From here, Spark, Hive, Pig, HBase and so on can take over. The feed will run once per day.
Using a configuration file listing URLs prevents us from hard-coding one InvokeHTTP processor for each URL and allows us to change the list on the fly. We can use this same flow identically for 100 URLs or for 1.
GetFile
This processor reads the content of a configuration file stored locally. Each line of the configuration file feedlist.conf is the URL of an RSS feed. It is scheduled to read the file each day at 23:00. Important configurations for this processor are:
- path and filename
- keep source file
SplitText
This processor takes the content from GetFile and splits each line into separate strings creating a list of URLs that are passed to the next processor as splits.
Extract Text
This processor takes each split and extracts its text, which I assign to a new property named target.url. Note that the text is extracted by regex. The matched regex for target.url property for each split is sent in sequence to the next processor. Note that in our case we are matching the original text, to get each URL as originally configured.
InvokeHTTP
Here is the where the magic of NiFi processors can be seen. The InvokeHTTP processor performs all the low-level work to send an HTTP request to each target.url property passed to it, and pull in its XML response (in the RSS case) which it passes to the next processor. This is done in sequence for each target.url it receives from upstream.
ExecuteScript
Here the XML response from each feed is transformed by a groovy script, in this case referenced as a locally stored file. Note that the script uses an external library (commons-io-2.5.jar) that is placed in a directory referenced by the configuration.
Let's take a look at the rss_etl.groovy script. One key feature is its use of XmlSlurper to easily parse XML, which is one of the many reasons that make Groovy so groovy.
import org.apache.commons.io.IOUtils
import java.nio.charset.*
def flowFile = session.get()
if(!flowFile) return
flowFile = session.write(flowFile, {inputStream, outputStream ->
def feedXml = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
def record = ""
try {
def rss = new XmlSlurper().parseText(feedXml)
def channelLastBuildDate = rss.channel.lastBuildDate
def channelTitle = rss.channel.title
def channelLink = rss.channel.link
rss.channel.item.each(){it ->
def itemTitle = it.title
def itemDescription = it.description
def itemLink = it.link
def itemPubDate = it.pubDate
// remove HTML from description
itemDescription = itemDescription.toString().replaceAll("\<.*?>|&#.*;", "").trim()
record = record + channelLastBuildDate + "\t" +
channelTitle + "\t" +
channelLink + "\t" +
itemTitle + "\t" +
itemDescription + "\t" +
itemLink + "\t" +
itemPubDate + "\n"
}
outputStream.write(record.getBytes(StandardCharsets.UTF_8))
}
catch(e) {
def channel = channelLink == null ? "UNK" : channelLink
log.error("Error during processing of RSS feed channel: ${channel}", e)
}
} as StreamCallback)
session.transfer(flowFile, REL_SUCCESS)
This scripts converts the below original RSS content from one feed:
<?xml version="1.0"?>
<rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:nyt="http://www.nytimes.com/namespaces/rss/2.0" version="2.0">
<channel>
<title>NYT > Technology</title>
<link>http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss</link>
<atom:link rel="self" type="application/rss+xml" href="http://www.nytimes.com/services/xml/rss/nyt/Technology.xml"/>
<description/>
<language>en-us</language>
<copyright>Copyright 2016 The New York Times Company</copyright>
<lastBuildDate>Tue, 26 Jul 2016 23:11:25 GMT </lastBuildDate>
<image>
<title>NYT > Technology</title>
<url>https://static01.nyt.com/images/misc/NYT_logo_rss_250x40.png</url>
<link>http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss</link>
</image>
<item>
<title>Apple Sales Continue to Drop, but Services Are a Bright Spot</title>
<link>http://www.nytimes.com/2016/07/27/technology/apple-earnings-iphone-sales.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/27/technology/apple-earnings-iphone-sales.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/27/technology/apple-earnings-iphone-sales.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/27/business/27APPLE-1/27APPLE-1-moth.jpg" medium="image" height="151" width="151"/>
<media:description>Apple’s new flagship store in San Francisco.</media:description>
<media:credit>Justin Sullivan/Getty Images</media:credit>
<description>A second quarter of dismal sales leaves analysts wondering: Are customers out of love with Apple, or just waiting for the latest products?</description>
<dc:creator>VINDU GOEL</dc:creator>
<pubDate>Tue, 26 Jul 2016 21:54:37 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Apple Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">iPhone</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Company Reports</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Cloud Computing</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">iPad</category>
</item>
<item>
<title>Marissa Mayer’s Media Problem at Yahoo Is Now Verizon’s to Solve</title>
<link>http://www.nytimes.com/2016/07/27/business/media/marissa-mayers-media-problem-at-yahoo-is-now-verizons-to-solve.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/27/business/media/marissa-mayers-media-problem-at-yahoo-is-now-verizons-to-solve.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/27/business/media/marissa-mayers-media-problem-at-yahoo-is-now-verizons-to-solve.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/27/business/27MARISSA/27MARISSA-moth.jpg" medium="image" height="151" width="151"/>
<media:description>Marissa Mayer, chief executive of Yahoo, and Adam Cahan, a senior vice president, at a mobile developer conference in February 2015 in San Francisco.</media:description>
<media:credit>Max Whittaker for The New York Times</media:credit>
<description>Before she led Yahoo, while still at Google, Ms. Mayer spoke of the difficulties for media companies of capitalizing on their own content.</description>
<dc:creator>JOHN HERRMAN</dc:creator>
<pubDate>Tue, 26 Jul 2016 20:14:48 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Yahoo! Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Verizon Communications Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Media</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_per">Mayer, Marissa</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_per">Schmidt, Eric E</category>
</item>
<item>
<title>F.B.I. Examining if Hackers Gained Access to Clinton Aides’ Emails</title>
<link>http://www.nytimes.com/2016/07/26/us/politics/fbi-investigating-intrusions-into-democratic-committees-emails.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/26/us/politics/fbi-investigating-intrusions-into-democratic-committees-emails.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/26/us/politics/fbi-investigating-intrusions-into-democratic-committees-emails.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/26/us/26dncemails-web/26dncemails-web-moth.jpg" medium="image" height="151" width="151"/>
<media:description>Workers wrapped up preparations on Monday at the Wells Fargo Arena in Philadelphia for the Democratic National Convention.</media:description>
<media:credit>Jim Wilson/The New York Times</media:credit>
<description>Federal officials say the inquiry has been underway since the spring, when the agency was first notified of the D.N.C.’s suspicions about hacking.</description>
<dc:creator>DAVID E. SANGER</dc:creator>
<pubDate>Tue, 26 Jul 2016 01:31:18 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_geo">Russia</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Cyberwarfare and Defense</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Democratic National Committee</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">United States Politics and Government</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Federal Bureau of Investigation</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_geo">United States</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_per">Clinton, Hillary Rodham</category>
</item>
<item>
<title>Robert Fano, 98, Dies; Engineer Who Helped Develop Interactive Computers</title>
<link>http://www.nytimes.com/2016/07/27/technology/robert-fano-98-dies-engineer-who-helped-develop-interactive-computers.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/27/technology/robert-fano-98-dies-engineer-who-helped-develop-interactive-computers.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/27/technology/robert-fano-98-dies-engineer-who-helped-develop-interactive-computers.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/27/business/27FanoObit/27FanoObit-moth.jpg" medium="image" height="151" width="151"/>
<media:description>Robert Fano.<br/><br/></media:description>
<media:credit>Jason Dorfman/M.I.T.</media:credit>
<description>In the early 1960s, Dr. Fano and others developed a time-sharing operating system, which signaled a shift to a more interactive computing world.</description>
<dc:creator>JOHN MARKOFF</dc:creator>
<pubDate>Tue, 26 Jul 2016 19:34:21 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_geo">Turin (Italy)</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Bell Laboratories</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">International Business Machines Corporation</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Computers and the Internet</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Mathematics</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_per">Fao, Robert</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Deaths (Obituaries)</category>
</item>
<item>
<title>Deal Professor: $1 Billion for Dollar Shave Club: Why Every Company Should Worry</title>
<link>http://www.nytimes.com/2016/07/27/business/dealbook/1-billion-for-dollar-shave-club-why-every-company-should-worry.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/27/business/dealbook/1-billion-for-dollar-shave-club-why-every-company-should-worry.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/27/business/dealbook/1-billion-for-dollar-shave-club-why-every-company-should-worry.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/27/business/27DB-DEALPROF/27DB-DEALPROF-moth.jpg" medium="image" height="151" width="151"/>
<media:description/>
<media:credit>Harry Campbell</media:credit>
<description>The internet, mass transportation and globalization allow decentralized companies to be smaller and leaner and have fewer employees.</description>
<dc:creator>STEVEN DAVIDOFF SOLOMON</dc:creator>
<pubDate>Tue, 26 Jul 2016 15:56:03 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Shaving and Shavers</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">E-Commerce</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Dollar Shave Club</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Unilever NV</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Start-ups</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Mergers, Acquisitions and Divestitures</category>
</item>
<item>
<title>Bits: What Goes Up Must Come Down: The End of Yahoo as We Know It</title>
<link>http://www.nytimes.com/2016/07/27/technology/what-goes-up-must-come-down-the-end-of-yahoo-as-we-know-it.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/27/technology/what-goes-up-must-come-down-the-end-of-yahoo-as-we-know-it.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/27/technology/what-goes-up-must-come-down-the-end-of-yahoo-as-we-know-it.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/27/technology/27bitsdaily-web/27bitsdaily-web-moth.jpg" medium="image" height="151" width="151"/>
<media:description>Marissa Mayer, the chief of Yahoo, at the company’s developers conference in San Francisco in February.</media:description>
<media:credit>Ramin Rahimian for The New York Times</media:credit>
<description>The sale of Yahoo’s core web business to Verizon for $4.83 billion caps a long downward spiral for the onetime pioneer, which at its height was valued at $125 billion.</description>
<dc:creator>PUI-WING TAM</dc:creator>
<pubDate>Tue, 26 Jul 2016 15:47:04 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Yahoo! Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Verizon Communications Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_per">Mayer, Marissa</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Facebook Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Google Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Appointments and Executive Changes</category>
</item>
<item>
<title>Tech Tip: Easy Editing for Mac Photos</title>
<link>http://www.nytimes.com/2016/07/27/technology/personaltech/easy-editing-for-mac-photos.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/27/technology/personaltech/easy-editing-for-mac-photos.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/27/technology/personaltech/easy-editing-for-mac-photos.html?partner=rss&emc=rss"/>
<description>Apple’s Photos for OS X has a button for one-click improvements to your pictures, but it also includes a toolbox for more detailed adjustments.</description>
<dc:creator>J. D. BIERSDORFER</dc:creator>
<pubDate>Tue, 26 Jul 2016 15:30:58 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Mac OS (Operating System)</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Photography</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Software</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Apple Inc</category>
</item>
<item>
<title>How a Homeless Teenager’s Viral Story Caused a Battle Over GoFundMe Money</title>
<link>http://www.nytimes.com/2016/07/27/us/how-a-homeless-teenagers-viral-story-caused-a-battle-over-gofundme-money.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/27/us/how-a-homeless-teenagers-viral-story-caused-a-battle-over-gofundme-money.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/27/us/how-a-homeless-teenagers-viral-story-caused-a-battle-over-gofundme-money.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/26/us/27xp-BARLEY/27xp-BARLEY-moth.jpg" medium="image" height="151" width="151"/>
<media:description>Fred Barley in an image from a YouTube video.</media:description>
<media:credit>WSBTV</media:credit>
<description>A fund to help Fred Barley, 19, an aspiring college student in Georgia, raised $184,000. Now there are suspicions about where the money was going.</description>
<dc:creator>KATIE ROGERS</dc:creator>
<pubDate>Tue, 26 Jul 2016 14:54:07 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Philanthropy</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Social Media</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">GoFundMe</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_per">Barley, Fred</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_per">Blaney, Casey</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_geo">Georgia</category>
</item>
<item>
<title>Open Blog: The Future of the Past: Modernizing the New York Times Archive</title>
<link>http://open.blogs.nytimes.com/2016/07/26/the-future-of-the-past-modernizing-the-new-york-times-archive/?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://open.blogs.nytimes.com/2016/07/26/the-future-of-the-past-modernizing-the-new-york-times-archive/</guid>
<atom:link rel="standout" href="http://open.blogs.nytimes.com/2016/07/26/the-future-of-the-past-modernizing-the-new-york-times-archive/?partner=rss&emc=rss"/>
<description>Thanks to a cross-team migration effort, we’re thrilled to announced that nearly every article published since 2004 is available to our readers with the most current New York Times digital article design.</description>
<dc:creator>SOPHIA VAN VALKENBURG and EVAN SANDHAUS</dc:creator>
<pubDate>Tue, 26 Jul 2016 16:50:38 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/">archives</category>
<category domain="http://www.nytimes.com/namespaces/">TimesMachine</category>
<category domain="http://www.nytimes.com/namespaces/">Archives and Records</category>
<category domain="http://www.nytimes.com/namespaces/">New York Times</category>
<category domain="http://www.nytimes.com/namespaces/">Projects</category>
</item>
<item>
<title>Editorial Notebook: Yahoo, a Web Pioneer, Cleared the Way for Many Innovations</title>
<link>http://www.nytimes.com/2016/07/26/opinion/yahoo-a-web-pioneer-cleared-the-way-for-many-innovations.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/26/opinion/yahoo-a-web-pioneer-cleared-the-way-for-many-innovations.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/26/opinion/yahoo-a-web-pioneer-cleared-the-way-for-many-innovations.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/26/opinion/26tue3web/26tue3web-moth.jpg" medium="image" height="151" width="151"/>
<media:description/>
<media:credit>Allan Tannenbaum/Getty Images</media:credit>
<description>A fond send-off for a company that brought order to the fast-growing internet forest.</description>
<dc:creator>VIKAS BAJAJ</dc:creator>
<pubDate>Tue, 26 Jul 2016 07:21:01 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Alphabet Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Verizon Communications Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Yahoo! Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Computers and the Internet</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Social Media</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Mergers, Acquisitions and Divestitures</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Editorials</category>
</item>
<item>
<title>Thinking About Suing Uber? Let This Be a Warning.</title>
<link>http://www.nytimes.com/2016/07/26/nyregion/investigation-of-conservationist-conducted-on-ubers-behalf-crossed-the-line-judge-rules.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/26/nyregion/investigation-of-conservationist-conducted-on-ubers-behalf-crossed-the-line-judge-rules.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/26/nyregion/investigation-of-conservationist-conducted-on-ubers-behalf-crossed-the-line-judge-rules.html?partner=rss&emc=rss"/>
<description>A Yale environmentalist filed a lawsuit against Uber accusing it of price-fixing. In response, Uber hired a Ergo, an outside agency, to look into his background.</description>
<dc:creator>BENJAMIN WEISER</dc:creator>
<pubDate>Tue, 26 Jul 2016 00:55:47 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Suits and Litigation (Civil)</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Car Services and Livery Cabs</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Uber Technologies Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_per">Rakoff, Jed S</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_per">Kalanick, Travis</category>
</item>
<item>
<title>Amazon Expands Drone Testing in Britain</title>
<link>http://www.nytimes.com/2016/07/26/technology/amazon-expands-drone-testing-in-britain.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/26/technology/amazon-expands-drone-testing-in-britain.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/26/technology/amazon-expands-drone-testing-in-britain.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/26/business/26AMAZON-1/26AMAZON-1-moth.jpg" medium="image" height="151" width="151"/>
<media:description>Amazon’s Prime Air drone.</media:description>
<media:credit>Amazon</media:credit>
<description>The British Civil Aviation Authority will allow the company to test several technologies that the United States has not permitted.</description>
<dc:creator>CECILIA KANG</dc:creator>
<pubDate>Mon, 25 Jul 2016 23:01:18 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Amazon.com Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Drones (Pilotless Planes)</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Delivery Services</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_geo">Great Britain</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Regulation and Deregulation of Industry</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_geo">United States</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Federal Aviation Administration</category>
</item>
<item>
<title>Comic-Con Makes a Fashion Statement</title>
<link>http://www.nytimes.com/2016/07/22/fashion/comic-con-makes-fashion-her-universe.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/22/fashion/comic-con-makes-fashion-her-universe.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/22/fashion/comic-con-makes-fashion-her-universe.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/24/fashion/22COMICCONFASHION1-WEB/22COMICCONFASHION1-WEB-moth.jpg" medium="image" height="151" width="151"/>
<media:description>Laura Cristina Ortiz’s 1980s-style cocktail dress and shrug inspired by the Disney Pixar film “Wall-E.”</media:description>
<media:credit>Jake Michaels for The New York Times</media:credit>
<description>A dress made from Legos and another made from recycled trash inspired by “Wall-E” were just some of the designs modeled on the Her Universe runway.</description>
<dc:creator>KAREN YOSSMAN</dc:creator>
<pubDate>Fri, 22 Jul 2016 20:23:53 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Women and Girls</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Comic-Con (Conventions)</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">MAC Cosmetics</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_per">Eckstein, Ashley</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_geo">San Diego (Calif)</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_keyword">Her Universe Fashion Show</category>
</item>
<item>
<title>For Yahoo, Question Is What to Do With $40 Billion in Leftovers</title>
<link>http://www.nytimes.com/2016/07/26/business/dealbook/for-yahoo-question-is-what-to-do-with-40-billion-in-leftovers.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/26/business/dealbook/for-yahoo-question-is-what-to-do-with-40-billion-in-leftovers.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/26/business/dealbook/for-yahoo-question-is-what-to-do-with-40-billion-in-leftovers.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/26/business/26alibabajpg/26alibabajpg-moth.jpg" medium="image" height="151" width="151"/>
<media:description>Among its remaining assets, Yahoo owns a 15 percent stake in Alibaba. But Yahoo would incur an enormous tax bill if it sold the stake outright — to the tune of billions of dollars — since its initial investment in the Chinese e-commerce titan has skyrocketed in value.</media:description>
<media:credit>Aly Song/Reuters</media:credit>
<description>After a sale of its core internet business to Verizon, Yahoo will still have stakes in Alibaba and Yahoo Japan, along with cash and a collection of patents.</description>
<dc:creator>MICHAEL J. de la MERCED</dc:creator>
<pubDate>Mon, 25 Jul 2016 21:29:22 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Yahoo! Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Alibaba Group Holding Ltd</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">SOFTBANK Corporation</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Mergers, Acquisitions and Divestitures</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Federal Taxes (US)</category>
</item>
<item>
<title>Virtual Reality Waits for the Music Industry to Catch Up</title>
<link>http://www.nytimes.com/2016/07/26/arts/music/virtual-reality-waits-for-the-music-industry-to-catch-up.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/26/arts/music/virtual-reality-waits-for-the-music-industry-to-catch-up.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/26/arts/music/virtual-reality-waits-for-the-music-industry-to-catch-up.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/26/arts/26VIRTURAL/26VIRTURAL-moth-v2.jpg" medium="image" height="151" width="151"/>
<media:description>Coldplay fans can experience part of a 2014 concert in virtual reality.</media:description>
<media:credit>NextVR</media:credit>
<description>Many companies are ready to create VR concerts, but not enough consumers have the headsets needed to experience the immersive performances.</description>
<dc:creator>GREGORY SCHMIDT</dc:creator>
<pubDate>Tue, 26 Jul 2016 19:06:21 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Virtual Reality (Computers)</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Computer and Video Games</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Pop and Rock Music</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_per">Deadmau5</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Absolut</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">iHeartMedia Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">NextVR Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Jaunt Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Universal Music Group</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Vrtify Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Music</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Duran Duran (Music Group)</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Coldplay (Music Group)</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">U2 (Music Group)</category>
</item>
<item>
<title>Bits: Tech Behemoths Report Their Results</title>
<link>http://www.nytimes.com/2016/07/26/technology/tech-behemoths-report-their-results.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/26/technology/tech-behemoths-report-their-results.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/26/technology/tech-behemoths-report-their-results.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/26/technology/26bitsdaily-web/26bitsdaily-web-moth.jpg" medium="image" height="151" width="151"/>
<media:description>Larry Page, the co-founder of Google and chief executive of Alphabet.</media:description>
<media:credit>Jeff Chiu/Associated Press</media:credit>
<description>Apple, Facebook, Amazon and Alphabet will post quarterly earnings this week, reflecting how they are doing in online ads, mobile and cloud computing.</description>
<dc:creator>PUI-WING TAM</dc:creator>
<pubDate>Mon, 25 Jul 2016 15:16:07 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Company Reports</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Computers and the Internet</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Alphabet Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Amazon.com Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Apple Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Calico (California Life Company)</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Facebook Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Google Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Nest Labs Inc</category>
</item>
<item>
<title>Yahoo Mail and Tumblr and Flickr: What Happens Now?</title>
<link>http://www.nytimes.com/2016/07/26/technology/yahoo-mail-verizon-sale-changes.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/26/technology/yahoo-mail-verizon-sale-changes.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/26/technology/yahoo-mail-verizon-sale-changes.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/26/business/26YAHOOSERVICES/26YAHOOSERVICES-moth.jpg" medium="image" height="151" width="151"/>
<media:description/>
<media:credit>Brendan Mcdermid/Reuters</media:credit>
<description>Here are answers to some common questions about the impact Verizon’s purchase will have on Yahoo’s numerous services.</description>
<dc:creator>VINDU GOEL</dc:creator>
<pubDate>Mon, 25 Jul 2016 21:02:24 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Verizon Communications Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Federal Communications Commission</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Mergers, Acquisitions and Divestitures</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Search Engines</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Mozilla Foundation</category>
</item>
<item>
<title>This Company Will Give a Peek Inside How Much Private Start-Ups Are Worth</title>
<link>http://www.nytimes.com/2016/07/26/technology/making-start-ups-financial-data-free-and-open.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/26/technology/making-start-ups-financial-data-free-and-open.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/26/technology/making-start-ups-financial-data-free-and-open.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/26/business/26transparent1/26transparent1-moth-v2.jpg" medium="image" height="151" width="151"/>
<media:description>Private companies like Uber are not required to share financial information.</media:description>
<media:credit>Mark Ralston/Agence France-Presse — Getty Images</media:credit>
<description>The lack of transparency around private companies has kept the market for start-up stocks small. Equidate, a San Francisco company, hopes to change that.</description>
<dc:creator>KATIE BENNER</dc:creator>
<pubDate>Mon, 25 Jul 2016 15:00:02 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Start-ups</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Equidate Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Stocks and Bonds</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Venture Capital</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Securities and Exchange Commission</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Uber Technologies Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Lyft Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Snapchat Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Entrepreneurship</category>
</item>
<item>
<title>Verizon Announces $4.8 Billion Deal for Yahoo’s Internet Business</title>
<link>http://www.nytimes.com/2016/07/26/business/verizon-yahoo-sale.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/26/business/verizon-yahoo-sale.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/26/business/verizon-yahoo-sale.html?partner=rss&emc=rss"/>
<description>Verizon is buying an entity that has made repeated missteps, but one that could help it be a stronger contender for digital advertising revenue.</description>
<dc:creator>VINDU GOEL</dc:creator>
<pubDate>Mon, 25 Jul 2016 17:09:34 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Computers and the Internet</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Mergers, Acquisitions and Divestitures</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Yahoo! Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Verizon Communications Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_per">Mayer, Marissa</category>
</item>
<item>
<title>Elon Musk of Tesla Sticks to Mission Despite Setbacks</title>
<link>http://www.nytimes.com/2016/07/25/business/despite-roadblocks-for-tesla-elon-musk-is-moving-full-speed-ahead.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/25/business/despite-roadblocks-for-tesla-elon-musk-is-moving-full-speed-ahead.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/25/business/despite-roadblocks-for-tesla-elon-musk-is-moving-full-speed-ahead.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/23/business/00musk8/00musk8-moth.jpg" medium="image" height="151" width="151"/>
<media:description>“The faster we can transition to low carbon, maybe, ultimately, to a negative carbon economy, the better,” Elon Musk said.</media:description>
<media:credit>Bobby Yip/Reuters</media:credit>
<description>The billionaire entrepreneur Elon Musk is running a private rocket company, doubling down on alternative fuels and pressing on with Tesla after a series of accidents.</description>
<dc:creator>MATT RICHTEL</dc:creator>
<pubDate>Mon, 25 Jul 2016 18:35:32 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_per">Musk, Elon</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Tesla Motors Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Space Exploration Technologies Corp</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">SolarCity Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Electric and Hybrid Vehicles</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Alternative and Renewable Energy</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Automobile Safety Features and Defects</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Global Warming</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Batteries</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Lithium (Metal)</category>
</item>
<item>
<title>Researchers Who Exposed VW Gain Little Reward From Success</title>
<link>http://www.nytimes.com/2016/07/25/business/vw-wvu-diesel-volkswagen-west-virginia.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/25/business/vw-wvu-diesel-volkswagen-west-virginia.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/25/business/vw-wvu-diesel-volkswagen-west-virginia.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/23/business/25emissionslab3/25emissionslab3-moth.jpg" medium="image" height="151" width="151"/>
<media:description>Dr. Arvind Thiruvengadam, an assistant professor, at the Center for Alternative Fuels Engines and Emissions at West Virginia University. As a graduate student, he helped test emissions using a mobile lab.</media:description>
<media:credit>Tom M. Johnson for The New York Times</media:credit>
<description>The team, at West Virginia University, often scrounges for grants. That financial pressure is unlikely to dissipate, despite uncovering the emissions fraud.</description>
<dc:creator>JACK EWING</dc:creator>
<pubDate>Mon, 25 Jul 2016 00:10:57 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Volkswagen AG</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Fuel Emissions (Transportation)</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Automobiles</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Frauds and Swindling</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">West Virginia University</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Research</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Automobile Safety Features and Defects</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Air Pollution</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Diesel Power</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Colleges and Universities</category>
</item>
<item>
<title>How Sponsored Content Is Becoming King in a Facebook World</title>
<link>http://www.nytimes.com/2016/07/25/business/sponsored-content-takes-larger-role-in-media-companies.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/25/business/sponsored-content-takes-larger-role-in-media-companies.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/25/business/sponsored-content-takes-larger-role-in-media-companies.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/26/business/26NATIVE/26NATIVE-moth.jpg" medium="image" height="151" width="151"/>
<media:description>Attendees at a Facebook developers conference in April. As it has for traditional editorial content, Facebook has become a primary distributor for many publications’ sponsored posts, even though outside sponsored content was not officially permitted until April, when the social network published formal guidelines.</media:description>
<media:credit>Michael Short/Bloomberg</media:credit>
<description>As native advertising gains prominence, publishers may find themselves competing not just with one another, but with the ad agencies that already exist.</description>
<dc:creator>JOHN HERRMAN</dc:creator>
<pubDate>Tue, 26 Jul 2016 15:47:02 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Online Advertising</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">News and News Media</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Social Media</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_keyword">Native advertising</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Facebook Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Advertising and Marketing</category>
</item>
<item>
<title>State of the Art: Live Streaming Breaks Through, and Cable News Has Much to Fear</title>
<link>http://www.nytimes.com/2016/07/14/technology/live-streaming-breaks-through-and-cable-news-has-much-to-fear.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/14/technology/live-streaming-breaks-through-and-cable-news-has-much-to-fear.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/14/technology/live-streaming-breaks-through-and-cable-news-has-much-to-fear.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/14/technology/14state-illo-larger/14state-illo-larger-moth.jpg" medium="image" height="151" width="151"/>
<media:description/>
<media:credit>Stuart Goldenberg</media:credit>
<description>The Philando Castile shooting and its aftermath have catapulted services like Facebook Live and Periscope into the center of the news, challenging cable to adapt.</description>
<dc:creator>FARHAD MANJOO</dc:creator>
<pubDate>Thu, 14 Jul 2016 20:57:58 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Social Media</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Video Recordings, Downloads and Streaming</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">News and News Media</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Television</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Police Brutality, Misconduct and Shootings</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Attacks on Police</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Facebook Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Twitter</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Dallas, Tex, Shooting of Police Officers (2016)</category>
</item>
<item>
<title>Another Night, Another Shooting on Facebook Live</title>
<link>http://www.nytimes.com/2016/07/14/us/another-night-another-shooting-on-facebook-live.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/14/us/another-night-another-shooting-on-facebook-live.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/14/us/another-night-another-shooting-on-facebook-live.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/13/us/14xp-shooting/14xp-shooting-moth.jpg" medium="image" height="151" width="151"/>
<media:description>An image from a video taken moments before a shooting in Norfolk, Va., on Tuesday.</media:description>
<media:credit>YouTube</media:credit>
<description>Three men are badly injured as video streams the gunfire and the aftermath in Norfolk, Va.</description>
<dc:creator>MIKE McPHATE</dc:creator>
<pubDate>Thu, 14 Jul 2016 11:25:20 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Social Media</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Computers and the Internet</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Video Recordings, Downloads and Streaming</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Facebook Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Periscope (Twitter Inc)</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Twitter</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_per">Zuckerberg, Mark E</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_geo">Norfolk (Va)</category>
</item>
<item>
<title>How to Protect Privacy While Using Pokémon Go and Other Apps</title>
<link>http://www.nytimes.com/2016/07/14/technology/personaltech/how-to-protect-privacy-while-using-pokemon-go-and-other-apps.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/14/technology/personaltech/how-to-protect-privacy-while-using-pokemon-go-and-other-apps.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/14/technology/personaltech/how-to-protect-privacy-while-using-pokemon-go-and-other-apps.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/13/business/14POKEMON/14POKEMON-moth.jpg" medium="image" height="151" width="151"/>
<media:description>A Pokémon appears next to a woman on a man’s phone screen as he plays the augmented reality mobile game Pokémon Go in New York.</media:description>
<media:credit>Mark Kauzlarich/Reuters</media:credit>
<description>The game’s developer made expansive permission requests — in error, it says, and it uses only basic data — but many apps make similar requests.</description>
<dc:creator>LAURA HUDSON</dc:creator>
<pubDate>Tue, 19 Jul 2016 18:41:16 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Mobile Applications</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Privacy</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Pokemon (Fictional Characters)</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Niantic Inc</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Nintendo Co Ltd</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Smartphones</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Computer and Video Games</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_ttl">Pokemon Go (Video Game)</category>
</item>
<item>
<title>Where Pokémon Should Not Go</title>
<link>http://www.nytimes.com/2016/07/13/technology/where-pokemon-should-not-go.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/13/technology/where-pokemon-should-not-go.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/13/technology/where-pokemon-should-not-go.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/12/world/13xp-pokenogo/13xp-pokenogo-moth.jpg" medium="image" height="151" width="151"/>
<media:description>A user plays Pokémon Go in Melbourne, Australia.</media:description>
<media:credit>Julian Smith/European Pressphoto Agency</media:credit>
<description>The digital beasts have been reported at Auschwitz and memorials for the Holocaust and the Sept. 11 attacks.</description>
<dc:creator>JONAH ENGEL BROMWICH</dc:creator>
<pubDate>Wed, 13 Jul 2016 18:41:39 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Holocaust and the Nazi Era</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">September 11 (2001)</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Computer and Video Games</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Concentration Camps</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Pokemon (Fictional Characters)</category>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Social Media</category>
</item>
<item>
<title>How to Make Pokémon Go Actually Useful</title>
<link>http://www.nytimes.com/2016/07/13/travel/pokemon-go-nyc-tourism.html?partner=rss&emc=rss</link>
<guid isPermaLink="true">http://www.nytimes.com/2016/07/13/travel/pokemon-go-nyc-tourism.html</guid>
<atom:link rel="standout" href="http://www.nytimes.com/2016/07/13/travel/pokemon-go-nyc-tourism.html?partner=rss&emc=rss"/>
<media:content url="https://static01.nyt.com/images/2016/07/13/travel/13pokemon-tournie/13pokemon-tourn-moth.jpg" medium="image" height="151" width="151"/>
<media:description>Capturing Pokémon characters in Central Park.</media:description>
<media:credit>George Etheredge/The New York Times</media:credit>
<description>Hunting Pokémon can be a fun way to explore a city, though it’s not without its drawbacks. Here are tips on how and where to play as a traveler.</description>
<dc:creator>JUSTIN SABLICH</dc:creator>
<pubDate>Thu, 14 Jul 2016 20:39:28 GMT</pubDate>
<category domain="http://www.nytimes.com/namespaces/keywords/mdes">Travel and Vacations</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Computer and Video Games</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Mobile Applications</category>
<category domain="http://www.nytimes.com/namespaces/keywords/des">Pokemon (Fictional Characters)</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_org_all">Nintendo Co Ltd</category>
<category domain="http://www.nytimes.com/namespaces/keywords/nyt_geo">New York City</category>
</item>
</channel>
</rss>
into our tsv records:
Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Apple Sales Continue to Drop, but Services Are a Bright Spot Apple’s new flagship store in San Francisco.A second quarter of dismal sales leaves analysts wondering: Are customers out of love with Apple, or just waiting for the latest products? http://www.nytimes.com/2016/07/27/technology/apple-earnings-iphone-sales.html?partner=rss&emc=rss Tue, 26 Jul 2016 21:54:37 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Marissa Mayer’s Media Problem at Yahoo Is Now Verizon’s to Solve Marissa Mayer, chief executive of Yahoo, and Adam Cahan, a senior vice president, at a mobile developer conference in February 2015 in San Francisco.Before she led Yahoo, while still at Google, Ms. Mayer spoke of the difficulties for media companies of capitalizing on their own content. http://www.nytimes.com/2016/07/27/business/media/marissa-mayers-media-problem-at-yahoo-is-now-verizo... Tue, 26 Jul 2016 20:14:48 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss F.B.I. Examining if Hackers Gained Access to Clinton Aides’ Emails Workers wrapped up preparations on Monday at the Wells Fargo Arena in Philadelphia for the Democratic National Convention.Federal officials say the inquiry has been underway since the spring, when the agency was first notified of the D.N.C.’s suspicions about hacking. http://www.nytimes.com/2016/07/26/us/politics/fbi-investigating-intrusions-into-democratic-committee... Tue, 26 Jul 2016 01:31:18 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Robert Fano, 98, Dies; Engineer Who Helped Develop Interactive Computers Robert Fano.In the early 1960s, Dr. Fano and others developed a time-sharing operating system, which signaled a shift to a more interactive computing world. http://www.nytimes.com/2016/07/27/technology/robert-fano-98-dies-engineer-who-helped-develop-interac... Tue, 26 Jul 2016 19:34:21 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Deal Professor: $1 Billion for Dollar Shave Club: Why Every Company Should Worry The internet, mass transportation and globalization allow decentralized companies to be smaller and leaner and have fewer employees. http://www.nytimes.com/2016/07/27/business/dealbook/1-billion-for-dollar-shave-club-why-every-compan... Tue, 26 Jul 2016 15:56:03 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Bits: What Goes Up Must Come Down: The End of Yahoo as We Know It Marissa Mayer, the chief of Yahoo, at the company’s developers conference in San Francisco in February.The sale of Yahoo’s core web business to Verizon for $4.83 billion caps a long downward spiral for the onetime pioneer, which at its height was valued at $125 billion. http://www.nytimes.com/2016/07/27/technology/what-goes-up-must-come-down-the-end-of-yahoo-as-we-know... Tue, 26 Jul 2016 15:47:04 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Tech Tip: Easy Editing for Mac Photos Apple’s Photos for OS X has a button for one-click improvements to your pictures, but it also includes a toolbox for more detailed adjustments. http://www.nytimes.com/2016/07/27/technology/personaltech/easy-editing-for-mac-photos.html?partner=r... Tue, 26 Jul 2016 15:30:58 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss How a Homeless Teenager’s Viral Story Caused a Battle Over GoFundMe Money Fred Barley in an image from a YouTube video.A fund to help Fred Barley, 19, an aspiring college student in Georgia, raised $184,000. Now there are suspicions about where the money was going. http://www.nytimes.com/2016/07/27/us/how-a-homeless-teenagers-viral-story-caused-a-battle-over-gofun... Tue, 26 Jul 2016 14:54:07 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Open Blog: The Future of the Past: Modernizing the New York Times Archive Thanks to a cross-team migration effort, we’re thrilled to announced that nearly every article published since 2004 is available to our readers with the most current New York Times digital article design. http://open.blogs.nytimes.com/2016/07/26/the-future-of-the-past-modernizing-the-new-york-times-archi... Tue, 26 Jul 2016 16:50:38 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Editorial Notebook: Yahoo, a Web Pioneer, Cleared the Way for Many Innovations A fond send-off for a company that brought order to the fast-growing internet forest. http://www.nytimes.com/2016/07/26/opinion/yahoo-a-web-pioneer-cleared-the-way-for-many-innovations.h... Tue, 26 Jul 2016 07:21:01 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Thinking About Suing Uber? Let This Be a Warning. A Yale environmentalist filed a lawsuit against Uber accusing it of price-fixing. In response, Uber hired a Ergo, an outside agency, to look into his background. http://www.nytimes.com/2016/07/26/nyregion/investigation-of-conservationist-conducted-on-ubers-behal... Tue, 26 Jul 2016 00:55:47 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Amazon Expands Drone Testing in Britain Amazon’s Prime Air drone.The British Civil Aviation Authority will allow the company to test several technologies that the United States has not permitted. http://www.nytimes.com/2016/07/26/technology/amazon-expands-drone-testing-in-britain.html?partner=rs... Mon, 25 Jul 2016 23:01:18 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Comic-Con Makes a Fashion Statement Laura Cristina Ortiz’s 1980s-style cocktail dress and shrug inspired by the Disney Pixar film “Wall-E.”A dress made from Legos and another made from recycled trash inspired by “Wall-E” were just some of the designs modeled on the Her Universe runway. http://www.nytimes.com/2016/07/22/fashion/comic-con-makes-fashion-her-universe.html?partner=rss&emc=... Fri, 22 Jul 2016 20:23:53 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss For Yahoo, Question Is What to Do With $40 Billion in Leftovers Among its remaining assets, Yahoo owns a 15 percent stake in Alibaba. But Yahoo would incur an enormous tax bill if it sold the stake outright — to the tune of billions of dollars — since its initial investment in the Chinese e-commerce titan has skyrocketed in value.After a sale of its core internet business to Verizon, Yahoo will still have stakes in Alibaba and Yahoo Japan, along with cash and a collection of patents. http://www.nytimes.com/2016/07/26/business/dealbook/for-yahoo-question-is-what-to-do-with-40-billion... Mon, 25 Jul 2016 21:29:22 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Virtual Reality Waits for the Music Industry to Catch Up Coldplay fans can experience part of a 2014 concert in virtual reality.Many companies are ready to create VR concerts, but not enough consumers have the headsets needed to experience the immersive performances. http://www.nytimes.com/2016/07/26/arts/music/virtual-reality-waits-for-the-music-industry-to-catch-u... Tue, 26 Jul 2016 19:06:21 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Bits: Tech Behemoths Report Their Results Larry Page, the co-founder of Google and chief executive of Alphabet.Apple, Facebook, Amazon and Alphabet will post quarterly earnings this week, reflecting how they are doing in online ads, mobile and cloud computing. http://www.nytimes.com/2016/07/26/technology/tech-behemoths-report-their-results.html?partner=rss&em... Mon, 25 Jul 2016 15:16:07 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Yahoo Mail and Tumblr and Flickr: What Happens Now? Here are answers to some common questions about the impact Verizon’s purchase will have on Yahoo’s numerous services. http://www.nytimes.com/2016/07/26/technology/yahoo-mail-verizon-sale-changes.html?partner=rss&emc=rs... Mon, 25 Jul 2016 21:02:24 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss This Company Will Give a Peek Inside How Much Private Start-Ups Are Worth Private companies like Uber are not required to share financial information.The lack of transparency around private companies has kept the market for start-up stocks small. Equidate, a San Francisco company, hopes to change that. http://www.nytimes.com/2016/07/26/technology/making-start-ups-financial-data-free-and-open.html?part... Mon, 25 Jul 2016 15:00:02 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Verizon Announces $4.8 Billion Deal for Yahoo’s Internet Business Verizon is buying an entity that has made repeated missteps, but one that could help it be a stronger contender for digital advertising revenue. http://www.nytimes.com/2016/07/26/business/verizon-yahoo-sale.html?partner=rss&emc=rss Mon, 25 Jul 2016 17:09:34 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Elon Musk of Tesla Sticks to Mission Despite Setbacks “The faster we can transition to low carbon, maybe, ultimately, to a negative carbon economy, the better,” Elon Musk said.The billionaire entrepreneur Elon Musk is running a private rocket company, doubling down on alternative fuels and pressing on with Tesla after a series of accidents. http://www.nytimes.com/2016/07/25/business/despite-roadblocks-for-tesla-elon-musk-is-moving-full-spe... Mon, 25 Jul 2016 18:35:32 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Researchers Who Exposed VW Gain Little Reward From Success Dr. Arvind Thiruvengadam, an assistant professor, at the Center for Alternative Fuels Engines and Emissions at West Virginia University. As a graduate student, he helped test emissions using a mobile lab.The team, at West Virginia University, often scrounges for grants. That financial pressure is unlikely to dissipate, despite uncovering the emissions fraud. http://www.nytimes.com/2016/07/25/business/vw-wvu-diesel-volkswagen-west-virginia.html?partner=rss&e... Mon, 25 Jul 2016 00:10:57 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss How Sponsored Content Is Becoming King in a Facebook World Attendees at a Facebook developers conference in April. As it has for traditional editorial content, Facebook has become a primary distributor for many publications’ sponsored posts, even though outside sponsored content was not officially permitted until April, when the social network published formal guidelines.As native advertising gains prominence, publishers may find themselves competing not just with one another, but with the ad agencies that already exist. http://www.nytimes.com/2016/07/25/business/sponsored-content-takes-larger-role-in-media-companies.ht... Tue, 26 Jul 2016 15:47:02 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss State of the Art: Live Streaming Breaks Through, and Cable News Has Much to Fear The Philando Castile shooting and its aftermath have catapulted services like Facebook Live and Periscope into the center of the news, challenging cable to adapt. http://www.nytimes.com/2016/07/14/technology/live-streaming-breaks-through-and-cable-news-has-much-t... Thu, 14 Jul 2016 20:57:58 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Another Night, Another Shooting on Facebook Live An image from a video taken moments before a shooting in Norfolk, Va., on Tuesday.Three men are badly injured as video streams the gunfire and the aftermath in Norfolk, Va. http://www.nytimes.com/2016/07/14/us/another-night-another-shooting-on-facebook-live.html?partner=rs... Thu, 14 Jul 2016 11:25:20 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss How to Protect Privacy While Using Pokémon Go and Other Apps A Pokémon appears next to a woman on a man’s phone screen as he plays the augmented reality mobile game Pokémon Go in New York.The game’s developer made expansive permission requests — in error, it says, and it uses only basic data — but many apps make similar requests. http://www.nytimes.com/2016/07/14/technology/personaltech/how-to-protect-privacy-while-using-pokemon... Tue, 19 Jul 2016 18:41:16 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss Where Pokémon Should Not Go A user plays Pokémon Go in Melbourne, Australia.The digital beasts have been reported at Auschwitz and memorials for the Holocaust and the Sept. 11 attacks. http://www.nytimes.com/2016/07/13/technology/where-pokemon-should-not-go.html?partner=rss&emc=rss Wed, 13 Jul 2016 18:41:39 GMT Tue, 26 Jul 2016 23:11:25 GMT NYT > Technology http://www.nytimes.com/pages/technology/index.html?partner=rss&emc=rss How to Make Pokémon Go Actually Useful Capturing Pokémon characters in Central Park.Hunting Pokémon can be a fun way to explore a city, though it’s not without its drawbacks. Here are tips on how and where to play as a traveler. http://www.nytimes.com/2016/07/13/travel/pokemon-go-nyc-tourism.html?partner=rss&emc=rss Thu, 14 Jul 2016 20:39:28 GMT
MergeContent
This processor merges the results of each ExecuteScript (one transformed tsv for each configured RSS feed) into one FlowFile (tsv for all RSS results combined). Thus, if we originally configured 100 RSS URLs at the beginning of the data flow, this processor combines the multi-lined results of each 100 into one unioned tsv. The MergeContent configurations here are all defaults. The important property is Delimiter Strategy = Filename (vs text delimiter).
UpdateAttribute
I want to put the file to HDFS but I want to name the HDFS file by timestamp so I need to update the filename attribute. Filename is an attribute for the FlowFile passed from one processor to the next. By default for my flow it is the name of the local file picked up by the first processor GetFile. I will use a simple implementation of the NiFi Expression Language to rename this attribute which will be passed and used by the next processor to put our merged result to HDFS.
Whenever we see ${} we are looking at the expression language. You can see above that the filename has been changed the current timestamp as formatted by the expression language.
PutHDFS
Now it is a simple matter of putting the file to HDFS. The important configuration here is path to core-site.xml stored locally, and HDFS directory to write the the output to.
Results
Conclusion
NiFi is powerful and agile way to ingest data from diverse sources into a data lake. In the case shown here, InvokeHTTP is the key processor in this flow but other processors that were used, with their expression language and regex capabilities, show how easy it is to manipulate the contents of a dataflow as well as the key value pairs (attributes) associated or added to it. Finally, the ability to include groovy scripting with its native java capabilities and 3rd-party library integration opens enormous possibilities in transforming data in the flow. And of course, this how-to article shows only the tip of the iceberg regarding NiFi capabilities.
The next time you are using flume, sqoop or an ETL tool ... think about the power, ease and reach of NiFi instead. Hortonworks distributes NiFi within its HDF data-in-motion platform -- a great integration with the HDP data-at-rest platform that we all know and love.
References
Quick shout-out to @Matt Burgess whose HCC NiFi and blog have been quite useful.
- http://hortonworks.com/apache/nifi/
- https://community.hortonworks.com/articles/7999/apache-nifi-part-1-introduction.html
- https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html
Announcements
What's New @ Cloudera
What's New @ Cloudera
What's New @ Cloudera
What's New @ Cloudera
What's New @ Cloudera
