Big Data News – 15 Sep 2016

Featured Article
This podcast features Doug Thompson, executive architect, Information Governance and master data management (MDM), at IBM, who discusses the various factors that make data integration challenging and how one can overcome them. Thompson also shares his views on hybrid- and cloud-based data integration and the opportunity to become highly efficient.

Top Stories
Chris Snow, a data and application architect, enjoys helping customers with their data architectures and is working extensively on an open source app project in his spare time. Hear what Snow has to say about his IT experience spanning several industries, his current efforts with customers and his involvement in the open source project in an interview with Andrea Braida, big data analytics product marketing at IBM.

Defining semantics is a matter of semantics, not less so in the Big Data space.

Facebook at Work has been in closed beta for the last 20 months, but the world's largest social network plans to officially push into enterprise later this year, and the company is adding features at a steady clip. Last month, for example, the team introduced group audio calling, a new call tab, a modular inbox for work chat that highlights frequent contacts and recent or unread conversations, search filters and live video. Weber Shandwick, a well-known public relations (PR) firm with offices in 78 cities in 34 countries, was one of the first organizations to start using Facebook's business product last year. Today, the company's entire workforce of roughly 4,000 employees uses Facebook at Work to collaborate, share tips with interested clients, and communicate in real time. When Facebook introduced live video to the platform last month, Weber Shandwick's CEO Andy Polansky was one of the first to use the feature for business communications.  

O'Reilly has released the results of the 2016 Data Science Salary Survey. This survey is based on data from over 900 respondents to a 64-question survey about data-related tasks, tools, and the salary they receive from doing/using them. The median salary reported in the survey was US$87,000; amongst data scientists in the US, the median salary was US$106,000. Appropriately for a survey about data science, O'Reilly doesn't merely report aggregate statistics from the survey; they fit a linear regression model for a data, and extact coefficients from the model indicative of salary "bumps" (or downgrades) attributable to demographic factors.

Has your business adopted a hybrid analytics architecture as part of its quest to compete? Listen as Martin Fleming, IBM's chief analytics officer and chief economist, explains why doing data analytics on the cloud is creating opportunities for modern businesses, and be sure to take notes as he outlines the things you should be keeping in mind as you move to the cloud.

Demand for data engineers is on the rise in enterprise, particularly in those organizations that aspire to be the next Uber or Airbnb. Here's a look at this quickly growing job category within IT.

Enterprise Productivity and Integration of Apache NiFi, Kafka and Storm, together with Ambari and Ranger We are pleased to announce that Hortonworks DataFlow (HDF™) Version 2.0 is now generally available for download!  As part of a Open and Connected Data Platforms offering from Hortonworks, HDF 2.0 provides a new level of enterprise integration for data… The post Announcing Hortonworks DataFlow (HDF) 2.0 GA! appeared first on Hortonworks.

Most businesses are no strangers to using video as a means to maintain good security, but that's only the tip of the iceberg in the benefits a good video system can provide. With advances in big data analytics and automated technologies, many organizations are starting to take advantage of video analytics. It's the direction much of the world is going these days, but analyzing video can be a challenge. Consider how much data is involved in a simple video.

Sophos Intercept X uses Big Data analytics applications to detect potential threats.

Spatial Big Data—be this natively geocoded content, geographical metadata, or data that itself refers to spaces and places—has become a pervasive presence in the spaces and practices of everyday life. Beyond preoccupations with "the geotag" and with mapping geocoded social media content, this special theme explores what it means to encounter and experience spatial Big Data as a quotidian phenomenon that is both spatial, characterized by and enacting of material spatialities, and spatializing, configuring relations between subjects, objects, and spaces in new and unprecedented ways.

We're seeing a huge move toward consumers buying products on a subscription basis. It used to be just electricity, phone lines and magazines, but now everything from underwear to razors, auto usage and software can be bought by subscription. The "subscription economy," as one player named it, really changes the way organizations do business. Indeed, a new survey from CFO Magazine has revealed that 70% of CFOs say more than half of their revenue now comes from services, which includes professional services, subscription-based services, service-level agreements, managed services and usage-based contract. Almost a third (28%) of the CFOs surveyed indicate all of their companies revenues are services-related.

What are some challenges retailers face as they develop omnichannel strategies to compete against the customer-experience expectations set by Amazon?

Memory management can be challenge enough on traditional data sets, but when big data enters the picture, things can slow way, way down. A new programming language announced by MIT this week aims to remedy that problem, and so far it's been found to deliver fourfold speed boosts on common algorithms. The principle of locality is what governs memory management in most computer chips today, meaning that if a program needs a chunk of data stored at some memory location, it's generally assumed to need the neighboring chunks as well. In big data, however, that's not always the case. Instead, programs often must act on just a few data items scattered across huge data sets.

In this paper, we describe how critical data designers have created projects that ‘push back’ against the eclipse of environmental problems by dominant orders: the pioneering pollution database Scorecard, released by the US NGO Environmental Defense Fund in 1997; the US Environmental Protection Agency’s EnviroAtlas that brings together numerous data sets and provides tools for valuing ecosystem services; and the Houston Clean Air Network’s maps of real-time ozone levels in Houston.




Every company wants to guarantee uptime and positive experiences for its customers. Behind the scenes, in increasingly complex IT environments, this means giving operations teams greater visibility into their systems — stretching the window of insight from hours or days to months and even multiple years. After all, how can IT leaders drive effective operations today if they don't have the full-scale visibility needed to align IT metrics with business results? Expanding the window of visibility has clear benefits in terms of identifying emerging problems anywhere in the environment, minimizing security risks, and surfacing opportunities for innovation. Yet it also has costs. From an IT operations standpoint, time is data: The further you want to see, the more data you have to collect and analyze. It is an enormous challenge to build a system that can ingest many terabytes of event data per day while maintaining years of data, all indexed and ready for search and analysis.

Gmail service for users of the Google for Work cloud-based productivity suite was down for over 12 hours on Wednesday, apparently affecting users in a number of countries including the U.S. Google reported early Thursday that the problem was resolved for the vast majority of affected customers, and it would be working individually with the rest of them. It said it would provide a more detailed analysis of the incident to customers once its internal investigation is completed. The company first acknowledged the problem on its Apps Status Dashboard at 8:16 a.m. Pacific Time on Wednesday, stating that it is investigating reports of an issue with Gmail. "Based on reports, it affects only Google for Work Gmail users," Google said. Affected users were redirected to a page with 'Service not available, contact your administrator.'

Survey of 200 executives indicates most manufacturers plan to increase investments in data analytics over next year — even while delaying other technology investments

For investors, deciding whether to invest money into renewable-energy projects can be difficult. The issue is volatility: Wind-powered energy production, for instance, changes annually — and even weekly or daily — which creates uncertainty and investment risks. With limited options to accurately quantify that volatility, today's investors tend to act conservatively. An MIT spinout, EverVest, has built a data-analytics platform whose goal is to give investors rapid, accurate cash-flow models and financial risk analyses for renewable-energy projects. Recently acquired by asset-management firm Ultra Capital, EverVest's platform could help boost investments in sustainable-infrastructure projects, including wind and solar power. Ultra Capital acquired the EverVest platform and team earlier this year in order to leverage the software for its own underwriting and risk analytics.

We introduce d2o, a Python module for cluster-distributed multi-dimensional numerical arrays. It acts as a layer of abstraction between the algorithm code and the data-distribution logic. The main goal is to achi…

This entry was posted in News and tagged , , , , , , , , . Bookmark the permalink.