The big data challenges

Products and services that receive widespread interest beyond technology circles tend to be those that people can touch and feel, or relate to: But Big Data, fundamentally, is… plumbing. Those companies were suddenly faced with unprecedented volume of data, had no legacy infrastructure and were able to recruit some of the best engineers around, so they essentially started building the technologies they needed. The ethos of open source was rapidly accelerating and a lot of those new technologies were shared with the broader world.

The big data challenges

A recurring problem I see is companies that put the technology ahead of the processes, people and specific outcomes.

Become a member

They essentially work forward from technology, instead of backwards from business outcomes. While many understand the value of information harnessed from a myriad of internal and external sources, fewer understand how to make that information accessible and actionable at the exact point where it can be used by knowledge workers across the organization.

These challenges to leveraging Big Data for SMART business objectives are no different than other information analysis methods and must start with The big data challenges integrated people, process and technology plan which includes the processes to identify and capture the data, the tools to manage access, sync, merge, store, tag, annotate, etc.

Another pervasive challenge with big data is data relevancy. With more data comes more noise.

The big data challenges

Business analysts will need to classify data into a spectrum running from noise to signals in large part based on the use case and weighted results of the data. Other challenges such as data privacy, information security, information distribution, data presentation and even data overload aka analysis paralysis are not unique to Big Data, and the risks and resolutions can be learned from the lessons and best practices of other business analytics solutions.

Top Big Data Challenges

The bigger cost is the labor needed for planning, cultural alignment, process definitions and deployment. And when labor is the largest cost involved in a technology deployment, that cost can be reduced or multiplied based on the specific resources allocated.

Clearly, untrained resources apply a trial and error approach while leveraging expert resources streamlines the effort and generally gets it right the first time. The likelihood of project success will increase dramatically if you allocate one or more creative thinkers or innovators, who can collaborate with executives and line of business managers to flush out decisions that will benefit from increased data, and quickly hypothesize the types and sources of information that will aid those decisions.

Huge sets of external data are available from Government and NGO bodies, social media and commercial services, and those people versed in these and other data sources will of course accelerate the process.

While deployment can be challenging, the really encouraging news about Big Data projects is that they generally deliver big paybacks. A recent Nucleus Research report titled The Big Returns from Big Data found that big data projects which connected internal data sets with social media earned, on average, percent ROI.

For example, one ROI analysis demonstrated how a vacation resort company drastically cut labor costs by syncing its scheduling process with data available from the National Weather Service.


In the research survey, IF the big data implementation was successful, it likely earned a significant ROI well in excess of percent. Seize the Data Big Data has not yet crossed the chasm to mainstream adoption, but is clearly delivering success for early adopters and is now at an inflexion point.

For most businesses, Big Data methods are as unique as their corporate cultures and business processes, and the technologies are more bespoke than packaged. This leaves Big Data success to those business champions that can act as change agents, rally staff around high payback projects and innovate new processes merged with enabling technology.

Business innovation and technology laggards will only acquire Big Data solutions when they are more packaged, easily deployable and no longer offer competitive advantage.

The ability for businesses to glean valuable nuggets of information from near limitless sources and apply that knowledge at just the right time to achieve tactical goals will clearly elevate those businesses over their competitors who operate without such knowledge.Planet Big Data is an aggregator of blogs about big data, Hadoop, and related topics.

Is Big Data Still a Thing? (The Big Data Landscape) – Matt Turck

We include posts by bloggers worldwide. Email us to have your blog included.. Subscribe to feed. Combining Operational and Analytical Technologies; Using Hadoop.

New technologies like NoSQL, MPP databases, and Hadoophave emerged to address Big Data challenges and to enable new types of products and services to be delivered by the business..

One of the most common ways companies are leveraging the capabilities of both systems is by integrating a NoSQL database such as MongoDB . World Bank Big Data Innovation Challenge Rethinking climate resilience through big data solutions. Applications closed World Bank Big Data Innovation.

BIG DATA: ITS POWER AND PERILS FOREWORD 5 Foreword ‘Big data’ has become a business buzzword, and its promise and pitfalls are gaining increasing attention from . Objectively, the main point of the V-based characterization of big data is to highlight its most serious challenges: the capture, cleaning, curation, integration, storage, processing, indexing, search, sharing, transfer, mining, analysis, and visualization of large volumes of fast-moving highly complex data.

Big Data (BD), with their potential to ascertain valued insights for enhanced decision-making process, have recently attracted substantial interest from both academics and practitioners.

Big Data Challenges and Opportunity | Qubole