I met with a client last week that was trying to determine what and how to deal with one of the most common and complex problems that organizations both small and large have to deal with – data management.  This oil and gas company had data coming out of its ears – production data, seismic data, HSE data, down-hole sensor data, well log data, financial data… you get the picture.  Each department was its own silo; a vast, sprawling archipelago of data islands completely disconnected from each other. Moving data from one department to another was painful and clunky, much like throwing a bunch of stuff on a boat and hoping whoever receives it finds something useful in the reams and ream of info.

I know I just used two or three different analogies to explain a problem; my apologies.  Data management isn’t really analogous to anything, given its size, scope, complexity, and the different systems and departments involved.  Data management is like data management, and that’s pretty much it.  When you ask most industry technology strategists what the big fuss with data management is about, they invariably respond with something to the effect of “companies must leverage data in order to be better informed for decision support, which will lead to higher profitability.” That’s why they are strategists and not CEOs; they can’t speak English coherently.

The problem isn’t what we want to do with the data – everyone knows that data, properly leveraged, can help make better business decisions.  The issue is how to tie the disparate pieces of data together such that they can be used to construct insights.  How do you get personnel records and field work orders tied together such that you can automatically track employees and their certifications to work on specific equipment, or how many times they have made mistakes on certified equipment?  How do you get the systems to share data?  Where will it be shared?  How can it be manipulated?  Once it is manipulated, is there any way to update the source data with insights?  That’s the problem – there’s a language barrier.  Systems can’t talk to each other.

There are some companies – smaller ones, mostly – that have the discipline and self-control to institute top down master data management (MDM) programs that essentialize the data into relevant quanta and then construct a standard format for that data to be shared with the rest of the company, usually a combination of a standard like PPDM and a bunch of SOA-enabled data adapters that sit on an enterprise service bus.  Others use tools like ETL (for tight coupling) or mediated schemas (for loose coupling) in order to overcome this problem, but you are stuck in rigid one-to-one database relationships, orphaned data, and a lack of agility.

Still other try to do it the “looks quick and easy but really isn’t” way with Informatica, Tibco, or InfoSphere. Sure, you get the enterprise bus and a nice data catalogue and promises of universal connectivity… but your corporate data governance, internal data models and processes, and organizational fluidity go to poop. Oh, and setting it up sucks.

At this aforementioned oil and gas client, each of the various departments had their own way of managing data.  Some had data warehouses where they threw the bulk of their stuff into.  Others had multiple data warehouses for multiple systems, or multiple data warehouses for the same system but for different end users.   Other departments lived in Excel or Access.  And anyone with even a hint of SQL experience would write queries. Many times, it seemed, just for the hell of it.

Directives for data management from executive leadership were unclear and not aligned to the vagaries of an oil patch employing a thousand people broken up into scores of different departments.  After about fourteen seconds of reflection, it occurred to me that overcoming organizational inertia, political realities, departmental lines, and resistance to change in order to create and implement a top down master data model for the enterprise would be, in a word, impossible.  No one was going to put together a data model that everyone would agree upon; understanding the relevancy of data specific to arcane technical disciplines was too complex an undertaking.  Understanding the systems, applications, current transmittal and ingestion technology, and the various reference architectures used for data management would take years.  Even if we could aggregate, document, and organize this data in a master data warehouse, by the time we were done it would have already changed.  Simply put, a curated data repository at an enterprise level was functionally impossible.

So we pitched a modified master data management strategy.  Here are the components:

• Bottom-Up Data Models: Who knows data best?  More than likely it is going to be the analysts that are creating, transforming, or ingesting the information on a regular basis.  They will know more about the characteristics, value, velocity, accuracy, and nuances associated with the data than any enterprise architect brought in to define a data model. There’s some data that needs to be shared with the rest of the company, and then there’s data that is noise to everyone but the department involved.  A solid data management strategy is to allow the decomponentization of data to occur at the lowest possible reasonable level, while ensuring that it conforms to an adopted industry standard like PPDM or whatever else strikes your fancy.  Let the boots on the ground decide what ontology or data integration methodology works best for the data or what data needs to be shared; as the architects, we just want to make sure that it conforms to a formatting standard and a top-level ontological vocabulary or schema.  The business should be responsible for managing the local data model/schema, updating it for new data types, and incorporating new definitions for specific data requests from other lines of business.   By creating a standard structural model for data in transit, we can describe a near-universal mechanism for data translation that only requires source knowledge rather than both source and target knowledge.  The goal is a universal mediated schema (for lack of a better term) that allows any-to-one data integration. As long as the ingested data conforms to the universal mediated schema, the need to create connectors between two disparate data sources should not be necessary.
• Enterprise Service Bus: This is where workflow and business processes are instantiated.  It also serves as a master communication hub for and between the various nodes connected to it. An ESB can carry event-driven messages that have the capability to kicking off real-time updates o both data and applications.  Although not strictly required, it extends the capabilities of connectors and applications immensely.
• SOA-enabled Data Connectors: In this scenario, connectors are universal and have only two functions, ingesting and sending.  The adapters, when sending data, transform the data to adhere to a data model that is hierarchically defined – like a master data model that everyone has to map to, and as you go down the food chain, it gets more complex and detailed. The transformation here is blind – the target system data format should be irrelevant. The adapters, when ingesting data that subscribe to the above-mentioned data model, are able to transform the data to suit the needs of the application that has requested it.  For instance, if we are sending data from SAP to Oracle, the SAP adapter would format the data according to the rules of the data hierarchy. The receiving Oracle adapter would ingest the data and transform it to fit the needs of the Oracle database based upon the hierarchy and information of the data model.

It’s important to note that for the near- to mid-term, most companies need master data management.  It is the only way to ensure data consistency and it simplifies business analytics.  There are many ways to skin this cat (pardon the metaphor), and each comes with varying degrees of effort, complexity, and agility.  Some may find that a pre-packaged solution from Informatica is good enough; others may determine that all data needs to be cleansed and curated from an executive level.  A path forward is predicated upon the structure, complexity, and volume of your current data sets, and success is not some imagined goal post but rather the slow and orderly integration of data and applications.  Ultimately, what you want is for your company to leverage that data in order to be better informed for decision support, which will lead to higher profitability.  Right?

Riiiight.

Follow @DataconomyMedia

Jamal is a regular commentator on the Big Data industry. He is an executive and entrepreneur with over 15 years of experience driving strategy for Fortune 500 companies. In addition to technology strategy, his concentrations include digital oil fields, the geo-mechanics of multilateral drilling, well-site operations and completions, integrated workflows, reservoir stimulation, and extraction techniques. He has held leadership positions in Technology, Sales and Marketing, R&D, and M&A in some of the largest corporations in the world. He is currently a senior manager at Wipro where he focuses on emerging technologies.

(Image Credit: Colleen Galvin)

Big Data has been making noise in almost every facet of corporate America.  From targeted marketing to reducing manufacturing defects, the promise of Big Data seems like the 21st century version of Warbug’s Tincture – a comprehensive, all-inclusive cure to the gross process negligence and overall corporate excesses of the 1990s.  What were you planning on doing with those data warehouses, anyways?  Was all that retained customer data going to be used for anything other than a bull’s-eye for hackers as they eyed your network and data infrastructure?

Utilities are facing the challenges of the big data deluge, but the opportunities and challenges that they face are fascinating to consider; the amount of data that is being collected by smart meters and other consumer-oriented electric monitoring devices has increased the amount of data collected by up to 180x, according to a 2012 survey by Oracle of utility senior executives across North America.

The “Big Data, Bigger Opportunities” survey identified a host of challenges – and opportunities – that utilities struggle with as they migrate from manual, once-a-month reporting for electric consumption to devices that take multiple measurements per minute and allow the aggregation and sharing of information across what were once departmental silos inside utility companies.  And it’s not just electric usage that is now being tracked; the data torrent now includes outages, voltages, tampering, and diagnostic data.

So far, value is being driven by homeowners and utility customers.  They are now able to analyze and trend their electricity usage across individual components in their home.  It’s one thing to know that incandescent bulbs are inefficient; knowing that those two 150-watt bulbs you leave on all night for security are costing you around $300/year in wasted electricity might encourage you to be a bit more frugal with your electric consumption.  And how about that desktop computer that stays on all the time?  According to my smart meter, every month I’m throwing away $20.  Click.

That’s not where the real value of this data resides.  It is good information, obviously; canny homeowners may be able to save $500 to $1000 off of their yearly electric bill.  In the future, these homeowners will be able to compare their usage against aggregates from their neighbors.  Sadly, not all homeowners are canny, and fewer still may take the time to take a look at their electricity usage in such detail.

It’s hard to put together an ROI model for a system that requires between ten and twenty percent of the capital spend  of a utility in the hope that some fraction of the customers will alter their consumption habits.  The real promise of smart meter-enabled big data lies in the external outcomes; it is in the aggregation of utility usage across a service area.  Imagine a city that can:

• Accurately predict energy usage in order to improve their performance on settlement markets
• Leverage operational and customer-facing data to reduce the number of customers that leave
• Measure and predict real-time loads so that energy grids can react intelligently to variations in supply and demand
• Identify anomalies in order to automatically reroute electric paths away from failed devices or links.
• Aggregate data across operational, transactional, and financial silos to garner a better, more intelligent understanding of the business
• Being able to analyze data in order to recognize fraud or predict maintenance or identify threats.
• Reducing the need for site visits through automated, remote meter reading and billing.
• Analyzing the generation and transmission costs of alternative energy sources such as wind and solar.

This list goes on.  It is limited only by the brilliance of the data scientists that construct the interrelationships between disparate fragments of data, and the agility of the business to leverage this data to make smarter decisions around strategy and operations.

Despite the many detractors and critics of smart metering, the writing is on the wall; as extremely accurate and demand-driven forecasts of electricity emerge, this information will invariably be tied to the hardware generating the data and the transmission elements of the grid.  The net result of this will be lower costs and higher availability.  Centerpoint Energy has already reduced the number of trucks on the road by hundreds; from meter readers to reconnection efforts, much of the work that once required a site visit is now completely automated.

According to Floyd LeBlanc, VP of Corporate Communications at Centerpoint Energy in Houston, Texas, “Centerpoint can now read meters remotely, which along with conducting nearly five million service connection and disconnection orders electronically to date – without sending a truck – has saved customers at least $24 million in eliminated fees as well as CNP nearly half a million gallons in fuel and removed over 4,000 tons of CO₂  from the atmosphere.”

Electric reclamation from savvy homeowners continues to rise and data analytics are fueling a deeper and more cogent understanding of the various relationships between electric demand, weather patterns, temporal statistics on usage, and a thousand other points of data that promise to reduce the cost of electricity for consumers.

Mr. LeBlanc goes on to say that “… more than 600,000 Texans now get more frequent and detailed information on their electricity use (down to 15-minute increments), and almost 15,000 get near-real time usage information from In-Home Display Energy (IHD) monitors.  Over 70% of surveyed customers have made energy-saving changes based on this information with some reporting savings up to $100 per month.”

All of this operational and performance data will end up tightening the operations of the grid.  In the coming years, we will see a more robust and elastic grid that has the ability to predict and react autonomously to demand fluctuations.  A grid that has the ability to heal itself when hurricanes down power lines or knock out transformers by rerouting power to homes and businesses; a grid that can purchase electricity with absolute accuracy on the open market in order to drive the best price for consumers; a grid that needs less hands-on maintenance and more data-intensive processing.

It is a grid where electricity is an on-demand facility rather than a utility service subject to the vagaries of fate and circumstance.

Follow @DataconomyMedia

Jamal is a regular commentator on the Big Data industry. He is an executive and entrepreneur with over 15 years of experience driving strategy for Fortune 500 companies. In addition to technology strategy, his concentrations include digital oil fields, the geo-mechanics of multilateral drilling, well-site operations and completions, integrated workflows, reservoir stimulation, and extraction techniques. He has held leadership positions in Technology, Sales and Marketing, R&D, and M&A in some of the largest corporations in the world. He is currently a senior manager at Wipro where he focuses on emerging technologies.

Interested in more content like this? Sign up to our newsletter, and you wont miss a thing!

[mc4wp_form]

(Image Credit: Portland General Electric)

There’s not a day that goes by where I don’t read about the CIO of some billion dollar company claiming that Big Data has saved his company untold sums of money.  A one percent efficiency increase identified by Big Data resulted in some few million in savings.  Segmentation of transactional sales mapped to a unified customer database revealed a 10% increase in alternate channel sales of related products.   Well, at least we know they’re not liars; a recent Bain and Company study found that adopters of Big Data analytics have gained a significant lead over the rest of the corporate world.  Companies with the most advanced analytics capabilities outperform competitors by wide margins. The leaders are:

• Twice as likely to be in the top quartile of financial performance within their industries
• Five times as likely to make decisions much faster than market peers
• Three times as likely to execute decisions as intended
• Twice as likely to use data very frequently when making decisions

However, this doesn’t mean that you should just jump into the deep side of the pool.  For every story about accelerated financial performance, I can point to ten that talk about mismanaged investments and a loss of interest by leadership in Big Data.  In my last blog, I talked about some of the things that Big Data isn’t.  As promised, here is a follow-up blog to help you know when the time is right to begin your platform, services, and organizational investment into a multi-year, multi-disciplinary practice that will change the way your organizations makes decisions.

1) You have some degree of mastery over business analytics.

Companies looking to exploit the promise of big data have a proving grounds that must first be mastered; business intelligence.  After all, if you don’t know what to do with the data you already have, chances are low that you will know how to proceed with mapping disparate data streams to find answers to questions you have yet to ask.  Mastery of your own data has two principles values; one, it gives your organization the expertise it needs to take analytics to the next level, and two, it suggests that you have driven some (hopefully most) of the financial benefit out of the data you already have.

2) You are collecting streams of data.

Without some sort of historical perspective, it will be difficult to baseline potential efficiencies. Even with real-time operations, historical data sets can be mined for co-related data that can indicate operational anomalies or financial opportunities.  Let’s imagine the following scenario:

As a part of the drilling and completions engineering team, you are looking to prevent differential sticking and possible twist-offs at a drill site.  There are multiple vectors that needs to analyzed; the change in differential pressure as a function of time, fluid loss to formation as a function of the geology of the reservoir, the lubricity of the mud cake as a function of friction, etc.  Operational components need to be kept in mind; the type of mud being used, collar shape, depth of drilling, and so on.

You decide to use Big Data to analyze what has happened in the past.  After ingesting the data from ten thousand twist-offs into Hadoop, you find that torque as a function of time has certain upper and lower boundaries that are suggestive of a twist off 30% of the time.  This increases to 60% when certain geological formations are present.  Maybe it’s 80% when you include temperature, pressure, and mud loss characteristics.  And maybe it’s even higher if you include the drill type, the manufacturer of the pipe, the incidence of earthquakes in a region, the age of the formation you are drilling into, the number of hours the crew has worked without a break, and the name of the drilling supervisor.

The data you are collecting in this scenario is manifold; SCADA devices are sending WITSML streams of data that give you down-hole information.  Asset management systems are giving you information about the type and characteristics of the machinery used.  Personnel records are giving you information about HSE and past performance of your crew.  GIS data is coming in through multiple public domain data sources.

Each one of those streams of data is valuable, but finding where all of these streams intersect… well, that’s Big Data.  But understanding where that data intersects cannot be properly identified without an initial analysis of historical data; essentially, understanding how something happens when it happens, and modelling that behavior for application to future use cases.

3) Your culture can embrace opportunistic analytics.

Big data does not have the same value proposition as other investments.  It is not, as a rule, requirements based.  There are no clearly-defined business problems that require resolution.  You can’t build a standard ROI or NPV model; you don’t know what the expected return is going to be or what sort of organizational gymnastics you will have to do in order to support it.  Big data is fundamentally opportunistic, best suited to solve problems that are ill-defined or not at all.  Your business analysts are going to have apoplectic fits and your finance team is going to hit you with the general ledger.  Be ready.

4) You have the nerd power. 

Playing with Hadoop, NoSQL, and Splunk sounds like something your first grader might be doing with play-doh and an unfortunate sense of humor, but these toys are for big kids.  These cutting edge tools have reinvented how data is assembled, analyzed, and correlated.  However, nerds with platform expertise is insufficient; for a big data strategy to work, you must also have business analysts that are domain experts with intimate knowledge of the operations of whatever it is that your company does.  Without this domain expertise, understanding the inter-relationships between disparate pieces of data will be impossible.  Another challenge is finding the data scientist; these people are tasked with understanding how to drive value out of data using creativity, intuition, and the types of higher-order calculus that you definitely have no interest in understanding.

Big Data has a lot to offer, but I have seen many leaders invest money, resources, and reputations in Big Data strategies, only to fall on their face when it became clear that the fundamentals were missing.  Maybe corporate data wasn’t being managed properly.  Perhaps they lacked the in-house expertise to aggregate or crunch the data.  Even worse were the organizations that expected results immediately – organizations that believed that Big Data could solve specific use cases or bring efficiencies into their organization, unaware that they probably already had the tools to solve such use cases using traditional analytics.  You’ve probably heard of the proverb “learn to crawl, then walk, and then you can run.”  Moving into Big Data without having a grasp on these four principles is like participating in a marathon when you’ve just learned how to scoot across a carpet.  It’s just not going to happen.

Follow @DataconomyMedia

Jamal is a regular commentator on the Big Data industry. He is an executive and entrepreneur with over 15 years of experience driving strategy for Fortune 500 companies. In addition to technology strategy, his concentrations include digital oil fields, the geo-mechanics of multilateral drilling, well-site operations and completions, integrated workflows, reservoir stimulation, and extraction techniques. He has held leadership positions in Technology, Sales and Marketing, R&D, and M&A in some of the largest corporations in the world. He is currently a senior manager at Wipro where he focuses on emerging technologies.

Interested in more content like this? Sign up to our newsletter, and you wont miss a thing!

[mc4wp_form]

If there is anything that has generated more noise than “Big Data,” it would be “The Internet of Things,” Cisco’s much-vaunted ideation of a world interconnected through the intelligence buried into the technology components that make up our lives. From cars to coffeemakers, one day everything will be connected to the internet in order to participate in a weird version of the Vulcan mind-meld; cars will tell air conditioners to turn on later than usual because the freeway has indicated a twenty minute delay due to congestion from a car accident. Your fridge will notice when you’re running low on eggs and will update the grocery list on your phablet. All kinds of insanely convenient things will happen once everything knows what everything else is doing.

Right?

Well, maybe, but probably not as soon as you’d think (or like). The technology manufacturers that are trying to market the Internet of Things to us gleefully create use cases that can make one wonder if humans will have any thinking to do whatsoever in the not-to-distant future, but their eye is on instrumentation and communications, not intelligence. Their devices are connected to the internet, and that’s about it.  When it comes to predicting outcomes, it’s still up in the air. There is a fundamental disconnect between being able to do something and understanding that something needs to be done. For instance, some engineer out there will realize that it takes 45 minutes to heat water up in a home’s water heater, and will write a subroutine tied to a car that informs the water heater to turn on at just the right time such that Joe Technophile will have hot water right when he pulls in from work, rather than hot water all day when no one is at home.  Another engineer will come up with another subroutine that geolocates members of a household in the event that smoke and heat sensors spike inside a home. And if we’re lucky, someone over at HP will figure out how to map a @#$%! printer to any device that comes within range without having to break your fingers in frustration. Maybe.

The problem here is with the word “subroutine.” Ok, go ahead and replace that with the word “app.” Or replace it with “functionality” or “integration” or whatever else strikes your fancy. At the end of the day, what it comes down to is this; the devices that we have instrumented and liberated through internet access are still nothing more than savants – incredibly efficient at what they are told to do, but not much else. For every awesome incident you can imagine your new high-tech toys doing, there must be a lateral use-case that enables that event to occur.  Your house won’t know to turn on the coffee maker or turn up the air conditioner when you are minutes away from your home; it has to be told. Your hi-definition integrated AV system won’t know to dim the lights and throw on some Sinatra when you bring over a date on a Friday night because it doesn’t know what a date or Friday night is. It has to be told. It has to be programmed. It is all very prescriptive.

Did you know that your smartphone can measure the speed of a moving object?  Or turn on your car? Did you know it is probably capable of telling you what planes are flying overhead, act as a leveler when you want to hang a picture, find the cheapest price of something you are merely looking at, and even control phone functions with a tilt of your head? Alas that there are ten million innovations buried beneath the inability of a human to aggregate and rationalize useful functionality. Maybe it’s laziness, or maybe it’s hubris; certainly, however, there is an element of capability overload.

All of this brings me back to the Internet of Things. It’s not, really. Without some sort of interpretive logic built across devices and across platforms, we are looking at a bunch of thingys that could talk to each other but probably won’t because we aren’t able to visualize the possibilities. Perhaps this is a play for Big Data – leveraging disparate sources of information and analyzing consumer trends in order to predict appropriate behavior.  However, even some sort of enterprise bus that provides intelligence between disparate devices would be a decent starting point – it’s not enough to just communicate; these devices need to be able to report and have their data analyzed within the context of their users. In any case, without the ability to predict what is needed and a common language such that disparate devices can share information with each other (and construct knowledge from that information), the internet of thingys is something that will continue to confound us. It will never mature to the Internet of Things.

Follow @DataconomyMedia

Jamal is a regular commentator on the Big Data industry. He is an executive and entrepreneur with over 15 years of experience driving strategy for Fortune 500 companies. In addition to technology strategy, his concentrations include digital oil fields, the geo-mechanics of multilateral drilling, well-site operations and completions, integrated workflows, reservoir stimulation, and extraction techniques. He has held leadership positions in Technology, Sales and Marketing, R&D, and M&A in some of the largest corporations in the world. He is currently a senior manager at Wipro where he focuses on emerging technologies.

Interested in more content like this? Sign up to our newsletter, and you wont miss a thing!

[mc4wp_form]

(Image Credit: Featured, Flikr; In text, Enterprise CIO Forum)

Follow @DataconomyMedia

In this article, I’m going to suggest three spaces that enterprise-class organizations should invest in over this year. The basis of these suggestions are as follows:

• Reduce operational costs over the long term
• Shift funds from operations to investment
• Orient the company for changes in the compute landscape
• Accelerate the ability to manage change

Lofty goals from a guy who can barely balance a checkbook, but I hope the reasoning I propose resonates with you. One downside is that I do not consider NPV in my suggestions; even if I could come up with a generalized formula to accommodate this overwhelmingly important factor (which I can’t), it still wouldn’t do any good. These suggestions are designed to change your posture to the marketplace, not assure return on investment in three years or less. They represent fundamental shifts in technology, process engineering, and the way your business is run. Reasonably speaking, each one of these suggestions should be viewed over a three year time frame. They are tectonic changes that require executive sponsorship, change control, risk assumption, and raw leadership. They are not to be taken lightly, and the return, although manifold, will be over the long term.

So here we go.

1) Your plan for cloud computing.
Cloud is not something you can just jump in to if you are a mid-market or enterprise class organization. There are steps you have to take, organizational changes you have to consider, and legalities you have to wrestle with. It is not simply a direction that a CIO can dictate; rather, it must be a staged, comprehensive plan that involves all executives in an organization.

• Planning for cloud computing will require a significant investment of your resources. Whether you are looking at private, public, or a hybrid, I cannot stress the importance of this investment as a precursor to any hardware or services investment you make in cloud computing. In my mind, there are three components that have to be defined before a conversation can occur with a cloud provider, assuming that you believe that you will move most of your workloads into the cloud.

• One, what is the strategy for moving interrelated workloads into the cloud, and how will it unfold. This will require you to analyze your software and compute infrastructure, establish SLAs with the business units, understand the interrelationship between workloads, and describe the risk of failure. Understanding this piece of the puzzle will clarify the challenges your lines of business experience with respect to both performance as well as process and will help stage your migration and deployment plan.

• Two, what does a post-cloud organizational design look like. Do you need server admins? What about infrastructure architects? How will moving to cloud affect software development? Do you need network admins and domain power users anymore? What role will security play in the future? Cloud fundamentally changes the way that business is done; it automates the provisioning process and abstracts the platform decision to business imperatives rather than the perspective of the manager. Many of the operational roles extant in your organization will be rendered obsolete, and many new roles will have to be recognized. It is vital to consider what the effect will be on how your organization is structured, the technical talent that you need on staff, and the responsibilities of the roles you have established.

• Three, what process changes will be introduced into your organization after the adoption of cloud computing. How are servers and storage requisitioned? How are they billed back to the business? How are exceptions handled? How will your new organizational design and manage these challenges? The modifications that are introduced into your provisioning and support functions must be understood and modeled out before deployment. The modeling that you create will function as the backbone of the workflow you will establish for these functions.

2) What role will mobile applications play in your organization going forward?Mobile access is not remarkable because it gives users the ability to access and manipulate data from edge devices; it is remarkable because users can manipulate and access data from edge devices. Value is problematic if a user has the option of leveraging data to fulfill a job function; it is an entirely different animal if the employee must leverage that data to make decisions. Mobile applications that shape workflow and define productivity will ensure a level of optimization and process automation that will virtually guarantee cost savings and a reduced instantiation cycle for whatever it is you are deploying. Mobile applications are not merely a way to free your employees from a desk; it is ensuring the resiliency of your organization’s communications, operations, and decision-making.

Real time operations are a good example of this collaboration. Decision-making can be accelerated logarithmically if your value chain is mobile-enabled. Cost savings can be dramatic. As employees in the field work on projects, they can update labor and material costs real-time and feed it directly into SAP. Goods can be identified, counted and tracked, ensuring both the elasticity of the supply chain and the accuracy of invoicing. Tethering time entry to geo-location services can ensure your employees are where they should be when they say they are. The possibilities are endless.

3) Get ready for Big Data.
Invest in two different areas specific to this space: the data science organization I am going to suggest you start building, and the instrumentation needed to start collecting Big Data.

Data science is a fledgling field in corporate America. In a nutshell, it is the science of extracting knowledge from data. From a technical perspective, it includes the disciplines of mathematics, signal processing, probability theory, statistics, pattern recognition, data warehousing, and even philosophy. However, a data science team is more than just a propeller head coming out of Yale. There are three roles that need to be filled:

• One, the business analyst: this is the individual that will capture business-specific requirements as well as existing processes. Without a consultant that can understand (and document) the as-is process and the business challenges of a line of business, it will be impossible to drive any real value from the plethora of information that Big Data will provide. This individual needs knowledge of not just the organization’s technical and business environment, but also must have some functional understanding of the promise and challenges of big data technologies like Hadoop.

• Two, the scientist: we’re talking about an individual with a desire to go beneath the surface of a problem, find the questions at its heart, and distill them into a very clear set of hypotheses that can be tested. This often entails the associative thinking that characterizes the most creative scientists in any field. Data scientists provide the basic intellectual, philosophical, and creative underpinning of any Big Data strategy. They are the ones that look at data and try to predict interrelationships that the rest of us might not see using algorithms, number crunching, and using raw intellectual and creative horsepower to make 1+1=3.

• Three, the Big Data engineer: you need an engineer that has the ability to use tools like Hadoop and HBase and Cassandra to capture, store, and process the data from various ingestion streams. These technical engineers work at the most fundamental level of data, working to aggregate, cleanse, and normalize data for ingestion.

The other area you need to think about is instrumentation. I’ve written blogs about this topic in the past, so suffice it to say that both hardware and software tools that capture data relevant to operations in your company are fundamental to expanding the breadth of information available to your Data Science team for analysis. The heavy lifting associated with big data is a function of instrumentation – the ability of an application or a device to collect runtime intelligence around usage levels, errors, user behavioral patterns, and ultimately the statistical analysis of this information. After all, without the “big” in big data, you’re missing half the story. An uninterrupted flow of data in massive sets across disparate data elements is critical to creating data associations that will ultimately drive sales, operational efficacy, and profits.

Obviously, this list is incomplete. There are many other sectors to invest in – refreshing hardware, investing in your people, accelerated hiring, marketing, database normalization, new technologies, and even leveraging partner relationships. There are many sectors available to invest in, and each one will have a compelling story for your organization. However, the three I have mentioned today are kind off “off-the-radar” in that they are long –term visions for CIOs rather than actionable value propositions. They are future-state technologies that sit on the border between marketing hype and market displacement. For the visionaries that elect to invest hard dollars in these technologies today, the future promises to be a leaner, more agile, and more profitable environment. For the others, I expect to see a bloody and agonizing conflict between those that catch-up and those that fall to the wayside.

Follow @DataconomyMedia

Jamal is a regular commentator on the Big Data industry. He is an executive and entrepreneur with over 15 years of experience driving strategy for Fortune 500 companies. In addition to technology strategy, his concentrations include digital oil fields, the geo-mechanics of multilateral drilling, well-site operations and completions, integrated workflows, reservoir stimulation, and extraction techniques. He has held leadership positions in Technology, Sales and Marketing, R&D, and M&A in some of the largest corporations in the world. He is currently a senior manager at Wipro where he focuses on emerging technologies.

Interested in more content like this? Sign up to our newsletter, and you wont miss a thing!

[mc4wp_form]

The hype around the term “big data” can be pernicious. The ubiquity of the term, coupled with a lack of understanding about what it actually means, often leads to overly-ambitious expectations of what big data projects can actually achieve.

This week, guest contributor Jamal Khawaja aimed to clear the fog around big data by busting some common myths. We’re intrigued to know which one of these myths our readers think is the biggest error, and most likely to lead to failing, ill-thought-out big data initiatives:

[socialpoll id=”2206164″]

We will be releasing the results of this poll in our newsletter next week! Sign up here:

[mc4wp_form]

Follow @DataconomyMedia

How many of you have watched the movie Fight Club?  If you haven’t, I strongly recommend you do.  It’s a movie worth watching, replete with violence, ennui, and a lament for the narcissistic failings of the baby boomers.  Anyways, there’s a scene in the beginning of the movie where Edward Norton talks about how crappy his job is.  He has to assess the risk associated with car accidents for an insurance company using actuarial data.  His job is to figure out whether a recall is the right decision for an auto manufacturer to make – from a financial perspective.  Let’s assume that there is a brake problem that will result in a financial liability associated with wrongful death.  Using actuarial tables, he puts a number to the total financial exposure that a company would have.  He then compares this to the costs associated with a recall.  If a brake problem will result in payouts of $50,000 per claim against 1,000 wrongful death claims (a total cost of $50,000,000), and the cost of a recall is $75,000,000, then the insurance company won’t authorize a recall.  Financially, it makes more sense to let people die in brake failure accidents than to correct the problem.

Hello, weird precursor to Big Data.

This is a small but substantive problem associated with Big Data – a company is given so much information that morality, ethics, and even the rule of law are subsumed to analytical realities.  Big data allows managers and executives to measure every facet of the customer experience, from marketing to returns.  They arm these decision makers with the power to base decisions about policies based upon data that may or may not be advantageous to the consumer or the society.  Actuarial tables are a type of decision-support analytics across a small subset of data, but data that arms insurance companies with information that forces them to make ethically challenging decisions.  As a person, such a decision is appalling – why let 1,000 people die when the fix is available?  As a computer, I the challenge tilts in the other direction, because the cost of each life has been valued at $50,000, and the cost of correcting the problem is $25 million more than the potential payout will be.  I’m not going to say most shareholders of financial institutions that cover insurance have the empathy of a computer, but I’ve worked in financial services. ‘Nuff said.

Ok, so actuarial tables exist for a reason; I get it.  There is a value associated with risk, even if that risk is distasteful to quantify.  But Big Data can be more insidious than cost analyzing car accidents that kill 1,000 people a year.  Much more insidious.  And much more relevant to you as a consumer.

Let’s fast forward ten years into the future.  Big Data is 10,000x more advanced than it is right now.  Companies use it to measure and manage customer satisfaction in every facet of its organization.  An imaginary company, Qualmart, is reviewing the costs associated with customer service.  Big Data is streaming information from a thousand different sources – cash registers, Twitter, Facebook, loyalty programs, credit cards, employee files, product recalls, and anything else you can imagine.  After correlating data from Facebook, Twitter, and its own returned merchandise receipts, it has found that Chinese consumers are 60% more likely to indicate negative experiences on the internet during a product return than the rest of the population.  This information is an algorithmic compendium of data captured from credit card transactions, age/sex/racial data compiled from its loyalty programs, and complaints and negative feedback from Facebook posts, tweets, and employee-documented information from the actual return itself.  As a result, a new corporate policy is issued for the customer service departments of Qualmarts located in Chinatowns of major cities to ensure that the return process is as quick and painless as possible, even if there is no real reason to honor the return.  Well, as someone who is not from China, I don’t think that seems fair.

Let’s look at it from a different perspective.  Let’s say that this retailer determines that white females under the age of 30 that live in Louisiana typically indicate negative experiences on the internet with respect to returns only 15% of the time.  According to sales data captured from stores located in Louisiana, the most returned products are baby carriages, medium sized t-shirts for women that cost less than $15, and women’s shoes in sizes 8 and 9.  Customer loyalty cards suggest that sales are made by young, married white women who make less than $50,000 annually.  The sales for this demographic are 15% of total store sales.  However, the costs associated with the return policy of this demographic exceed 75% of total returns of all returned products.  Since the percentages of returns in this hypothetical situation in Louisiana are from women under the age of 30, Qualmart issues a corporate policy in Louisiana to make the return process in Louisiana more cumbersome.

It can get uglier.  Let’s take it a step further.  Let’s say that our retailer finds that 50% of their returns in Texas are from African-Americans. This information is compared against the internet browing habits of shoppers on qualmart.com; their customers tend to look at competing products on their website after purchasing the product in question.  They decide that the return policy is an issue of “buyer’s remorse” and the impulse buy by the consumer was mitigated by other options available on the website.  Qualmart decides that returns from African-Americans should be challenged.  To ensure compliance with this new directive, they decide to incentivize their Texas employees: if only 35% of all attempted product returns are ultimately returned, they will all get $100 bonuses.  This will have a disproportionate effect on African American consumers in the Texas region.  However, Qualmart will save millions of dollars by instituting this new policy.  And they have data to back up their decision-making.

How about Big Data that has life-or-death repercussions?  Big Data could prove out that Korean pilots are 35% more likely to experience a crash.  Analyzing personnel data from all crashes originating from the US might reveal a disproportionate number of crashes can be attributed to pilots born in South Korea.  However, it might be a causation that has deeper roots; it might be something specific to training or the types of aircraft they fly or cultural barriers.  It could be that Korean pilots fly poorly designed aircraft.  Maybe all of the aircraft shipped to Korea in a specific year came with faulty altimeters or something.

The magic we are looking at is the promise of Big Data – companies are able to predict the behavior of customers even before they exercise the power of their wallet.  If you’re buying online, they can give you recommendation based upon your previous buying history or the buying history of people that fit your demographic so that you are fully satisfied with your purchase.  The insight this provides retailers is almost frightening to consider.  If 75% of people buy one type of widget, only to return it for one or two other styles of widget, what would that suggest?  It is a problem with the first widget, obviously.  Why even present that as an option?  Why not drop the product and offer only the alternative?  Even better, why not claim it is “no longer available” and offer the top two alternatives?  After all, we can measure and manage the customer experience more precisely than ever before. We can make better predictions and smarter decisions. We can target interventions that surpass the gut instinct of executives that came before us, guided by data rather than intuition.

What we are looking at is cultural clichés that are validated by empirical data.   Once we overlay Big Data against predictive analytics, the result is (apologies for the pun) predictable.  Customer segmentation is no longer just a function of increasing sales; it becomes a relevant factor in driving policies that discriminate between racial, gender, age, or socio-economic classes.

In America, we have laws that mitigate the effect of relevant data on the employment process.  It is called “disparate impact.”  In fact, slew of laws exist to mitigate the effect of data on hiring policies.  For instance, it is illegal to refuse to hire a woman if she is pregnant.  Similarly, you cannot refuse service to someone based upon a protected class.  Even if you have empirical data to suggest that gay urban males are more likely to complain about service on social media than other customer segments, you cannot refuse to sell them a product.

But can you do the opposite? It is fair (or legal) to provide a protected class advantageous policies, such as reward programs, special discounts and promotions, or other preferential treatment, because of (or despite) their protected class? Should pregnant women get mommy discounts? Should veterans get discounted fares on airlines?  Should gay urban males receive special discounts in order to mitigate the potential effects of negative social media comments? After all, theirs is a difference between fair and profitable.  If companies were fair, we wouldn’t have to worry about jobs being exported to China.  Companies exist to make a profit.  Although it is illegal to discriminate against a protected class, should it be illegal to give preferential treatment to one if it will result in higher profits?

I don’t know what the first rule of Big Data is, because there are no real rules for Big Data.  It is an endless sea of information that can be used for the forces of good or the forces of evil.  We need to start thinking about Big Data issues as they relate to how our society functions and is governed: security, privacy, and advantageous or discriminatory policies.  The potential for the abuse Big Data is enormous, even as it promises a better tomorrow.

Jamal is a regular commentator on the Big Data industry. He is an executive and entrepreneur with over 15 years of experience driving strategy for Fortune 500 companies. In addition to technology strategy, his concentrations include digital oil fields, the geo-mechanics of multilateral drilling, well-site operations and completions, integrated workflows, reservoir stimulation, and extraction techniques. He has held leadership positions in Technology, Sales and Marketing, R&D, and M&A in some of the largest corporations in the world. He is currently a senior manager at Wipro where he focuses on emerging technologies.

For more articles:

Follow @DataconomyMedia

(Image Credit: Feature Image, Derek Gavey.First Picture:Graham. Second Picture:Official U.S. Navy Page)