Data Scientist- Our unicorn in the Big Data world - 2 (ver Aug 2014)

I wrote my initial DS attributes blog in Jan 2013 what I call DS version 1. This blog is version 2 and we are sharing our experience and reviewing what changed over the last year and a half.

Note added on Aug 14th: based on the recent ASUG poll from SAP users on HANA here is an additional item I would like to add.
1. Make sure your data scientist does not take a Technocratic solution in total isolation of  business needs, business benefits or business inclusion.
2. Your data scientist must have a Business Value Architect background with extensive business interaction or you could end with a model that it technically perfect but has little business relevance.
3. Your perfect data scientist is one that can model a business benefit model, has extensive erperience modeling to meet business expectations. has extensive experience co-innovating with business users. And, has a pure business benefit focus (Cant say this often enough for it seems to simply float by the technocratic ears.

Take Chris Farrell who spent five years mining data from a giant particle accelerator and now spends his time analyzing ratings from Yelp (WSJ 8/9/14). The job title of Data Scientist barely existed three years ago and today has become the hottest platform for the hi-tech market. Retailers, banks, heavy equipment manufacturers, ministry of defense, armed forces, match-makers, Netflix, and almost everyone we talk to today want to extract the diamonds, i.e. EDGE data ERI (extraction reduction à Interpretation) from the great explosion of data-explosion that surrounds us today.

Your big data vortex

This explosion of data is like a Data Vortex consisting of Exabyte’s and Yottabyte’s of free flowing data, a vortex that never touches the ground, is not destructive and most humans are not even aware of it even though they are completely surrounded by it. It’s a vortex that we first need to visualize, and then place a device into its stream to get direct access to nuggets that are important to us. In our image the cloud above is the data that surrounds us and the slim vortex represents data that we need to extract and interpret into real-time decision enhancers. This Big-Data is disruptively beneficial and needs EDGE thinking for the data scientist to remotely succeed.
 At a very shallow level this data-growth is being propelled by your smart phone, your likes and dislikes on Facebook or twitter, internet clicks, upstream appliances-machines and devices, downstream appliances-machines and devices, the media, blogs like this one, blogosphere and the world-wide-web itself.

According to Jonathan Goldman, who ran LinkedIn data science team, (WSJ 8/11) good data scientists are already being referred to as ‘Unicorns’ due to their extreme rarity. This is because as we mature we realize that the combination of skills required is so rare that they are worth their weight in potential value explosion (way beyond platinum).

Data scientists need to have more than technical intelligence. Ideal candidates must possess a surgical passion for more than traditional market-research skills. They need to possess a business benefit focus then use that as a foundation to identify patterns in millions of available data elements from different data sources, infer the interactive patterns prior to building statistical models that pinpoint desired triggers.

Just as an examples a biostatistician (actual business use) who earlier spent years mining medical records to identify patterns for early identification of breast cancer now writes statistical models to figure out the terms people use when they search etsy website for a new fashion they see on the street.  At Square, the new e-payment application, a cognitive Ph.D scientist who built statistical models to identify triggers on how people change political affiliations today is working on identifying which customers are more inclined to have clients that will demand their money back.  Another Ph.D at yelp who worked on genetic mapping today is building models to measure the effect on consumers when multiple small changes are made to online advertisements.

The key in all this big data analysis is the ability to make and measure small changes that cumulatively has very big impacts.

On the negative side Facebook and dating sites are being accused of making small changes, data manipulation and alterations, to ensure higher connections and likelihood of two people getting together in order that these consumers would use the service more frequently.

While a six figure salary is becoming rather common on the Silicon Valley anyone with ‘Data-Scientist’ skills with just two years of data experience can easily earn between $200,000 to $300,000 per year. Anyone with Data Scientist in their resume in LinkedIn can expect to get around 100 or so emails a day from potential recruiters.

This scarcity is muddied by the some potential candidates using ‘Data Scientist’ in appropriately either out of ignorance or by intent. However, once the spin is isolated we are left with pure experience and skills. Just as an example LinkedIn today has around 24,000 to 36,000 positions open for data scientists. In 2012 there were approximately 2,500 doctoral degrees awarded to biostatisticians, Statistics, Particle physics and computer science all of which are on the trajectory to finding your data scientist. Over the last year many universities have initiated programs to launch certificate and masters programs in data science to fill this new demand-supply gap.  Taking it one step further, close to Stanford University a new program called Insight Data Science Fellows program takes doctoral candidates and funnels them into data science programs. This program is funded by tech companies close by and has a 100% placement record.

Five years ago statisticians and data scientists who would have gone for banking  or become wall street quants now feel this new pull for their skills from companies as diverse as Airnb, Palantir, Jawbone,  to Capital One Finance Group, New York times and a host of companies who plan to build applications to meet new demands in this world of micro-segmentation and B2P direct solutions. The biggest apps are the ones that are consumer facing.  We all need to go out and build applications that directly affect people’s lives and make their day-to-day living a lot easier.

140808_ Moving users from Report to Analytics to informatics

Old methodologies were never designed with current technologies in mind. However there is a 0.95 percent probability your current partner advisors are taking you down the HANA path with old methodologies. One of these is the need to move from reports to analytics and from Analytics to Informatics.

this year have assisted many companies move their users from Reports to Analytics and executives from Analytics to informatics as Step 1 for decision enhancement. Thought of sharing this ideation.

From report to Analytics to informativcs

0.     Report: concept is probably as old as when humans could write or draw. So reports are an ancient method of tracking events. Reports have modernized with the advent of computers but still contain some fairly standard variables.

a.     Is Data intensive and based on a row/column page orientation

b.    Normally 1 report per query topic, low graphical content

c.     Used extensively by operational users for day to day activities

d.    Can be information or list reports

Users: Operators, Operational users to track day-to-day activities

Type: Sales person, Production Planner, Stock mover, etc.

1.     Query: BW introduced the ability to run multiple slice and dice capabilities that allowed running multiple reports from one sheet in excel. For example by changing navigation parameters a user could flip from total sales by region to states. Or flip from Value to margins.

a.     Is Data intensive or aggregative

b.    Ability to meet multiple reports in 1 query, medium graphical content

c.     Used extensively by modern operational users

d.    Can be information or list reports

Users: Operators, Operational users to track day-to-day activities

Type: Sales person, Production Planner, Stock mover, etc.

2.     Analytics: By the early to mid 80’s the concept of graphs attached to data sheets became quite common. Since then Analytics evolved from graphs to Data Visualizations. These were reports used by management.

a.     Is Data aggregative

b.    Has lots of visualization and graphs, high visualization content.

c.     Has many navigation layers with options to drill into data layers

d.    Designed as roll-up from operational reporting data and KPI’s

Users: Management, directors and VP’s

Type: Sales management, Production Management, Logistics management.

3.     Informatics: By 2009-10 Gartner introduced the Informatics concepts for C level executives. This layer needs to see global operations, trends, Investments and financial flows.

a.     Is Information intensive

b.    Mainly visual and graphical with very few aggregated data KPI’s

c.     Has few to no navigation layers except for experienced users.

d.    Normally drill downs handed to management for clarifications.

e.     Designed as roll-up from management KPI’s

Users: C level executives

Type: Executives

When deploying HANA as a solution for 'Time-to-information' layer with Real-time decision capabilities it becomes imperative to architect on FEDW (Introduced in ‘BI Valuenomics’ in 2010, and move from reports to Analytics for the operational layers. And move from Analytics to informatics at the executive decision layer.

The visualization below is the Decision Matrix visualizing reports, analytics and Informatics in an enterprise decision enablement. Audience is the Ministry of defense or the Armed Forces.

 

 

How to Maximize Business Value from SAP HANA Initiatives

Old methodologies were never designed for large data volumes. Manual modeling or optimizing of ABAP transforms cannot handle big data. Automation with a Business Value Attainment focus is the only path to assure Business Benefits.

The chart on the left indicates that despite IT and business lowering their expectations from BI projects, Gartner reports that the 'Meet Business Expectations' score is dropping more rapidly. We now need to demand 80% and over Business User Satisfaction from our partners.

Level the playing field

Fact 1- Business Value Attainment: See figure above. If we look as Gartner’s CIO BI reports from 2003 to 2012 we are faced with a strange trend. In 2003 customers expected a lot from their BI investments but only 50% of BI projects met Business Expectations. By 2012 IT and business somehow were convinced to lower their expectations and the result was a worse Value attainment. From 2003 to 2012-14 BI projects capability to meet business expectations went down from under 50% to under 30%.

Fact 2- HANA is a business solution: Most customers continue to treat BW on HANA migration just like another technical upgrade or migration. It is critical to mandate HANA as a strategic business solution. HANA can deliver ‘Time to Information’ in 20-25 weeks as never before. Most companies have simply taken their BW environment ‘As-Is’ and simply migrated it to SAP HANA. Take Fact 1 statistics and we need to realize that these customers will be hit with a double whammy after go live. On one hand they will migrate a lot of junk data and objects into HANA. On the other side someone needs to tell me the “…benefit of accelerating a 715 second to sub seconds if business is never going to use it..”

Fact 3- HANA costs can be reduced by over 40%: Most customers believe that the HANA migration is a rather fixed cost. No matter how we look at HANA it is not a low cost option and one of the biggest barriers to HANA migration remains the ‘sticker-shock’. It is important to realize the HANA installation, migration and annual support costs can be reduced by over 40% in most cases. In a recent BoH migration we managed to reduce the HANA migration costs by as much as 68%. Another goal we managed to achieve is our proposal for a ‘Near-Net-Zero’ HANA migration by keeping the FHRE (Future HANA Run Rate) about as close to the CBRR (Current BW Run Rate). In our final solution we actually managed to drop the FHRE below the CBRR.

Fact 4: What got you here will not get you there: We are in the Fact 1 reality because of extremely faulty methodologies on one side, and ‘BI Experts’ that do not adapt to new standards and processes. The writing is clear that customers should not use current methodologies, standards, processes, resources and in some cases even partners if they want to succeed in SAP HANA, i.e. highest Business Value, Highest Information Quality at the lowest cost. In the current state of BI confusion we are witnessing a spurt of very dissatisfied customers changing from one implementation partner to another not realizing that they are simply stepping from one frying pan to another. Demand BVA deliverables, BVA based payments and add BVA project reporting attributes.

Fact 5: HANA Appliance modes becoming obsolete: There is adequate evidence that the HANA Appliance model may have reached a point of obsolescence.  The appliance model allows a few partners to assume a monopolistic attitude towards pricing and support. SAP's confidence in the HW architecture is demonstrated by their new TDI HANA model, where customers and data-center owners can now choose to mix- and-match their HANA components based on who does that components better than the rest and what their DC accepts. It also allows market competitive forces thereby assuring higher quality at lower costs. We have in the past month built and certified TDI devices for two customers.

Proactive companies can and are achieving enormous financial and decision-capability advantages by focusing on HANA BVA (Business Value Attainment) methodologies.

A.    Focus on Business Value Attainment: BI implementations must be owned and architecture by IT. However, not in total isolation of business stakeholders. The only way to meet business expectations is to involve business stakeholders and users all through the process. Ask business what they would like from HANA and then try to deliver in accordance to their exact needs. One way is to conduct a zero-baseline Design thinking workshop to identify pure business value. In one project we managed to deliver a net new set of reports that increased revenues to make the whole HANA migration break even in 8 months.  Start with a 2 hour business executive HANA workshops that comes with tips & tricks on business ownership and checklists to ensure that the wool some partners may be pulling over your eves is not just cotton.

B.    Optimize your BW prior to moving to HANA: BoH projects can leverage various automated options for reducing TCO, eliminating redundancies and dead objects, rewriting the ABAP transformations, remodeling cubes, merging multiple BW landscapes and other small things that can together reduce TCO by 40 to 60%. In one of our recent BoH projects we managed to reduce initial BW HW costs by 68%. Reduced HNRR below past BCRR. Targets most thought were unachievable. We also launched the IQDCT methodology with a target to Increase Quality, Decrease Costs and Decrease Time for migration to HANA.

C.     Point of Convergence: Our technologies are as a point of disruptive convergence. For any technology to qualify as disruptive it needs to deliver ‘higher quality at lower TCO’. However, if your partner, or their advisors, do not comprehend the disruptive capabilities or continues to use old methodologies then it becomes a lose-lose situation. We are currently at a critical point of convergence where four disruptive technologies are coming together with tremendous capabilities to enhance true real-time decision capabilities.  With the right advisors, BVA methodologies it is now a simple process to turn your HANA migrations into a win-win future. Find a partner that can design your future state with all these four technology convergences covered from a tactical, mid term and strategic. Find a partner with a Business Value focus and capabilities to help meander between all these convergence technologies and alternatives available.

D.    Plan with the future in mind: The best migration is done that aligns to the SAP HANA roadmap 5-10 years from today. It optimizes, models, architects and models for the long haul. Whereas the worst SAP HANA migrations simply treat it as just another technical upgrade or migration.

Demand

1.     Get your business stakeholders a 2 hr HANA workshop with HANA checklists by each selection and phase

2.     A minimum of 40% reduction in your HANA TCO (initial and annual)

3.     Clean your BW prior to moving to HANA

4.     Use automation for remodeling, system mergers and ABAP optimization

5.     Consider a custom RDS solution using a ‘zero-baseline’ HANA Design Thinking workshop

6.     Review if your old standards, processes, architecture, resources or partner can guarantee HANA Business Value Attainment.

Is TDI going to make the HANA appliance obsolete-1?

Up until recently your only option for moving to HANA was to buy a HANA appliance. However, we know all too well that no single company does all things better than any other company. to solve this dilemma SAP introduced the TDI (Tailored Datacenter Integration) model which allows companies to mix and match their leveraged vendors and datacenter skills, without compromising the overall quality of the HANA solution. TDI is way ahead of its time and customers need to adapt to the future rather than hesitate. TDI allows customers to mix and match certified servers into existing enterprise storage systems. This reduces time-to-value, risks, and costs for strategic HANA adoption, and conforms to your datacenter standards rather effortlessly.
Note 1: At the current time we can clearly state that scale-out HANA systems are not truly an appliance as they are not enclosed into a single environment but interconnected in some way or the other. The same reasoning establishes that only the single node HANA Scale-Up actually qualifies as an appliance. 
Note 2: TDI is a concept ahead of its time. I predict that in another 3-5 years most of the HANA appliances will be replaced with generic TDI architecture devices with medium to large customers. So start thinking and planning for a cost mitigated HANA TDI architecture.
Note 3: Appliances do not allow companies to leverage their existing assets. for example if you have UCS servers from Cisco in your data center the appliance model from most companies will not allow companies to integrate. The same goes for Storage from EMC or Net App. TDI actually allows companies to leverage assets, lower costs and in my opinion get an overall quality that is higher.
Note 4: HANA-for-IOT (HANA4IoT) is also a concept ahead of its time. Most customers that adopted HANA in 2010 are now being faced with IoT (Internet of Things) architecture and infrastructure concerns. So plan your HANA4IoT along with TDI discussions

LETS FIRST LEVELTHE PLAYING FIELD- Core Benchmark's- Overall the quest here is to establish that no one company can do all things better that all others. My BVA (Business Value Attainment) question is asked towards the end of this section


Symantec leader in Archiving (for HANA-DR)

A. QUALITY Overall: The overall quality of the HANA appliance or TDI mix is certified by a common denominator - SAP Certification. Once a HANA HW is certified by SAP then its quality is a level playing field and should no longer be a concern for any enterprise or buyer.
A.1 Component Quality: It is a well known fact that no supplier manufactures the best of everything. Some are good in one area and other's good in another.  Lets look at what Gartner states.
Servers: If we look at X82 servers Cisco is the leading provider for Servers in North America as of June 2014. They just beat HP in 2014 and had beaten IBM by April 2014. So your best bet is to get your HANA Compute (Servers) from Cisco today
Backup and DR: There is only one partner certified for HANA DR and that is Symantec. Thus Symantec is your best partner for HANA Backup and DR. So your best bet for HANA Backup is Symantic today.
Storage: Once again the global leaders in storage are EMC and NetApp. So your best bet is to get your HANA storage from EMC or NetApp.
QUALITY- Takeaway: If you want the best you need to partner with the best. We have just built a TDI model for a HANA customer with EMC Storage, Hitachi Compute and Symantec backup. The reason for this was that this very large customer has existing leveraged contracts with these partners and could put together the appliance at around 50% of buying an appliance. However, they did not know about TDI, its certification or support. So we provide that. HANA with better quality at lower costs. We provide the KBS support (K-Keep Lights On; B=Basis; S=Security)


Cisco leader in wireless Devices & Access (for HANA4IoT)

,  

B. COST Overall: Our experience so far is that whn you buy TDI the Quality is at par, and the costs are far lower. Just as an example (this is not cast in stone) I can get customers a TDI appliance of 1TB at around $250k with  3 years of 'Keep-Lights-On' support built in today. Rty that with an appliance today and then come talk to us.

BVA Question: So this TDI combination is as good in quality (certified by SAP) could cost you around 40-50% less; is under no threat of obsolescence (as it is a growth area). Then the important question is should you consider the TDI selection right-now- or simply pay more for less?

EMC & Netappare the global leaders for Storage



I firmly believe that the TDI model, is driven by, and establishes the confidence SAP has in their HANA platform by allowing open-ware configurations in a controlled manner for now. It allows customers to move from current HANA 'black-box' appliances towards a more IT determined open-ware environment based on leverage advantages of each company and their data center management.

As we proceed on the path of TDI we see models where customers mix and match their HANA components with their own leveraged partners for their world-class capabilities rather than buying a black-box that IT is not allowed to touch.

It is important to note that Typically customers have started choosing their HANA PRD environments on bare metal and non-PRD on virtual servers. However, a new kid on the block called ‘Docker’ is promising to even make virtual servers kind of obsolete. Docker with their disruptive technology is leapfrogging to provide the next generation of SW based virtual possibilities. Watch this one closely.

SAP HANA, though quite mature, is still a new kid on the block with exceptional business value and business benefit potentials. In its first step it came with very tight controls in the form of a lock-down appliance. As SAP HANA stabilizes we are seeing an adoption towards open-ware and TDI is that future option available today. One of the indications of maturity is SAP allowing TDI HANA systems that take customers from a single vendor monopolistic chokehold to a more open platform design. As SAP HANA continues to mature the market is clearly moving from HANA appliances to HANA TDI systems that leverage customer strengths and get out of the ‘Black-Box’ appliance model. We will start to see the best practices of HANA installation shift from a single vendor appliance model to a ‘mix-&-match' TDI one.

Historically the SAP Go-T0-Market decision- to start with an appliance model has solid reasons. BWA was an appliance and SAP HANA was a disruptive extension that took the same appliance route. However, as HANA matures and becomes more main stream  SAP is already allowing customers to opt for a more open platform. Customers can today chose an IBM or Cisco compute (HANA servers), and mix that with an EMC or NetApp storage. TDI is the first step to open platform of a maturing SAP HANA. It is also the first step to increasing ROI and decreasing TCO in HANA implementations.

Some facts about HANA (+ and -)

1.     The standard recommendation is scale-out for BI and scale-up for ERP.  

2.     (+) HANA comes on a shared cluster architecture allowing administrators to split tables (or partitions of columnar tables) across multiple nodes in a cluster. This is very different from an Oracle cluster but that is another discussion

3.     (+) HANA can run on both RDBMS and Columnar simultaneously allowing operations and analytics to run off the same database. This leads to large-scale Real-Time operational decision enhancement.

4.     (+) HANA is currently on SP 9 and so far we can store row tables on a single master node of the cluster. SP 9 now supports multi tenancy and a host of new functionalities.

5.     (-) today it is only possible to store row tables on a single master node of the cluster

6.     (-) All connections to the HANA system are controlled by a single ‘Master nameserver’ making this a potential bottleneck for very large scale-out deployments

7.     (-) Single commits from different HANA nodes, i.e. joins of data across different HANA nodes, and translates into performance costs due to distribution of tables across the various servers.

8.     (-) Business continuity of SAP HANA in a scale-out of clusters will always be more complex than operating out of a single node system

9.     (-) TREX is an essential part of HANA that SAP continues to leverage from their existing portfolio. TREX leverages the ‘shared filesystem’ to store all the data when in a multi host scale-out mode. TREX predates BWA and MAXDB. It was built on a NFS access to persistency.

10.  HANA, over time has evolved from being an analytic engine to become the SAP platform. Its historical evolutionary path still clings to some legacy considerations as we move forward beyond it being ‘just’ an analytic engine.

11.  SAP continues to make rapid developments and over time most of the current issues will become obsolete. HANA Scale out is still on a maturity path. HANA is evolving extremely rapidly and most appliance partners clearly state that all HANA scale-out appliances need to be on the same version control. So though 12-18 months today seems to be an eternity it will come very rapidly in real life.

12.  HANA appliances are evolving very rapidly with newer versions coming out every 6 months or so. So if you bought a HANA appliance 6 months ago then you have the following options today. [1] Continue with the current appliance that your appliance partner may stop supporting shortly, or [2] Totally reinstall/change all the procured appliances and change to the new appliances and new support contracts. Imaging doing this every 18-20 months. If you think this sounds far-fetched ask your appliance vendor as to 'till when will they continue to support BWA appliances" and therein you have your answer about proceeding on the appliance or TDI models

 Evolution of SAP HANA

·         Started as an analytics database

·         Evolved as the preferred database for SAP BW

·         Becoming the preferred platform for operational Suite on HANA (ERP, CRM and SCM)

·         Stage 1:  – Platform safeguarding- HANA Appliances only

·         Stage 2: - Platform maturity and open-ware = TDI HANA model
-         Stage 3: - S/4HANA the new SAP platform currently enabling Finance
 

a.     TDI releases IT from the choke-hold of a single appliance vendor by allowing mix-and-match by established leveraged customer vendors. Currently this is limited to certified partners.

b.    There is already a large enough SAP HANA customer base that is selecting the HANA TDI options. Which allows customers to choose their preferred, server/ storage/ networking/ IoT vendors independently.

c.     Critical Note: when thinking TDI also consider your strategic IoT data acquisition in the partner mix.

Strategically TDI is the first sign that SAP HANA is maturing and opening to meet the terms of the customers focused solutions based on an open market adoption. There is clear evidence that HANA adoption will continue to grow. So while some customers choose the appliance route to HANA we see a clear long-term adoption to best-in-class, multi vendor solutions.

As more and more customers adapt to HANA we are also seeing many of them take the TDI route to HANA. This allows more and more customers manage to extricate themselves from the choke-hold of the ‘black-box’ approach of HANA appliances. In the 2014-16 timeframe we predict that SAP HANA will  become just another SAP application in your datacenter freeing customers from the chokehold of HANA appliances.  One that can be deployed on approved infrastructure of the TDI model.

What we are providing to customers today:

Customer 1 is a large financial institution that loves their HW partner but does not like their support experience. So they have chosen the TDI route with neutral support capabilities, they are choosing their own preferred vendors for server and storage and allowing us to integrate the two components along with a SAP TDI certification. This directly impacts their initial costs, support costs and strategic costs without compromising quality of support.

Customer 2 is a large data-center where ROI and TCO are critical considerations. They plan to run hundreds of HANA customer as hosted or on the HANA cloud model. They already have strategic alliances with Compute/server, storage and network partners and clearly see the benefits of not going on the ‘Black-Box’ monopoly chokehold of the appliance model. TDI is every customer’s answer to break free from current monopolistic control appliance model to a more open infrastructure model. Open architecture allows customers to leverage market competitive forces. Appliances, on the other-hand, locks customer into monopolistic command and control scenarios.  

TDI is as close to how you run your current net weaver platform today, where the customer chooses their partners-A procures the components-B and installs their SAP applications for SAP HANA.

Takeaway:

1. There is no single vendor that does all thing best on the planet. Server, Storage, network devices, communications, IoT and IoE security fabric, certified HANA backup, etc.
2. TDI allows customers to get out of the choke-grasp of a single vendor monopoly and launches them on a path of open-ware selections based on world class providers.
3. Appliances are surely on their way out and TDI is on the way in- so plan your HANA Infra with the future in mind.
4. VM servers could be on the way out with the Docker disruptive technology options as it promises to both enhances the quality and capabilities and lowers TCO.

If unclear then reach out for your TDI solution and launch your HANA Infrastructure with world class end-to-end leveraged partners and not get into a choke-hold grasp of an
appliance and what it truly represents.