Di in the age of digital disruptions v1.0
•Als PPTX, PDF herunterladen•
0 gefällt mir•342 views
This document discusses how digital disruptions are changing businesses and the need for data integration (DI) modernization. It emphasizes that data is crucial for digital businesses and an efficient DI platform is key to success. The document outlines strategies like the big bang or 2-speed approach for DI modernization. It also highlights capabilities needed like API-based integration, stream computing, cloud infrastructure and logical data warehousing. Finally, it stresses the importance of adopting an agile operating model and DevOps culture for lean execution of the DI transformation.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (19)
Andere mochten auch
Andere mochten auch (15)
Ähnlich wie Di in the age of digital disruptions v1.0
Ähnlich wie Di in the age of digital disruptions v1.0 (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Di in the age of digital disruptions v1.0
- 1. www.LnTInfotech.com DI Modernization at the Age of Digital Disruptions Amar Roy L&T Infotech Ltd.
- 2. 2 About Myself I help my customers in their digital journey by developing their big, small & fast data solution.
- 3. 3 About L&T Infotech L&T Infotech is a global IT services and solutions provider. We provide the winning edge to our clients by leveraging our Business-to-IT Connect and Deeply Committed People. Our clients have found in us a right-size partner who combines scale, stability and customer-centricity.
- 4. 4 Digital businesses - disrupting “business as usual”!
- 5. 5 Data is the glue of all digital businesses! And an efficient data integration platform is the logical starting point towards success
- 6. 6 Today we will discuss… What is a digital business? Needs and challenges of DI modernization Strategies an organization can adopt Capabilities to be acquired Platform & toolset and Operating model to make the transformation to work
- 7. 7 What does it mean to be Digital?
- 8. 8 Being Digital Means*… And it demands innovation, speed & agility! creating value at the new frontiers of the business world creating value in the processes that execute a vision of customer experiences and building foundational capabilities that support the entire structure. *Source: What ‘digital’ really means | McKinsey & Company
- 9. 9 This is new normal of customer service and satisfaction! Uber runs its business on a platform which uses enables this level of customer service and more…
- 10. 10 Image Source: http://www.datanami.com/2015/10/05/how-uber-uses-spark-and-hadoop-to-optimize-customer-experience/ It’s fundamentally a data problem! - Aaron Schildkrout, Head of Data @ Uber
- 11. 11 However, this is not easy for every organizations!
- 12. 12 Their present Data Integration landscape looks like this! Where to start?
- 13. 13 Clearly , it is not only about technology!
- 14. 14 You will need to choose the right modernization strategy, acquire the right capabilities and integrate with existing ones, implement state of the art platform & toolset, adopt to an agile operating model which makes everything work seamlessly to get your DI ready for Digital Transformation…
- 15. 15 What are the right strategy an organizations can adopt? Depending upon your situation, you can choose either of these 2 strategies, Big Bang Approach 2-Speed Transformation Most of our customer adopts the 2nd strategy
- 16. 16 Identify the candidates which need to change at faster speed Not all DI applications need to change at faster speed
- 17. 17 Back office adopts slowly
- 18. 18 Customer facing systems needs to change faster!
- 19. 19 Big Data is only a piece in the DI puzzle… Stream Computing will make real-time integration possible Cloud Infrastructure will provide one- click provisioning and elasticity API Based Integration will enable a data as Microservices Logical Data Warehouse will contain Small/Big & Fast Data
- 20. 20 API based integration provides seamless cross channel experience by enabling data as a microservice… design of data services around business capabilities, automated deployment and decentralized control of language & data. REST is the de facto standard Image Source: https://blog.karmawifi.com/how-we-build-microservices-at-karma-71497a89bfb4#.ixbgljju0
- 21. 21 Real time streaming for responsiveness… Image Source: https://spark-summit.org/2014/wp-content/.../Lambda-Architecture-Jim-Scott..pdf
- 22. 22 Logical DW extends existing infrastructure to cater new data and related workloads… Data Quality ILM Data Governance Data Integration MDM Source Security Web Services APIs SQL Queries Search Custom Applications ERP SCM CRM Image Audio & Video Machine Logs Emails Web & Social Sensor Data Documents Eg Pdf, Docs StructuredUn-structured Data Acquisition Layer Data Consumption Layer Marketing Executives Operational Managers Data Scientists Business Analysts Engineers Customers Partners Frontline Workers Data Lake Archived Data Master Data Staging Data Non Prod Data Metadata Management Data Virtualization Analytical Data Data Fill Platform Data Mill Platform Data Drill Platform Data Warehousing Platform Near Real Time Load Batch Load Advanced Analytics Prescriptive Predictive Descriptive Business Intelligence BI Reports Ana Reports Dashboards Visualization OLAPODS Data Marts
- 23. 23 One-click provisioning & elasticity through a cloud infrastructure; allowing faster innovation “…Cloud Computing is multi-tenant, it’s faster, half the cost, pay as you go, it grows as you grow or shrinks as you shrink. It is extremely efficient…” - Marc Benioff, CEO SFDC [Oracle Open World 2010]
- 24. 24 Plethora of tools to architect your agile & scalable data platform Data Sources Flow Controller Stream Processor Storage Analytics Enterprise Applications Mobile Devices IoT Sensors System logs Web Scrapping Messaging Queue i.e. Kafka, Tibco EMS Amazon Kinesys Storm Spark Streaming Informatica CDC Flink Azure Streaming RDBMS NoSQL HDFS Cloud Storage S3, HDInsight Cloud DB - Redshift Custom Application – D3 Tools – Tableau, Qlik Elastic Search Many more
- 25. 25 Because of the speed and agility digital initiatives will usher a cultural shift in the way organization builds and deliver software…
- 26. 26 DevOps & Continuous Integration – Lean Execution Model for DI TransformationStart Continuous Integration Server Source code Repositor y Resolve Dependenci es Get Latest job/packag e Integration of components Static Analysis Deploy in QA Environme nt Unit & Functional Test Regression TestResult Validation Update Matching Defects Send Notificatio n (Start) (End) Build Tool (xml test cases) (Coding Standard, Sanity checks) (xml test cases) Java API Trigger Mechanis m Code Check-in Matrix Dashboar d (pearl, shell, js) Test Management Server SMTP Server Bug Reporting Tool Automated Test Script Raise Defect Feedback ETL jobs BI Dashboard s
- 27. 27 “Companies rarely die from moving too fast, and they frequently die from moving too slowly.” - Reed Hastings, CEO of Netflix
- 28. www.LnTInfotech.com Please send your questions to amar.roy@lntinfotech.com Let’s Solve
Hinweis der Redaktion
- Good afternoon everyone! My name is Amar Roy and I am a principal solution architect at L&T Infotech. I help my customers in their digital journey by developing their big, small & fast data solution.
- Among other service lines, we have keen focus in Analytics and Information management in the Context of digital transformation of our customers.
- Let’s get started with today’s topic, but before we get their I have a question. What is common among these companies? Yes, these are all digital businesses; but they are digital disruptors as well. Today digital businesses are everywhere. Every day they are not only developing new business models but disrupting those of the incumbents. Amazon did this to Border’s brick and mortar bookstores. Uber is virtually wiping out the traditional Taxi industry by allowing people to provide taxi service if they have a smart phone and a car. And they are coming out in all segments of businesses. So in a way posing threat to the incumbents with the data centric innovation & speed. So clearly there is a need of change to be in business and be competitive!
- All these digital companies have one more thing in common; they are master in using data for running their business. They use data for product innovation, customer insight, process optimization and speed. In fact , data is used as Glue of all digital businesses. And before any bit of data can be used, it must be ingested, processed and presented to to internal/external consumers. This is the reason DI plays the central role in the digital journey of an Organization.
- My goal of this presentation, is to elaborate a clear framework for DI modernization which can sustain the digital initiative of a company…
- But even before we discuss the importance of data integration modernization in the context of digital disruption, let’s understand what it means to be digital. And understand why it is disruptive in nature.
- Last year, I took an Uber from my office in Edison to Newark Airport. When I got down at Newark, I looked at my Uber app and Realized the driver did not closed the trip immediately! He drove another 2-3 miles before he closed the trip. I was literally pissed and gave a bad rating with description of what happened. Within next 6 min, I got this email… This is the benchmark digital companies are setting… Digital businesses are setting new benchmarks for speed, agility, and user-friendliness, consumers expect similar online performance from banks, retailers, and telecommunications companies.
- Before we move on to how DI modernization can be done for Digital Age, lets pause for a moment to examine what kind of systems and infrastructure enables Uber to provide such un parallel customer service with speed and Agility. As expected, Uber Head of Data, Aaron Schildkrout says “It’s fundamentally a data problem!” It is evident from this architecture , Uber has focused on building a robust DI architecture which enable processing huge amount of data real time. This is the core of their business…
- But unlike the Uber, incumbent organizations does not have the luxury of starting with a clean slate. Their Data Integration landscape has grown overly complex over time and can’t be changed overnight.
- So what do they do if their DI systems looks like this? Where should they start to make their DI more agile and responsive without disturbing all functions at once?
- So technology can’t alone solve this problem. They need a more holistic approach for a successful DI modernization.
- This transformation is possible if these 4 activities executed well choose the right modernization strategy, acquire the right capabilities and integrate with existing ones, implement state of the art platform & toolset, adopt to an agile operating model which makes everything work seamlessly We will examine these in next few slides and understand how these will facilitate the DI modernization in the digital journey of the organization.
- We have found there are two ways organization can make this happen Big Bang Approach – Change the entire DI landscape as part of multi year transformation program. However, most often that is not practical 2-Speed Transformation – In today’s competitive environment, you need to adopt an incremental approach which McKinsey termed as two – speed approach. Most of our customer adopts the 2nd strategy. One of our Oil & Gas customer actually followed the approach #1 But one large US bank has chosen to go for approach #2
- In an organization not all DI application need to change at the same speed or to be touched at all during DI modernization initiative. To start with, create a distinction between these two set of applications and build a plan on how these two set of applications will be handled in your DI transformation program.
- Core transactional system of records needs to be stable & consistent for various reasons Like risk & regulatory compliance etc. These are typically your ERP, Financial and Inventory management systems. They need Accuracy over speed. And should not be changed often. We can leave them alone if they are doing what they are expected to do…
- a. However, your customer facing systems needs to change fast and often multiple time within a day. So these should be your primary focus for the DI transformation project. b. These systems needs to be agile and change often. Think about major retailer like Amazon, Media and Entertainment Company like HBO/Netflix. They changes often – even multiple time during the same day… c. These application should be focused during the DI transformation…
- a. There is a notion that Digital is all about big data analytics. Not completely true! To provide the speed & agility data integration part is equally important as storing And analyzing big data. You need to make your DI platform capable of handling big, small & fast data. We have seen these are the broad DI capabilities which will make the customer facing systems, Provide seamless experience across channels Real-time insight on customer behavior Zero down time Auto-scale up/down based on load
- Amazon and Netflix are among the biggest users of Microservices. Amazon has more than 100 micro-services in their site. Because of the modular nature micro-services enables automated deployment with zero down time of business Critical websites. You can deploy multiple changes within normal business hours with out disrupting your Customer experience. Most important from DI perspective, you can reduce the need to move data across systems and save valuable ETL development time. Finally you, can virtually remove the nightly FTP jobs , make the data available on-demand freeing up the Network traffic.
- Real-time streaming data integration is the other component of speed, agility and customer experience. We have seen the mail that Uber customer support sent within 6 min of submitting my grievance! Traditionally , all data integration were designed for processing batches. Even, in HDFS in essence a Batch processing system with huge throughput but high latency. That was good, but today most of your data comes in streams, be it sensor data or tweets from customer. So , besides batch processing you need the ability of processing data streams as well. Lambda Architecture , is an architecture style which implements both batch as well as stream processing Side by side. You can see the technology components of such an architecture. Finally, you need a database Which will provide the data to applications with extremely low latency
- However, most of the organizations has invested heavily on the Enterprise Data Warehouse ecosystem. Most often, they need to be expanded to include new data sources or new types of data. This gives rise To what we call is Logical Data Warehouse. There is another school of thought which calls for a Hadoop DataLake. But we have seen mixed success Hadoop Data Lake.
- In a digital business workload varies , often unexpectedly. Organizations should look for an elastic infrastructure which expands and contracts with load. And cloud infrastructure Provides exactly the same. If you see , most of the digital businesses born in cloud and it has given them extra-ordinary scale With minimal investment. They can provision a cluster within minute, ingest data and try out an Idea. They can fail fast and innovate faster… In a two speed DI transformation, cloud infrastructure is ideal for the fast moving applications.
- For your data pipeline you have lots of choice from open source or Informatica to choose from
- Digital businesses breaks silos within organizations…It brings level of transparency in the way software is built, tested and deployed. Most often software development is tied up with product launch and success of an initiative. So these often draws more attention of Senior executives. On the team front, project mindset gets dissolved paving the way for product mindset. A team of competent engineers owns and Improves the code base. People is rewarded for failing faster and innovation…
- While the system of records follow waterfall release methodology, customer facing systems needs to be agile. Changes to be deployed in days not hours. So DevOps and Continuous Delivery is the way forward. Automated build solution Automated Deployment Solution Automated testing solution – Test driven development Capture metrics and improve