The Cloud, an accelerator of "data deluge", Are there any solutions?

 

The Cloud, an accelerator  of  "data deluge",  Are there any  solutions?

According to a 2022 Salesforce global survey, 33% of CIOs said they were unable to generate value-added insights from their data, and 30% said they were simply overwhelmed by the volume of data produced within the company….   Businesses are ingesting more data than ever before largely thanks to the infinite scalability of the Cloud. The idea is to exploit them to obtain new “business insights”, to personalize experiences, or to meet regulatory requirements, the best known of which is the GDPR. Connected objects are also a fantastic accelerator for data generation.   But in many cases, these volumes end up being counterproductive, “drowning” truly useful data in terabytes of cold, obsolete, redundant data. IT teams end up spending most of their time introspecting systems to try to master their assets, and IT is no longer doing anything but maintaining systems whose seams are failing one after the other! And speaking on trending topics, companies that don't have a good handle on their data will struggle to exploit the possibilities of artificial intelligence, including generative AI models.   Furthermore, most large companies, in order not to put all their eggs in one basket, have adopted Multicloud strategies. But when companies consolidate this data for analysis, they often lose much of its contextual information, so much so that it is common for 80% of any data project to be devoted to simply cleaning the data and re- creation of context!   The complex software landscape also contributes to this difficulty: on average, companies have around 120 Cloud applications!   CIOs have been driven despite themselves by the accelerated pace of innovation and have not thought strategically about its consequences such as data management or Cloud costs.  One solution could simply be to collect less data, because much of this data remains “cold” or is never used: this share is estimated at 50% (Gartner).   A certain number of companies are starting to sort the data collected upstream to avoid downstream congestion. “We've pretty much stopped bringing in some completely cold data. When we realize that no one is using them, we close the pipes,” said Shanti Lyer, CIO of DocuSign, for example.    More statistically: 7 out of 10 companies do not know exactly what they are spending their Cloud budget on (Fortinet 2021). 131 IT professionals say that undue cloud spending could represent up to 47% of a cloud budget (Stormforge 2022).   Other options?    We decided to implement a tooled approach which is based on our software {openAudit} and which can help solve a large part of this equation, without “closing the pipes” upstream, without disrupting the richness of the systems.  How ?      1. By mapping systems in an automated way,  and sharing this knowledge to everyone   We believe that teams must have tools at hand allowing them to understand in a few clicks how each piece of data circulates within the information system as well as knowing all the uses that are made of it.  In short, each piece of data could be diagnosed. This is the technical “data lineage”. Thus, even if the information system becomes more complex, these automated “cards” would make it possible to control flows, to intervene in the right place when there is an error or an inoperative flow or even an incorrect formula in the upper layers ( dataviz).      2. By identifying dead matter dynamically to simplify systems

We recommend eliminating this “dead matter” on a continuous basis. By identifying unnecessary data in the business layers, then all their sources via data lineage, it is possible for us to identify and put aside all these materials which unnecessarily create opacity.      Conclusion Informational inflation and all its corollaries are not inevitable. We believe that an information system must both be mastered by everyone and brought back to the essentials to make it possible over time to control Cloud costs, to restore intelligibility and energy sobriety.  The task is sufficiently arduous that it is necessary to implement tools that automate processes to support this virtuous circle. This is our proposal through {openAudit}.     #datalineage #itdebt #finops #dataops  www.ellipsys-lab.com