From source to dashboard cell: Mapping the IS to rebuild it intelligently

From source to dashboard cell:     Mapping the IS to rebuild it intelligently

In IT transformation projects, there is one injunction that keeps coming up: “We must migrate!” … to the Cloud, to modern architectures, to more agile solutions, etc.    In recent years, we have seen the emergence of new data solutions that are sweeping everything away in their path: we are thinking, for example, of Power BI for dataviz, dbt for transformation management or Snowflake for storage: simple, efficient, scalable, Cloud native, often inexpensive, etc. When you decide to take action, a few questions come to mind: "  What exactly should be migrated  ?" "  So, what can we throw away   ?" "  And how can we rebuild what's valuable in the target without breaking everything?  " Below is an approach based primarily on real-time analysis of all internal processes within a company.

To modernize, to migrate, it will be necessary to understand how information circulates, from operational sources to its final exposure, column by column, to the dashboard cell accessed by the business.   And knowing what the information produced is used for (in concrete terms)   ! The observation is often quite surprising...

Action #1:  Inventory all continuous flows, from sources to the dashboard cell, with complete reverse engineering of the IS

In the power layers This will involve analyzing stored procedures, ETL/ELT jobs, nested views, encapsulated SQL transformations, FTP transfers, subqueries, cursors, etc.

Output ;  At each stage, technical data lineage will highlight the transformation tools used (such as ETLs, scripts, SQL queries, etc.), the nature of the processing applied (filters, aggregations, joins, etc.) as well as the sequence of operations or flows (via schedulers, execution scripts, etc.):

In the data visualization layer This will involve pushing the analysis down to the cell of a dashboard, even in complex multi-tech environments: analysis of management rules embedded in dataviz objects, the intermediate semantic layer, expressions, filters, etc.

Output: A data lineage in the data visualization layer, which allows you to zoom from a source field in the DWH to the cell accessed by the business. This data lineage is connected to the data lineage in the sources, to have a complete view of a data flow.

Action #2:  Define usage levels: Know what each power flow is used for, from operational sources to the dashboard cell.

It will be necessary to analyze the main technical stack  to know all the data consumed in and outside the batch chains.

Data consumed by satellites (non-parsed applications) will also be analyzed  to identify the completeness of useful information.

This dual analysis will be configured to take into consideration the business target  : regulatory information can be consumed very periodically, while having significant added value.

Combine action #1 and action #2:  Combining data lineage and usage analysis, the winning combination for transforming a system!

Reassemble the streams to isolate unnecessary channels: Data lineage allows you to trace the pipeline from unused data to the first table at the origin of information consumed in another branch. From this branch, it will be possible to delete the unnecessary fraction of the chain  without impact: our various missions allow us to advance that 50% of the tables and processes can be eliminated on average. The same is true (often well beyond) for dashboards… Go! In the "remainder to migrate" section, each flow and dashboard can be scored based on internal considerations.  This will allow for precise migration prioritization.

Automated Migration to the Cloud:  Technical reverse engineering (continuous) will allow the processing chains to be exposed (ETL/ELT jobs, procedures in the feed layers, management rules in the data visualization layer). This ultra-granular knowledge of the source, its upstream rationalization, will allow for intelligent reconstruction of flows in the target technologies,  in the feeds and in the dataviz (in addition to the template and the possible semantic layer). The pivot will be largely operated via SQL. This will be a real gateway to the transformation of the IS.

Migrate power supplies:  Technological scope addressed by {openAudit}:  Migrate the dataviz layer:  Technological scope addressed by {openAudit}:

Understanding to better transform: this is the whole role of data lineage and usage analysis. With  {openAudit} , you can have a robust and automated solution for mapping, streamlining, and securing IT modernization projects—from the first column to the last cell of the dashboard. Our diagnostic and migration projects are offered as a fixed price.