CloudEXPO 2019

Addressing Critical Shortcomings of ETL Tools for Better Data Analytics

Poor data quality and analytics drive down business value. In fact, Gartner estimated that the average financial impact of poor data quality on organizations is $9.7 million per year. But bad data is much more than a cost center. By eroding trust in information, analytics and the business decisions based on these, it is a serious impediment to digital transformation.

Extract, transform and load (ETL) tools like AWC Glue bring much needed functionality. This tool enables new approaches to pulling, processing and pushing data from source to target, and introduces concepts such as performing data transformation tasks using SparkSQL scripts in Apache spark environment. However, there are shortcomings with AWS Glue, leading to a number of challenges and questions:

Where are the necessary coding techniques when it comes to dealing with specific data types and many such technical aspects of traditional ETL tools?

How can data analysts rely on the Glue-Dynamic frame concept for some of the key design aspects when it comes to incremental loads design and data conversion process?

How can accurate reporting be achieved when the data processing step truncates decimals points, creating huge data discrepancies?

At Infostretch, we believe these challenges can be handled by utilizing specific database level functions. In this session, we will showcase real-life experience with such deal-breaking scenarios, and demonstrate how to mitigate these issues without jeopardizing reporting accuracy, compromising on quality, or endlessly waiting for new releases.

Attendees will learn how to overcome the critical shortcomings of the AWS Glue ETL tool to achieve success:

How to use specific database level functions without jeopardizing quality

Address issue of truncating decimals to produce accurate reports

Ensure source system KPIs match up with target system KPIs for complete business insights.