Authors:
Arunkumar Thirunagalingam

Addresses:
Department of Business Intelligence and Reporting, Santander Consumer, Texas, United States of America. arunkumar.thirunagalingam@gmail.com

Abstract:

Modern data-driven applications demand efficient real-time data processing, driving innovation across sectors like e-commerce, banking, and healthcare. However, the manual construction and optimization of data pipelines struggle to meet the challenges of today’s high-velocity, dynamic data environments. Automated machine learning (AutoML) emerges as a transformative technology by automating and enhancing the creation, optimization, and management of data pipelines. This study explores the profound impact of AutoML on dynamic data pipelines, highlighting its role in improving efficiency, adaptability, and scalability in real-time data processing. AutoML streamlines the development process, reducing manual intervention and enabling faster, more accurate decisions. It empowers organizations to adapt swiftly to changing data patterns and business needs, facilitating more responsive and resilient data systems. Additionally, the study delves into the challenges and opportunities of integrating AutoML into real-time data pipelines. Key obstacles, such as ensuring data quality and managing computational resources, are discussed alongside the potential for AutoML to overcome these issues through advanced algorithms and automation. Case studies are presented to demonstrate the practical benefits of AutoML integration, showcasing real-world improvements in pipeline performance and operational efficiency. The findings underline AutoML’s pivotal role in shaping the future of dynamic, real-time data-driven applications.

Keywords: Digital Platforms; Technological Advancements; Machine Learning; Analysis Tools; Data Engineering; Performance; Technical Difficulties; Auto ML; Dynamic Data Pipelines; Data Processing.

Received on: 03/02/2024, Revised on: 07/04/2024, Accepted on: 25/05/2024, Published on: 07/06/2024

DOI: 10.69888/FTSIN.2024.000213

FMDB Transactions on Sustainable Intelligent Networks, 2024 Vol. 1 No. 2, Pages: 110-119

• Views : 115
• Downloads : 4

Download PDF