AI-driven ETL Optimization for Security and Performance Tuning in Big Data Architectures

Shiva Kumar Vuppala

AI-driven ETL Optimization for Security and Performance Tuning in Big Data Architectures

Author(s)	Shiva Kumar Vuppala
Country	United States
Abstract	In today’s era of big data, Extract, Transform, and Load (ETL) pipelines form the backbone of enterprise data processing. Traditional ETL systems, while foundational, often suffer from major limitations such as poor scalability, vulnerability to security breaches, slow transformation speeds, and heavy reliance on manual tuning and monitoring. These inefficiencies lead to increased human errors, delayed analytics, and difficulties in maintaining compliance with stringent security standards like PCI 4.0 and GDPR. Furthermore, conventional anomaly detection techniques used within ETL pipelines often fail to accurately identify complex, time-dependent irregularities in large and heterogeneous datasets. To overcome these challenges, this work proposes an AI-driven ETL optimization framework that enhances performance, security, and compliance capabilities. The methodology integrates Attention-LSTM networks for real-time anomaly detection, enabling the system to dynamically focus on critical sequence patterns for higher detection accuracy. Secure data extraction is enforced using TLS 1.3 encryption protocols, ensuring data confidentiality during transfer, while intelligent data transformation is achieved through Random Forest algorithms to automate and optimize transformation operations efficiently. Role-Based Access Control (RBAC) mechanisms are used to strengthen secure data loading, and the final deployment is seamlessly integrated with AWS Glue for scalable orchestration. Extensive experimental results demonstrate that the proposed AI-enhanced ETL pipeline significantly outperforms traditional methods in processing speed, anomaly detection accuracy, data throughput, and security compliance, while simultaneously reducing the risk of human error. This research highlights the transformative potential of AI technologies in modernizing ETL architectures, paving the way for more resilient and intelligent big data systems in enterprise environments.
Keywords	AI-driven ETL, Anomaly Detection, Attention, Secure Data Extraction, Data Transformation, Compliance Automation
Field	Computer > Data / Information
Published In	Volume 6, Issue 5, May 2025
Published On	2025-05-10
Cite This	AI-driven ETL Optimization for Security and Performance Tuning in Big Data Architectures - Shiva Kumar Vuppala - IJLRP Volume 6, Issue 5, May 2025.

View / Download PDF File

doi

CrossRef DOI is assigned to each research paper published in our journal.

IJLRP DOI prefix is
10.70528/IJLRP

Downloads

Research Paper Format Copyright Permission Form and Undertaking Form Cover Page Vol 5 Isu 12 Cover Page Vol 5 Isu 11 Cover Page Vol 5 Isu 10

All research papers published on this website are licensed under Creative Commons Attribution-ShareAlike 4.0 International License, and all rights belong to their respective authors/researchers.

CC-BY-SA

About IJLRP Fees & Payment Current Issue Publication Archive	Submit Research Paper Track Submission Status Publication Guidelines Publication Ethics Peer Review & Plagiarism	Join as a Reviewer Editors & Reviewers Reviewer Referral Program Get Reviewer Membership Certi.	Website/Journal Policies Usage Policy Content Policies Privacy Policy

Contact Us		+91-9687-828-838	editor@ijlrp.com

International Journal of Leading Research Publication

A Widely Indexed Open Access Peer Reviewed Multidisciplinary Bi-monthly Scholarly International Journal

AI-driven ETL Optimization for Security and Performance Tuning in Big Data Architectures

Share this