Yusuf Ganiyu
Senior Data Engineer
About
Yusuf Ganiyu is a Senior Data Engineer at AstraZeneca, where he architects AI-powered big data solutions that transform pharmaceutical data into actionable insights at enterprise scale. As Founder of Data Mastery Lab—recognized as London's Best Data Engineering and AI Training Platform in 2025—he has established himself as a leading voice in big data education.With over 50,000 students taught globally through platforms like Udemy, YouTube (CodeWithYu, 1M+ views), and his own training platform, Yusuf excels at making complex big data concepts practical and implementable. His end-to-end projects, ranging from real-time streaming pipelines to complete data platform implementations, serve as reference architectures for engineering teams worldwide.Holding an MSc in Computational Intelligence from Cranfield University, Yusuf is triple-certified across AWS, Azure, and GCP. His expertise spans the complete big data stack, including Apache Kafka, Spark, Airflow, Cassandra, Elasticsearch, and modern cloud data services.As an active contributor to the global big data community, Yusuf was a 2023 Elastic Silver Contributor with a 2M+ reach on Stack Overflow. His unique position, bridging enterprise implementation and large-scale education, provides practical insights into what truly works at production scale.
Talk
Yusuf Ganiyu | Case Study: Implementing Self-Healing Data Pipelines with Agentic AI
Self-Healing Data Pipelines, AI Agents, Apache Airflow, Data Quality, Human-in-the-Loop
<p>This talk presents a detailed case study of implementing agentic AI workflows in production data systems at AstraZeneca, one of the world's largest pharmaceutical companies.</p>
<p>THE CHALLENGE:<br />
AstraZeneca's data platform processes petabytes of pharmaceutical data across thousands of pipelines. Traditional monitoring generated alert fatigue, with teams spending 40% of their time on reactive incident response. Schema changes caused cascading failures. Mean time to recovery (MTTR) averaged 4+ hours.</p>
<p>THE IMPLEMENTATION:<br />
Yusuf Ganiyu led the implementation of agentic AI systems that autonomously:</p>
<ul>
<li>Investigate data quality anomalies, reducing investigation time by 70%</li>
<li>Detect and adapt to schema changes before failures occur</li>
<li>Diagnose root causes and suggest remediations</li>
<li>Execute approved fixes with human-in-the-loop governance</li>
</ul>
<p>ARCHITECTURE DETAILS:<br />
The talk covers the complete technical implementation:</p>
<ul>
<li>Large language model (LLM) integration patterns with existing Airflow orchestration </li>
<li>Tool design enabling agents to query metadata, run validations, and execute fixes</li>
<li>Guardrails to prevent hallucination in data-critical, pharmaceutical contexts</li>
<li>Cost management practices achieving an 85% reduction compared to initial projections </li>
</ul>
<p>MEASURABLE RESULTS:</p>
<ul>
<li>70% reduction in anomaly investigation time</li>
<li>60% decrease in schema-related incidents</li>
<li>MTTR improved from 4+ hours to under 45 minutes</li>
<li>Zero compliance violations from agent actions in a regulated industry </li>
</ul>
<p>Attendees will receive a reference architecture and an adoption roadmap template. This session is led by Yusuf Ganiyu, who brings hands-on experience in deploying production-grade agentic AI systems in highly regulated enterprise environments.</p>
2026-11-27
10:10
10:55