1.   Abstract
In the time of data-driven decision-making, the data validation of ETL (Extract, Transform, Load) pipelines is crucial for delivering high quality information. This study explores the design and testing of an ETL data pipeline built with the top market technologies as:

• Prefect, Python Pandas, and Pytest. Prefect orchestrates pipeline workflows, ensuring transformation dependencies and scheduling are managed correctly. Pandas handles data manipulation, offering robust tools for efficient transformations. Pytest enables a structured framework for data attributes testing like reliability, accuracy, consistency, completeness and uniqueness.
• Dagster, DBT (Data Build Tool) with DBT Test. Dagster orchestrates data pipeline, DBT is a T in ETL so performs the Transformation and finally DBT Test is an integrated test module in DBT which provides very useful ready to use data tests in SQL and YAML.
  Overall, the presented approaches demonstrate how integrating workflow orchestration, data transformation, and automated testing creates a reliable foundation for trustworthy analytics. By embedding validation directly into the ETL process, potential data quality issues can be detected early, reducing downstream risks and improving decision-making confidence. The combination of integrated technologies in both scenarios provide a scalable and maintainable framework that supports continuous monitoring of data quality, highlighting the importance of test-driven practices in modern data engineering pipelines.

2. Agenda

1. Introduction to Data Testing
2. Environment configuration: Pycharm IDE + Python libraries (Prefect, Pytest, Pandas), Dagster + DBT
3. Create simple Data Pipeline with:

• Prefect and Pandas
• Dagster and DB

4. Data Testing (Design, Execution, Report) with:

• Pytest
• DBT Test

5. Q&A and Summary

3.   Objectives

• To design and implement an ETL data pipeline using modern data engineering technologies, including Prefect, Pandas, and Pytest, as well as Dagster, DBT, and DBT Test.
• To evaluate data quality by testing key data attributes, including accuracy, completeness, consistency, uniqueness, and reliability throughout the ETL process.
• To implement automated data validation using Pytest and DBT Test to identify data quality issues and detect errors early in the pipeline.
• To compare two ETL testing approaches in terms of testing capabilities, workflow orchestration, maintainability, scalability, and ease of implementation.
• To assess the effectiveness of integrating automated testing into ETL workflows to improve data reliability and minimize the risk of inaccurate analytical results.
• To propose a scalable and maintainable testing framework that supports continuous data quality monitoring and ensures trustworthy data for decision-making.

4. Target audience and Prerequisites
Workshop for Beginners (QA Engineers, Automation Testers, Data Developers). Basics of Python, SQL, YAML is recommended.

5. Technical requirements
The best choice is to bring your own private laptop. Corporate laptops have security limitations of installing new software.

All tools and software will be installed in the workshop.
Internet access is required in the room.