Biography

Valdas is an IT Architect at Cognizant, where he leads Data Lake and Data Science platform implementations on Azure Cloud for various sectors in the Nordics. He helped design and develop product recommendation, sales optimization, product pricing and churn reduction use cases. Valdas is passionate about AI, Formula 1 and Space Exploration.

Workshop

Build a Modern Data Platform in Azure

Abstract

In this 1-day workshop, you will learn about the main concepts related to data processing and data engineering with Azure Data Services. You will understand what Azure services you can leverage to establish a solid data platform to quickly ingest, process and visualize data from a large variety of data sources. Together we will implement a modern data platform architecture – a proven setup to give you flexibility and scalability to grow and handle large volumes of data and keep an optimal level of performance. 

Technologies you will use: Azure Data Factory, Databricks (w/Spark), Azure SQL, Azure Data Lake Gen 2, Event Hub, Stream Analytics, Power BI

Agenda

• Workshop Overview | 15 minutes
• Modern Data Platform Concepts: Part I (Data Storages and Data Movement) | 15 mins
• Lab 1: Load Data into Azure Data Lake Gen 2 and Azure SQL using Azure Data Factory Pipelines | 60 mins
• Modern Data Platform Concepts: Part II (Data Exploration and Transformation) | 15 mins
• Lab 2: Explore and Transform Big Data using Azure Databricks | 60 mins
• Modern Data Platform Concepts: Part III (ML and Cognitive Services) | 15 mins
• Lab 3: Add AI to your Big Data Pipeline with Cognitive Services | 60 mins
• Modern Data Platform Concepts: Part IV (Real-time pipelines) | 15 mins
• Lab 4: Ingest and Analyse real-time data with Event Hubs and Stream Analytics | 60 mins

Throughout a series of 4 labs, you will progressively implement a modern data platform architecture.

First, we introduce the concepts of the data lake and big data challenges. You start ingesting various data types to your Azure Data Lake Gen 2 storage. Then, you have to use Databricks and the power of Spark clusters to explore data files. Later, you incorporate AI into your data pipeline by invoking Cognitive Services and store results in Azure SQL database. Finally, you use car telemetry events as a source of streaming events that you will capture, store and process in real-time with Event Hubs, Stream Analytics, and Power BI.

Prior working knowledge of the Azure Data Platform components is not required, but attendees should be familiar with the services at the theoretical level. Each lab will contain multiple exercises of different difficulty levels.

Target audience: BI Developers, Data Engineers, Data Scientists, Data Analysts, Database/Sys Admins

• The following prerequisites must be completed before you start these labs:

  – You must have an Azure account with administrator- or contributor-level access to your subscription. If you don’t have an account, you can sign up for an account following the instructions here:
  https://azure.microsoft.com/en-au/pricing/purchase-options/pay-as-you-go/
  https://azure.microsoft.com/en-in/pricing/member-offers/credit-for-visual-studio-subscribers/
  https://www.microsoftazurepass.com/

  – Lab 4 requires you to have a Power BI Pro account. If you don’t have an account you can sign up for a 60-day trial for free here: https://powerbi.microsoft.com/en-us/power-bi-pro/

  – You must have Visual Studio Code installed and admin rights on your laptop