Biography

Daniel Wrigley has works as a Senior Search & Analytics Consultant. He mostly deals with search and big data applications with a strong focus on modern open source projects such as NiFi, Solr, Spark or Zeppelin. His experience as a Solr trainer enabled him to co-author the first German book on Solr.

Workshop

Introduction to Data Shipment and Processing with Apache NiFi

Abstract

Huge amounts of data are being generated every second. Humans produce data by writing e-mails, documents or text messages, presidents produce data by tweeting and machines or applications produce data by logging information. These are just a few examples of how data is generated nowadays.
As data in its original place and form often is no use at all it needs to be transferred to other systems or applications and it also needs to be transformed on its way to these. And this is where Apache NiFi has its strengths. In this workshop you will not only learn the fundamentals of Apache NiFi but also learn in hands-on activities best practice approaches to unleash NiFi’s power for your business cases and become acquainted in reading data from data sources, transforming data and write it to other systems.
After this workshop you will be able to set up a distributed system that is able to process thousands of records or even more per second to gain actionable insights from your data.

Agenda

Part 1: Intro to Apache NiFi

• What is Apache NiFi?
• NiFi Terminology & Components

Part 2: Designing DataFlows

• DataFlow Basics
• Record-based Processing:
  • When and why to use Records
  • Defining a Schema
• Extending NiFi: Scripting & Custom Processors

Part 4: Scaling NiFi & Beyond

• NiFi Cluster Mode
• NiFi Registry: Version Control for your Flows
• NiFI API & Toolkit: Interact with NiFi via REST API & the NiFi Toolkit

Part 5: NiFi Best Practices

• Repository Handling, Configuration & Important Settings
• Monitoring NiFi

The main objective of this workshop is to introduce Apache NiFi to the audience and how to use it for data shipment and processing tasks. After this workshop attendees will have a broad overview of what NiFi is and how it can be leveraged in data-related projects using best practices.

The target audience includes anyone dealing with one of the following topics: Data shipment, data processing, data engineering, data validation, data cleansing, … Basically, anyone working with data is a good fit for this workshop. Those new to NiFi will profit most as we start our journey gently. But those who already have some NiFi knowledge will profit from the more advanced topics.

• Installations:
  • A laptop with Windows/MacOS/Linux operating system.
  • Java Runtime Environment 1.8 or 1.9
• Technical Knowledge: Some experience with the Linux command line helps

Note: Lab & hands-on material including the necessary binaries (NiFi, Registry, ZooKeeper) will be distributed on USB-Sticks.