Biography

Cristian is an hands-on chief architect, fully dedicated to build Previnet private cloud infrastructure through custom made devops tools, used to deliver and rollout highly performant and scalable applications based on reactive microservices and kubernetes.

Talk

Real-Time Stream Processing for Insurance & Health Care With Kafka, Kafka Streams and Multi-Runtime Microservices

Insurance companies traditionally heavily used batch processing to manage their data. However, nowadays we expect to have the data ready in near real time. Moreover systems increased in complexity to satisfy new challenging requirements like usage of AI and ML components.

As a service provider for insurance companies, pension & healthcare funds we rolled out a resilient stream processing platform running in kubernetes that we can scale out horizontally to integrate different microservices developed in different languages like java, scala or python.

To communicate between microservices kafka is used, since it’s scalable and resilient. All microservices are usually stateless and are governed by scala based orchestrators developed with kafka-streams. For python microservices we adopted pykafka, while for scala microservices we heavily uses akka and akka-streams to be able to scale vertically inside a single microservice instance.

The state is usually passed along each message down the data processing pipeline, but in case of big payloads (like documents to be classified by ML components) a redis cluster is used for in memory object storage or a mongo cluster if persisted storage is required. In house connectors based on akka are used to operate with traditional relational databases.

Session Keywords

« Back