Event-Driven Data Consolidation for Global Streaming via Passive ETL Middleware
We unified a global creator platform's product databases with Customer.io via the Kraken passive ETL. Event data from eight distinct streaming products flows through it. That job demands dependable middleware stability.
Complex subscription milestones and onboarding events got mapped. Automated, persona-driven lifecycle communications now reach millions of global users.
Executive Summary
Context
A global leader in live streaming software and creator tools ran multiple email platforms. They needed one single instance of Customer.io to support a localized, worldwide audience.
What We Built
A unified data pipeline built on Kraken (a passive ETL tool). It formats, filters, and publishes core product and seed product events directly to Customer.io.
Tech Stack
- Custom Kraken ETL tool (API-driven), Customer.io, and eight approx distinct seed products including Video Editor, Talk Studio, and Cross Clip.
Not a fit if you want a "no-code" or "plug-and-play" integration. Teams without internal dev resources to maintain middleware source code should also pass.
The Challenge
A source product database schema update broke the integration immediately. The only fix was mirroring the logic inside the Kraken middleware by hand. The architecture is "passive." The ETL tool couldn't poll for changes or self-heal. It simply processed whatever was pushed to it. Every schema migration carried that high-risk dependency.
The target release date was February 1, 2024. Legacy functionality had to be documented while the expansion was architected in parallel. The new build had to handle complex subscription lifecycle events like NON_ULTRA_SUBSCRIPTION_CANCELLED and USER_SETTINGS_UPDATED. Missing a sync across the eight independent seed products meant fragmented user data. It also meant broken automated billing communications for millions of global creators.
Our Approach
An exhaustive audit of the Kraken payload events came first, hunting for data gaps. One finding stood out: critical affiliate status data lived on a third-party tool that wasn't connected to the pipeline, which blocked several high-value lifecycle series. We didn't rebuild the ETL. We focused on documentation and stabilization instead, creating a detailed Entity Relationship Diagram (ERD) that maps how user data flows from the core product into Customer.io. We then updated the logic for the onboarding and creator-type email series. Onboarding questionnaire results now pass correctly through Kraken and branch the email logic for different personas, such as "Livestreamer" versus "Podcaster."
Impact
Platform Consolidation
Multiple disparate email systems were migrated and consolidated into a single Customer.io instance. The global communications stack got simpler to run.
Unified Product Data
Transactional data from eight "seed" products, including Link Space and Charity, now routes through Kraken to CIO. The result is a complete view of user activity.
Standardized Deployment
We delivered a comprehensive Standard Operating Procedure (SOP) for campaign creation. The client's internal team can now run campaigns on their own, using live product data.
18 Lifecycle Emails Redesigned and Localized
Over 18 lifecycle emails were redesigned or developed to match a new, vibrant brand voice. Each was localized for 5, 6 languages.
Kraken operates as "wait-and-process" middleware. It receives API calls from the product databases, then cleans and formats the JSON payloads before pushing them to the Customer.io REST API.
Specific subscription states (e.g., ULTRA_SUBSCRIPTION_STARTED) map as triggered events in CIO. Billing changes get an immediate automated response.
Onboarding questionnaire data gets captured at the source. It passes as attributes to the user profile in CIO. That drives dynamic content delivery based on the creator's specific platform (Video Creator, Podcaster, etc.).
The email system supports 5 or 6 languages. Liquid logic within Customer.io serves the correct language variant based on the user's browser settings or profile settings.
A middleware integration architecture consolidating event data from eight disparate streaming products into Customer.io via Kraken ETL. By mapping high-volume event payloads and utilizing liquid logic for persona-based branching, the system stabilizes automated transactional messaging across multiple languages, ensuring event-trigger parity with the core products.
FAQ
A unified user ID serves as the primary key. Each product sends its events under distinct event names. The marketing team can trigger product-specific sequences while the user's total engagement stays on a single, consolidated record.
