Event-Driven Data Consolidation for Global Streaming via Passive ETL Middleware
A global creator platform runs eight distinct streaming products. Event data from all of them reaches Customer.io through a custom passive ETL, which puts a premium on middleware stability. We mapped the platform's complex subscription milestones and onboarding events through that pipeline, so automated, persona-driven lifecycle communications now reach millions of users worldwide.
Executive Summary
Context
A global leader in live streaming software and creator tools ran multiple email platforms. They needed a single instance of Customer.io to support a localized, worldwide audience.
What We Built
A unified data pipeline built on a passive custom ETL tool. It formats, filters, and publishes core product and seed product events directly to Customer.io.
Tech Stack
- Custom ETL tool (API-driven)
- Customer.io
- Eight creator products
Not a fit if you want a "no-code" or "plug-and-play" integration. Teams without internal dev resources to maintain middleware source code should also pass.
The Challenge
The custom ETL that feeds Customer.io is passive by design. It cannot poll the source products for changes and it cannot repair itself; it processes whatever is pushed to it. When a schema update went through in one of the source product databases, the integration broke immediately. The only fix was to mirror the new logic inside the middleware by hand. Every schema migration across the eight products created the same exposure.
The expansion had to be architected against a release date of February 1, 2024, while the legacy functionality was documented in parallel. The new build needed to handle complex subscription lifecycle events such as a subscription cancellation or a settings change. A missed sync on any of the eight independent seed products would mean fragmented user data and broken billing communications for millions of creators.
Our Approach
We began with an exhaustive audit of the ETL payload events, looking for data gaps. The audit turned up one significant gap: critical affiliate status data was held in a third-party tool not connected to the pipeline, which blocked several high-value lifecycle series.
We chose not to rebuild the ETL and concentrated on documentation and stabilization instead. A detailed Entity Relationship Diagram (ERD) now maps how user data flows from the core product into Customer.io. With that in place we updated the logic for the onboarding and creator-type email series. Onboarding questionnaire results now pass correctly through the ETL and branch the email logic by persona, Livestreamer or Podcaster for example.
This engagement is part of our systems integration work for telecommunications & media organisations.
Impact
Platform Consolidation
We migrated and consolidated multiple disparate email systems into a single Customer.io instance, so the global communications stack got simpler to run.
Unified Product Data
Transactional data from eight "seed" products, now routes through the ETL to Customer.io, so each user has one view of activity across products.
Standardized Deployment
We delivered a comprehensive Standard Operating Procedure (SOP) for campaign creation. The client's internal team can now run campaigns on their own, using live product data.
18+ Lifecycle Emails Redesigned and Localized
We redesigned or developed more than 18 lifecycle emails to match a new, vibrant brand voice, and localized each one for 5 or 6 languages.
The ETL operates as "wait-and-process" middleware. It receives API calls from the product databases, then cleans and formats the JSON payloads before pushing them to the Customer.io REST API.
Specific subscription states (e.g., SUBSCRIPTION_STARTED) map as triggered events in CIO. Billing changes get an immediate automated response.
Onboarding questionnaire data gets captured at the source. It passes as attributes to the user profile in CIO. That drives dynamic content delivery based on the creator's specific platform (Video Creator, Podcaster, etc.).
The email system supports 5 or 6 languages. Liquid logic within Customer.io serves the correct language variant based on the user's browser settings or profile settings.
The custom ETL that feeds Customer.io is passive. It can't poll the source products or repair itself. A schema change in any of the eight products used to break the integration until someone mirrored the new logic by hand. We chose not to rebuild the ETL, so we audited the ETL payload events and documented the data flow into Customer.io in an Entity Relationship Diagram. Onboarding questionnaire results now pass through the ETL and branch the email logic by persona.
FAQ
A passive ETL requires dual-active maintenance: if you update your product database schema, you must also manually update the ETL code to recognize the new fields. This is more rigid than a standard sync but allows for much higher control over data filtering and formatting before it reaches your CRM or marketing tool.
A unified user ID serves as the primary key. Each product sends its events under distinct event names. The marketing team can trigger product-specific sequences while the user's total engagement stays on a single, consolidated record.
