Adobe Launch Monitor: Real-time Data Capture
Looking into analytics tag managers reveals just how fascinating they are. They (can) capture data on what users of your website are up to — what pages they visit, which buttons they click on, how long they spend on each page, and more. And they (can eventually) send that data to reporting tools, Adobe Analytics for example, for when you want to observe trends or generate insights on users’ behaviour. Quite the feat!
At Telus Digital, our tag manager is Adobe Launch, and all of our data eventually lands in Adobe Analytics. Magical though it may seem, this is far from automatic. It works in principle because, in Launch, we create rules/tags and specify the conditions that must be met for them to be triggered — and where we want the data sent to. It is not enough to trigger a rule though. We pay attention to rules that are successfully triggered as well as unsuccessfully triggered rules, the latter implying that at least one condition failed.
Anytime a rule is successfully triggered, we capture data that tells us what the website user was up to. For the most part, this data shows information relevant to the user. Reporting tools provide insights into what data is captured, but they do not show any data on what rules (in Launch) were triggered to facilitate the user data capture — at least not by default. This is partly because Launch does not capture any data on what rules were triggered — again, at least not by default. Matter-of-factly, the Launch API, despite its awesomeness, does not (have endpoints that) show any statistics on how many times all rules were collectively triggered, nor is it granular enough to show how many times each individual rule was triggered.
Fundamental and valid question then: what about the data that shows all the rules that are successfully triggered — and when? Say you have 10 rules in Launch: how to determine how many times each rule was successfully triggered? How to determine when each rule was successfully triggered? And how to capture all this data?
Quick answer: we could not quite say. Until now! We set out to fill this gap thusly: as soon as a rule is triggered by a user’s activity on our website, we capture some data for that rule if the conditions specified in the rule are satisfied. That is, if the rule is triggered with success. Think of this as a real-time monitor that keeps an eye on all the rules/tags that are live on our website.
Under the hood, one key element to actualizing this is Satellite Monitors. They are essentially hooks that are tied to the Launch runtime library, a not-so-plain way of saying they listen in for triggered rules based on users’ activities.
It is, however, never guaranteed that the conditions for every triggered rule will be met. This results in two groups of triggered rules: those whose conditions are satisfied, and those whose conditions are not satisfied. Let us refer to the former as Completed Rules (the group we want our monitoring system to capture), and the latter as Not-completed Rules (i.e. Failed Rules). For Completed Rules, we capture data like the (unique) ID of the rule, the name of the rule, the time that the rule was triggered (and completed), the page URL where the rule was triggered, and so on.
Now, leveraging Satellite Monitors, we are able to build a real-time, fault-tolerant system that captures all that data in a way that scales — we are speaking of +80 Completed Rules per second on average. Or +4K per minute. Or +290K per hour. Or +7M per day. Or +200M per month. Or +2B per year. This means based on users’ activities on our website, some (if not all) of our rules are collectively triggered that many times. And we capture them all. How? You ask. Let me tell you:
Completed Rules matter to us. When a user visits any page on our website (and interacts with it), several rules are triggered — some fail, others do not. Using Satellite Monitors, we capture the data for Completed Rules. We then send this data to a HTTPs endpoint defined in a Cloud Run app. This endpoint pushes the data to a Pub/Sub topic, where it queues up for eventual processing by our streaming pipeline.
The pipeline is built on Apache Beam, and runs on Cloud Dataflow. High level, it is an Extract, Transform, and Load (ETL) pipeline that obtains data from a Source, transforms it, and then outputs it to someplace called a Sink. Visualize this as a running tap over your sink, with the water coming from a source. Our Source is a Pub/Sub subscription, and our Sink is BigQuery, and the subscription constantly pulls data from the topic queue.
Having this streaming pipeline provides us a constant flood of data. As long as users keep visiting our website, the data never stops pouring. Much like you could decide to leave your tap running forever. With the data now in BigQuery, we do some additional ETL to clean things up. This creates a view that then feeds our reports, providing us a fluid, live picture of how our rules are trending. Nothing falls through the cracks. And life is good!
