Domain monitoring explained

Looking at the vastness and richness of web pages today, I still find it surprising that standards from the origins of internet have stood the test of time? When you look at the source of any given web page, you still find HTML, styled with CSS and code to change HTML and CSS dynamically. These standards are still open even if the GAFA have managed to put fences around large parts of the internet. So you, as a domain owner or an employee of a company with an online presence, are still in control. Right?

Open standards underpin the API economy. It is far easier to compile small building blocks of specific functionality – such as A/B testing, user engagement, behavioural profiling - than to build it out yourself. Here’s the rub. The scalability, flexibility and user experience of modular web components come at the cost of proliferating data to hundreds of third party servers. You are in control of what you embed. Are you also in control of what happens after?

This is where domain monitoring comes in. It refers to observing data flows on your digital domains and validating compliance with agreements and regulations. You want to know which components make requests to servers (‘requests’), which data are collected and whether these have a valid basis.

To be clear, these requests are not hiding. You can open the inspector panel of your browser any time and observe what happens. Making sense of requests is another story. You need context to map a domain name to a company or decipher a flurry of query string parameters. And consent management platforms haven’t made this easier. Every combination of a purpose and vendor presents a unique data collection condition.

To be clear, these requests are not hiding. You can open the inspector panel of your browser any time and observe what happens. Making sense of requests is another story. You need context to map a domain name to a company or decipher a flurry of query string parameters. And consent management platforms haven’t made this easier. Every combination of a purpose and vendor presents a unique data collection condition.

We believe domain monitoring should involve three components:

• Automated monitoring of requests for a range of consent scenarios and browsers
• Reading meaningful signals from request domains and metadata
• Validating conformity rules

You can observe domains from a browser (client-side) or a server. The first one involves automating website visits and capturing every data point that the browser comes across. Server-side data collection intercepts network traffic to do the same. At Webclew, we prefer the first approach. Client-side monitoring does not require upfront configuration, allows simulating specific user actions and captures every aspect of the browser. Server-side monitoring literally intercepts all requests of your domain users. This comes in handy when looking for a comprehensive overview of third-party vendors. Yet this setup is expensive, requires consent from users and does not allow capture all browser storage.

Regardless of the approach, the main take-away is that you either automate domain monitoring or you just don’t monitor. The combination of many simple components yields incredibly rich behaviour, that no single person can track with their own eyes.

If you use specific tools or approaches for domain monitoring, let us know. We would love to get in touch to learn about your experiences and needs.