How Virgin Media O2's internal data interchange was expedited using BigQuery Analytics Hub
Simple data exchange is now an essential tool for any business trying to make educated decisions. However, many firms still struggle to share data in a way that is both efficient and compliant with the law. Data teams frequently face challenges like unclear governance, version control issues, data silos, access restrictions, and a lack of knowledge about data management in the broader company.
The media and telecom firm Virgin Media O2 leverages internal data sharing to improve operations, drive strategy, and empower decision makers. From marketing to finance, the data team provides timely and accurate information to all departments.
Virgin Media O2 needed a solution that would help business divisions with data access and governance. Without it, it wouldn't have a centralized way to share data and wouldn't be able to get organization-level visibility and efficiency.
Resolving issues with internal data sharing
Teams were already working on Google Cloud-based projects, thus strict version control was required to ensure that the data was always accurate, consistent, and current. Nevertheless, this often increased the amount of time needed to produce fresh discoveries. Virgin Media O2 already had its corporate data in BigQuery to fulfill their enterprise and AI demands. Therefore, using Analytics Hub, BigQuery's data sharing function, was one potential approach to expand on their current architecture.Analytics Hub for BigQuery
The data platform team decided to test BigQuery Analytics Hub after learning about its scalability, self-service features, and simple governance mechanism for data tagging and quality. Specifically, this last element aligned with improving the application of privacy by design.Virgin Media O2 had created a clear onboarding and training process after a successful pilot. To make it easier to follow any actions within BigQuery, two owners were assigned to the subscriber side and two owners to each new data exchange. Over the next nine months, this approach was extended to 25 teams, more than 50 exchanges, 100 postings, 500 tables, and around 300 daily users.
One of the key benefits the team found is that BigQuery Analytics Hub eliminates data duplication, which reduces network and storage costs. By creating a shareable real-time pointer to the underlying dataset known as a Linked Dataset it does this. As a result, it is easy to audit, trace, and restore the original data source, and any subscriber can get updated data quickly. This approach also incorporates a safety net for disaster recovery.
BigQuery Analytics Hub also fixes the complexity problems that come with developing views, such as the fact that allowed views sometimes result in the loss of original table metadata when data is accessed. Subscribers with direct access to the original dataset can still see all table descriptions and columns.
By linking data directly from the data publisher to a data subscriber, Virgin Media O2 was able to reduce latency, save time, and enhance administration and usability for both publishers and subscribers. Furthermore, a centralized place for managing data access and quality was provided via the platform's improved governance.
BigQuery Analytics Hub streamlines data sharing between teams and business divisions, reducing human labor and errors. The platform has been particularly beneficial to software developers, analytics engineers, data scientists, and analysts at Virgin Media O2. It ensures that everyone gets immediate access to the information they need for their various jobs.
Saving time by utilizing info that is more accessible than ever
After BigQuery Analytics Hub was deployed to roughly 25 squads, the solution helped squads utilizing the previous method save up to 30 hours a week on time spent on training, support, pipeline deployment issues, and communication overhead. All teams now spend as little as thirty minutes a week because of the almost nonexistent issues. The team estimates that this saves roughly 95% of the effort. Since data is no longer kept in silos, it is now broadly available to the several departments who need it.By developing a dashboard, the team was able to democratize data access for subscribers and their larger teams without needing them to use BigQuery Analytics Hub directly. Allowing consumers to subscribe to datasets enables self-service while maintaining a strong governance approach. Eliminating the middleman streamlines and speeds up this time-consuming process while maintaining a robust governance structure.
The main benefits of safe data transmission
Data Integrity and Security
Secure, zero-copy sharing ensures uniform data integrity across departments by protecting against metadata loss and unauthorized access.
Cost-effectiveness and Simplified Management
By eliminating data transfer, the platform reduces long-term expenses and operational overhead, and a small personnel can effectively manage data supervision.Centralized Management and Monitoring
A single dashboard allows for real-time control over data sharing processes, enforces strict authorization and access rules, and facilitates timely problem identification.Simplifying four crucial areas data ownership, data catalog, data quality measures, and more effective sensitive data tagging is one of the group's top priorities for the future. To accomplish the goal of automating the entire data activity, these four areas must be properly defined and rigorously followed as a policy via BigQuery Analytics Hub after the data has been verified.
- The previously mentioned process, known as "data certification," has two primary benefits:
- It is feasible to detect uncertified data assets and track out issues with the quality of the data in a matter of minutes by using data quality measures (at the column level) and data lineage.
- Real-time audit logs that track the consumption of sensitive data and identify it in real-time enable proactive control of data privacy issues.
BigQuery is the ideal starting point for people who are unfamiliar with Google Cloud. Analytics Hub is the next greatest thing for users after BigQuery.
0 Comments