A data funnel for financial reporting

To achieve this required us to migrate processes and business logic from a legacy data warehouse to a more effective cloud solution.

We embedded our experts within the in-house team and primarily focused on making changes and additions rapidly and flexibly in response to pressing needs. We also focused on knowledge transfer and upskilling internal analysts to better self-serve data.

Pipelines were constructed in Google Cloud Platform (GCP) using Apache Airflow DAGs to schedule an orchestrated data flow. BigQuery was used to transform, process and store data in excess of 40Tb. GCPComposer and Github were used for rigorous testing and analysis. Data quality was monitored with the Great Expectations Python package.

Our work has led to increased usage of data informed decision making within the organisation. Robust automated pipelines pull rich data together from a wide range of sources across the business including the huge customer base.

The Table delivered is used by several client teams downstream for business-critical reporting, insight analytics and prediction models. The Dataset is also used in a wide range of applications: from financial reporting and powering predictive Machine Learning models, all the way to informing board level decisions about pricing.