How we do BI at Cytora - Part I
About the authors:
Xiao Liu is a Senior Software Engineer and backend tech lead at Cytora Underwriting Productivity Team. He led the Business Intelligence Team in Cytora before joining Underwriting Productivity.
Nicole Gillett is a Software Engineer at Cytora. She has been part of the Underwriting Productivity Team, as well as Business Intelligence.
1. Introduction
At Cytora, our Business Intelligence service has processed millions of messages so far. By the time you have finished reading this blog post, a few hundred messages would have flowed through our system, helping underwriters and Cytorians to make data-driven decisions.
Cytora operates in the Insurtech space, and our mission is to use technology to transform the insurance industry. Specifically, we build software for underwriters to turbocharge their workflow. We have both been at Cytora for about three years. During that time, we have designed and implemented the architecture of our BI platform, as well as been users of the system.
In Part I of this BI blog post series, we will give a technical explanation of the BI architecture that we use at Cytora and some examples of how BI has made a difference to the lives of Cytorians, along with our customers. We will also touch upon topics like billing (shout out to our readers from Finance!) and a case study of how BI has been pivotal to one of our products, Underwriter Productivity.
Whether you are a Software Engineer looking for architectural inspiration, or a user of BI tools who wants to leverage the most out of data, this series will give you an insight into how we do BI at Cytora and hopefully inspire you to try something new.
2. Architecture
This diagram shows the whole lifecycle of BI messages from various Cytora services into our BI data lake. They are then used for business analytics. For individual components, we use Serverless resources where possible. There are various benefits to using Serverless resources, which we will explain in more detail in Part II.
2.1 Architecture walkthrough
Let’s begin the walkthrough from the left-hand side. Cytora services publish messages with BI Publisher, which is a client written in Python and Golang. Each service has its own topic to which it writes BI events. Here we use SNS topics as the underlying resource for these topics.
There is a single SQS queue that is subscribed to each of the topics of different services. The BI ingestion service then processes the queue. The BI ingestion service is responsible for validating the incoming messages, processing them based on the metadata, and then adding them to a corresponding path in the data lake. We are using a Serverless architecture for the BI ingestion service where processing is triggered by the messages from the queue.
We use GCS buckets and BigQuery for our data lake. After the messages have arrived in the data lake, they are ready to be further transformed or analysed by our BI analytics. We use dbt Cloud to take care of further transformations and Looker for data modelling and visualisation.
At the bottom of the diagram is our BI message definition store. It’s a Protobuf definition, version-controlled and name-spaced by service. It is used throughout the BI lifecycle. Golang or Python code is generated from the Protobuf definition and used to compose the BI message in individual services. In the BI ingestion service, the definition is used to validate the incoming messages to make sure they match the defined schema. BigQuery tables are created using the schema generated from the definition in the BI data lake. In BI analytics, the schema is used to build data models. As mentioned before, it is a single source of truth, ensuring that all BI messages arrive in BI in good shape.
3. Business impact
One of the key purposes of BI is to drive business impact, and we have seen this in multiple areas at Cytora. Specifically, we will touch upon how BI has provided business insight into one of our products, Underwriter Productivity (UWP), and more administrative processes, such as billing.
3.1 Underwriter Productivity
Underwriter Productivity is a product that aims to improve the underwriter workflow by digitising risks and processing them most efficiently. As with most of our other services at Cytora, we integrated the app with BI to ensure that we were recording data across all areas of the app: how it was being used and insights about the data itself.
It has been fascinating to watch how the usage of our app has grown by looking at the number of underwriters that are using our platform over time. We can even observe how risk-averse one underwriter is compared to another, even when they work in the same team. We are also monitoring how many submissions are created in our app and whether they were created manually or via our automatic email integration system. Sharing these business insights with the company has been an extremely valuable tool in providing context across all business areas.
We can also provide insights to the underwriters by noticing trends in their underwriting patterns, which have not been mentioned before. In the future, we plan to deliver the most business-critical insights back to the underwriters to help them make data-driven decisions.
BI has also improved the developer experience, especially when it comes to debugging because we can easily check the data that BI is collecting in BigQuery. We recently used dbt Cloud to build a semantic layer, which is a view in BigQuery that transformed the data into an even more accessible format so that people from other teams could work with it. It’s where we make data make sense.
3.2 Billing
Another critical area in which BI drives business impact is empowering the Finance team to use BI self-service. The Finance department at Cytora forecast costs based on usage tracked in BI dashboards. This helps to track and explore anomalies and reduce any element of surprise at month-end. A specific feature that Tom Coward, our CFO, really likes is the simplicity of adding new customers. In Tom’s words:
“For new customers, as long as they’re set up correctly in BigQuery, their usage shows up automatically in the Looker dashboards, so it’s really simple from my perspective.”
Tom has also been impressed with the analytics side and the ease with which different Finance team members can work with the data. Specifically,
“the ability to democratise the data across the team to explore how they want to build their reports without having to use a central BI team to run the queries for them.”
4. Conclusions
We hope that you have enjoyed this adventure through the architecture of BI at Cytora! We also touched upon some of the users of BI, both external and internal Cytorians. We hope that we have inspired you to try out something new with your BI architecture!
Stay tuned for Part II where we will do a deep dive into the Serverless technology that underpins our infrastructure and discuss the challenges and technical decisions that we have made on our BI journey.
