MGS to Databricks connector

Analytics teams in banks and insurance companies use platforms like Databricks for data science, ML/AI model development, model testing, and more.

These same companies also have teams such as Senior Risk Management, Internal Audit, or Model Validation to deal with internal models, but all these teams have different requirements. They focus more on high-level portfolio overview, dependencies between models, team organisation, monitoring regulatory requirements, and advanced model risk dashboards or reports. 

These two described worlds have much in common. Both groups work with the firm’s model portfolio, and they need to interact on several levels. Model development kick-off is often characterized by several decisions and approvals. The model validation process needs to be organised through several teams. Furthermore, searchable documentation should be stored in a reliable and secure place.

While Databricks is ideal for analytics teams, the Model Governance Suite (MGS) offers enterprise-level model inventory perfect for high-level teams. It provides a flexible and open solution that goes far beyond the requirements defined by the SR 11-7 regulatory standard. MGS serves as a center and a single source of truth for your firm’s model portfolio while providing organisation of your teams around model governance, documentation management, automation of repetitive processes, validation process support, reporting, and much more.

The MGS team is now introducing a connector between MGS and Databricks to maximise the value for teams working in both worlds. Teams governing the firm’s portfolio in MGS - definitely not only ML/AI models - get access to the details of model development and monitoring right at their fingertips. Teams using the Databricks platform can write code interacting with the inventory. Examples include registering new models or their revisions, updating model status, adjusting risk scoring measures, registering identified model issues, transitioning internal MGS workflows, uploading generated documentation, and whatever else you’ll need. Thanks to our connector, MGS can easily orchestrate jobs executed by Databricks such as performing regular monitoring tasks at a large scale. The combination of both tools guarantees audibility and repeatability.

The ModelOps (model operations) concept maintained by the Databricks platform is naturally extended towards teams which don’t need access to the analytics platform but rather interact with the inventory. For example, a model that has been through the formal approval process can be directly deployed to production use and served by the Databricks cluster. Monthly backtest reports will be enclosed with the model itself in an auditable place available for further reporting. 

The combination of these tools provides the top-down experience from high-level firm’s model portfolio overview to the model’s source code and production runtime environment.

Contact us to learn more.