MAP Insights

(2nd of 2 Parts)

 

Layer 2: Governing Predictive Machine Learning

Objective: ensure that probabilistic outputs remain accurate, stable, and ethically sound over time.

 

Predictive systems are used to score credit applicants, price insurance contracts, detect financial fraud, forecast demand, and screen job candidates. Unlike BI, they generate probabilities rather than facts and degrade over time as customer behavior and real-world conditions change.

 

Core controls

• Drift monitoring. Two types of drift require attention. Data drift occurs when the statistical properties of input data change, such as when a new customer segment is added. Concept drift occurs when the relationship between inputs and outcomes shifts, for example, due to changing economic conditions. Both can gradually reduce model effectiveness.
• Explainability (XAI). Techniques, such as SHAP and LIME, help identify which factors drove a particular outcome. For high-impact decisions, such as those in credit, underwriting, and hiring, the organization must be able to justify the outcome to a customer, a regulator, or a court.
• Fairness and bias audits. Test models across population segments before deployment to detect disparate impact, defined as a disproportionately adverse outcome for a protected group, even if unintentional. Ongoing testing is required after deployment.

 

Typical failures include silent performance decay, in which model accuracy declines without detection; feedback loops that reinforce historical human bias; overfitting; and excessive false positives, which raise operational and customer costs.

 

Board and Risk/Audit focus: The central issues are algorithmic bias liability, consumer protection compliance, and model risk management. In the Philippines, the BSP’s credit risk management guidelines under Circular No. 855 address credit scoring models and risk-rating systems. Beyond the circular’s specific requirements, BSP supervisory expectations in recent years have pointed banks toward full life cycle model risk management: an inventory of models, independent validation, ongoing monitoring, and clear accountability. To support effective oversight, Boards should require regular management reporting on the complete model inventory, the status of independent validation, and the results of ongoing model monitoring and bias audits. This keeps Directors informed about how risks are managed in practice and enables timely, risk-based oversight. (The BSP has also issued guidance on the ethical and responsible use of AI. Boards should confirm whether it is binding or an expectation.)

 

Layer 3: Governing Generative AI

Objective: reduce hallucinations, safeguard proprietary intellectual property and personal data, and prevent confidential information from leaving the enterprise.

 

Generative AI is easy to adopt because any employee with a browser (or by downloading the free AI mobile apps) can enter sensitive information into external tools. The primary governance challenge is preventing data from leaving the organization, not model effectiveness.

 

Core controls

• Model gateway. A model gateway sits between users or applications and foundation models. It screens prompts and files to ensure that confidential information and customer personally identifiable information (PII) do not reach external models or become part of vendor training data. It also provides centralized logging and access control.
• Guardrails. Real-time guardrails screen inputs and outputs for toxicity, jailbreak attempts, prompt injection attacks, and likely hallucinations. While these controls reduce risk, they do not eliminate it. Generative models may still produce convincing but fabricated content, so critical use cases require human review and validation against verified enterprise sources.

 

Typical failures include hallucinated or fabricated data, such as fabricated citations and figures; data exfiltration and loss through unauthorized tools; and copyright infringement, either from generated content or from employee-submitted material.

 

Board and Risk/Audit focus: Boards should align governance with strategy and confirm that AI investments support business objectives. Oversight of AI vendors is critical. Boards must require robust due diligence on any external AI vendor, focusing on three key contract terms: (1) clear data-use restrictions to prevent vendors from retaining or reusing enterprise data, (2) enforceable commitments to zero data retention, and (3) strong indemnification and liability provisions for intellectual property infringement. Boards should also expect management to regularly review vendors’ security practices and audit rights as part of ongoing oversight.

 

Enterprise risk policies should address “Shadow AI,” defined as employee use of unapproved AI tools, and management should demonstrate its ability to detect and manage such use. The Board should also expect a plan to comply with emerging AI regulations in the Philippines and in other jurisdictions where the organization operates. The 2024 EU AI Act is binding only where the organization operates in or serves the EU; elsewhere, it serves as a reference point.

 

Layer 4: Governing AI Agents

Objective: enforce boundaries on autonomous decisions and actions.

 

Agents differ from earlier layers by taking action, as systems that combine models, tools, permissions, and instructions, not just models. They can query systems, execute code, modify records, and send communications, often chaining multiple steps without human intervention. The risk shifts from incorrect answers to incorrect actions executed rapidly across systems.

 

Core controls

• Agent registry – Records which agents are live, who owns each agent, the authority each agent holds, and which data and systems each agent may access. An agent not in the registry should not be running.
• Human oversight – Scaled to risk, two common models are Human-in-the-Loop (HITL), where a person approves actions before execution, and Human-on-the-Loop (HOTL), where a person supervises and can intervene or stop the agent. Governance should set verification thresholds so that high-risk actions, such as executing transactional code or altering databases, require approval, while lower-risk activities are monitored.
• Observability – Maintain complete logs of what each agent did, why, and with what data, so the enterprise can reconstruct and audit actions.
• Failover and shutdown process – If an agent must be stopped, the supported business process must continue via manual fallback, rollback procedures, or a degraded mode. A “kill switch” is effective only if it has been tested and the process remains operational.

 

Typical failures include unconstrained agent behavior and processes that fail when agents are unavailable. The latter is often overlooked; organizations that automate processes and lose manual capabilities trade resilience for efficiency.

 

Board and Risk/Audit focus: Boards should ask about agent observability, guardrails, “kill switches”, and process resilience. Internal audit should verify that these controls exist and have been tested.

 

Cross-Cutting Priorities for the Board

 

Several themes cut across all four layers:

 

Maintain an inventory. Organizations cannot govern what they cannot see. Register reports, models, generative AI use cases, and agents, and assign owners.

 

Tier controls by materiality. Not every dashboard requires the rigor of a credit model, and not every generative AI use case requires the oversight of an autonomous agent. Classify use cases by potential impact and apply controls proportionally. This aligns with risk-based approaches, such as ISO 31000.

 

Assign clear accountability. Governance fails when responsibility is diffused. Specify who owns data quality, validates models, approves AI use cases, and has the authority to halt an agent’s operation.

 

Plan for the entire life cycle. Failures in Layers 2 and 4 often surface after deployment. One-time approval is insufficient; ongoing monitoring, periodic revalidation, and a defined retirement process are essential.

 

Treat vendors as part of the control environment. Many of the AI capabilities an enterprise uses are delivered by third parties. Contract terms, vendor assurance reports, and exit options serve as governance controls.

 

Know what binds you. Distinguish three tiers: binding law and regulation (for example, the Data Privacy Act of 2012, SEC reporting rules, and BSP circulars for banks); supervisory expectations and guidance; and voluntary frameworks (for example, ISO 31000 and the NIST AI Risk Management Framework, or AI RMF 1.0). Management should identify which layer each obligation comes from and who owns compliance.

 

Protect personal data by design. AI governance also requires lawful processing of personal information. Before personal data enters a model, tool, or vendor, management should confirm the lawful basis, purpose limitation, data minimization, and retention in accordance with the 2012 Data Privacy Act and National Privacy Commission guidance. This applies to training data, prompts, and outputs, not just to actual leaks.

 

Divide oversight without diluting accountability. Board Committees enable deeper review but do not replace the full Board’s accountability. Subject to each company’s charters and Board Committee workloads, the Risk Committee can lead on the AI risk profile, appetite, and scenarios; the Audit Committee on reporting reliability, controls, and independent assurance; and the Corporate Governance Committee on policies, decision rights, ethics, and Director capability. Mandates will likely overlap, so coordination is needed. Some Boards may designate one Director to coordinate AI oversight.

 

Questions for Directors and Executives to Ask

• Can we trace the figures in our financial and regulatory reports to their source, and who attests to their accuracy?
• Do we have a complete inventory of the models in use, and which ones influence customer or employment decisions?
• How would we know if a model’s performance was silently deteriorating or if it was producing disparate outcomes?
• What happens to confidential data when an employee uses a generative AI tool, and what does our contract with the vendor allow them to do with it?
• How many AI agents are operating today, who owns each one, and what are they authorized to do?
• If we had to turn off an agent tomorrow, would the process it supported continue to run?
• What evidence shows that our controls are working on an ongoing basis, and who has independently tested them?
• What material AI risks remain after controls, and what is management doing about them?
• Are the business benefits we expected from our AI investments materializing?

 

If management cannot promptly answer these questions, it indicates a governance gap or an unclear owner.

 

Conclusion

 

AI governance should not be separated from data governance. As systems become more autonomous, AI governance is an extension of the same discipline. Each layer introduces new risks while retaining previous ones. Effective organizations build data quality, add model reliability and fairness controls, protect data in generative AI, and set clear boundaries for agent autonomy.

 

Boards do not need to understand the mechanics of neural networks or the state management and orchestration for AI agents. But they must ensure that management can demonstrate accountability, define controls, test them, and show how the organization would respond to failures.

 

Ultimately, Boards should:

• Demand clear accountability across all layers of data and AI governance.
• Require evidence that strong controls are in place and functioning as intended across all layers.
• Insist that management regularly test response plans for failures or incidents and report the results to the Board.
• Track both sides of the mandate: that AI delivers the expected business value and that residual risks remain within an organization’s risk appetite.

 

By following this checklist, Directors can proactively safeguard the organization and foster responsible innovation.

 

Thanks to Anton Bonifacio, Pam Cabudoy, Pete Maniego, and Dan Natindim for their valuable comments.

 

(The author is Governor and Secretary of the Management Association of the Philippines or MAP.  He is a retired banker and Globe and Ayala executive, and a member of the Analytics and AI Association of the Philippines and the Institute of Corporate Directors. He serves as an Independent Director at GT Capital Holdings, Puregold Price Club and Megawide Construction. Feedback at <map@map.org.ph> and <iamgilgenio@gmail.com>.)