What We’re Seeing
In our engagements with leaders across financial services, healthcare, and the public sector, we see a consistent pattern: the initial euphoria around public cloud generative AI is meeting the hard reality of enterprise security and data sovereignty. While foundation models from major cloud providers offer incredible capabilities, the prospect of sending sensitive customer data or proprietary intellectual property outside the corporate firewall for inference remains a non-starter for many. This friction has stalled numerous high-value AI projects, leaving leaders caught between innovation pressure and their core duty to manage risk.
This is why a recent research paper from IBM, detailing a complete Retrieval-Augmented Generation (RAG) system running on a mainframe, is such a significant signal. The paper, titled Spyre-Accelerated Retrieval-Augmented Generation on IBM LinuxONE: A Cloud-Native Architecture for Secure, High-Throughput Enterprise AI Inference, describes an architecture where the entire AI pipeline—from data retrieval to model inference—is contained within a single, trusted hardware platform. It validates an emerging trend we’ve been tracking closely: the strategic necessity of secure on-premises AI for the most critical enterprise use cases.
The Number That Changes Everything
0
The number of times sensitive enterprise data leaves the secure hardware perimeter in IBM’s new on-premises RAG architecture.
Who’s Ahead and Why
The push for secure, local AI is creating a new competitive dynamic. On one side, you have incumbents like IBM, who are leveraging their deep legacy in secure, high-transaction enterprise computing to build a defensible niche. By positioning the mainframe not as a legacy system but as a purpose-built ‘secure AI appliance,’ they are playing to their strengths in trust, reliability, and compliance. For their existing customer base in banking and insurance, this is a compelling proposition that minimizes architectural disruption and simplifies regulatory audits.
On the other side are the public cloud hyperscalers—AWS, Google, and Microsoft. While they dominate the broader AI market, they face headwinds with these specific high-security use cases. Their solutions, such as AWS PrivateLink or Google’s Confidential Computing, are attempts to address these concerns, but they often introduce complexity. Auditing a distributed system that stitches together multiple cloud services and private network connections is inherently more challenging than validating a single, integrated hardware box. As noted in research from McKinsey on hybrid cloud, managing security and compliance across hybrid environments remains a top challenge for enterprises. IBM’s approach effectively short-circuits that complexity for a specific, high-value problem.
The Gap Most Teams Miss
Many enterprise AI teams are discovering a painful gap between a model’s functional capability and its compliant deployability. A proof-of-concept using a public API with anonymized data can look incredibly promising, showing massive potential for efficiency gains or new customer experiences. However, the project often grinds to a halt when the security and legal teams are engaged to approve its use with real, sensitive production data. The conversation shifts from model accuracy to data residency, encryption standards, and the chain of custody for data in transit and at rest.
The gap most teams miss is in underestimating the total cost of compliance for cloud-based AI. The cost of inference per token is just one small part of the equation. The real, often hidden, costs are the person-hours from security, legal, and compliance teams required to vet and continuously monitor a complex, multi-service cloud architecture. An integrated, on-premises solution dramatically reduces this surface area of risk and, consequently, the overhead of governance. It transforms the security conversation from a distributed systems problem to a single-platform validation exercise, which is far more manageable for regulated organizations.
How to Close the Gap
Closing the gap between AI ambition and security reality requires a pragmatic, risk-based approach to infrastructure. First, we recommend that technology leaders explicitly re-evaluate the ‘cloud-only’ assumption that has dominated IT strategy for the past decade. The unique demands of generative AI, particularly concerning data gravity and security, necessitate a more nuanced, hybrid strategy.
Second, enterprises must segment AI use cases by data sensitivity and regulatory impact. A marketing chatbot using public product information has a vastly different risk profile from a wealth management advisor bot accessing client financial data. This segmentation allows for a right-sized infrastructure strategy: low-risk workloads can leverage the scale and flexibility of public cloud models, while high-risk workloads are directed to a secure on-premises AI environment. This tiered approach is critical for any effective AI Governance & Risk framework.
Finally, security and compliance teams must be partners in the AI development lifecycle from day one, not a final checkpoint. By involving them in the initial architectural design, teams can proactively select platforms and patterns that are pre-vetted for high-risk data, avoiding months of rework and frustration when a promising pilot fails its final security review. This is especially true for firms navigating the complexities of AI in finance, as detailed in our Generative AI in Financial Services whitepaper.
| Maturity Level | Current State | Next Action | Timeline |
|---|---|---|---|
| Exploring | Running PoCs on public cloud APIs with anonymized or synthetic data. | Classify potential production use cases by data sensitivity and regulatory risk. | 1-2 months |
| Piloting | Building a pilot with sensitive data, often encountering security and compliance roadblocks. | Evaluate on-prem or virtual private cloud AI architectures for the specific high-risk use case. | 3-6 months |
| Scaling | A cloud-based AI service is in production, but only for low-risk, non-sensitive data. | Develop and ratify a formal hybrid AI infrastructure strategy that includes a secure on-prem component. | 6-9 months |
| Optimising | Running a mix of on-prem and cloud AI workloads based on a clear risk framework. | Standardize governance, MLOps, and model monitoring tools across the entire hybrid environment. | 9-12 months |
Watch These Signals
- Specialized AI Accelerators: Keep a close watch on the development of AI hardware beyond GPUs. Chips and cards purpose-built for specific tasks like secure inference (e.g., IBM’s Spyre) or low-power edge computing will signal a maturation of the market beyond raw training performance.
- Cloud Provider ‘On-Prem’ Offerings: Monitor the evolution of AWS Outposts, Google Anthos, and Azure Arc. The degree to which they can deliver their full-stack, managed AI services in a truly air-gapped, customer-controlled environment will indicate how seriously they are taking the enterprise demand for data sovereignty.
- Regulatory Scrutiny: Pay attention to new data residency laws and sector-specific AI regulations. Any new legislation that increases the compliance burden for cross-border data flows will directly accelerate the demand for on-premises AI solutions.
Our Take
We believe the future of enterprise AI is fundamentally hybrid. The notion that all workloads will migrate to a handful of public clouds is being replaced by a more sophisticated understanding of risk, cost, and competitive differentiation. For the most valuable and sensitive enterprise use cases—the ones that touch core intellectual property and customer trust—secure on-premises AI will be a non-negotiable architectural pattern.
IBM’s LinuxONE RAG architecture is a powerful data point, not because it signals a return to a bygone era of computing, but because it represents a forward-looking strategy that meets a durable enterprise need. It proves that secure, compliant, and high-performance AI can be delivered within the enterprise perimeter. For leaders in regulated industries, this approach offers a path to unlock the value of generative AI without compromising on their foundational commitments to security and trust. At Thinkia, we help enterprise leaders navigate these critical infrastructure decisions, building AI strategies that are both ambitious and achievable.