6 days ago
Navigating The Generative AI Technology Stack
Bill Wong - AI Research Fellow, Info-Tech Research Group.
Generative AI is transforming the technology landscape, introducing new large language models (LLMs), development tools and a range of new or enhanced applications. As adoption grows, organizations must focus on building a robust technology stack to support these applications—ensuring they meet performance and scalability demands.
When evaluating vendor solutions for generative AI, it's important to understand the core components of the supporting technology stack. This stack includes several layers, with applications sitting at the top. Key layers and examples include:
• Application Layer: Business applications such as CRM, ERP and marketing/sales tools, along with industry-specific solutions for sectors like healthcare, legal and financial services.
• Data And AI Tools: Platforms and tools used for managing data and developing machine learning models, including LLMs.
• LLMs: The foundational AI models that generate outputs, insights and recommendations.
• Data Layer: Systems for storing and managing both structured and unstructured data, such as databases and data lakes.
• Infrastructure Layer: The hardware and core software needed to support AI workloads—this includes compute, storage and networking resources.
There are two primary approaches for delivering generative AI applications:
1. Component-Based (Loosely Coupled) Approach: In this model, the vendor provides the application itself but allows the client to choose the underlying components—such as the LLM, data platform and infrastructure. This approach offers maximum flexibility, enabling organizations to tailor the solution to their specific needs. However, it also requires more time, effort and resources to integrate the components and optimize performance. This is currently the most common approach in the market.
2. Integrated (Tightly Coupled) Approach: This model provides a complete, pre-integrated solution that includes the application, LLM, data platform and infrastructure. Often delivered as a software-as-a-service (SaaS) offering, this approach prioritizes ease of deployment and speed to value. While it offers less customization than the component-based approach, it reduces complexity for the client. While not an exhaustive list, examples of this model include Microsoft 365 Copilot, Salesforce Einstein and Amazon Q.
Criteria For Evaluating Generative AI Applications
When evaluating generative AI applications, it's essential to assess them against a core set of criteria to ensure they align with business goals and technical requirements. Key categories include:
Selection should be driven by specific business needs and prioritized use cases. For example, when evaluating AI-powered CRM solutions, relevant capabilities might include customer segmentation, sentiment analysis, personalized content generation, predictive lead scoring and user experience factors (such as flexibility or ease of integration).
Applications should meet required performance and scalability standards. Evaluation criteria may include response time or latency, ability to support concurrent users (concurrency), scalability across workloads and resource utilization.
Cost considerations should factor in both initial and ongoing expenses, including licensing fees, support and maintenance costs, and infrastructure or usage-based charges (if applicable).
Alignment To AI Guiding Principles
Generative AI applications should align with your organization's broader AI and business strategies. These guiding principles help ensure that AI systems are deployed responsibly and effectively, while also providing a framework that IT can use to implement risk-mitigation measures. These should be tailored to fit the specific values and goals of your organization. Core principles typically include:
• Safety And Security: AI systems must be resilient, secure and safe throughout their entire life cycle—from development through deployment and operation.
• Data Privacy: Personal and sensitive company data must be protected to ensure anonymity, confidentiality and compliance with data protection regulations.
• Explainability And Transparency: AI systems should be as transparent as possible in their operations and offer explanations that end users can understand and trust.
• Fairness And Bias Detection: AI systems should be designed to identify and mitigate bias in data and algorithms, promoting fairness and improving decision accuracy.
• Validity And Reliability: AI-generated outputs must be consistently accurate, reliable and valid.
• Accountability: Clear responsibility must be established for AI system outcomes. Organizations should define who is accountable for the design, performance and oversight of each system.
While some organizations may refer to these as "responsible AI principles," the specific terminology is less important than ensuring these principles are customized and aligned with organizational strategies and values.
In Summary
The evaluation framework should be tailored to reflect your organization's unique context—including its AI guiding principles, the specific use cases the solution is expected to address, performance and scalability SLAs, and the available budget for acquiring and operating the application.
The degree of flexibility to optimize application performance will vary depending on the architecture of the solution—some applications offer limited configurability, while others provide significant control over components such as infrastructure and model selection.
When assessing vendor solutions, it's important to prioritize evaluation criteria based on the organization's strategic goals. In my experience working with C-level executives, two factors consistently stand out as most critical in the selection process: business capabilities and the alignment with AI guiding principles, ensuring the solution reflects the organization's ethical, governance and risk frameworks for AI.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
