Architecting AI for the Enterprise
Written by Malith Jayasinghe, VP of Research and AI, WSO2
The rapid evolution of AI, particularly in generative AI, has significantly reshaped the way we work and the broader technological landscape. Central to this transformation is the advancements in Large Language Models (LLMs), which have greatly expanded the capabilities and applications of AI. AI can now efficiently accomplish tasks that once required extensive effort and years of model training.
Integrating these AI models with organisational data can lead to the creation of innovative digital experiences and enhance overall efficiency. AI offers transformative possibilities, which is one of the reasons organisations are increasingly adopting AI technologies. Strategic integration of AI can enhance efficiency, improve user experience, provide a competitive advantage, and prepare the business for future growth.
According to Forbes, the global AI market was valued at $136 billion in 2022, and recent research by Grand View predicts that AI will grow to $1,811 billion by 2030. Gartner predicts that by 2026, over 80% of companies will be using Gen AI API/model or will have deployed a GenAI app in production.
While many companies are experimenting, the number of AI applications entering production is still relatively low, at less than 5% in 2023. That’s because integrating AI into an organisation and architecting it involves overcoming a new set of challenges and acquiring knowledge in emerging technologies.
Building and launching an AI-powered application or feature has numerous challenges, such as accuracy issues resulting from model hallucinations, data security-related challenges, and speed problems caused by slow models. So, how should you go about tackling your AI integration initiative?
Starting the AI Journey: Defining Problems and Prioritising Projects
Before your organisation embarks on its AI journey, it must start with a clear definition of the problem and a keen understanding of your application’s value to guide the development journey. This understanding is essential to unlocking AI’s potential. Organisations often struggle to choose from multiple potential AI projects, making it challenging to decide which projects to prioritise. It is crucial to evaluate the value that each project will bring compared to the effort required for implementation. Priority should go to projects that offer high value with relatively low effort. Additionally, organisations should prioritise projects that demand significant effort but offer high value, as these are important strategic investments for long-term benefits.
Assessing the Value of AI Projects
Assessing the value of AI projects can be challenging, but organisations can achieve this by considering various factors. These factors include the number of users who will benefit from the project, the percentage of time users spend on the task, and the amount of time they can save by utilising the AI solution. We should also consider the direct financial benefits resulting from using the AI solution.
Technical execution
When developing your AI application, you will encounter several challenges. These may include architectural issues in designing the solution, ensuring model accuracy to prevent errors, and addressing concerns related to data privacy, security, and speed. Organisations need to carefully consider the trade-offs between accuracy, speed, cost, and associated risks. Understanding these trade-offs is crucial, as is ensuring that the chosen AI solution aligns with the organisation’s goals and constraints.
Dataset utilisation
The organisation will need to identify available data sources which they may need to fine-tune and train models. When training and fine-tuning models, organisations need to perform data cleaning and preprocessing to ensure there is no data leakage and that your LLM doesn’t accidentally reveal sensitive Personally Identifiable Information (PII) or proprietary algorithms. You will also need to consider whether to collect any new data.
AI’s interaction with APIs
AI and APIs can collaborate to generate powerful experiences, replacing graphical user interfaces (GUIs) with natural language interfaces. This approach enables the development of universal interfaces that interact with multiple APIs. To create this type of experience, it is essential to understand the APIs that the AI must interact with, including their specifications. APIs come in various types and representations, such as RESTful APIs, OpenAPIs, GraphQL APIs, and more. Well-documented APIs tend to perform better when integrated with AI, but AI can also help bridge the gap when documentation is lacking.
Prompt engineering
Prompt engineering guides generative AI to produce the desired outputs. A prompt is a natural language instruction describing how generative AI should perform. It acts as the interface between human intent and machine output and includes aspects such as Prompting, Zero-Shot Prompting, Few-Shot Prompting, Chain-of-Thought (COT), Tree of Thoughts (TOT) and ReAct (Reason and Act).
LLM generation provides information, guidance, and/or suggestions. Their output doesn’t inherently drive actions, therefore, ReAct Agents can perform actions, for example, calling APIs. ReAct Agents, with the assistance of the model, can execute natural language commands using a collection of tools registered with them. Tools can retrieve data or perform tasks, and they come in different forms, such as functions, APIs and so forth.
Accuracy evaluation and continuous feedback
It is important to create a dataset for evaluation purposes and select a metric by either using an existing metric or by developing new metrics. Here you can automate the accuracy evaluation and re-evaluate after any changes to confirm that accuracy has not been diminished. It is worth noting that even minor changes to prompts can significantly affect accuracy.
By continuously collecting user feedback you will improve the accuracy of AI outputs. It is important for compliance purposes that you ensure you obtain user consent and follow privacy laws, anonymising sensitive data where appropriate. As you fine-tune models, so the organisation will improve prompts.
Dealing with Slow AI Models
AI systems that execute complex tasks, require multiple reasoning iterations, and involve invoking multiple APIs that can take a long time to process. Techniques such as caching can be used, but latency might still be high. Prolonged round-trip times can result in user dissatisfaction; therefore, the organisation should think about implementing strategies to improve user experience such as incremental display. This is where you progressively display partial results as AI systems process different APIs, thereby reducing waiting times for final outcomes.
AI is a dynamic ongoing process
The development of generative AI applications is a dynamic, ongoing process requiring constant learning and adjustment to meet evolving technological trends and user demands.
Every organisation is at a different stage in its AI journey. Some have made significant progress, while others are just getting started. The unique problems each organisation seeks to solve to gain a competitive advantage will take time to address fully. As you progress in your AI journey, you will continue to enhance your products and solutions. At the same time, new AI technologies will emerge, including more accurate and faster models. By integrating these new developments into your products and solutions, you can stay ahead of the curve.
