Reflections on OpenAI DevDay 2023 Keynote

Open AI announcement#

We’re at Dev Day this week with Open AI and quite frankly this is an early contender for AI Christmas 2023 (if we hadn’t had enough already!).

What we heard from the Keynote.

• GPT-4 Turbo (longer context (128k), faster, cheaper).
• GPT agent builder in ChatGPT
• Agent Assist agent builder in API
• New ChatGPT ‘all-in’ function calling Whisper v3
• Stateful APIs to reduce token expense Lower prices for GPT models

That’s right, the big news is Open AI talking more about agents. Designing them, interacting with them, building them.

Our reflections#

Before we dive into specific reflections, it’s worth highlighting what we stand for at Tomoro to ground our view; we believe that:

1. Applied AI is the largest potential driver of competitive advantage for every knowledge business today
2. The fastest way to achieve competitive advantage is through the introduction of autonomous AI agents in your business

Open AI further reducing the barrier to entry for building autonomous AI agents (or “GPTs”) massively reinforces this as the direction that we believe all knowledge-based businesses will go on to achieve meaningful competitive advantage with deeply embedded AI.

What would you need for a business-ready AI knowledge worker?

Core ChatGPT (e.g. GPT-4) gave us the foundational capability to:

• Apply linguistic reasoning
• Generate new stuff
• Apply numerical reasoning (with advanced data analytics function in GPT-4).

This new agent capability introduces the foundational capability to:

• Access trusted knowledge (see Retrieval section in the Open AI image above)
• Communicate and act (see Functions section in OpenAI image above).

Therefore for the first time from one vendor the primary capabilities required to develop a meaningful knowledge worker are available in one place.

Does that mean it's solved?

Sadly not quite yet.

This capability is suitable to build knowledge workers for consumers, and you can quickly imagine your own personal agent which can execute simple tasks for you across many modern tech platforms.

However for complex organisations there remain key questions which will need to be resolved before large scale adoption of AI agents can succeed in their organisation.

1. How to make sense of this technology and apply it valuable and securely in you business as quickly as possible?
2. How to capture, classify and access organisational knowledge to optimise agent ‘memory’ and speed to access critical reference data? AND how to ensure you resolve hallucinations and understand why it returned a particular answer?
3. How to create a two-way cycle of self-improvement for agents to update knowledge and allow humans to ‘teach’ their AI counterparts.
4. How to create agents that sound and act like long-term members of the business, deeply ingrained in the culture, as opposed to new colleagues who’ve joined from OpenAI?
5. How to provide agents with advanced numerical reasoning capabilities, such as proprietary models developed over years which run the business today?
6. How to integrate agents with critical business function systems, which may not be available via standard APIs?
7. How to test and validate AI solutions prior to taking them live through DevOps pipelines and testing?

At Tomoro, we build solutions to these problems to realise the value of generative AI at enterprise scale.

Making use of this new capability valuably within business

It is hard to recollect a technology transformation occurring faster, deeper and with less time between academic research and enterprise application than the advent of generative AI.

Every single business is struggling to maintain pace with understanding what their business looks like in the age of deeply embedded AI, and the goalposts move every 3 months with a new technology and engineering breakthrough.

At Tomoro we have deep experience in:

1. Large language model driven applications at scale
2. Making significant change in complex organisations work effectively, repeatably

The need for strategic guidance on how to use AI appropriately has never been greater, and we use our people (and our agents!) to do this for our clients.

Solving for enterprise-level knowledge management, hallucinations and decision audibility

In the Agent Assist API, you can see a section upload files which the agent (or GPT) can interact with. This is based on a vector similarity search based method to query and return knowledge alongside a request to the agent.

We have written a lot more about enterprise knowledge management on our blogs on this subject. But suffice to say here the key problems you have once you go beyond small amount of content are:

1. Limiting yourself to a small amount of content to reference doesn’t realise the value, you have to bolster the agent with your own knowledge too often to be truly useful
2. Why content is significantly increased distinguishing the right content from similar similar items becomes non-trivial for the model (which is the latest, which is up to date, what content needs joining together across different documents etc.)
3. Lack of explainability of the results (e.g. why did it return this knowledge and not other knowledge)
4. Non-performance at large volumes of knowledge due to vector query time and retrieval time

At Tomoro we have pioneered a proprietary knowledge storage model which uses both embeddings and knowledge graph technology to enable businesses to deliver scaled agents which can access organisation-scale knowledge, rather than a few pdfs at a time.

Creating self-learning agents

Powering agents with trusted knowledge is critical to ensuring real value from implementation. However, it is only part of the story. No organisation has all their tacit knowledge written down today, and certainly almost all organisations learn new things over time which they want to commit to their long term organisational knowledge.

This is where two-way knowledge capture becomes critical. Ensuring that not only you can quickly extract the latest most relevant knowledge, but when something new comes up, you have the ability to capture this as new knowledge that you want to keep, and govern the answer is appropriate and correct before committing it to long term memory.

At Tomoro we’ve built a method for our agents to not only retrieve the latest knowledge of the organisation (in an explainable, auditable way) but built a similar solution to update the agent with new knowledge, and assure the contributions to this knowledge.

This will be critical to performant knowledge worker AI agents at scale within complex organisations.

Creating agent colleagues that belong in your culture

If you’ve used ChatGPT for some time, you’ll notice it’s style can be quite ‘vanilla’ in terms of its word choices and communication style. The reason for this is that it’s underlying training (be that fine-tuning, reinforcement learning with human feedback (RLHF) and even prompt engineering) is targeted to make it as useful as possible to the widest range of consumers globally.

For a specific organisation, much like when you hire new people, you don’t want everyone to be generic, you want them to espouse your particular culture, behaviours and language to fit seamlessly into the rest of your colleagues and how you present your brand to market.

At Tomoro we specialise in going beyond ‘system prompts’ (which are an excellent but basic way to do this - and you can see some of these in the ‘instructions’ section in the agent video above, and focus on enabling imbuing deeper personalities and behaviours into AI agent colleagues.

1. Retrieval augmented generation (RAG) to include tone of voice and brand awareness (using the knowledge techniques outlined above)
2. Model fine-tuning - we work with Open AI, Scale AI, Nvidia and others to develop the right fine-tuning approach for our clients and then execute these on the right partners platform for any given client
3. Agent meta-prompts - we have a curated set of tested meta-prompts that can instantiate agents to perform valuable tasks in the specific way of working for a particular organisation.

Providing agents advanced numerical reasoning capability

For those who’ve used ChatGPT plus or enterprise editions, you may be familiar with the Advanced Data Analysis (or Code Interpreter as it’s called it the above video) function. This is an INCREDIBLE function which enables ChatGPT to not only write code to solve a problem but execute that code and present back answers and/or visualisations to solve data analysis questions (in addition to much else).

However, many businesses don’t need to do data analysis for the first time, they need to execute complex algorithms (from rules-based to predictive machine learning) which drive core operational process within their business.

At Tomoro we integrate large-scale enterprise AI models with generative AI agent assistants to facilitate end-to-end decision making within organisational workflows. This means:

• Establishing complex models as function calls for the agent / GPT model to call when it recognises the appropriate situation
• Templating the question / response framework to ensure that all the relevant information is captured and utilised in the algorithm
• Ensuring the appropriate failure scenarios are accounted for and minimising potential for hallucinations in business critical responses.

Integrating agents with business critical systems

In the screenshot above, it mentions ‘Functions’ where you can teach the agent (or GPT) to take action dependent on a specific scenario.

That might be as simple as ‘send an email from outlook’ when I type “send that email” or something more complicated dependent on the routine you set up.

This is a critical part of creating agents which aren’t simply an extended memory bank but can operate on your behalf for complex tasks that aren’t suitable for Robot Process Automation (RPA) today.

However, as we know many businesses don’t operate on systems with an architecture that is particularly easy to integrate with; from mainframes to other types of legacy technology there are significant complications already today to exchange and transfer data across boundaries.

At Tomoro we have developed a proprietary framework to enable GPT models to communicate with key business systems, which we can either provide as an asset or use to develop a custom integration pattern dependent on the business value of the solution you’re trying to build.

Taking agents to Production safely

Using an agent as a consumer is a relatively easy process, I certainly don’t have a DevOps pipeline set up for the widgets I build for my personal life (though I expect our CTO does!). However when you are working in an enterprise context having a secure, controlled way to take solutions to live becomes critical.

One of the key areas we work closely with Open AI on is building the ‘DevOps’ like pipelines to take AI solutions to scale safely and securely, so everyone knows the business critical function they take part in is not going to become a risk in an AI-native approach.

To do this we have pioneered:

1. Native implementation of vendor agnostic agents eval framework 2. BDD based test cases - written in natural language.
2. Automated integration in CI / CD pipelines.
3. Marketplace of red team prompts to validate products
4. Implementation platform for responsible AI practices

Summary#

Open AI DevDay has delivered another step change of capability for the world when frankly most people are still getting use to GPT-3.5.

At Tomoro we continue to focus on enabling our clients to:

1. Use applied AI to drive significant competitive advantage for their business today
2. Design, build and deploy autonomous AI agents in their business as the fastest way to do this

We do this through:

• Strategically partnering with business leaders to help them understand what modern AI agents can do for their business, and developing their strategy to realise that vision securely
• Providing enterprise-class tools and methods which take scaled solutions to production efficiently, securely and valuably.

The Open AI announcement, and our alliance partnership, is a critical part of us realising this vision for every business to become AI-native, and the core problems we solve remain as critical today as they were before DevDay:

• Setting the right AI, agent-ready, strategy
• Optimising knowledge to ensure accurate, fast and self-learning AI solutions
• Establishing the frameworks and tests which take solutions to live safely and securely, operating within existing architectual choices