Generative AI has been all over the news throughout 2023 and will continue to be in 2024. Most Gen AI discussions have revolved around using it as a tool for employees to answer questions, edit images, or summarize content. As an intelligent automation company, Reveal Group has spent extensive time trying to find applications for AI in the automation universe. We aim to automate and add intelligence to some or all processes to reduce the manual effort in completing a business process – providing immense value back to the business.

Although the applicability of Gen AI is extensive, the main questions that concern potential implementors revolve around the risk of an uncaptured error in a deterministic process and the ability to remediate those errors. Additionally, the extent of an error, if interpreted as a small mistake or something egregiously wrong and outside of the context of what was asked – so-called hallucinations – adds to hesitation in approaching Gen AI.

Reveal Group’s experts continue to evaluate the viability of Generative AI and are steadfast in identifying the potential benefits and drawbacks of pairing it with Intelligent Automation.  Here’s what we’ve uncovered so far…

Determining GenAI Value and Understanding Risks

Before implementing any use case, a detailed statistical analysis of the potential risk should be made so that the business is fully aware of what an end state could look like and where controls should be put in place. Specifically, if the use case should be moderated by human review – where value is provided by completing most of the work – or if a programmatic control is possible. The method for measuring the efficacy and risk of an AI solution is highly dependent on the use case proposed.

Examples of testing methodologies include error matrix analysis, calculating the Rouge metric, quantifying matching characters between true and predicted outputs, and Levenstein distances, among other methods. An informed decision on implementation can be made once risk and efficacy are quantified.

It is also worth comparing an AI algorithm’s performance against a human’s performance in an implementation scenario. If, for example, the false negativity rate using Gen AI is 3% – the quadrant that poses the highest risk to the business – but the false negativity rate of a person performing the same task is at 5%, then implementing the automation not only provides time value but also reduces the risk of erroneous output.

The Good – Real-Life Use Cases of GenAI + Automation

Reveal Group sees Gen AI as another highly valuable tool when approaching an intelligent automation implementation. It can be the centerpiece of a process or used to improve the automation rate of larger existing automation by replacing an existing rules-based approach with a high margin of error.

Examples can include entity extraction using regex, address parsing using Levenstein distance, name matching between databases, and form extraction using simple string matching. These are commonly seen in automations and are usually the culprit for reducing the overall automation rate of a process by forcing human review.

When an intelligent automation process has many error-prone steps, each will individually contribute to a reduction in the automation rate of the overall process. Improving the performance of each of these steps by 50-60% by substituting with a higher-performing Gen AI engine can dramatically affect the business value returned.

One promising use case identified by our team is the use of Gen AI for ETL tasks of highly unstructured data. We have had incredibly positive results transforming raw data extracted from PDFs, excel, and web formats and standardizing and condensing the information to the desired format of the business. This is a common ask in automation use cases, where employees spend large amounts of time in non-value-added tasks of pulling information from tables to produce a report or create a business artifact on a routine basis.

Previous automation of these processes where the incoming data is unstructured has yet to be possible with rules-based automation or ML tools on the market due to the highly variable nature of the transformed content. Still, barriers remain.

The Bad – Limitations of Current GenAI Models on the Market

Like humans, context is needed in prompt engineering to achieve a desired output. The more complex the ETL step, for example, if we are trying to standardize 100 columns rather than just 5-10 columns, the degradation in output quality is very clear. This can be mitigated slightly by adding the necessary context for the column matching, but the business value is considerably diminished overall. The complexity of the use case will determine how much use intervention will be required and the percentage of use cases that will require extensive review.

Another limitation affecting the complexity dimension is the size of the requests that a given Gen AI model permits. Models that can accept larger data sets and more extensive context cost considerably more. The infrastructure required to host these more capable models is also much greater.

Finally, where math is required, there is consensus in the community that Gen AI should not be used, as it does not perform math calculations and only considers the most statistically likely answer, which may or may not be the same.

The Potential – Choosing a Trusted Partner to Implement AI for Your Organization

Reveal Group’s Innovation Team continues to assess the strengths and limitations of models in the market, such as Open AI’s GPT and Meta’s LLAMA2 and their different model tiers. Price points, performance, infrastructure requirements, business risks, and testing methodologies are some of the many considerations required when considering bringing Gen AI to your business effectively. Let us help you design and implement the GenAI and intelligent automation path that is best for your enterprise.