Not a day has gone by recently where I do not hear about the new AI craze, ChatGPT. I am sure you have the same sentiment.
If it is not on the Internet, someone is talking about it on the TV, Radio, in the office, or at an event you’ve attended recently. Today, it was brought up in my little sphere four separate times: at a conference I attended (numerous times), by colleagues in passing, and by my children. Yes, my eight- and eleven-year-old know what ChatGPT is and how to use it.
ChatGPT can do many things, undoubtedly. Like all artificial intelligence engines, though, most people forgot something critical:
The data in the system is only as good as what the system learns and is provided. If there is no data available, you are out of luck. If the data lacks quality, you’ll get the wrong answer.
Read the Disclaimers
People are quick to ask the engine a question that they seldom read the system limitations and heed the warnings. ChatGPT’s data is limited past 2021. That’s a huge drawback for the smart engine. Don’t you think?
Sure, if you want to generate information related to something general purpose, the engine is likely to produce reasonable results most of the time. But, if you are looking for time-bound details, such as changes in Government Regulations, ChatGPT may not be your handy dandy trusted source of truth. The reason goes back to my first thought: the system is only as good as the data gathered within its annotated context.
The second point to consider is data accuracy and reliability. Again, ChatGPT practically screams to the user that the information presented may not be accurate.
Reality check: You need to conduct due diligence and check the data before you use it. Do everyday folks do this? Likely not. The reason is that if it sounds legitimate and resembles factual accuracy, it must be right. Wrong!
History Repeats Itself, With a New Spiffy Look
Remember when IBM Watson played Jeopardy in 2011 and beat out the two reigning champs?
The system indeed sounded smart. But was Watson as brilliant as everyone thought? Not quite.
Watson used targeted data sources to win each Jeopardy game. It didn’t search Google or come up with answers by osmosis in a record-breaking time. The Watson Research Team at #ibm had to tell the system precisely where to go and exactly how it had to evaluate the data, including the appropriate manner to respond to Alex Trebek’s questions. It took years (five, to be exact) to get the process just right. Once the IBM Research Team had perfected the engine enough, it was showtime to commercialize the solution.
IBM believed it could solve medical miracles with Watson, hence the flood of investments in healthcare between 2013 – 2019. But there was one problem, no matter how many acquisitions and partnerships were established, the data available would still yield “oldish” results.
In the medical world, data recency and accuracy are predicated on the sheer volume of new datasets Watson had to learn. IBM simply couldn’t ingest and acquire enough data to make Watson Health a rockstar product, because no matter how hard the company tried, medicine doesn’t follow the IT worlds #mooreslaw proclamation. Medicine transforms almost daily, even hourly. And new product attempts in the market today all face the same challenges Big Blue realized a decade ago.
If you’ve taken a high school statistics course, you know that more data yields confirmation of data accuracy and reliability. Or it can work against you too, because if you find data anomalies, data integrity issues become front and center.
With the AI search tools occupying the media at hype levels, the problems the legacy IT vendors noted years ago still exists, perhaps to a greater degree. If ChatGPT, Bing AI, or Google’s Bard AI aren’t programmed to know the answer or the data provided is incomplete, or even worse, inaccurate, the results you’ll get fall short.
When you think about it, Watson was AI 1.0, and ChatGPT, Bing AI, and Bard AI are AI 2.0. The difference: The masses have access to these tools versus having to spend a cool $1 Million Dollars plus to tinker or explore. Now, it’s free.
Identifying Flaws: Data Recency and Relevancy Issues
Here are two examples of where the AI engines fall short. The first points to the death of Lisa Marie Presley. ChatGPT took about 2 minutes to get back to me with invalid data. Look at the reason: the most recent data available to ChatGPT is September 2021. A lot has happened since 2021, including the January 2023 passing of Lisa Marie. The lack of data proves that AI is only as good as the volume of recent data fed.
A second example I searched for with ChatGPT had to do with #federalcontracting concepts, specifically FAR Part 12. If you do business with the government, you know that Part 12 of the FAR is the baseline rule for Simplified Acquisitions Purchasing (SAP). What does ChatGPT tell me? Surely not the specific detail that I need to understand the SAP Acquisition Parameters.
It gives me a generic book report on what the Federal Acquisition Regulation standards are for, including a basic definition of small purchase acquisitions. But does it state the main point of FAR Part 12, the acquisition thresholds? No. Therefore, the data can be deemed incomplete or perhaps even inaccurate.
Takeaway: Be mindful of the recency, quality, and accuracy of the data coming from the robots because they can be flawed.
Suddenly The Perfect Writer or Brilliant Scholar?
I’ve written a few books in my lifetime. If you’ve read any of my technical books, which you can preview on Amazon, you know I have a particular structure when presenting examples, whether images or tables. Even when crafting exercises in each book, I generally use the same sentence structure to call out directions. Similarly, I write each newsletter for LinkedIn following a reasonably consistent style: state a fact, present research, and defend my points. Call it the Ph.D. in me, but it’s hard to change a person’s writing style, no matter how much you try.
And to my point, this where I say, roger, we have a problem with using ChatGPT to be your secretary.
With ChatGPT came a newfound way to plagiarize. Imagine a student who doesn’t know where the periods, commas, and semi-colons go. Then, suddenly, writing a grammatically perfect submission was within reach. Or a student who is always curt with their responses now prepares highly technical answers to a targeted question. Yes, these scenarios are all commonplace with ChatGPT and other platforms.
To prove my point about highly researched responses that are grammatically perfect, I asked ChatGPT to tell me about UX Design. The result is presented in the image below.
ChatGPT gave me a lengthy write up regarding UX Design. It was a well-crafted two-page essay full of technical jargon. A good researcher can eek out each of these terms and expand upon the initial submission. The byproduct is an even longer paper without having to think about actually writing a sentence.
My point to this example is that data quality and accuracy can be questionable, even if the write up is grammatically accurate and uses the right terminology (or so you think). Do most users care if they don’t have to do the work? I think you already know the response.
Outsmarting the User, Thanks To AI
Here is one last interesting point on how to outsmart potential abusers of AI.
All of these next generation AI engines easily re-engineer their work and tell you if the writing is authentic. Here is an example with slight modifications to two sentences regarding the UX Design inquiry.
ChatGPT has confirmed that the AI engine wrote this sample with slight modifications. It even told me what lines were modified versus verbatim from its own internal system.
The Possibilities of Summarizing Data with Precision and Accuracy
You can also ask: Did you write a specific passage and to summarize its findings simultaneously. That’s exactly what ChatGPT did, with precision and accuracy, using just the first few sentences of this article.
Final Thoughts
For all of the negatives I’ve pointed out, AI engines are pretty awesome for specific use cases. You can make a case for AI usage in virtually any industry. You just need to go into using AI solutions with eyes wide open.
These solutions are far from perfect and require constant “love and care” to be factually accurate. You need to remember that any of these systems are only as good as the quality of the data, and the volume of data it is fed as well as the retention and training capabilities available.
