By: Fred with the “direction” of Emma Eight out of ten.
That’s how many ISM questions AI got right when I asked it for the most common ones in the OOW oral. Not bad! But the other two would have sent you to the wrong regulations. And in front of an MCA examiner, that’s not where you want to be.
Now, don’t get me wrong. I love AI. It’s changed how I run Fred’s Oral Prep, and I use it every day. But our subject isn’t black and white. It’s full of grey – the vessel, the flag, which regulations apply – and grey is exactly where AI trips you up.
So before you start studying with it, let me show you its weaknesses.
What Does AI Actually Get Wrong?
Where it gets its answers
It copies the crowd. A chatbot (ChatGPT, Gemini, Claude, take your pick) reads websites, forums and FAQ pages, then gives you what most of them agree on.
The crowd is often wrong. Twenty websites copying the same mistake look like the truth to a machine.
It can be out of date. Maritime regulations and guidance change. Unless AI checks the latest official publications, it may give you information that’s no longer current.
How it answers
It can sound incredibly confident. Even when it’s wrong! You might get a beautifully structured answer, complete with regulation numbers and references. Unfortunately, some of those references might not even exist.
It can hallucinate! Yes, that’s actually what it’s called when AI makes things up. It can invent regulations, quote references that don’t exist or give you a beautifully written explanation that’s completely wrong. And the worrying part? It can sound incredibly convincing!
It can mix up regulations. Different regulations can cover related subjects, and AI doesn’t always distinguish between them correctly.
It rarely says “I don’t know”. It would rather guess than admit it isn’t sure.
It gives a different answer each time. Ask the same question twice and you might get two different answers. Not ideal when you’re trying to learn the right one!
It struggles with pictures. Lights, shapes, day signals and radar plots are a big part of your oral. AI can misread or describe them wrongly, and it can’t reliably plot a radar problem.
How it affects your learning
It can do the thinking for you. This is the big one. Reading a perfect AI answer feels like learning. But in your oral, you have to explain it yourself, out loud, under pressure. If AI has always given you the words, you might freeze when the examiner asks “why?”
It waffles. AI loves a long answer. Learn its five-paragraph version and you’ll give the examiner an essay when they wanted two clear sentences.
It can be a little too eager to agree with you. AI sometimes tells you what you want to hear rather than challenging your answer. It might congratulate you for getting something right when you’ve actually missed something important. Not particularly helpful when you’re preparing for an examiner who certainly won’t be doing that!
It doesn’t learn from its mistakes. Crew tell me this one all the time! You correct it, it agrees and fixes its answer. Then you start a new chat and it makes exactly the same mistake again. Unless memory is switched on (often only in paid versions), each chat starts from scratch, and even then, correcting it doesn’t actually teach it.
I even asked the AI how it finds its answers. Fair play, it was honest:
“I look for what most sources agree on, but I can’t always tell which one carries legal authority. An MCA notice and a forum post can look equally convincing to me. So ask me for the regulation and paragraph, then go and check it.”
Why I’m Spending So Much Time Testing AI
Remember those eight out of ten? Here are the two that weren’t strictly ISM questions at all:
Safety officers, safety reps and safety meetings, which come under the Merchant Shipping and Fishing Vessels (Health and Safety at Work) Regulations 1997.
Risk assessments and permits to work, which draw on that legislation, COSWP and the vessel’s SMS.
These subjects are closely connected, so I can see why AI grouped them together. But knowing subjects are related isn’t the same as knowing where the legal requirement comes from. Tell the examiner the safety officer comes from ISM, and watch the eyebrow go up.
I can spot these mistakes because of my experience. But what if you’re learning the subject for the first time? How would you know?
That’s exactly why I’m spending so much time training, testing and refining the AI tools I use in Fred’s Oral Prep. Because as useful as AI is, I know its limitations. And I want my students to understand them too.
It’s also why the AI in my Crew Portal is restricted to my own materials, rather than searching the whole internet. And I’m continually testing and refining it, checking the results against the regulations and my own teaching materials.
Because when it comes to preparing students for their MCA oral exams, getting the answers right matters! - funny that.
How to Use AI Safely
Ask for the source. If AI gives you a regulatory answer, ask for the actual regulation and paragraph. Then check it yourself.
Be specific. Tell it you’re preparing for a UK MCA oral exam, which qualification you’re studying for and what type of vessel you’re discussing.
Get it to challenge you. Rather than asking, “Is my answer correct?”, tell it to identify anything wrong or missing.
Use it to test your knowledge. Ask it to give you questions and scenarios rather than simply supplying answers to memorise.
Keep it short. Ask for the answer in two or three sentences – the way you’d actually say it in the exam.
Say it out loud. Once you’ve checked an answer, close the chat and explain it in your own words, as if I’m the examiner.
Don’t blindly trust its corrections. AI can mark a perfectly good answer as wrong too!
Ask me! If you’re one of my students and something doesn’t sound right, bring it to our Sunday webinar. We can work through it together.
Final Thought
Rule 5 says keep a proper lookout by sight and hearing, as well as by all available means. AI is a brilliant “available means”.
But you still look out of the window.
Happy studying!
