Hedge fund says it's not easy being a prompt engineer
Prompt engineering isn't likely to become a job in-and-of its own right, but prompt engineering as a skill is becoming increasingly valuable. Hedge fund Man Group, which has been implementing its in-house AI chatbot ManGPT, said at the Quant Strats 2024 conference in London yesterday that using LLMs effectively isn't easy.
Click here to follow our new WhatsApp channel, and get instant news updates straight to your phone 📱
Tim Mace, head of data and machine learning at Man, said generative AI "isn't entirely delivering on the hype." One of the most frustrating aspects of AI bots is that they are "designed to give a plausible answer, not a factual answer," which can lead to costly misinformation.
People at Man Group are using the technology effectively, but Mace says they are also "writing very long prompts in order to get the model to do what they want it to do." This runs its own set of risks; previous research into prompt engineering showed that prompts are exponentially harder to execute as they get longer, and can often disregard information in the middle of a prompt to focus on the start and the end.
There are ways to ensure accuracy, however. Man Group says implementing 'chain-of-thought prompting' has been "very successful." This is a technique where you encourage the chatbot to explain its reasoning rather than provide you with a straight answer. Sometimes, this will directly lead to a more accurate answer, other times, it will let you see where the bot has hallucinated, and allow you to easily correct it. Mace said "the longer you allow a model to reason over its answers, the more accurate it will be."
ManGPT allows its users to switch between different LLMs depending on the task. This can be particularly important for software developers; Anthropic's Sonnet models are widely thought to be much better for coding than OpenAI's GPT-4o, the default model used by the ManGPT.
Mace also said Man Group has taken an interest in implementing reinforcement learning. This is a different way of training LLMs advocated by Goldman Sachs, which focuses on trial and error. Its thought to be more reliable and accurate than current methods, but more costly to train.
Have a confidential story, tip, or comment you’d like to share? Contact: Telegram: @AlexMcMurray, WhatsApp: (+1 269 237 3950). Click here to fill in our anonymous form, or email editortips@efinancialcareers.com.
Bear with us if you leave a comment at the bottom of this article: all our comments are moderated by human beings. Sometimes these humans might be asleep, or away from their desks, so it may take a while for your comment to appear. Eventually it will – unless it’s offensive or libelous (in which case it won’t.)
