With the release of OpenAI’s latest model, i.e., GPT-4, and Google’s Gemini, the field of Artificial Intelligence (AI) has gathered significant attention. The capabilities of Large Multimodal Models (LMMs) have been greatly extended beyond their typical uses in tasks like picture captioning and visual question answering.
In recent research, a team of researchers from Ohio State University has studied the possibilities of large multimodal models, particularly GPT-4V, as flexible online agents that can carry out tasks on any website by adhering to natural language instructions. The team has introduced SEEACT, a generalist web agent created to use LMMs’ capabilities for integrated online visual comprehension and action.
SEEACT has been evaluated using the most recent MIND2WEB benchmark. The researchers have presented a unique online evaluation environment in addition to the conventional offline evaluation carried out on cached websites. They have accomplished this by creating a tool that makes it easier to run web agents on active websites, resulting in an evaluation that is more dynamic and applicable to real-world scenarios.
The team has shared that GPT-4V is great for online agents as when its textual plans are manually converted into actions on the websites, it can effectively complete 50% of the jobs on real websites. This success rate is significantly greater than that of smaller models like FLAN-T5 and BLIP-2 which are specifically optimized for online agents, as well as text-only Large Language Models (LLMs) like GPT-4.
The team has also demonstrated that in the context of web agents, grounding continues to be a significant difficulty despite good results. It has been discovered that current LMM grounding techniques, including set-of-mark prompting, are useless for online agents.
The team has summarized their primary contributions as follows.
- Good results have been seen when oracle grounding is given to SEEACT with GPT-4V. It has demonstrated strong generalist web agent capabilities.
- It performs 50% of the tasks across several websites successfully in online evaluations, which significantly improves over the current approaches, such as GPT-4, which is 20%, and FLAN-T5, which is 18%.
- Grounding has been seen as a major challenge even with success; the optimal technique still performs 20–25% worse than oracle grounding.
- The best grounding technique integrates data from images and HTML text, outperforming picture annotation techniques by as much as 30%.
- Large models for in-context learning have shown better generalization to unknown websites. However, supervised fine-tuning has an advantage on websites visited during the training period.
- There is a significant distinction between online and offline assessments, mostly because several workable strategies exist for doing the same assignment. Since online evaluation considers the variations in task completion strategies on live websites, it is a more accurate representation of a model’s actual performance.
Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our 35k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.
If you like our work, you will love our newsletter..
Tanya Malhotra is a final year undergrad from the University of Petroleum & Energy Studies, Dehradun, pursuing BTech in Computer Science Engineering with a specialization in Artificial Intelligence and Machine Learning.
She is a Data Science enthusiast with good analytical and critical thinking, along with an ardent interest in acquiring new skills, leading groups, and managing work in an organized manner.
