AI is revolutionizing how we search and consume content on the web. In the near future is likely to turn current search upside down, and change everything that content creators have learned about the web.
At this point I don’t think it is a bold statement. The progress is absurdly fast. And the pace keeps growing with each new release.
The OpenAI devday in November came with an array of new features. Not a stretch to say that each of the features are more significant that what had happened to legacy search engines in years.
Certain users have been testing the ability to search the web through ChatGPT for a few months now. It gives the platform the ability to access real-time information. You can search and browse. You can find quotes, you can research a certain topic, or cross-check something that ChatGPT spit out.
The feature has received mixed critisism. Some reviews are claiming that it is better to use external plugins. I agree that the tool needs to improve. But in my eyes the current version of the browsing ability is only the infant version. The potential is limitless.
In this article I am going to play around and explore what we can do with search in ChatGPT. My hope is that I can show you the rules, functions and limitations that can help you use this feature effectively.
Firstly, let’s start with the prompt instructions that ChatGPT follows when working with the browser tool.
Disclaimer: I am a firm believer in following terms of service, and I think GPTs instructions are not part of models, algorithm, or systems that fall under TOS. Even still I will not reveal the instructions. My intention for the article is to help you understand what prompts are allowed, what options you have and how to efficiently use browser tool through ChatGPT.
Here is a summary of prompt policies for browser tool in ChatGPT:
Do not render any links
ChatGPT is instructed to not render any links except for quotes returned from the search. This makes sense as an obvious security measure. Rendering unknown links is a surefire way to sign up for cross-site scripting. Nobody should do it.
However I don’t think it is as safe as it should be. I tested this by asking to find a link in one of my websites, it had no trouble returning the exact link I asked for.
Perhaps this can be considered as a quote. But my intuition says that someone with bad intentions could find a way how to exploit this.
Do not regurgitate content
The tool is not allowed to regurgitate content that it finds on the web. It is interesting that the instructions use the word regurgitate, as it is a common word to describe the overall content returned by AI models.
No rephrasing or translating whole content
ChatGPT is not allowed to translate, rephrase or paraphrase whole content returned from the web. However it is ok to do it with a fraction of the content. In my tests ChatGPT was able to follow this instruction very well.
Surely, there are jailbreaking techniques that might bypass this. But this is good enough for the common user.
Summary can be up to 80 words long
It is not allowed to write a summary of more than 80 words. If asked, it should default to writing an 80 word summary.
In my experience ChatGPT usually returns ~120 words. However it does try to keep it as short as possible. LLMs have trouble following the word count and are more precise with paragraphs, so I am not surprised that the 80 word limit is rarely followed.
Analysis, synthesis, comparisons, etc, are all acceptable.
I don’t clearly understand what is meant with each of the examples. However the general idea is that ChatGPT can use the content for various manipulations, as long as they don’t overlap with the forbidden actions.
Do not repeat, instead point to the source
Instead of repeating content point the user to the source and ask them to click.
I think this is a nice gesture to content creators. We can never be sure that they will keep this forever but it is nice to see that content creators are being thought about. In my tests ChatGPT always included at least 1 source, and encouraged to read it for more information.
Include multiple sources
ChatGPT is instructed to include multiple distinct sources in your response, at LEAST 3–4. It does a poor job in this regard. Usually we get 1 or 2 sources.
I find this feature a bit disappointing out of the box. If I want to get a more comprehensive research of recent articles I would expect multiple sources. It takes multiple prompts and a lot of search attempts to get there.
My hope would be that eventually the feature would allow you to do a comprehensive search. Replicate what a human would do when searching. Go through multiple articles, compare, make notes, evaluate, search again and then summarize.
At the moment ChatGPT jumps to conclusions based on the initial search. I wish it would investigate more.
Be thorough, search till you find
When there are no good results ChatGPT is instructed to keep searching. If it can’t find anything, it should give up only as a last resort .
I think this is a useful rule. ChatGPT tries it’s best. Without it, I suspect it would have real trouble with more difficult queries. I would like to combine this approach with real-time analysis of each article — then we would truly have something.
In addition, ChatGPT is supposed to consult all relevant sources you found during browsing. And at the same time it should keep the answer concise and don’t include superfluous information.
All I will say about this is — it tries it’s best.
Organize responses by flow
ChatGPT is instructed to focus on making the response flow. Instead of returning exactly what the search results showed, it adds a bit of it’s own synthesizing that is supposed to help the user.
This seems more like one of those instructions that are meant to discourage repeating and plain regurgitating. Instead it promotes regurgitated repeating.
You can work around this by asking for quotes directly, or giving instructions to minimize synthesizing.
Do not be thorough in the case of lyrics or recipes
One instruction that was a bit surprising was the emphasis on working with lyrics or recipes. The instruction contained bold text with — EXTREMELY IMPORTANT.
I suspect that this has to do with some of the lawsuits happening in the industry. For example, Universal Music sued Anthropic for distributing song lyrics. It is obvious that song lyrics where a part of the training data. There are hundreds of sites that do this. The ethical and law side of this feature deserves a separate article that I should explore in the future.
Reading the whole prompts, I was a bit surprised by the quality. The style of the language was a bit careless. The prompts did not seem well thought-out. I would have expected more advanced prompt engineering from the team that created the tool.
For example, capitalizing extremely important instructions does nothing in my experience when creating custom GPTs. I suspect that these prompts are merely a convenience or a work in progress. For the truly important policies such as ethics the tool does not rely on its prompt instructions.
When you ask ChatGPT to search something, under the hood it starts by creating a search query. It tries to be helpful and specify a good query that should bring you what you are looking for. The prompt is not perfect at the moment but this part of the functionality is where AI search has the ability to kill the existing search engines. It can understand the context of the problem, use it’s knowledge to help you find what you need. Faster.
As a beta feature there are really no pointers to help you use the tool better. For me understanding the available inputs and functions was essential to use the tool efficiently.
ChatGPT Browser Tool Inputs and Functions
Browser tool has the following internal functions available:
search(query: str, recency_days: int)
Description: Issues a query to a search engine and displays the results.
This is the main function that performs the search through Bing. Once you have asked ChatGPT to search for something it will trigger this function.
Besides the search query you are also able to suggest the recency of the search. You can specify search for articles published yesterday, or last year.
click(id: str)
Description: Opens the webpage with the given id, displaying it. The ID within the displayed results maps to a URL.
This is an internal function that ChatGPT uses to click on page elements. In my tests I was not able to get it to do anything other than open links.
But talking with ChatGPT it seemed to indicate that it could do more.
Potential hallucination alert!
When using the browser tool, I don’t manually find these IDs. Instead, the system provides a list of clickable elements and their corresponding IDs based on the current view of the webpage. I select the appropriate ID from this list to interact with the page, such as opening a link or activating a page element. This process is abstracted from the traditional method of inspecting a webpage’s source code to find element IDs.
You can also ask for clickable elements on a page. In my case I asked to show clickable elements for my homepage.
I learned a few things from this:
- While it was not able to click on Subscribe or Sign In buttons, it did recognize them. I suspect that this is filtered out in the browser tool.
- It does not look for any specific elements or identificators. As long as the user can click it without a lot of Javascript it should work.
- Seeing that the navigation is visible only on desktop it is clear that ChatGPT is browsing with a desktop version of the site.
- Browser tool has the potential to become much more interactive soon.
back()
Description: Returns to the previous page and displays it.
This is basically used only internally when browser tool is searching for something specific and needs to navigate multiple pages. I did not think of anything interesting to experiment here.
scroll(amt: int)
Description: Scrolls up or down in the open webpage by the given amount.
We can make it scroll to a specific part of the page, read it and return something.
Could this be used to manipulate ChatGPT somehow? Just off the top of my head, you might be able to set up events that listen to scroll and trigger other browser events that are not allowed. Overall pointless exercise but I feel a potential for vulnerability.
open_url(url: str)
Description: Opens the given URL and displays it.
Pretty simple, browser tool uses this to open pages from search results. By default it seems that ChatGPT would like to use the search but it turns out it is also possible to visit a website directly using this function. You just have to ask.
quote_lines(start: int, end: int)
Description: Stores a text span from an open webpage. Specifies a text span by a starting int `start` and an (inclusive) ending int `end`. To quote a single line, use `start` = `end`. For citing quotes from the ‘browser’ tool: please render in this format: `{message idx}:{link text}`.
In my tests I was able to ask ChatGPT to give me the quotes it has stored from it’s search, and it returned some additional information that previously was not mentioned. Potentially a useful feature to get a more thorough research response.
In a way, this bypasses the instructions that prevent it from repeating the content, however since these are only quotes I believe there is no foul play. It can be really useful if you want to see the search references directly, without the added GPT noise.
The quotes are stored only for the current session.
Limitations of ChatGPT Browser Tool
Browsing ability inside the chat is a huge step. However at the moment there are some significant limitations. I made some notes for a few of them that I stumbled upon during my testing.
ChatGPT has trouble with PWA Sites
ChatGPT has limited Javascript capabilities and is unable to render sites that completely rely on Javascript. I tested this with multiple PWA sites and indeed it was unable to view them.
This is a problem because a lot of newer sites rely on Javascript to improve their user experience. In recent years web has become truly javascript dependent, and virtually every browser and devices fully supported this. So ChatGPT Browser tool might need to catch up here.
Authority and source selection bias
Another thing to consider is that AI models are still heavily biased by their training data. While testing the tool I got curious about how ChatGPT chooses the sources to include in the response.
ChatGPT explained that it’s evaluation of source credibility is primarily based on it’s training knowledge. And yes, the training data set is almost always lagging behind. For ChatGPT Plus at the time of writing it is April 2023, which is more than half a year.
This introduces an interesting topic of source referral. Until the model updates start happening monthly, newer websites might have no chance of being referred. In most cases this is fine, because newer websites are unlikely to rank first so quickly. But it does happen.
I decided to explore this hypothetical scenario:
Priority is given to sources known to be credible and trustworthy. If the latter three sources in your example are known to be highly credible and were recognized as such in my training data, they would likely be preferred… / ChatGPT
At the moment ChatGPT does not have any real-time evaluation capabilities, and it can not follow up to date information.
So the real time browsing is only partially real-time, because the evaluation is still happening based on outdated biases.
Before ending, I want to mention that there are ChatGPT plugins that provide similar functionality, for example, Webpilot. The general idea of these plugins are similar but there are less restrictions due to not following the same policies. I can’t rule out that someone might build a GPT that would be truly irreplaceable or unique.
My gut feeling is that the native browser feature will eventually catch up, and likely surpass anything built by 3rd party vendors. I say this with some confidence because OpenAI can set the rules, they can align their future to make the environment as they wish.
Everyone else is just playing by their rules.
Regardless of the current limitations I believe that the browsing ability will be game changing.
It is a step closer to a better search experience. A step closer to the next generation web.
