Building your own Internet Chat
Does your LLM work with the Internet? If you are a ChatGPT Plus customer you probably have tried the web browsing capabilities, and given the relationship between OpenAI and Microsoft you may have seen this type of experience built into Bing. With Google there are similar features rolled out, for example the AI Generated Results taking up the top of the search results page.
Similar to how Web Browsers and Search Engines were combined there is a new paradigm of how your AI is combined with the two. What if you aren’t a fan of Bing and Bard and just want Google and ChatGPT?
More broadly, how do we integrate APIs with AI chat in 2023?
Google Search API
There are two options you can go with for Google, the official Google Custom Search API or SerpAPI (there may be similar competitors). SerpAPI is a third party which offers a complete Google search including Knowledge Graph data like business info and news. There are endless APIs out there that offer more specific searches I’m sure.
URL Navigation
The second part to an Internet chat is being able to navigate to a URL and extract text from the page. This is harder than it seems for many reasons, if you are familiar with web scraping then you know plenty of them. Basically whenever you access a website programmatically you risk being identified as a bot. Sometimes websites can be accessed but it requires a headless browser that can run javascript.
All of these factors are multiplied if you are running them in the cloud as a service for users. Sometimes simply being on a cloud provider with a headless browser gets you blocked.
The good news is for simple websites optimized for SEO and accept the traffic, it is simple to extract the text. I’ve noticed the LLM will sometimes call the URL function immediately after the search function, or it might fetch a url that it has ingested not from search, like from wikipedia.
Handling LLM Context
In my previous post I tested out language models with increased context. This becomes important here as we need to consider how much data we can feed without going over the context limit.
In my prototype I’m simply creating an in-memory vector store. This likely doesn’t scale but works fine for local use. There was a lot of complexity to explore here around how many results to return. OpenAI’s storage solution with their new GPTs feature handles this well, as well as lots of new projects in the space.
Adding APIs
Google Search and web scraping have their own challenges, but when it comes to simple APIs the process of integrating into chat is seamless, for example the NBA API I added on my project didn’t require a vector store and is free.
I’m working on building this out more as my personal chat to learn more about AI UX.
You can check out my “Web Chat” project here.