The internet has been established as the primary access point for information in our lives. We are increasingly exclusively reliant on web-based information and services. However, the way we access information on the internet is still evolving rapidly.
From an enthusiast network of personal websites to a mainstream commercial enterprise shaped by search engines, the web continues to evolve in ways we can only attempt to predict.
As we stand on the cusp of an AI revolution, we will look at how content online has changed and will continue to.
The New Frontier
During its formative years, personal websites formed the backbone of the web, with significant traffic routed via links between each of them.
Search engines were an important tool, but they might typically form the starting point instead of the centre point from which information was found. Browsing (or perhaps ‘surfing’ then) felt much more exploratory, which fitted with the feeling of a pioneering experience full of possibilities and unknown potential.
Website building was a new and unique way to communicate, simple enough to do in a text editor or supplemented with basic site builder tools. Site networks like Geocities made this even easier and free, although many sites were hosted on university networks at no cost to the creator. There was a community focus of individuals with common interests, linking between similar sites, and collaborating on forums and chat networks.
Before Google became established, search engines provided a mixed experience, not always ranking results usefully and struggling with the exponential growth of the web. Yahoo, among others, attempted to categorise all indexed sites through a vast taxonomy. These attempts at categorisation inevitably become too large and inaccurate.
However, as most sites would routinely link to similar content, there were pathways to content discovery that would always lead to good or at least consistent quality content. Almost every website would have a ‘links’ page, something almost unheard of now.
Established sites became increasingly popular due to their ranking, crowding out those less popular. The convenience of better quality search results meant the kind of casual browsing for content was no longer necessary, and we lost the appetite to explore. The loss of traffic to these sites meant they became deserted, being difficult or impossible to discover.
As the commercial possibilities of the Internet began to gain speed, large corporations and organisations would find it easy to come online and then cement themselves towards the top of search results. Combined with the widespread growth in the web’s user base, this first generation of websites would start to look unofficial and amateur, gradually fading into obscurity.
There have been community-developed websites that have been successful and stood the test of time. Wikipedia in particular defied what intuitively should have been an unworkable concept, with enough contributors to keep adding value and maintain sufficient quality. Forums still today support huge communities with a common interest, providing immensely useful, if scattered, in-depth information.
SEO Eats the Web
In a relentless race to chase search result positioning, the SEO industry was created. As knowledge of preferential page characteristics grew, SEO rules were laid down.
With a wide range of requirements like content style, length, linking and page appearance, websites as we know them are adapted and controlled by their effect on search engine results pages (SERPs).
Being discoverable and attracting visitors is almost always beneficial to a site, whether it has commercial aims, or is an online community. But with the great reach of the web and the commercial potential, there are many individuals that seek to exploit this to their own gain.
As SEO has become more sophisticated in exploiting search engines over the years, the web has become swamped with low-quality content, cluttering out genuine, useful resources. This undermines the principle of a search engine itself; no longer a resource to find content, but a battleground of paid advertising and spam pages.
With few established search engines, there are limited ways for users to find a way through this maze of increasingly irrelevant results. Some may find content through other channels like word of mouth or social media, but discovering new content is much harder. By ranking pages on a metric tied to their popularity, it’s inevitable that traffic will be pushed into an increasingly small number of sites. The odds are stacked against anyone establishing a site with a new domain, and trying to grow organically without paid promotion.
We have now reached a strange point in time where web content is frequently dictated by search results, instead of the other way around.
This effect can be seen more acutely on networks like YouTube, Instagram, and TikTok where content is often created to get hits alone, an essentially commercial endeavour to draw from the huge pool of users and subsequent advertising and sponsorship. This is a market in the form of an entertainment platform instead of a resource of knowledge and services. The two can co-exist together, but the challenge is building search tools to help us navigate between them.
Social Networks vs The Web
With the emergence of large-scale social networks, we began to see a split forming - with content either on the web accessible via search engines, or siloed into a social network.
With a focus on apps, social networks may now expose little to none of their content on the web. There is now no effective way to search everything at once. Content creators might duplicate content across networks as a workaround to this fragmentation.
This marks a shift away from the philosophy of the web as an open platform with open standards; apps are typically closed source and their user-driven content is locked away, owned by a small number of tech giants.
As the focus on the traditional web diminishes, the newest generation of users is adopting different habits of consuming content as well. Prabhakar Raghavan, Google’s Senior Vice President noted that 40% of young people would look to TikTok or Instagram to find a place for lunch. There’s no getting away from the immediate visual feedback these platforms deliver, perhaps easier than navigating through a map to try to find some photos that are often low quality or not even of the place itself.
This transition of content to social networks includes Twitter, which was founded on the premise of short-form messages. Instead, users post long threads of tweets when they have more to say; this might have been a blog post one time, but the outreach is much more effective, knowing their followers will see it immediately. This raises the question as to how discoverable will this be in the future though, indeed will it even continue to be available with Twitter’s uncertain future?
One of the most remarkable lessons learned since the birth of the web is that content is transient and rarely enduring. Websites disappear due to no longer being hosted, pages change constantly, and rarely provided with a revision history. Large content platforms may simply lose huge amounts of data one day, as happened to MySpace.
Countless web pages are left empty by embedded retracted tweets or YouTube videos no longer public. The Internet Archive is the only small offering we have to give some insight into the past of the web right now, a fantastically undervalued service.
The Rise of AI
The recent explosion of AI chatbots such as those from OpenAI, Microsoft, and Google will clearly have a profound impact on the way information online is accessed.
Instead of looking through sites offered in search results, we may rely on an AI-generated answer, based on those same results. Given the prevalence of misinformation in recent years, this sounds like a daunting prospect. While some chatbots are now starting to cite their references, such as Bing, these can still be based on wholly inaccurate sources.
There is also the problem of traffic (and income) being taken away from the sites offering the content AI is being trained from. This happened before with search engines offering news excerpts directly in their results, but AI will affect content publishers of all types, not just news.
Some commentators have been dismissive, observing the limited accuracy of the current generation of chatbots, but we are witnessing only the earliest generations emerging now. Investors and tech companies will race to establish themselves in a market that will rapidly evolve.
The data returned by chatbots now has an eerie plausibility about it, even though it does not withstand much scrutiny. This should be taken as an early warning signal that less enquiring minds will too readily accept what they are told by an AI assistant.
On the positive side, it seems likely AI has great potential to find useful information in a much more efficient way compared to traditional search results. If it removes too much context, we may easily be misled, but it will be a powerful tool to access the massive volume of information available on the internet today.
Would you like to join the conversation on the latest tech and digital developments? Potato collaborate with clients looking to launch purposeful digital products that help make everyday life better. Reach out to explore how we leverage emerging technologies to create innovative digital solutions.