Nova Spivack – Minding the Planet

Here at Radar Networks we are working on practical ways to bring the Semantic Web to end-users. One of the interesting themes that has come up a lot, both internally, as well as in discussions with VC’s, is the coming plateau in the productivity of keyword search. As the Web gets increasingly large and complex, keyword search becomes less effective as a means for making sense of it. In fact, it will even decline in productivity in the future. Natural language search will be a bit better than keyword search, but ultimately won’t solve the problem either — because like keyword search it cannot really see or make use of the structure of information.

I’ve put together a new diagram showing how the Semantic Web will enable the next step-function in productivity on the Web. It’s still a work in progress and may change frequently for a bit, so if you want to blog it, please link to this post, or at least the .JPG image behind the thumbnail below so that people get the latest image. As always your comments are appreciated. (Click the thumbnail below for a larger version).

Today a typical Google search returns up to hundreds of thousands or even millions of results — but we only really look at the first page or two of results. What about the other results we don’t look at? There is a lot of room to improve the productivity of search, and the help people deal with increasingly large collections of information.

Keyword search doesn’t understand the meaning of information, let alone its structure. Natural language search is a little better at understanding the meaning of information — but it still won’t help with the structure of information. To really improve productivity significantly as the Web scales, we will need forms of search that are data-structure-aware — that are able to search within and across data structures, not just unstructured text or semistructured HTML. This is one of the key benefits of the coming Semantic Web: it will enable the Web to be navigated and searched just like a database.

Starting with the "data web" enabled by RDF, OWL, ontologies and SPARQL, structured data is becoming increasingly accessible, searchable and mashable. This in turn sets the stage for a better form of search: semantic search. Semantic search combines the best of keyword, natural language, database and associative search capabilities together.

Without the Semantic Web, productivity will plateau and then gradually decline as the Web, desktop and enterprise continue to grow in size and complexity. I believe that with the appropriate combination of technology and user-experience we can flip this around so that productivity actually increases as the size and complexity of the Web increase.

See Also: A Visual Timeline of the Past, Present and Future of the Web