Twine has been growing at 50% per month since launch in October. We've been keeping that quiet while we wait to see if it holds. VentureBeat just noticed and did an article about it. It turns out our January numbers are higher than Compete.com estimates and February is looking strong too. We have a slew of cool viral features coming out in the next few months too as we start to integrate with other social networks. Should be an interesting season.

UPDATE: There’s already a lot of good discussion going on around this post in my public twine.

I’ve been writing about a new trend that I call “interest networking” for a while now. But I wanted to take the opportunity before the public launch of Twine on Tuesday (tomorrow) to reflect on the state of this new category of applications, which I think is quickly reaching its tipping point. The concept is starting to catch on as people reach for more depth around their online interactions.

In fact – that’s the ultimate value proposition of interest networks – they move us beyond the super poke and towards something more meaningful. In the long-term view, interest networks are about building a global knowledge commons. But in the short term, the difference between social networks and interest networks is a lot like the difference between fast food and a home-cooked meal – interest networks are all about substance.

At a time when social media fatigue is setting in, the news cycle is growing shorter and shorter, and the world is delivered to us in soundbytes and catchphrases, we crave substance. We go to great lengths in pursuit of substance. Interest networks solve this problem – they deliver substance.t

So, what is an interest network?

In short, if a social network is about who you are interested in, an interest network is about what you are interested in. It’s the logical next step.

Twine for example, is an interest network that helps you share information with friends, family, colleagues and groups, based on mutual interests. Individual “twines” are created for content around specific subjects. This content might include bookmarks, videos, photos, articles, e-mails, notes or even documents. Twines may be public or private and can serve individuals, small groups or even very large groups of members.

I have also written quite a bit about the Semantic Web and the Semantic Graph, and Tim Berners-Lee has recently started talking about what he calls the GGG (Giant Global Graph). Tim and I are in agreement that social networks merely articulate the relationships between people. Social networks do not surface the equally, if not more important, relationships between people and places, places and organizations, places and other places, organization and other organizations, organization and events, documents and documents, and so on.

This is where interest networks come in. It’s still early days to be clear, but interest networks are operating on the premise of tapping into a multi–dimensional graph that manifests the complexity and substance of our world, and delivers the best of that world to you, every day.

We’re seeing more and more companies think about how to capitalize on this trend. There are suddenly (it seems, but this category has been building for many months) lots of different services that can be viewed as interest networks in one way or another, and here are some examples:

• Twine (my site)
• Friendfeed
• Strands
• Intuu
• SocialMedian
• Zimesh
• Popego
• YourVersion

What all of these interest networks have in common is some sort of a bottom-up, user-driven crawl of the Web, which is the way that I’ve described Twine when we get the question about how we propose to index the entire Web (the answer: we don’t.

We let our users tell us what they’re most interested in, and we follow their lead).

Most interest networks exhibit the following characteristics as well:

• They have some sort of bookmarking/submission/markup function to store and map data (often using existing metaphors, even if what’s under the hood is new)
• They also have some sort of social sharing function to provide the network benefit (this isn’t exclusive to interest networks, obviously, but it is characteristic)
• And in most cases, interest networks look to add some sort of “smarts” or “recommendations” capability to the mix (that is, you get more out than you put in)

This last bullet point is where I see next-generation interest networks really providing the most benefit over social bookmarking tools, wikis, collaboration suites and pure social networks of one kind or another.

To that end, we think that Twine is the first of a new breed of intelligent applications that really get to know you better and better over time – and that the more you use Twine, the more useful it will become. Adding your content to Twine is an investment in the future of your data, and in the future of your interests.

At first Twine begins to enrich your data with semantic tags and links to related content via our recommendations engine that learns over time. Twine also crawls any links it sees in your content and gathers related content for you automatically – adding it to your personal or group search engine for you, and further fleshing out the semantic graph of your interests which in turn results in even more relevant recommendations.

The point here is that adding content to Twine, or other next-generation interest networks, should result in increasing returns. That’s a key characteristic, in fact, of the interest networks of the future – the idea that the ratio of work (input) to utility (output) has no established ceiling.

Another key characteristic of interest networks may be in how they monetize. Instead of being advertising-driven, I think they will focus more on a marketing paradigm. They will be to marketing what search engines were to advertising. For example, Twine will be monetizing our rich model of individual and group interests, using our recommendation engine. When we roll this capability out in 2009, we will deliver extremely relevant, useful content, products and offers directly to users who have demonstrated they are really interested in such information, according to their established and ongoing preferences.

6 months ago, you could not really prove that “interest networking” was a trend, and certainly it wasn’t a clearly defined space. It was just an idea, and a goal. But like I said, I think that we’re at a tipping point, where the technology is getting to a point at which we can deliver greater substance to the user, and where the culture is starting to crave exactly this kind of service as a way of making the Web meaningful again.

I think that interest networks are a huge market opportunity for many startups thinking about what the future of the Web will be like, and I think that we’ll start to see the term used more and more widely. We may even start to see some attention from analysts — Carla, Jeremiah, and others, are you listening?

Now, I obviously think that Twine is THE interest network of choice. After all we helped to define the category, and we’re using the Semantic Web to do it. There’s a lot of potential in our engine and our application, and the growing community of passionate users we’ve attracted.

Our 1.0 release really focuses on UE/usability, which was a huge goal for us based on user feedback from our private beta, which began in March of this year. I’ll do another post soon talking about what’s new in Twine. But our TOS (time on site) at 6 minutes/user (all time) and 12 minutes/user (over the last month) is something that the team here is most proud of – it tells us that Twine is sticky, and that “the dogs are eating the dog food.”

Now that anyone can join, it will be fun and gratifying to watch Twine grow.

Still, there is a lot more to come, and in 2009 our focus is going to shift back to extending our Semantic Web platform and turning on more of the next-generation intelligence that we’ve been building along the way. We’re going to take interest networking to a whole new level.

Stay tuned!

This is an older version of this article. The most recent version is located here:

http://www.readwriteweb.com/archives/future_of_the_desktop.php

—————

I have spent the last year really thinking about the future of the Web. But lately I have been thinking more about the future of the desktop. In particular, here are some questions I am thinking about and some answers I’ve come up so far.

(Author’s Note: This is a raw, first-draft of what I think it will be like. Please forgive any typos — I am still working on this and editing it…)

What Will Happen to the Desktop?

As we enter the third decade of the Web we are seeing an increasing shift from local desktop applications towards Web-hosted software-as-a-service (SaaS). The full range of standard desktop office tools (word processors, spreadsheets, presentation tools, databases, project management, drawing tools, and more) can now be accessed as Web-hosted apps within the browser. The same is true for an increasing range of enterprise applications. This process seems to be accelerating.

As more kinds of applications become available in Web-based form, the Web browser is becoming the primary framework in which end-users work and interact. But what will happen to the desktop? Will it too eventually become a Web-hosted application? Will the Web browser swallow up the desktop? Where is the desktop headed?

Is the desktop of the future going to just be a web-hosted version of the same old-fashioned desktop metaphors we have today?

No. There have already been several attempts at doing this — and they never catch on. People don’t want to manage all their information on the Web in the same interface they use to manage data and apps on their local PC.

Partly this is due to the difference in user experience between using files and folders on a local machine and doing that in “simulated” fashion via some Flash-based or HTML-based imitation of a desktop. Imitations desktops to-date have simply been clunky and slow imitations of the real-thing at best. Others have been overly slick. But one thing they all have in common: None of them have nailed it. The desktop of the future – what some have called “the Webtop” – still has yet to be invented.

It’s going to be a hosted web service

Is the desktop even going to exist anymore as the Web becomes increasingly important? Yes, there will have to be some kind of interface that we consider to be our personal “home” and “workspace” — but ultimately it will have to be a unified space that all our devices connect to and share. This requires that it be a hosted online service.

Currently we have different information spaces on different devices (laptop, mobile device, PC). These will merge. Native local clients could be created for various devices, but ultimately the simplest and therefore most likely choice is to just use the browser as the client. This coming “Webtop” will provide an interface to your local devices, applications and information, as well as to your online life and information.

Today we think of our Web browser running inside our desktop as an applicaiton. But actually it will be the other way around in the future: Our desktop will run inside our browser as an application.

Instead of the browser running inside, or being launched from, some kind of next-generation desktop web interface technology, it’s will be the other way around: The browser will be the shell and the desktop application will run within it either as a browser add-in, or as a web-based application.

The Web 3.0 desktop is going to be completely merged with the Web — it is going to be part of the Web. In fact there may eventually be no distinction between the desktop and the Web anymore.

The focus shifts from information to attention

As our digital lives shift from being focused on the old fashioned desktop to the Web environment we will see a shift from organizing information spatially (directories, folders, desktops, etc.) to organizing information temporally (feeds, lifestreams, microblogs, timelines, etc.).

Instead of being just a directory, the desktop of the future is going to be more like a feed reader or social news site. The focus will be on keeping up with all the stuff flowing in and out of the user’s environment. The interface will be tuned to help the user understand what the trends are, rather than just on how things are organized.

The focus will be on helping the user to manage their attention rather than just their information. This is a leap to the meta-level: A second-order desktop. Instead of just being about the information (the first-order), it is going to be about what is happening with the information (the second-order).

Users are going to shift from acting as librarians to acting as daytraders.

Our digital roles are already shifting from acting as librarians to becoming more like daytraders. In the PC era we were all focused on trying to manage the stuff on our computers — in other words, we were acting as librarians. But this is going to shift. Librarians organize stuff, but daytraders are focused on discovering and keeping track of trends. It’s a very different focus and activity, and it’s what we are all moving towards.

We are already spending more of our time keeping up with change and detecting trends, than on organizing information. In the coming decade the shelf-life of information is going to become vanishingly short and the focus will shift from storage and recall to real-time filtering, trend detection and prediction.

The Webtop will be more social and will leverage and integrate collective intelligence

The Webtop is going to be more socially oriented than desktops of today — it will have built-in messaging and social networking, as well as social-media sharing, collaborative filtering, discussions, and other community features.

The social dimension of our lives is becoming perhaps our most important source of information. We get information via email from friends, family and colleagues. We get information via social networks and social media sharing services. We co-create information with others in communities.

The social dimension is also starting to play a more important role in our information management and discovery activities. Instead of those activities remaining as solitary, they are becoming more communal. For example many social bookmarking and social news sites use community sentiment and collaborative filtering to help to highlight what is most interesting, useful or important.

It’s going to have powerful semantic search and social search capabilities built-in

The Webtop is going to have more powerful search built-in. This search will combine both social and semantic search features. Users will be able to search their information and rank it by social sentiment (for example, “find documents about x and rank them by how many of my friends liked them.”)

Semantic search will enable highly granular search and navigation of information along a potentially open-ended range of properties and relationships.

For example you will be able to search in a highly structured way — for example, search for products you once bookmarked that have a price of $10.95 and are on-sale this week. Or search for documents you read which were authored by Sue and related to project X, in the last month.

The semantics of the future desktop will be open-ended. That is to say that users as well as other application and information providers will be able to extend it with custom schemas, new data types, and custom fields to any piece of information.

Interactive shared spaces instead of folders

Forget about shared folders — that is an outmoded paradigm. Instead, the  new metaphor will be interactive shared spaces.

The need for shared community space is currently being provided for online by forums, blogs, social network profile pages, wikis, and new community sites. But as we move into Web 3.0 these will be replaced by something that combines their best features into one. These next-generation shared spaces will be like blogs, wikis, communities, social networks, databases, workspaces and search engines in one.

Any group of two or more individuals will be able to participate in a shared space that connects their desktops for a particular purpose. These new shared spaces will not only provide richer semantics in the underlying data, social network, and search, but they will also enable groups to seamlessly and collectively add, organize, track, manage, discuss, distribute, and search for information of mutual interest.

The personal cloud

The future desktop will function like a “personal cloud” for users. It will connect all their identities, data, relationships, services and activities in one virtual integrated space. All incoming and outgoing activity will flow through this space. All applications and services that a user makes use of will connect to it.

The personal cloud may not have a center, but rather may be comprised of many separate sub-spaces, federated around the Web and hosted by different service-providers. Yet from an end-user perspective it will function as a seamlessly integrated service. Users will be able to see and navigate all their information and applications, as if they were in one connected space, regardless of where they are actually hosted. Users will be able to search their personal cloud from any point within it.

Open data, linked data and open-standards based semantics

The underlying data in the future desktop, and in all associated services it connects, will be represented using open-standard data formats. Not only will the data be open, but the semantics of the data – the schema – will also be defined in an open way. The emerigng Semantic Web provides a good infrastructure for enabling this to happen.

The value of open linked-data and open semantics is that data will not be held prisoner anywhere and can easily be integrated with other data.

Users will be able to seamlessly move and integrate their data, or parts of their data, in different services. This means that your Webtop might even be portable to a different competing Webtop provider someday. If and when that becomes possible, how will Webtop providers compete to add value?

It’s going to be smart

One of the most important aspects of the coming desktop is that it’s going to be smart. It’s going to learn and help users to be more productive. Artificial intelligence is one of the key ways that competing Webtop providers will differentiate their offerings.

As you use it, it’s going to learn about your interests, relationships, current activities, information and preferences. It will adaptively self-organize to help you focus your attention on what is most important to whatever context you are in.

When reading something while you are taking a trip to Milan it may organize itself to be more contextually relevant to that time, place and context. When you later return home to San Francisco it will automatically adapt and shift to your home context. When you do a lot of searches about a certain product it will realize your context and intent has to do with that product and will adapt to help you with that activity for a while, until your behavior changes.

Your desktop will actually be a semantic knowledge base on the back-end. It will encode a rich semantic graph of your information, relationships, interests, behavior and preferences. You will be able to permit other applications to access part or all of your graph to datamine it and provide you with value-added views and even automated intelligent assistance.

For example, you might allow an agent that cross-links things to see all your data: it would go and add cross links to relevant things onto all the things you have created or collected. Another agent that makes personalized buying recommendations might only get to see your shopping history across all shopping sites you use.

Your desktop may also function as a simple personal assistant at times. You will be able to converse with your desktop eventually — through a conversational agent interface. While on the road you will be able to email or SMS in questions to it and get back immediate intelligent answers. You will even be able to do this via a voice interface.

For example, you might ask, “where is my next meeting?” or “what Japanese restaurants do I like in LA?” or “What is Sue’s Smith’s phone number?” and you would get back answers. You could also command it to do things for you — like reminding you to do something, or helping you keep track of an interest, or monitoring for something and alerting you when it happens.

Because your future desktop will connect all the relationships in your digital life — relationships connecting people, information, behavior, prefences and applications — it will be the ultimate place to learn about your interests and preferences.

Federated, open policies and permissions

This rich graph of meta-data that comprises your future desktop will enable the next-generation of smart services to learn about you and help you in an incredibly personalized manner. It will also of course be rife with potential for abuse and privacy will be a major function and concern.

One of the biggest enabling technologies that will be necessary is a federated model for sharing meta-data about policies and permissions on data. Information that is considered to be personal and private in Web site X should be recognized and treated as such by other applications and websites you choose to share that information with. This will require a way for sharing meta-data about your policies and permissions between different accounts and applicaitons you use.

The semantic web provides a good infrastructure for building and deploying a decentralized framework for policy and privacy integration, but it has yet to be developed, let alone adopted. For the full vision of the future desktop to emerge a universally accepted standard for exchanging policy and permission data will be a necessary enabling technology.

Who is most likely to own the future desktop?

When I think about what the future desktop is going to look like it seems to be a convergence of several different kinds of services that we currently view as separate.

It will be hosted on the cloud and accessible across all devices. It will place more emphasis on social interaction, social filtering, and collective intelligence. It will provide a very powerful and extensible data model with support for both unstructured and arbitrarily structured information. It will enable almost peer-to-peer like search federation, yet still have a unified home page and user-experience. It will be smart and personalized. It will be highly decentralized yet will manage identity, policies and permissions in an integrated cohesive and transparent manner across services.

By cobbling together a number of different services that exist today you could build something like this in a decentralized fashion. Is that how the desktop of the future will come about? Or will it be a new application provided by one player with a lot of centralized market power? Or could an upstart suddently emerge with the key enabling technologies to make this possible? It’s hard to predict, but one thing is certain: It will be an interesting process to watch.

My company, Radar Networks, has just come out of stealth. We’ve announced what we’ve been working on all these years: It’s called Twine.com. We’re going to be showing Twine publicly for the first time at the Web 2.0 Summit tomorrow. There’s lot’s of press coming out where you can read about what we’re doing in more detail. The team is extremely psyched and we’re all working really hard right now so I’ll be brief for now. I’ll write a lot more about this later.

Twine

The Business 2.0 Article on Radar Networks and the Semantic Web just came online. It’s a huge article. In many ways it’s one of the best popular articles written about the Semantic Web in the mainstream press. It also goes into a lot of detail about what Radar Networks is working on.

One point of clarification, just in case anyone is wondering…

Web 3.0 is not just about machines — it’s actually all about humans — it leverages social networks, folksonomies, communities and social filtering AS WELL AS the Semantic Web, data mining, and artificial intelligence. The combination of the two is more powerful than either one on it’s own. Web 3.0 is Web 2.0 + 1. It’s NOT Web 2.0 – people. The "+ 1" is the
addition of software and metadata that help people and other
applications organize and make better sense of the Web. That new layer
of semantics — often called "The Semantic Web" — will add to and
build on the existing value provided by social networks, folksonomies,
and collaborative filtering that are already on the Web.

So at least here at Radar Networks, we are focusing much of our effort on facilitating people to help them help themselves, and to help each other, make sense of the Web. We leverage the amazing intelligence of the human brain, and we augment that using the Semantic Web, data mining, and artificial intelligence. We really believe that the next generation of collective intelligence is about creating systems of experts not expert systems.

Robert Scoble spent 2 hours with us looking at our app yesterday. We had a great conversation and he had many terrific ideas and suggestions for us. We are still in stealth, so we asked him to agree not say much about what we showed him yet. He blogged a very nice post about us today, providing a few hints.

We had a bunch of press hits today for my startup, Radar
Networks…

PC World  Article on  Web 3.0 and Radar Networks

Entrepreneur Magazine interview

We’re also proud to announce that Jim
Hendler, one of the founding gurus of the Semantic Web, has joined our technical advisory board.

This is just a brief post because I am actually slammed with VC meetings right now. But I wanted to congratulate our friends at Metaweb for their pre-launch announcement. My company, Radar Networks, is the only other major venture-funded play working on the Semantic Web for consumers so we are thrilled to see more action in this sector.

Metaweb and Radar Networks are working on two very different applications (fortunately!). Metaweb is essentially making the Wikipedia of the Semantic Web. Here at Radar Networks we are making something else — but equally big — and in a different category. Just as Metaweb is making a semantic analogue to something that exists and is big, so are we: but we’re more focused on the social web — we’re building something that everyone will use. But we are still in stealth so that’s all I can say for now.

This is now an exciting two-horse space. We look forward to others joining the excitement too. Web 3.0 is really taking off this year.

An interesting side note: Danny Hillis (founder of Metaweb), myself (founder of Radar Networks) and Lew Tucker (CTO of Radar Networks) all worked together at Thinking Machines (an early AI massively parallel computer company). It’s fascinating that we’ve all somehow come to think that the only practical way to move machine intelligence forward is by having us humans and applications start to employ real semantics in what we record in the digital world.

It’s been a while since I posted about what my stealth venture, Radar Networks, is working on. Lately I’ve been seeing growing buzz in the industry around the "semantics" meme — for example at the recent DEMO conference, several companies used the word "semantics" in their pitches. And of course there have been some fundings in this area in the last year, including Radar Networks and other companies.

Clearly the "semantic" sector is starting to heat up. As a result, I’ve been getting a lot of questions from reporters and VC’s about how what we are doing compares to other companies such as for example, Powerset, Textdigger, and Metaweb. There was even a rumor that we had already closed our series B round! (That rumor is not true; in fact the round hasn’t started yet, although I am getting very strong VC interest and we will start the round pretty soon).

In light of all this I thought it might be helpful to clarify what we are doing, how we understand what other leading players in this space are doing, and how we look at this sector.

Indexing the Decades of the Web

First of all, before we get started, there is one thing to clear up. The Semantic Web is part of what is being called "Web 3.0" by some, but it is in my opinion really just one of several converging technologies and trends that will define this coming era of the Web. I’ve written here about a proposed definition of Web 3.0, in more detail.

For those of you who don’t like terms like Web 2.0, and Web 3.0, I also want to mention that  I agree — we all want to avoid a rapid series of such labels or an arms-race of companies claiming to be > x.0. So I have a practical proposal: Let’s use these terms to index decades since the Web began. This is objective — we can all agree on when decades begin and end, and if we look at history each decade is characterized by various trends. 

I think this is reasonable proposal and actually useful (and also avoids endless new x.0′s being announced every year). Web 1.0 was therefore the first decade of the Web: 1990 – 2000. Web 2.0 is the second decade, 2000 – 2010. Web 3.0 is the coming third decade, 2010 – 2020 and so on. Each of these decades is (or will be) characterized by particular technology movements, themes and trends, and these indices, 1.0, 2.0, etc. are just a convenient way of referencing them. This is a useful way to discuss history, and it’s not without precedent. For example, various dynasties and historical periods are also given names and this provides shorthand way of referring to those periods and their unique flavors. To see my timeline of these decades, click here.

So with that said, what is Radar Networks actually working on? First of all, Radar Networks is still in stealth, although we are
planning to go beta in 2007. Until we get closer to launch what I can
say without an NDA is still limited. But at least I can give some
helpful hints for those who are interested. This article provides some hints, as well as what I hope is a helpful tutorial about natural language search and the Semantic Web, and how they differ. I’ll also discuss how Radar Networks compares some of the key startup ventures working with semantics in various ways today (there are many other companies in this sector — if you know of any interesting ones, please let me know in the comments; I’m starting to compile a list).

(click the link below to keep reading the rest of this article…)

Semantic Social Software: The Semantic Web for Consumers

Here at Radar Networks, we are building a next-generation Web-based
online service that will bring the Semantic Web to consumers and
professionals across the Web. This application is focused on enabling
the next generation of social software (note that
social software is not necessarily social networking — that is subset
of social software). It is an example of what "the
Intelligent Web" will be like. We are very excited about this service
and what it already does, but there’s still more to do before we
release it.

Our app is based on the Semantic Web. It will
enrich and facilitate more intelligent online relationships, community,
content, collaboration and even commerce. It will help to bring the
Semantic Web from research to reality by making it user-friendly,
accessible and most of all, directly useful and valuable, to ordinary
people. We are focused on providing value to consumers — not just developers or early-adopters. But like I said, I can’t really provide more details until we
get closer to launch.

Our Web 3.0 Applications Platform

In order to build our product we had to first build a new platform
to support the kinds of features and capabilities we designed — we
could not find any existing platform that could do what we wanted to
do. Existing platforms for the Semantic Web were too research-oriented
and did not provide the levels of scalability, performance and
ease-of-use that we required.

We have been working on this platform
over several years and several generations of our codebase. It is now
very robust and sophisticated. We believe it is also significantly more
scalable and performant than any platform we’ve seen in the Semantic
Web space to-date.

Our platform is a comprehensive, Java-based framework for semantic
web applications and services that has some similarities to Ruby on
Rails (although it is also very different from RoR and we are not going after the platform market — we’re really more focused on our application right now). Our platform also includes a lot of other technology such as our
extremely fast and scaleable storage layer for semantic data tuples,
powerful semantic query capabilities, and a range of algorithms for
analyzing data and doing intelligent things for users.

The platform
could be called a "Web 3.0" applications platform because it is
inherently based around RDF/OWL and the emerging Semantic Web. In
addition to the "Web 3.0" aspects of what we are doing, our platform
also makes heavy use of "Web 2.0" methods and technologies such as
AJAX, REST, widgets, and RSS/ATOM, to name a few.

What We are Not Doing: Natural Language Search

First of all, we at Radar Networks are NOT building a new search engine to compete with Google, like Powerset and TextDigger are doing — so we’re not competing with them. Companies like Powerset and TextDigger are working on natural language search. Natural language search is not equivalent to the Semantic Web, although the Semantic Web can certainly help that process.

Companies working specifically on natural language search are making
use of semantics, but at the word-level only. They use networks of words that are linked to synonyms, antonyms, homonyms and other variations. These are sometimes called semantic networks. Based on these networks of word meanings, they can understand the meaning of various words and expressions.

More sophisticated natural language search algorithms don’t just look at the words alone, they look at them in context, by analyzing the grammar and the rest of the content around them. The point of natural language search is ultimately to try to match the meaning of words in search queries to the content of various documents — and to do this better than Google, which basically just matches keywords without paying attention to the meaning of the words.

Essentially natural language search requires at least some level of artificial intelligence. Machine
understanding of natural language is a difficult problem and there has
been a lot of work on this over the last few decades. Today there are
many technologies that focus on this but the majority of them are based
on the assumption that software should do all the work to figure out
the meaning of information.

What We Are Doing: Semantic Web

In contrast to natural language search which focuses on trying to derive the meanings of words, the approach of the emerging Semantic Web makes use of metadata to encode the meaning of information.

In this approach, the meaning of the information can be explicitly
coded into the information just as HTML codes are added into content today — and this can be done by people or software, and even by communities. Once this meaning — or semantics — is explicitly encoded into content, it can then be re-used by other applications to make sense of the content. It’s worth noting that explicit semantics in content can also help natural language processing apps, as well as apps that don’t understand natural language.

In the Semantic Web approach, the meaning of the information is encoded using markup
languages such as RDF and OWL, which are W3C open standards. Words and concepts in the content of documents and data records can be marked up with RDF/OWL expressions to indicate what they mean — does a certain word or phrase such as "Lotus" for example, mean a software company, a software product, an exotic sportscar brand, or some other kind of concept? Without sophisticated natural language processing it is often difficult for software to determine this on its own. The Semantic Web provides markup codes that explicitly indicate the intended meaning of information in an unambiguous, machine-readable format.

Marking up content with additional metadata was possible before the Semantic Web using XML: you could just say <sportscar>Lotus</sportscar> but the problem is that the meaning of "sportscar" still had to be coded into applications in order for them know what it implies. With RDF/OWL that meaning can be formally encoded outside of applications in a set of definitions called an ontology. An ontology defines facts such as "a sportscar is a kind of car," "a car is a ground vehicle," "a car is a product," "a car is a device," "a sportscar is a recreational or competitive vehicle," etc.

Semantic Markup

By marking up content with OWL indicating that it is a sportscar, that meaning refers to the appropriate definitions in an ontology, from which any application that can read the ontology can then then infer these various specific intended meanings. The point here is that semantics are less ambiguous — they are explicitly encoded by the ontology which functions as a kind of more advanced data schema of sorts.

But this is really an oversimplification — OWL and ontologies can actually go a lot further than just defining the meaning of concepts — they can also define their logical relations as well. For example, how exactly are two things connected and are there any special restrictions on that connection? For instance, an ontology can define that a person’s sister must be female, or that a person can only have 1 biological mother, etc.

All kinds of apps can benefit from the extra hints about the meaning of
the information that can be provided by Semantic Web metadata around
content. For example, even a natural language search engine could do
less analysis and would need less intelligence, if it could leverage
existing semantic metadata that was already in content.

It’s important to note that applications and people don’t have to necessarily ever look at RDF or
OWL code (thank heavens!) — they can just work with objects and forms
like they already do on the Web and the underlying markup can be
created automatically for them. Nobody should have to look at raw RDF and OWL (unless they really want to), and the Semantic Web doesn’t force anyone to. For example, most of us don’t write HTML or XML or CSS by hand — but if we are using blogs or wikis or even posting listing on sites like job boards and auctions, we we are doing things that result in HTML, XML and CSS being created.

It should be clear from the above section that natural language search is a specific process that makes use of word-level semantics, but the Semantic Web is a broad set of technologies for defining the meaning of any kind of information (including, but not limited to words). The Semantic Web can help improve the process of natural language search, but today many natural language search algorithms do not make use of the Semantic Web or RDF/OWL data structures. However, as these technologies begin to converge (as they are here at Radar Networks, in fact) we will see new levels of accuracy become possible — the combination of traditional natural language processing and the richer semantics of RDF/OWL markup enables even more powerful machine-understanding and processing of text. That said, once again, I want to be clear that Radar Networks is not a search company — although we do use next-generation semantic search quite extensively in our application and platform.

Any application that can understand RDF/OWL can correctly interpret
the meaning of any content that is marked up with RDF/OWL metadata. If
a news article that mentions "Paris" many times is marked up with
RDF/OWL metadata then any app that can understand that metadata can for
example, correctly determine that the article is about the place Paris,
Texas, not the place Paris, France, and not the person Paris Hilton
either. The application doesn’t have to do any fancy natural language
processing to know this. Even a relatively "dumb" application that has
no ability to do natural language processing can still make sense of a
document if it can at least understand RDF/OWL.

So how does this explicit semantic markup in the form of
RDF/OWL metadata get into the document in the first place? Well it could have been added
automatically by some other software app that did  natural language
processing on it, or it could have resulted from newspaper editors and/or
even readers categorizing and/or tagging the document with tags for places, people, etc. in
a manner not unlike how they tag content in services like Flickr today.

The main point here is that adding the semantic metadata does not require
the apps that create or consume consume the content to understand natural language, nor does it require people to be XML coders — even regular end-users can help to define
the semantics of content by simply tagging it. The Semantic Web provides a much
richer and more expressive framework for doing this than is currently available in Web 2.0 "tags," but it’s not that far off either.

The Semantic Web can enhance word-level understanding and processing
of text in many ways, but note that it is not limited only to
word-level applications. The Semantic Web provides a way to make any
information more understandable to other applications — including data
records in databases, documents on the desktop and the Web, enterprise
data, photos, videos, music, and even Web services and software code.

Simple Examples of Semantics

For example, today there is a big problem in integrating data across
applications. In the enterprise for example, one application might define a record called a "Customer"
while another might call that concept by the term "Client." If a user
then searches for "Customers" they won’t necessarily also find records
for Clients. But using the Semantic Web the data records for Customers
and Clients can be mapped together so that applications can treat them
as equivalent. Any search for one will return the other as well. Not only can records be mapped to each other, but also the fields of those records can be
mapped together. For instance, the Customer record might have a field
named "Referred by" while the Client record might have a field called
"Introduced by" — these can be mapped together as well.

A similar example could apply to a consumer use-case — for example shopping: different stores describe the same product differently — with different terms. In one store a laptop is called a "laptop computer" and in another it is called a "portable computer," while another calls it a "desktop replacement." A search for any of these terms should return products that use any of these. Within a single commerce site this is not so hard, but what about searching across many commerce sites (which isn’t really even that easy to do at all today…)? If different commerce sites used the same underlying semantic metadata definitions to markup their various products, then users could search across their products with less trial-and-error, and they would get better results.

Of course the
technology for mapping between databases is not new — there are many
ways to do this — but the Semantic Web provides a way to do it that
may be more open and efficient in the long-run. Central to this approach is that an organization or
online service can use ontologies that centrally define key
concepts in a rigorous way. So instead of every different app and data
record having to be individually mapped to every other, they can
potentially all just map to the central ontology which functions as a
kind of semantic switchboard of sorts. All applications and queries can
use a common ontology (or set of them) to unify access to data records across many
different online services and databases. In a sense ontologies provide a way to define and share common languages for data, content, relationships and applications.

SPARQL and the Emerging Data-Web

More recently a new Semantic Web technology called SPARQL has also started to
emerge. SPARQL provides a common query language, like SQL, for querying
data that is stored in RDF. Any site or database that has RDF data and
that provides a SPARQL interface can be searched by any application
that speaks SPARQL. This means that the dream of "deep web search" is
finally going to become a reality. There is a huge amount of interest
in SPARQL at the moment and there are already a growing number of
SPARQL endpoints popping up around the Web. These new SPARQL endpoints
are to data what websites were to documents. It’s the beginning of what
some call "The Data Web" — which is the first step to the full-blown
Semantic Web. SPARQL is also a big piece of what we are doing.

Reasoning: The Next Frontier After Search

Another key benefit of using RDF/OWL is that these languages are
designed to support formal logical reasoning. By marking up information
with RDF/OWL sophisticated search and inferencing can then take place
around it. For example, by marking up various people and their social
connections it is then possible to infer for example, that Sue is
Jane’s cousin, that Bob and Dave are colleagues, and that product A is
incompatible with product B, etc.

This kind of logical reasoning and
inference is essential to enable the next-generation of the Web — an
Intelligent Web — where software and online services start to help
people work, communicate, socialize and shop more productively. For
example it will enable something beyond search — it will enable
services that provide answers or suggestions. This is not necessarily important for all applications today, but it
will become increasingly important in the future. Content that exists
in RDF/OWL essentially has a longer shelf-life and will be easier to
reuse, integrate and reason across in the future.

Differentiating The Players

The Semantic Web provides a comprehensive and growing framework of
technologies that enable the next evolution of the Web — it is therefore a
much broader and farther-reaching vision than natural language search,
even though that is certainly one area that it will benefit. Natural language search is really just about matching search queries to documents, by analyzing the meaning of words. The Semantic Web is about defining the meaning of data — any data — words, data records, documents, social relationships, product listings, etc. — and providing a way to query that data, integrate it, and reason across it.

In our own
application and platform we make use of a lot of natural-language
processing (NLP) and we also provide semantic search capabilities, but
our focus is on something quite different than searching the Web — yet
equally useful and important to everyone. Frankly, I’m glad we are not
working on search, as big an opportunity as that is — I think
competing directly with Google is a daunting task and not one I would
want to be on! Instead, we are providing a new environment in which
people can start to benefit from the power of the Semantic Web in areas
that Google is very weak in today or is not in at all in some cases;
it’s really quite orthogonal to Google and other search engines.

So from the above conversation it should be clear that we are
working on The Semantic Web, not just natural language search and
so we are quite different from companies like Powerset, Textdigger and
others who are working on word-level semantic understanding of text.
But what about Metaweb — how do
we differ from them? — Well from what we can glean so far, what we are
doing is also very different from them as well but perhaps not as
different as we are from Powerset.

Radar Networks and Metaweb are
frequently cited as the two main startups working to bring
semantically-driven Web 3.0 online services to consumers. My guess is
that there will be some similarities but even more differences. There
may even be opportunities for us to work together someday. But we’re
all still in stealth, so it’s hard to get very specific about our
similarities and differences today. One thing is for sure, 2007 is
going to be an exciting year for both our companies, and for the
emerging Web 3.0 generation of companies and products.

Web 3.0 is just beginning

In any case the next-evolution of the Web — what we call "The
Intelligent Web" (and what many are also calling "Web 3.0") is still in
the very early stages and I don’t think it will really hit big until
2010 (for a graphical timeline of how I think this will unfold, click here). In the meantime we are all putting the pieces in place.

Fortunately Web 3.0 is a big space with a lot of opportunity and there is
room for a many different players and business models to co-exist and
compete. The fact that there are now several ventures in this space is
a good thing for all of us, for as one person said to me the other day,
"a rising tide lifts all boats." I’m happy that there is enough action
for there to actually be some confusion for me to clear up! Only a year
ago it felt like we were the only commercial voice a wilderness of
academic research. Today VC’s are lining up to speak to us and the other
companies in the space, and we are literally having to keep them at bay
until we start our B round.

Solving Information Overload

The key realization behind all this recent interest in semantics is that keyword
search and traditional content and data representations are declining
in productivity. As the Web gets vaster and more complex, and as
consumers must work with a growing array of content and services,
productivity is seriously being threatened — not only in search, but
also in every other area of our digital lives. Most of us who work
intensively with knowledge and information already have a direct and
intuitive experience of how information overload has grown, even in the
last decade. Clearly something must be done about this or in another
few years we will all be buried in our own information.

The Semantic Web provides the best (and really the only) long-term
solution to information overload and complexity. By starting to add
richer semantics to data, and by enabling applications to start
leveraging this, it will make it possible to help people regain more of
their productivity and to make software smarter — without having to
attempt to create super-duper science fiction artificial intelligence.

It’s very important to keep in mind that The Semantic Web does not require that machines understand or reason as well as people — the semantics of the Semantic Web can be created by people and/or machines, and it doesn’t have to be perfect, it simply has to add hints that make content less ambiguous and more structured. By contrast, both the keyword approach of Google and the natural language search approach of companies like Powerset — if they are to keep up with the growing complexity of the Web — will require increasingly intelligent software, because basically in such systems the software has to do all the work by itself.

The Semantic Web actually is really more about leveraging the
collective intelligence of people and applications to enrich content — rather than trying
to make applications do all the work on their own — but this will be a lot clearer later in the process when there are several Semantic Web apps that demonstrate this.

Here at Radar Networks we have
been working towards this vision steadily — and we’re proud of the fact that we started working with semantics long before it was "cool" — we know this space inside out
and we think that our first application on our platform will be an "Aha
experience" for users.

It certainly has taken some time to bring the Semantic Web to
fruition, but when you think about it, Web 1.0 took about 5 years to
really get started, so it’s not without precedent. A new generation of
the Web is a big undertaking. For now, all of us working on anything having to do with "semantics"
or Web 3.0 need to work together to start mapping out this space and educating the marketplace so
that people (including the press and VC’s, and early-adopters) can
understand the companies and technologies more clearly. The rather
humorous irony for all of us, is that the meaning of the term "semantic" is still so
ambiguous today!

NOTES

• Master Copy can be found at this URL or http://tinyurl.com/yynb93
• Last Update: Tuesday, November 7, 2006, 10:17AM PST
• License — This article is distributed under the Creative Commons Deed. If you would like to distribute a version of thisarticle, please link back to http://www.mindingtheplanet.net from yourversion, thanks.
• Printable version — Click here to download the printable PDF version of this article
• Illustrated Version — See the version by the Lifeboat Foundation

Prelude

Many years ago, in the late 1980s, while I was still a college student, I visited my late grandfather, Peter F. Drucker, at his home in Claremont, California. He lived near the campus of Claremont College where he was a professor emeritus. On that particular day, I handed him a manuscript of a book I was trying to write, entitled, “Minding the Planet” about how the Internet would enable the evolution of higher forms of collective intelligence.

My grandfather read my manuscript and later that afternoon we sat together on the outside back porch and he said to me, “One thing is certain: Someday, you will write this book.” We both knew that the manuscript I had handed him was not that book, a fact that was later verified when I tried to get it published. I gave up for a while and focused on college, where I was studying philosophy with a focus on artificial intelligence. And soon I started working in the fields of artificial intelligence and supercomputing at companies like Kurzweil, Thinking Machines, and Individual.

A few years later, I co-founded one of the early Web companies, EarthWeb, where among other things we built many of the first large commercial Websites and later helped to pioneer Java by creating several large knowledge-sharing communities for software developers. Along the way I continued to think about collective intelligence. EarthWeb and the first wave of the Web came and went. But this interest and vision continued to grow. In 2000 I started researching the necessary technologies to begin building a more intelligent Web. And eventually that led me to start my present company, Radar Networks, where we are now focused on enabling the next-generation of collective intelligence on the Web, using the new technologies of the Semantic Web.

But ever since that day on the porch with my grandfather, I remembered what he said: “Someday, you will write this book.” I’ve tried many times since then to write it. But it never came out the way I had hoped. So I tried again. Eventually I let go of the book form and created this weblog instead. And as many of my readers know, I’ve continued to write here about my observations and evolving understanding of this idea over the years. This article is my latest installment, and I think it’s the first one that meets my own standards for what I really wanted to communicate. And so I dedicate this article to my grandfather, who inspired me to keep writing this, and who gave me his prediction that I would one day complete it.

This is an article about a new generation of technology that is sometimes called the Semantic Web, and which could also be called the Intelligent Web, or the global mind. But what is the Semantic Web, and why does it matter, and how does it enable collective intelligence? And where is this all headed? And what is the long-term far future going to be like? Is the global mind just science-fiction? Will a world that has a global mind be good place to live in, or will it be some kind of technological nightmare?

I’ve often joked that it is ironic that a term that contains theword “semantic” has such an ambiguous meaning for most people. Mostpeople just have no idea what this means, they have no context for it,it is not connected to their experience and knowledge. This is aproblem that people who are deeply immersed in the trenches of theSemantic Web have not been able to solve adequately — they have notfound the words to communicate what they can clearly see, what they areworking on, and why it matters for everyone. In this article I havetried, and hopefully succeeded, in providing a detailed introductionand context for the Semantic Web fornon-technical people. But even technical people working in the fieldmay find something of interest here as I piece together the fragmentsinto a Big Picture and a vision for what might be called “Semantic Web2.0.”

I hope the reader will bear with me as Ibounce around across different scales of technology and time, and fromthe extremes of core technology to wild speculation in order to tellthis story. If you are looking for the cold hardscience of it all, this article will provide an understanding but willnot satisfy your need for seeing the actual code; there are otherplaceswhere you can find that level of detail and rigor. But if you want tounderstand what it all really means and what the opportunity and futurelookslike – this may be what you are looking for.

I should also note that all of this is my personal view of what I’vebeen working on,and what it really means to me. It is not necessarily the official viewof the mainstream academic Semantic Web community — although there arecertainly many places where we all agree. But I’m sure that somereaders will certainly disagree or raise objections to some of myassertions, and certainly to my many far-flung speculations about thefuture. I welcome those different perspectives; we’re all trying tomake sense of this and the more of us who do that together, the more wecan collectively start to really understand it. So please feel free towrite your own vision or response, and please let me know so I can linkto it!

So with this Prelude in mind, let’s get started…

The Semantic Web Vision

The Semantic Web is a set of technologies which are designed toenable aparticular vision for the future of the Web – a future in which allknowledge exists on the Web in a format that software applications canunderstand andreason about. By making knowledge more accessible to software, softwarewillessentially become able to understand knowledge, think about knowledge,and createnew knowledge. In other words, software will be able to be moreintelligent –not as intelligent as humans perhaps, but more intelligent than say,your wordprocessor is today.

The dream of making software more intelligent has been around almost as longas software itself. And although it is taking longer to materialize than past experts hadpredicted, progress towards this goal is being steadilymade. At the same time, the shape of this dream is changing. It is becomingmore realistic and pragmatic. The original dream of artificial intelligence wasthat we would all have personal robot assistants doing all the work we don’twant to do for us. That is not the dream of the Semantic Web. Instead, today’sSemantic Web is about facilitating what humans do – it is about helping humansdo things more intelligently. It’s not a vision in which humans do nothing andsoftware does everything.

The Semantic Web vision is not just about helping software become smarter –it is about providing new technologies that enable people, groups,organizations and communities to be smarter.

For example, by providing individuals with tools that learn about what theyknow, and what they want, search can be much more accurate and productive.

Using software that is able to understand and automatically organize largecollections of knowledge, groups, organizations and communities can reachhigher levels of collective intelligence and they can cope with volumes ofinformation that are just too great for individuals or even groups tocomprehend on their own.

Another example: more efficient marketplaces can be enabled by software thatlearns about products, services, vendors, transactions and market trends andunderstands how to connect them together in optimal ways.

In short, the Semantic Web aims to make software smarter, not just for itsown sake, but in order to help make people, and groups of people, smarter. Inthe original Semantic Web vision this fact was under-emphasized, leading to theimpression that Semantic Web was only about automating the world. In fact, it isreally about facilitating the world.

The Semantic Web Opportunity

The Semantic Web is one of the most significant things to happen since theWeb itself. But it will not appear overnight. It will take decades. It willgrow in a bottom-up, grassroots, emergent, community-driven manner just likethe Web itself. Many things have to converge for this trend to really take off.

The core open standards already exist, but the necessary development tools haveto mature, the ontologies that define human knowledge have to come into beingand mature, and most importantly we need a few real “killer apps” to prove thevalue and drive adoption of the Semantic Web paradigm. The first generation ofthe Web had its Mozilla, Netscape, Internet Explorer, and Apache – and it alsohad HTML, HTTP, a bunch of good development tools, and a few killer apps andservices such as Yahoo! and thousands of popular Web sites. The same things arenecessary for the Semantic Web to take off.

And this is where we are today – this all just about to start emerging.There are several companies racing to get this technology, or applications ofit, to market in various forms. Within a year or two you will see mass-consumerSemantic Web products and services hit the market, and within 5 years therewill be at least a few “killer apps” of the Semantic Web. Ten years from nowthe Semantic Web will have spread into many of the most popular sites andapplications on the Web. Within 20 years all content and applications on theInternet will be integrated with the Semantic Web. This is a sea-change. A bigevolutionary step for the Web.

The Semantic Web is an opportunity to redefine, or perhaps to better define,all the content and applications on the Web. That’s a big opportunity. Andwithin it there are many business opportunities and a lot of money to be made. It’snot unlike the opportunity of the first generation of the Web. There areplatform opportunities, content opportunities, commerce opportunities, searchopportunities, community and social networking opportunities, and collaborationopportunities in this space. There is room for a lot of players to compete andat this point the field is wide open.

The Semantic Web is a blue ocean waiting to be explored. And like anyunexplored ocean its also has its share of reefs, pirate islands, hidden treasure, shoals,whirlpools, sea monsters and typhoons. But there are new worlds out there to be discovered,and they exert an irresistible pull on the imagination. This is an excitingfrontier – and also one fraught with hard technical and social challenges thathave yet to be solved. For early ventures in the Semantic Web arena, it’s notgoing to be easy, but the intellectual and technological challenges, and the potentialfinancial rewards, glory, and benefit to society, are worth the effort andrisk. And this is what all great technological revolutions are made of.

Semantic Web 2.0

Some people who have heard the term “Semantic Web” thrown around too muchmay think it is a buzzword, and they are right. But it is not just a buzzword –it actually has some substance behind it. That substance hasn’t emerged yet,but it will. Early critiques of the Semantic Web were right – the early visiondid not leverage concepts such as folksonomy and user-contributed content atall. But that is largely because when the Semantic Web was originally conceivedof Web 2.0 hadn’t happened yet. The early experiments that came out of researchlabs were geeky, to put it lightly, and impractical, but they are already beingfollowed up by more pragmatic, user-friendly approaches.

Today’s Semantic Web – what we might call “Semantic Web 2.0” is a kinder,gentler, more social Semantic Web. It combines the best of the original visionwith what we have all learned about social software and community in the last10 years. Although much of this is still in the lab, it is already starting totrickle out. For example, recently Yahoo! started a pilot of the Semantic Webbehind their food vertical. Other organizations are experimenting with usingSemantic Web technology in parts of their applications, or to store or mapdata. But that’s just the beginning.

The Google Factor

Entrepreneurs, venture capitalists and technologists are increasinglystarting to see these opportunities. Who will be the “Google of the SemanticWeb?” – will it be Google itself? That’s doubtful. Like any entrenchedincumbent, Google is heavily tied to a particular technology and worldview. Andin Google’s case it is anything but semantic today. It would be easier for anupstart to take this position than for Google to port their entireinfrastructure and worldview to a Semantic Web way of thinking.

If it is goingto be Google it will most likely be by acquisition rather than by internal origination. Andthis makes more sense anyway – for Google is in a position where they can just wait and buy the winner,at almost any price, rather than competing in the playing field. One thing to note however is that Google has at least one product offering that shows some potential for becoming a key part of the Semantic Web. I am speaking of Google Base, Google’s open database which is meant to be a registry for structured data so that it can be found in Google search. But Google Base does not conform to or make use of the many open standards of the Semantic Web community. That may or may not be a good thing, depending on your perspective.

Of course the downside of Google waiting to join the mainstream Semantic Web community until after the winner is announced is very large – once there is a winner it may be too late for Google to beat them. Thewinner of the Semantic Web race could very well unseat Google. The strategistsat Google are probably not yet aware of this but as soon as they seesignificant traction around a major Semantic Web play it will become of interestto them.

In any case, I think there won’t be just one winner, there will be severalmajor Semantic Web companies in the future, focusing on different parts of theopportunity. And you can be sure that if Google gets into the game, every majorportal will need to get into this space at some point or risk becomingirrelevant. There will be demand and many acquisitions. In many ways the Semantic Web will not be controlled by just one company — it will be more like a fabric that connects them all together.

Context is King — The Nature ofKnowledge

It should be clear by now that the Semantic Web is all about enablingsoftware (and people) to work with knowledge more intelligently. But what isknowledge? Knowledge is not just information. It is meaningful information – itis information plus context. For example, if I simply say the word “sem” toyou, it is just raw information, it is not knowledge. It probably has nomeaning to you other than a particular set of letters that you recognize and asound you can pronounce, and the mere fact that this information was stated byme.

But if I tell you that “sem” it is the Tibetan word for “mind” then suddenly,“sem means mind in Tibetan” to you. If I further tell you that Tibetans have about as many words for “mind” as Eskimos have for “snow,” this is further meaning. Thisis context, in other words, knowledge, about the sound “sem.” The sound is raw information. When it is given context itbecomes a word, a word that has meaning, a word that is connected to conceptsin your mind – it becomes knowledge. By connecting raw information to context,knowledge is formed.

Once you have acquired a piece of knowledge such as “sem means mind in Tibetan,” you may then also form further knowledgeabout it. For example, you may form the memory, “Nova said that ‘sem means mind in Tibetan.’” You mightalso connect the word “sem” to networks of further concepts you have about Tibet and your understanding of what the word “mind” means.

The mind is the organ of meaning – mind is where meaning is stored,interpreted and created. Meaning is not “out there” in the world, it is purelysubjective, it is purely mental. Meaning is almost equivalent to mind in fact.For the two never occur separately. Each of our individual minds has some way of internally representing meaning — when we read or hear a word that we know, our minds connect that to a network of concepts about it and at that moment it means something to us.

Digging deeper, if you are really curious,or you happen to know Greek, you may also find that a similar sound occurs inthe Greek word, sēmantikós – which means “having meaning” and in turn is the root of the English word “semantic”which means “pertaining to or arising from meaning.” That’s an odd coincidence!“Sem” occurs in Tibetan word for mind, and the English and Greek words that allrelate to the concepts of “meaning” and “mind.” Even stranger is that not only do these words have a similar sound, they have a similar meaning.

With all this knowledge at yourdisposal, when you then see the term “Semantic Web” you may be able to inferthat it has something to do with adding “meaning” to the Web. However, if youwere a Tibetan, perhaps you might instead think the term had something to dowith adding “mind” to the Web. In either case you would be right!

Discovering New Connections

We’ve discovered a new connection — namely that there is an implicit connectionbetween “sem” in Greek, English and Tibetan: they all relate to meaning andmind. It’s not a direct, explicit connection – it’s not evident unless you digfor it. But it’s a useful tidbit of knowledge once it’s found. Unlike the direct migration of the sound “sem” from Greek to English,there may not have ever been a direct transfer of this sound from Greek toSanskrit to Tibetan. But in a strange and unexpected way, they are all connected. This connectionwasn’t necessarily explicitly stated by anyone before, but was uncovered byexploring our network of concepts and making inferences.

The sequence of thought about “sem”above is quite similar to kind of intellectual reasoning and discovery that theactual Semantic Web seeks to enable software to do automatically.  How is this kind of reasoning and discovery enabled? The Semantic Web providesa set of technologies for formally defining the context of information. Just asthe Web relies on a standard formal specification for “marking up” informationwith formatting codes that enable any applications that understand those codesto format the information in the same way, the Semantic Web relies on newstandards for “marking up” information with statements about its context – itsmeaning – that enable any applications to understand, and reason about, the meaning of those statements in the same way.

By applying semantic reasoning agents to large collections of semantically enhanced content, all sorts of new connections may be inferred, leading to new knowledge, unexpected discoveries and useful additional context around content. This kind of reasoning and discovery is already taking place in fields from drug discovery and medical research, to homeland security and intelligence. The Semantic Web is not the only way to do this — but it certainly will improve the process dramatically. And of course, with this improvement will come new questions about how to assess and explain how various inferences were made, and how to protect privacy as our inferencing capabilities begin to extend across ever more sources of public and private data. I don’t have the answers to these questions, but others are working on them and I have confidence that solutions will be arrived at over time.

Smart Data

By marking up information with metadata that formally codifies its context, we can make the data itself “smarter.” The data becomes self-describing. When you get a piece of data you also get the necessary metadata for understanding it. For example, if I sent you a document containing the word “sem” in it, I could add markup around that word indicating that it is the word for “mind” in the Tibetan language.

Similarly, a document containing mentions of “Radar Networks” could contain metadata indicating that “Radar Networks” is an Internet company, not a product or a type of radar technology. A document about a person could contain semantic markup indicating that they are residents of a certain city, experts on Italian cooking, and members of a certain profession. All of this could be encoded as metadata in a form that software could easily understand. The data carries more information about its own meaning.

The alternative to smart data would be for software to actually read and understand natural language as well as humans. But that’s really hard. To correctly interpret raw natural language, software would have to be developed that knew as much as a human being. But think about how much teaching and learning is required to raise a human being to the point where they can read at an adult level. It is likely that similar training would be necessary to build software that could do that. So far that goal has not been achieved, although some attempts have been made. While decent progress in natural language understanding has been made, most software that can do this is limited around particular vertical domains, and it’s brittle — it doesn’t do a good job of making sense of terms and forms of speech that it wasn’t trained to parse and make sense of.

Instead of trying to make software a million times smarter than it is today, it is much easier to just encode more metadata about what our information means. That turns out to be less work in the end. And there’s an added benefit to this approach — the meaning exists with the data and travels with it. It is independent of any one software program — all software can access it. And because the meaning of information is stored with the information itself, rather than in the software, the software doesn’t have to be enormous to be smart. It just has to know the basic language for interpreting the semantic metadata it finds on the information it works with.

Smart data enables relatively dumb software to be smarter with less work. That’s an immediate benefit. And in the long-term as software actually gets smarter, smart data will make it easier for it to start learning and exploring on its own. So it’s a win-win approach. Start with by adding semantic metadata to data, end up with smarter software.

Making Statements About the World

Metadata comes down to making statements about the world in a manner that machines, and perhaps even humans, can understand unambiguously. The same piece of metadata should be interpreted in the same way by different applications and readers.

There are many kinds of statementsthat can be made about information to provide it with context. For example, youcan state a definition such as “person” means “a human being or a legalentity.” You can state an assertion such as “Sue is a human being.” You canstate a rule such that “if x is a human being, then x is a person.”

From thesestatements it can then be inferred that “Sue is a person.” This inference is soobvious to you and me that it seems trivial, but most software today cannot dothis. It doesn’t know what a person is, let alone what a name is. But ifsoftware could do this, then it could for example, automatically organizedocuments by the people they are related to, or discover connections betweenpeople who were mentioned in a set of documents, or it could find documentsabout people who were related to particular topics, or it could give you a listof all the people mentioned in a set of documents, or all the documents relatedto a person.

Of course this is a very basicexample. But imagine if your software didn’t just know about people – it knewabout most of the common concepts that occur in your life. Your software wouldthen be able to help you work with your documents just about as intelligentlyas you are able to do by yourself, or perhaps even more intelligently, becauseyou are just one person and you have limited time and energy but your softwarecould work all the time, and in parallel, to help you.

Examples and Benefits

How could the existence of the Semantic Web and all the semantic metadata that defines it be really useful toeveryone in the near-term?

Well, for example, the problem of email spam would finally be cured:your software would be able to look at a message and know whether it wasmeaningful and/or relevant to you or not.

Similarly, you would never have to file anything by hand again. Your software could atuomate all filing and information organization tasks for you because it would understand your information and your interests. It would be able to figure out when to file something in a single folder, multiple folders, or new ones. It would organize everything — documents, photos, contacts, bookmarks, notes, products, music, video, data records — and it would do it even better and more consistently than you could on your own. Your software wouldn’t just organize stuff, it would turn it into knowledge by connecting it to more context. It could this not just for individuals, but for groups, organizations and entire communities.

Another example: search would bevastly better: you could search conversationally by typing in everyday naturallanguage and you would get precisely what you asked for, or even what youneeded but didn’t know how to ask for correctly, and nothing else. Your searchengine could even ask you questions to help you narrow what you want. You wouldfinally be able to converse with software in ordinary speech and it would understandyou.

The process of discovery would be easier too. You could have software agent that worked as your personal recommendation agent. It would constantly be looking in all the places you read or participate in for things that are relevant to your past, present and potential future interests and needs. It could then alert you in a contextually sensitive way, knowing how to reach you and how urgently to mark things. As you gave it feedback it could learn and do a better job over time.

Going even further with this,semantically-aware software – software that is aware of context, software thatunderstands knowledge – isn’t just for helping you with your information, itcan also help to enrich and facilitate, and even partially automate, yourcommunication and commerce (when you want it to). So for example, your software could help you with your email. It would be able to recommend responses to messages for you, or automate the process. It would be able to enrich your messaging anddiscussions by automatically cross-linking what you are speaking about withrelated messages, discussions, documents, Web sites, subject categories,people, organizations, places, events, etc.

Shopping and marketplaces wouldalso become better – you could search precisely for any kind of product, withany specific attributes, and find it anywhere on the Web, in any store. You could post classified ads and automatically get relevant matches according to your priorities, from all over the Web, or only from specific places and parties that match your criteria for who you trust. You could also easily invent a new custom datastructure for posting classified ads for a new kind of product or service and publishit to the Web in a format that other Web services and applications couldimmediately mine and index without having to necessarily integrate with yoursoftware or data schema directly.

You could publish an entiredatabase to the Web and other applications and services could immediately startto integrate your data with their data, without having to migrate your schemaor their own. You could merge data from different data sources together to create new data sources without having to ever touch or look at an actual database schema.

Bumps on the Road

The above examples illustrate thepotential of the Semantic Web today, but the reality on the ground is that the technology isstill in the early phases of evolution. Even for experienced software engineersand Web developers, it is difficult to apply in practice. The main obstaclesare twofold:

(1) The Tools Problem:

There are very few commercial-gradetools for doing anything with the Semantic Web today – Most of the tools forbuilding semantically-aware applications, or for adding semantics toinformation are still in the research phase and were designed for expertcomputer scientists who specialize in knowledge representation, artificialintelligence, and machine learning.

These tools require a largelearning curve to work with and they don’t generally support large-scaleapplications – they were designed mainly to test theories and frameworks, notto actually apply them. But if the Semantic Web is ever going to becomemainstream, it has to be made easier to apply – it has to be made moreproductive and accessible for ordinary software and content developers.

Fortunately, the tools problem isalready on the verge of being solved. Companies such as my own venture, RadarNetworks, are developing the next generation of tools for building Semantic Webapplications and Semantic Web sites. These tools will hide most of thecomplexity, enabling ordinary mortals to build applications and content thatleverage the power of semantics without needing PhD’s in knowledge representation.

(2) The Ontology Problem:

The Semantic Web providesframeworks for defining systems of formally defined concepts called “ontologies,”that can then be used to connect information to context in an unambiguous way. Withoutontologies, there really can be no semantics. The ontologies ARE the semantics,they define the meanings that are so essential for connecting information tocontext.

But there are still few widely used or standardized ontologies. Andgetting people to agree on common ontologies is not generally easy. Everyonehas their own way of describing things, their own worldview, and let’s face itnobody wants to use somebody else’s worldview instead of their own.Furthermore, the world is very complex and to adequately describe all the knowledgethat comprises what is thought of as “common sense” would require a very largeontology (and in fact, such an ontology exists – it’s called Cyc and it is solarge and complex that only experts can really use it today).

Even to describe the knowledge ofjust a single vertical domain, such as medicine, is extremely challenging. Tomake matters worse, the tools for authoring ontologies are still very hard touse – one has to understand the OWL language and difficult, buggy ontologyauthoring tools in order to use them. Domain experts who are non-technical andnot trained in formal reasoning or knowledge representation may find theprocess of designing ontologies frustrating using current tools. What is needed are commercial quality tools for buildingontologies that hide the underlying complexity so that people can just pourtheir knowledge into them as easily as they speak. That’s still a ways off, butnot far off. Perhaps ten years at the most.

Of course the difficulty ofdefining ontologies would be irrelevant if the necessary ontologies alreadyexisted. Perhaps experts could define them and then everyone else could justuse them? There are numerous ontologies already in existence, both on thegeneral level as well as about specific verticals. However in my own opinion,having looked at many of them, I still haven’t found one that has the rightbalance of coverage of the necessary concepts most applications need, andaccessibility and ease-of-use by non-experts. That kind of balance is arequirement for any ontology to really go mainstream.

Furthermore, regarding the presentcrop of ontologies, what is still lacking is standardization. Ontologists havenot agreed on which ontologies to use. As a result it’s anybody’s guess whichontology to use when writing a semantic application and thus there is a highdegree of ontology diversity today. Diversity is good, but too much diversityis chaos.

Applications that use differentontologies about the same things don’t automatically interoperate unless theirontologies have been integrated. This is similar to the problem of databaseintegration in the enterprise. In order to interoperate, different applicationsthat use different data schemas for records about the same things, have to bemapped to each other somehow – either at the application-level or the data-level.This mapping can be direct or through some form of middleware.

Ontologies canbe used as a form of semantic middleware, enabling applications to be mapped atthe data-level instead of the applications-level. Ontologies can also be usedto map applications at the applications level, by making ontologies of Webservices and capabilities, by the way. This is an area in which a lot ofresearch is presently taking place.

The OWL language can expressmappings between concepts in different ontologies. But if there are manyontologies, and many of them partially overlap, it is a non-trivial task toactually make the mappings between their concepts.

Even though concept A inontology one and concept B in ontology two may have the same names, and evensome of the same properties, in the context of the rest of the concepts intheir respective ontologies they may imply very different meanings. So simplymapping them as equivalent on the basis of their names is not adequate, theirconnections to all the other concepts in their respective ontologies have to beconsidered as well. It quickly becomes complex. There are some potential waysto automate the construction of mappings between ontologies however – but theyare still experimental. Today, integrating ontologies requires the help ofexpert ontologists, and to be honest, I’m not sure even the experts have itfigured out. It’s more of an art than a science at this point.

Darwinian Selection of Ontologies

All that is needed for mainstream adoption to begin is for a largebody of mainstream content to become semantically tagged andaccessible. This will cause whatever ontology is behind that content to become popular.

When developers see that there is significant content andtraction around aparticular ontology, they will use that ontology for their ownapplicationsabout similar concepts, or at least they will do the work of mappingtheir ownontology to it, and in this way the world will converge in a Darwinianfashionaround a few main ontologies over time.

These main ontologies will then beworth thetime and effort necessary to integrate them on a semantic level,resulting in acohesive Semantic Web. We may in fact see Darwinian natural selection take place not just at the ontology level, but at the level of pieces of ontologies.

A certain ontology may do a good job of defining what a person is, while another may do a good job of defining what a company is. These definitions may be used for a lot of content, and gradually they will become common parts of an emergent meta-ontology comprised of the most-popular pieces from thousands of ontologies. This could be great or it could be a total mess. Nobody knows yet. It’s a subject for further research.

Making Sense of Ontologies

Since ontologies are so important,it is helpful to actually understand what an ontology really is, and what itlooks like. An ontology is a system of formally defined related concepts. Forexample, a simple ontology is this set of statements such as this:

A human is a living thing.

A person is a human.

A person may have a first name.

A person may have a last name.

A person must have one and only onedate of birth.

A person must have a gender.

A person may be socially related toanother person.

A friendship is a kind of socialrelationship.

A romantic relationship is a kindof friendship.

A marriage is a kind of romanticrelationship.

A person may be in a marriage withonly one other person at a time.

A person may be employed by anemployer.

An employer may be a person or anorganization.

An organization is a group ofpeople.

An organization may have a productor a service.

A company is a type organization.

We’ve just built a simple ontologyabout a few concepts: humans, living things, persons, names, socialrelationships, marriages, employment, employers, organizations, groups,products and services. Within this system of concepts there is particular logic,some constraints, and some structure. It may or may not correspond to yourworldview, but it is a worldview that is unambiguously defined, can becommunicated, and is internally logically consistent, and that is what isimportant.

The Semantic Web approach providesan open-standard language, OWL, for defining ontologies. OWL also provides fora way to define instances of ontologies. Instances are assertions within theworldview that a given ontology provides. In other words OWL provides a meansto make statements that connect information to the ontology so that softwarecan understand its meaning unambiguously. For example, below is a set ofstatements based on the above ontology:

There exists a person x.

Person x has a first name “Sue”

Person x  has a last name “Smith”

Person x has a full name “Sue Smith”

Sue Smith was born on June 1, 2005

Sue Smith has a gender: female

Sue Smith has a friend: Jane, who isanother person.

Sue Smith is married to: Bob, anotherperson.

Sue Smith is employed by Acme, Inc, a company

Acme Inc. has a product, Widget2.0.

The set of statements above, plusthe ontology they are connected to, collectively comprise a knowledge basethat, if represented formally in the OWL markup language, could be understoodby any application that speaks OWL in the precise manner that it was intendedto be understood.

Making Metadata

The OWL language provides a way tomarkup any information such as a data record, an email message or a Web pagewith metadata in the form of statements that link particular words or phrasesto concepts in the ontology. When software applications that understand OWLencounter the information they can then reference the ontology and figure outexactly what the information means – or at least what the ontology says that itmeans.

But something has to add thesesemantic metadata statements to the information – and if it doesn’t add them or adds thewrong ones, then software applications that look at the information will getthe wrong idea. And this is another challenge – how will all this metadata getcreated and added into content? People certainly aren’t going to add it all byhand!

Fortunately there are many ways tomake this easier. The best approach is to automate it using special softwarethat goes through information, analyzes the meaning and adds semantic metadataautomatically. This works today, but the software has to be trained or providedwith rules and that takes some time. It also doesn’t scale cost-effectively tovast data-sets.

Alternatively, individuals can beprovided with ways to add semantics themselves as they author information. Whenyou post your resume in a semantically-aware job board, you could fill out aform about each of your past jobs, and the job board would connect that data toappropriate semantic concepts in an underlying employment ontology. As anend-user you would just fill out a form like you are used to doing;under-the-hood the job board would add the semantics for you.

Another approach is to leveragecommunities to get the semantics. We already see communities that are addingbasic metadata “tags” to photos, news articles and maps. Already a few simpletypes of tags are being used pseudo-semantically: subject tags and geographicaltags. These are primitive forms of semantic metadata. Although they are notexpressed in OWL or connected to formal ontologies, they are at leastsemantically typed with prefixes or by being entered into fields or specificnamespaces that define their types.

Tagging by Example

There may also be another solution to the problem of how to add semantics to content in the not to distant future. Once asuitable amount of content has been marked up with semantic metadata,it may be possible, through purely statistical forms of machinelearning, for software to begin to learn how to do a pretty good job ofmarking up new content with semantic metadata.

For example, if thestring “Nova Spivack” is often marked up with semantic metadata statingthat it indicates a person, and not just any person but a specificperson that is abstractly represented in a knowledge base somewhere,then when software applications encounter a new non-semanticallyenhanced document containing strings such as “Nova Spivack” or”Spivack, Nova” they can make a reasonably good guess that thisindicates that same specific person, and they can add the necessarysemantic metadata to that effect automatically.

As more and more semanticmetadata is added to the Web and made accessible it constitutes a statisticaltraining set that can be learned and generalized from. Although humansmay need to jump-start the process with some manually semantic tagging,it might not be long before software could assist them and eventuallydo all the tagging for them. Only in special cases would software needto ask a human for assistance — for example when totally new terms orexpressions were encountered for the first several times.

The technology for doing this learning already exists — and actually it’s not very different from how search engines like Google measure the community sentiment around web pages. Each time something is semantically tagged with a certain meaning that constitutes a “vote” for it having that meaning. The meaning that gets the most votes wins. It’s an elegant, Darwinian, emergent approach to learning how to automatically tag the Web.

One this is certain, if communities were able to tagthings with more types of tags, and these tags were connected to ontologies andknowledge bases, that would result in a lot of semantic metadata being added tocontent in a completely bottom-up, grassroots manner, and this in turn would enable this process to start to become automated or at least machine-augmented.

Getting the Process Started

But making the userexperience of semantic tagging easy (and immediately beneficial) enough that regular people will do it, is a challenge that has yet to be solved.However, it will be solved shortly. It has to be. And many companies andresearchers know this and are working on it right now. This does have to be solved to get the process of jump-starting the Semantic Web started.

I believe that the Tools Problem – the lack of commercial grade tools forbuilding semantic applications – is essentially solved already (although theproducts have not hit the market yet; they will within a few years at most).The Ontology Problem is further from being solved. I think the way this problemwill be solved is through a few “killer apps” that result in the building up ofa large amount of content around particular ontologies within particular onlineservices.

Where might we see this content initially arising? In my opinion it will most likely be within vertical communities of interest, communities of practice, and communities of purpose. Within such communities there is a need to create a common body of knowledge and to make that knowledge more accessible, connected and useful.

The Semantic Web can really improve the quality of knowledge and user-experience within these domains. Because they are communities, not just static content services, these organizations are driven by user-contributed content — users play a key role in building content and tagging it. We already see this process starting to take place in communities such as Flickr, del.icio.us, the Wikipedia and Digg. We know that communities of people do tag content, and consume tagged content, if it is easy and beneficial enough for to them to do so.

In the near future we may see miniature Semantic Webs arising around particular places, topics and subject areas, projects, and other organizations. Or perhaps, like almost every form of new media in recent times, we may see early adoption of the Semantic Web around online porn — what might be called “the sementic web.”

Whether you like it or not, it is a fact that pornography was one of the biggest drivers of early mainstream adoption of personal video technology, CD-ROMs, and also of the Internet and the Web.

But I think it probably is not necessary this time around. While, I’m sure that the so-called “sementic web” could become better from the Semantic Web, it isn’t going to be the primary driver of adoption of the Semantic Web. That’s probably a good thing — the world can just skip over that phase of development and benefit from this technology with both hands so to speak.

The World Wide Database

In some ways one could think of theSemantic Web as “the world wide database” – it does for the meaning of data records what theWeb did for the formatting documents. But that’s just the beginning. It actually turnsdocuments into richer data records. It turns unstructured data into structureddata. All data becomes structured data in fact. The structure is not merelydefined structurally, but it is defined semantically.

In other words, it’s notmerely that for example, a data record or document can be defined in such a wayas to specify that it contains a certain field of data with a certain label ata certain location – it defines what that field of data actually means in anunambiguous, machine understandable way. If all you want is a Web of data,XML is good enough. But if you want to make that data interoperable and machineunderstandable then you need RDF and OWL – the Semantic Web.

Like any database,the Semantic Web, or rather the myriad mini-semantic-webs that will comprise it,have to overcome the challenge of data integration. Ontologies provide a betterway to describe and map data, but the data still has to be described andmapped, and this does take some work. It’s not a magic bullet.

The Semantic Webmakes it easier to integrate data, but it doesn’t completely remove the dataintegration problem altogether. I think the eventual solution to this problemwill combine technology and community folksonomy oriented approaches.

The Semantic Web in HistoricalContext

Let’s transition now and zoom out to see the bigger picture. The Semantic Webprovides technologies for representing and sharing knowledge in new ways. Inparticular, it makes knowledge more accessible to software, and thus to otherpeople. Another way of saying this is that it liberates knowledge fromparticular human minds and organizations – it provides a way to make knowledgeexplicit, in a standardized format that any application can understand. This isquite significant. Let’s put this in historical perspective.

Before the invention of the printing press, there were two ways to spreadknowledge – one was orally, the other was in some symbolic form such as art orwritten manuscripts. The oral transmission of knowledge had limited range and ahigh error-rate, and the only way to learn something was to meet someone whoknew it and get them to tell you. The other option, symbolic communicationthrough art and writing, provided a means to communicate knowledgeindependently of particular people – but it was only feasible to produce a fewcopies of any given artwork or manuscript because they had to be copied byhand. So the transmission of knowledge was limited to small groups or at leastsmall audiences. Basically, the only way to get access to this knowledge was tobe one of the lucky few who could acquire one of its rare physical copies.

The invention of the printing press changed this – for the first timeknowledge could be rapidly and cost-effectively mass-produced and mass-distributed.Printing made it possible to share knowledge with ever-larger audiences. Thisenabled a huge transformation for human knowledge, society, government,technology – really every area of human life was transformed by thisinnovation.

The World Wide Web made the replication and distribution of knowledge eveneasier – With the Web you don’t even have to physically print or distributeknowledge anymore, the cost of distribution is effectively zero, and everyonehas instant access to everything from anywhere, anytime. That’s a lot betterthan having to lug around a stack of physical books. Everyone potentially haswhatever knowledge they need with no physical barriers. This has been anotherhuge transformation for humanity – and it has affected every area of humanlife. Like the printing press, the Web fundamentally changed the economics ofknowledge.

The Semantic Web is the next big step in this process – it will make all theknowledge of the human race accessible to software. For the first time,non-human things (software applications) will be able to start working withhuman knowledge to do things (for humans) on their own. This is a big leap – aleap like the emergence of a new species, or the symbiosis of two existingspecies into a new form of life.

The printing press and the Web changed the economics of replicating,distributing and accessing knowledge. The Semantic Web changes the economics ofprocessing knowledge. Unlike the printing press and the Web, the Semantic Webenables knowledge to be processed by non-human things.

In other words, humans don’t have to do all the thinking on their own, theycan be assisted by software. Of course we humans have to at least first createthe software (until we someday learn to create software that is smart enough tocreate software too), and we have to create the ontologies necessary for thesoftware to actually understand anything (until we learn to create software thatis smart enough to create ontologies too), and we have to add the semanticmetadata to our content in various ways (until our software is smart enough todo this for us, which it almost is already). But once we do the initial work ofmaking the ontologies and software, and adding semantic metadata, the systemstarts to pick up speed on its own, and over time the amount of work we humanshave to do to make it all function decreases. Eventually, once the system hasencoded enough knowledge and intelligence, it starts to function withoutneeding much help, and when it does need our help, it will simply ask us andlearn from our answers.

This may sound like science-fiction today, but in fact it a lot of this isalready built and working in the lab. The big hurdle is figuring out how to getthis technology to mass-market. That is probably as hard as inventing thetechnology in the first place. But I’m confident that someone will solve iteventually.

Once this happens the economics of processing knowledge will truly bedifferent than it is today. Instead of needing an actual real-live expert, theknowledge of that expert will be accessible to software that can act as theirproxy – and anyone will be able to access this virtual expert, anywhere,anytime. It will be like the Web – but instead of just information beingaccessible, the combined knowledge and expertise of all of humanity will alsobe accessible, and not just to people but also to software applications.

The Question of Consciousness

The Semantic Web literally enables humans to share their knowledge with eachother and with machines. It enables the virtualization of human knowledge andintelligence. With respect to machines, in doing this, it will lend machines“minds” in a certain sense – namely in that they will at least be able tocorrectly interpret the meaning of information and replicate the expertise ofexperts.

But will these machine-minds be conscious? Will they be aware of themeanings they interpret, or will they just be automatons that are simplyfollowing instructions without any awareness of the meanings they areprocessing? I doubt that software will ever be conscious, because from what Ican tell consciousness — or what might be called the sentient awareness ofawareness itself as well as other things that are sensed — is an immaterialphenomena that is as fundamental as space, time and energy — or perhaps evenmore fundamental. But this is just my personal opinion after having searchedfor consciousness through every means possible for decades. It just cannot befound to be something, yet it is definitely and undeniably taking place.

Consciousness can be exemplified through the analogy of space (but unlikespace, consciousness has this property of being aware, it’s not a mere lifelessvoid). We all agree space is there, but nobody can actually point to itsomewhere, and nobody can synthesize space. Space is immaterial andfundamental. It is primordial. So is electricity. Nobody really knows whatelectricity is ultimately, but if you build the right kind of circuit you canchannel it and we’ve learned a lot about how to do that.

Perhaps we may figure out how to channel consciousness like we channelelectricity with some sort of synthetic device someday, but I think that ishighly unlikely. I think if you really want to create consciousness it’s mucheasier and more effective to just have children. That’s something ordinarymortals can do today with the technology they were born with. Of course whenyou have children you don’t really “create” their consciousness, it seems to bethere on its own. We don’t really know what it is or where it comes from, orwhen it arises there. We know very little about consciousness today.Considering that it is the most fundamental human experience of all, it isactually surprising how little we know about it!

In any case, until we truly delve far more deeply into the nature of themind, consciousness will be barely understood or recognized, let aloneexplained or synthesized by anyone. In many eastern civilizations there aremulti-thousand year traditions that focus quite precisely on the nature ofconsciousness. The major religions have all universally concluded thatconsciousness is beyond the reach of science, beyond the reach of concepts,beyond the mind entirely. All those smart people analyzing consciousness for solong, and with such precision, and so many methods of inquiry, may have a pointworth listening to.

Whether or not machines will ever actually “know” or be capable of beingconscious of that meaning or expertise is a big debate, but at least we can allagree that they will be able to interpret the meaning of information and rulesif given the right instructions. Without having to be conscious, software willbe able to process semantics quite well — this has already been proven. It’sworking today.

While consciousness is and may always be a mystery that we cannot synthesize– the ability for software to follow instructions is an established fact. Inits most reduced form, the Semantic Web just makes it possible to providericher kinds of instructions. There’s no magic to it. Just a lot of details. Infact, to play on a famous line, “it’s semantics all the way down.”

The Semantic Web does not require that we make conscious software. It justprovides a way to make slightly more intelligent software. There’s a bigdifference. Intelligence is simply a form of information processing, for themost part. It does not require consciousness — the actual awareness of what isgoing on — which is something else altogether.

While highly intelligentsoftware may need to sense its environment and its own internal state andreason about these, it does not actually have to be conscious to do this. Theseoperations are for the most part simple procedures applied vast numbers of timeand in complex patterns. Nowhere in them is there any consciousness nor doesconsciousness suddenly emerge when suitable levels of complexity are reached.

Consciousness is something quite special and mysterious. And fortunately forhumans, it is not necessary for the creation of more intelligent software, noris it a byproduct of the creation of more intelligent software, in my opinion.

The Intelligence of the Web

So the real point of the Semantic Web is that it enables the Web to becomemore intelligent. At first this may seem like a rather outlandish statement,but in fact the Web is already becoming intelligent, even without the SemanticWeb.

Although the intelligence of the Web is not very evident at first glance,nonetheless it can be found if you look for it. This intelligence doesn’t existacross the entire Web yet, it only exists in islands that are few and farbetween compared to the vast amount of information on the Web as a whole. Butthese islands are growing, and more are appearing every year, and they arestarting to connect together. And as this happens the collective intelligenceof the Web is increasing.

Perhaps the premier example of an “island of intelligence” is theWikipedia, but there are many others: The Open Directory, portals such as Yahooand Google, vertical content providers such as CNET and WebMD, commercecommunities such as Craigslist and Amazon, content oriented communities such asLiveJournal, Slashdot, Flickr and Digg and of course the millions of discussionboards scattered around the Web, and social communities such as MySpace andFacebook. There are also large numbers of private islands of intelligence onthe Web within enterprises — for example the many online knowledge andcollaboration portals that exist within businesses, non-profits, andgovernments.

What makes these islands “intelligent” is that they are places where people(and sometimes applications as well) are able to interact with each other tohelp grow and evolve collections of knowledge. When you look at them close-upthey appear to be just like any other Web site, but when you look at what theyare doing as a whole – these services are thinking.They are learning, self-organizing, sensing their environments, interpreting,reasoning, understanding, introspecting, and building knowledge. These are theactivities of minds, of intelligent systems.

The intelligence of a system such as the Wikipedia exists on several levels– the individuals who author and edit it are intelligent, the groups that helpto manage it are intelligent, and the community as a whole – which isconstantly growing, changing, and learning – is intelligent.

Flickr and Digg also exhibit intelligence. Flickr’s growing system of tagsis the beginnings of something resembling a collective visual sense organ onthe Web. Images are perceived, stored, interpreted, and connected to conceptsand other images. This is what the human visual system does. Similarly, Digg isa community that collectively detects, focuses attention on, and interpretscurrent news. It’s not unlike a primitive collective analogue to the humanfacility for situational awareness.

There are many other examples of collective intelligence emerging on theWeb. The Semantic Web will add one more form of intelligent actor to the mix –intelligent applications. In the future, after the Wikipedia is connected tothe Semantic Web, as well as humans, it will be authored and edited by smartapplications that constantly look for new information, new connections, and newinferences to add to it.

Although the knowledge on the Web today is still mostly organized withindifferent islands of intelligence, these islands are starting to reach out andconnect together. They are forming trade-routes, connecting their economies,and learning each other’s languages and cultures. The next-step will be forthese islands of knowledge to begin to share not just content and services, butalso their knowledge — what they know about their content and services. The SemanticWeb will make this possible, by providing an open format for the representationand exchange of knowledge and expertise.

When applications integrate their content using the Semantic Web they willalso be able to integrate their context, their knowledge – this will make thecontent much more useful and the integration much deeper. For example, when anapplication imports photos from another application it will also be able toimport semantic metadata about the meaning and connections of those photos.Everything that the community and application know about the photos in theservice that provides the content (the photos) can be shared with the servicethat receives the content. Better yet, there will be no need for customapplication integration in order for this to happen: as long as both servicesconform to the open standards of the Semantic Web the knowledge is instantlyportable and reusable.

Freeing Intelligence from Silos

Today much of the real value of the Web (and in the world) is still lockedaway in the minds of individuals, the cultures of groups and organizations, andapplication-specific data-silos. The emerging Semantic Web will begin to unlockthe intelligence in these silos by making the knowledge and expertise theyrepresent more accessible and understandable.

It will free knowledge and expertise from the narrow confines of individualminds, groups and organizations, and applications, and make them not only moreinteroperable, but more portable. It will be possible for example for a personor an application to share everything they know about a subject of interest aseasily as we share documents today. In essence the Semantic Web provides acommon language (or at least a common set of languages) for sharing knowledgeand intelligence as easily as we share content today.

The Semantic Web also provides standards for searching and reasoning moreintelligently. The SPARQL query language enables any application to ask forknowledge from any other application that speaks SPARQL. Instead of merekeyword search, this enables semantic search. Applications can search forspecific types of things that have particular attributes and relationships toother things.

In addition, standards such as SWRL provide formalisms for representing andsharing axioms, or rules, as well. Rules are a particular kind of knowledge –and there is a lot of it to represent and share, for example proceduralknowledge, and logical structures about the world. An ontology provides a meansto describe the basic entities, their attributes and relations, but rulesenable you to also make logical assertions and inferences about them. Withoutgoing into a lot of detail about rules and how they work here, the importantpoint to realize is that they are also included in the framework. All forms ofknowledge can be represented by the Semantic Web.

Zooming Way, Waaaay Out

So far in this article, I’ve spenta lot of time talking about plumbing – the pipes, fluids, valves, fixtures,specifications and tools of the Semantic Web. I’ve also spent some time onillustrations of how it might be useful in the very near future to individuals,groups and organizations. But where is it heading after this? What is thelong-term potential of this and what might it mean for the human race on ahistorical time-scale?

For those of you who would prefer not to speculate, stop reading here. Forthe rest of you, I believe that the true significance of the Semantic Web, on along-term timescale is that it provides an infrastructure that will enable theevolution of increasingly sophisticated forms of collective intelligence. Ultimatelythis will result in the Web itself becoming more and more intelligent, untilone day the entire human species together with all of its software andknowledge will function as something like a single worldwide distributed mind –a global mind.

Just the like the mind of a single human individual, the global mind will bevery chaotic, yet out of that chaos will emerge cohesive patterns of thoughtand decision. Just like in an individual human mind, there will be feedbackbetween different levels of order – from individuals to groups to systems ofgroups and back down from systems of groups to groups to individuals. Becauseof these feedback loops the system will adapt to its environment, and to itsown internal state.

The coming global mind will collectively exhibit forms of cognition andbehavior that are the signs of higher-forms of intelligence. It will form andreact to concepts about its “self” – just like an individual human mind. Itwill learn and introspect and explore the universe. The thoughts it thinks maysometimes be too big for any one person to understand or even recognize them –they will be comprised of shifting patterns of millions of pieces of knowledge.

The Role of Humanity

Every person on the Internet will be a part of the global mind. Andcollectively they will function as its consciousness. I do not believe some newform of consciousness will suddenly emerge when the Web passes some thresholdof complexity. I believe that humanity IS the consciousness of the Web anduntil and unless we ever find a way to connect other lifeforms to the Web, orwe build conscious machines, humans will be the only form of consciousness ofthe Web.

When I say that humans will function as the consciousness of the Web I meanthat we will be the things in the system that know. The knowledge of theSemantic Web is what is known, but what knows that knowledge has to besomething other than knowledge. A thought is knowledge, but what knows thatthought is not knowledge, it is consciousness, whatever that is. We can figureout how to enable machines to represent and use knowledge, but we don’t knowhow to make them conscious, and we don’t have to. Because we are alreadyconscious.

As we’ve discussed earlier in this article, we don’t need conscious machines, we just need more intelligent machines.Intelligence – at least basic forms of it – does not require consciousness. It may be the case that the very highest forms of intelligence require or are capable of consciousness. This may mean that software will never achieve the highest levels of intelligence and probably guaranteesthat humans (and other conscious things) will always play a special role in theworld; a role that no computer system will be able to compete with. We providethe consciousness to the system. There may be all sorts of other intelligent,non-conscious software applications and communities on the Web; in fact therealready are, with varying degrees of intelligence. But individual humans, andgroups of humans, will be the only consciousness on the Web.

The Collective Self

Although the software of the Semantic Web will not be conscious we can say that system as a whole contains or is conscious to the extent that human consciousnesses are part of it. And like most conscious entities, it may also start to be self-conscious.

If the Web ever becomes a global mind as I am predicting, will it have a“self?” Will there be a part of the Web that functions as its central self-representation?Perhaps someone will build something like that someday, or perhaps it will evolve.Perhaps it will function by collecting reports from applications and people inreal-time – a giant collective zeitgeist.

In the early days of the Web portals such as Yahoo! provided this function — they were almost real-time maps of the Web and what was happening. Today making such a map is nearly impossible, but services such as Google Zeitgeist at least attempt to provide approximations of it. Perhaps through random sampling it can be done on a broader scale.

My guess is that the global mind will need a self-representation at somepoint. All forms of higher intelligence seem to have one. It’s necessary forunderstanding, learning and planning. It may evolve at first as a bunch ofcompeting self-representations within particular services or subsystems withinthe collective. Eventually they will converge or at least narrow down to just afew major perspectives. There may also be millions of minor perspectives thatcan be drilled down into for particular viewpoints from these top-level “portals.”

The collective self, will function much like the individual self – as amirror of sorts. Its function is simply to reflect. As soon as it exists theentire system will make a shift to a greater form of intelligence – because forthe first time it will be able to see itself, to measure itself, as a whole. Itis at this phase transition when the first truly global collective self-mirroring function evolves, that we can say that the transition from a bunch of cooperating intelligent parts toa new intelligent whole in its own right has taken place.

I think that the collective self, even if it converges on a few majorperspectives that group and summarize millions of minor perspectives, will becommunity-driven and highly decentralized. At least I hope so – because theself-concept is the most important part of any mind and it should be designedin a way that protects it from being manipulated for nefarious ends. At least Ihope that is how it is designed.

Programming the Global Mind

On the other hand, there are times when a little bit of adjustment or guidance iswarranted – just as in the case of an individual mind, the collective selfdoesn’t merely reflect, it effectively guides the interpretation of the pastand present, and planning for the future.

One way to change the direction ofthe collective mind, is to change what is appearing in the mirror of thecollective self. This is a form of programming on a vast scale – When thisprogramming is dishonest or used for negative purposes it is called “propaganda,” but there are cases whereit can be done for beneficial purposes as well. An example of this today ispublic service advertising and educational public television programming. Allforms of mass-media today are in fact collective social programming. When yourealize this it is not surprising that our present culture is violent andmessed up – just look at our mass-media!

In terms of the global mind, ideally one would hope that it would be able tolearn and improve over time. One would hope that it would not have the collective equivalent of psycho-social disorders. To facilitate this, just like any form of higherintelligence, it may need to be taught, and even parented a bit. It also mayneed a form of therapy now and then. These functions could be provided by thepeople who participate in it. Again, I believe that humans serve a vital and irreplaceablerole in this process.

How It All Might Unfold

Now how is this all going to unfold? I believe that there are a number ofkey evolutionary steps that Semantic Web will go through as the Web evolvestowards a true global mind:

1. Representing individual knowledge. The first step is to make individuals’knowledge accessible to themselves. As individuals become inundated withincreasing amounts of information, they will need better ways of managing it,keeping track of it, and re-using it. They will (or already do) need”personal knowledge management.”

2. Connecting individual knowledge. Next, once individual knowledge isrepresented, it becomes possible to start connecting it and sharing it acrossindividuals. This stage could be called “interpersonal knowledgemanagement.”

3. Representing group knowledge. Groups of individuals also need ways ofcollectively representing their knowledge, making sense of it, and growing itover time. Wikis and community portals are just the beginning. The Semantic Webwill take these “group minds” to the next level — it will make the collective knowledge ofgroups far richer and more re-usable.

4. Connecting group knowledge. This step is analogous to connectingindividual knowledge. Here, groups become able to connect their knowledge togetherto form larger collectives, and it becomes possible to more easily access andshare knowledge between different groups in very different areas of interest.

5. Representing the knowledge of the entire Web. This stage — what might becalled “the global mind” — is still in the distant future, but atthis point in the future we will begin to be able to view, search, and navigatethe knowledge of the entire Web as a whole. The distinction here is thatinstead of a collection of interoperating but separate intelligentapplications, individuals and groups, the entire Web itself will begin tofunction as one cohesive intelligent system. The crucial step that enables thisto happen is the formation of a collective self-representation. This enablesthe system to see itself as a whole for the first time.

How it May be Organized

I believe the global mind will be organized mainly in the form of bottom-up and lateral, distributed emergent computation andcommunity — but it will be facilitated by certain key top-down services thathelp to organize and make sense of it as a whole. I think this future Web willbe highly distributed, but will have certain large services within it as well– much like the human brain itself, which is organized into functionalsub-systems for processes like vision, hearing, language, planning, memory,learning, etc.

As the Web gets more complex there will come a day when nobody understandsit anymore – after that point we will probably learn more about how the Web isorganized by learning about the human mind and brain – they will be quitesimilar in my opinion. Likewise we will probably learn a tremendous amountabout the functioning of the human brain and mind by observing how the Webfunctions, grows and evolves over time, because they really are quite similarin at least an abstract sense.

The internet and its software and content is like a brain, and the state ofits software and the content is like its mind. The people on the Internet arelike its consciousness. Although these are just analogies, they are actuallyuseful, at least in helping us to envision and understand this complex system. Asthe field of general systems theory has shown us in the past, systems at verydifferent levels of scale tend to share the same basic characteristics and obeythe same basic laws of behavior. Not only that, but evolution tends to convergeon similar solutions for similar problems. So these analogies may be more thanjust rough approximations, they may be quite accurate in fact.

The future global brain will require tremendous computing and storageresources — far beyond even what Google provides today. Fortunately as Moore’s Law advances thecost of computing and storage will eventually be low enough to do thiscost-effectively. However even with much cheaper and more powerful computingresources it will still have to be a distributed system. I doubt that therewill be any central node because quite simply no central solution will be ableto keep up with all the distributed change taking place. Highly distributed problemsrequire distributed solutions and that is probably what will eventually emergeon the future Web.

Someday perhaps it will be more like a peer-to-peer network, comprised ofapplications and people who function sort of like the neurons in the human brain.Perhaps they will be connected and organized by higher-level super-peers orsuper-nodes which bring things together, make sense of what is going on andcoordinate mass collective activities. But even these higher-level serviceswill probably have to be highly distributed as well. It really will bedifficult to draw boundaries between parts of this system, they will all beconnected as an integral whole.

In fact it may look very much like a grid computing architecture – in whichall the services are dynamically distributed across all the nodes such that atany one time any node might be working on a variety of tasks for differentservices. My guess is that because this is the simplest, most fault-tolerant,and most efficient way to do mass computation, it is probably what will evolvehere on Earth.

The Ecology of Mind

Where we are today in this evolutionary process is perhaps equivalent to therise of early forms of hominids. Perhaps Austrolapithecus or Cro-Magnon, ormaybe the first Homo Sapiens. Compared to early man, the global mind is like the rise of 21^stcentury mega-cities. A lot of evolution has to happen to get there. But itprobably will happen, unless humanity self-destructs first,which I sincerely hope we somehow manage to avoid. And this brings me to afinal point. This vision of the future global mind is highly technological;however I don’t think we’ll ever accomplish it without a new focus on ecology.

Ecology probably conjures up images of hippies and biologists, or maybehippies who are biologists, or at least organic farmers, for most people, but infact it is really the science of living systems and how they work. And anysystem that includes living things is a living system. This means that the Webis a living system and the global mind will be a living system too. As a living system, the Web is an ecosystem and is alsoconnected to other ecosystems. In short, ecology is absolutely essential tomaking sense of the Web, let alone helping to grow and evolve it.

In many ways the Semantic Web and the collective minds, and the global mind,that it enables, can be seen as an ecosystem of people, applications,information and knowledge. This ecosystem is very complex, much like naturalecosystems in the physical world. An ecosystem isn’t built, it’s grown, andevolved. And similarly the Semantic Web, and the coming global mind, will notreally be built, they will be grown and evolved. The people and organizationsthat end up playing a leading role in this process will be the ones thatunderstand and adapt to the ecology most effectively.

In my opinion ecology is going to be the most important science anddiscipline of the 21^st century – it is the science of healthysystems. What nature teaches us about complex systems can be applied to everykind of system – and especially the systems we are evolving on the Web. Inorder to ever have a hope of evolving a global mind, and all the wonderfullevels of species-level collective intelligence that it will enable, we have tonot destroy the planet before we get there. Ecology is the science that cansave us, not the Semantic Web (although perhaps by improving collectiveintelligence, it can help).

Ecology is essentially the science of community – whether biological,technological or social. And community is a key part of the Semantic Web atevery level: communities of software, communities of people, and communities ofgroups. In the end the global mind is the ultimate human community. It is thereward we get for finally learning how to live together in peace and balancewith our environment.

The Necessity of Sustainability

The point of this discussion of the relevance of ecology to the future ofthe Web, and my vision for the global mind, is that I think that it is clearthat if the global mind ever emerges it will not be in a world that is anythinglike what we might imagine. It won’t be like the Borg in Star Trek, it won’t belike living inside of a machine. Humans won’t be relegated to the roles ofslaves or drones. Robots won’t be doing all the work. The entire world won’t becoated with silicon. We won’t all live in a virtual reality. It won’t be one ofthese technological dystopias.

In fact, I think the global mind can only come to pass in a much greener,more organic, healthier, more balanced and sustainable world. Because it willtake a long time for the global mind to emerge, if humanity doesn’t figure outhow to create that sort of a world, it will wipe itself out sooner or later,but certainly long before the global mind really happens. Not only that, butthe global mind will be smart by definition, and hopefully this intelligencewill extend to helping humanity manage its resources, civilizations andrelationships to the natural environment.

The Smart Environment

The global mind also needs a global body so to speak. It’s not going to bean isolated homunculus floating in a vat of liquid that replaces the physicalworld! It will be a smart environment that ubiquitously integrates with ourphysical world. We won’t have to sit in front of computers or deliberatelylogon to the network to interact with the global mind. It will be everywhere.

The global mind will be physically integrated into furniture, houses,vehicles, devices, artworks, and even the natural environment. It will sensethe state of the world and different ecosystems in real-time and alert humansand applications to emerging threats. It will also be able to allocateresources intelligently to compensate for natural disasters, storms, andenvironmental damage – much in the way that the air traffic control systemsallocates and manages airplane traffic. It won’t do it all on its own, humansand organizations will be a key part of the process.

Someday the global mind may even be physically integrated into our bodiesand brains, even down the level of our DNA. It may in fact learn how to curediseases and improve the design of the human body, extending our lives, sensorycapabilities, and cognitive abilities. We may be able to interact with it bythought alone. At that point it will become indistinguishable from a limitedfrom of omniscience, and everyone may have access to it. Although it will onlyextend to wherever humanity has a presence in the universe, within thatboundary it will know everything there is to know, and everyone will be able toknow any of it they are interested in.

Enabling a Better World

By enabling greater forms of collective intelligence to emerge we really arehelping to make a better world, a world that learns and hopefully understandsitself well enough to find a way to survive. We’re building something thatsomeday will be wonderful – far greater than any of us can imagine. We’re helpingto make the species and the whole planet more intelligent. We’re building thetools for the future of human community. And that future community, if it ever arrives,will be better, more self-aware, more sustainable than the one we live intoday.

I should also mention that knowledge is power, and power can be used forgood or evil. The Semantic Web makes knowledge more accessible. This puts more power in the hands of the many, not just the few. As long as we stick to this vision — we stick to making knowledge open and accessible, using open standards, in as distributed a fashion as we can devise, then the potential power of the Semantic Web will be protected against being coopted or controlled by the few at the expense of the many. This is where technologists really have to be socially responsible when making development decisions. It’s important that we build a more open world, not a less open world. It’s important that we build a world where knowledge, integration and unification are balanced with respect for privacy, individuality, diversity and freedom of opinion.

But I am not particularly worried that the Semantic Web and the future globalmind will be the ultimate evil – I don’t think it is likely that we will end upwith a system of total control dominated by evil masterminds with powerfulSemantic Web computer systems to do their dirty work. Statistically speaking, criminal empires don’t last very long because theyare run by criminals who tend to be very short-sighted and who also surroundthemselves with other criminals who eventually unseat them, or theyself-destruct. It’s possible that the Semantic Web, like any other technology,may be used by the bad guys to spy on citizens, manipulate the world, and doevil things. But only in the short-term.

In the long-term either our civilization will get tired of endlesssuccessions of criminal empires and realize that the only way to actuallysurvive as a species is to invent a form of government that is immune to beingtaken over by evil people and organizations, or it will self-destruct. Eitherway, that is a hurdle we have to cross before the global mind that I envisioncan ever come about. Many civilizations came before ours, and it is likely thatours will not be the last one on this planet. It may in fact be the case that adifferent form of civilization is necessary for the global mind to emerge, andis the natural byproduct of the emergence of the global mind.

We know that the global mind cannot emerge anytime soon, and therefore, ifit ever emerges then by definition it must be in the context of a civilizationthat has learned to become sustainable. A long-term sustainable civilization is a non-evil civilization. And that is why I think it is a safebet to be so optimistic about the long-term future of this trend.

Today A-List blogger and emerging "media 2.0" mogul, Om Malik, dropped by our offices to get a confidential demo of what we are building. We’ve asked Om to keep a tight lid on what we showed him, but he may be releasing at least a few hints in the near future.

Om was there in the early days of the Web and really understands the industry and the content ecosystem. I remember running into him in NYC when I was a co-founder of EarthWeb. He’s seen a lot of technologies come and go, and he has a huge knowledgebase in his head. So he was an excellent person to speak to about what we are doing.

He gave us some of the most useful user-feedback about our product that we’ve ever gotten. One of our target audiences is content creators, and what Om is building over at Gigaom is a perfect example. He is a hard-core content creator. So he really understands deeply the market pain that we are addressing. And he had some incredibly useful comments, tweaks and suggestions for us. During the meeting there were quite a few Aha’s for me personally — Several new angles and benefits of our product. Meeting with folks like Om, who represent potential users of what we are building, is really helpful to us in understanding what the needs and preferences of content creators are today. I’m really excited to start doing some design around some of the suggestions he made.

Of course, the needs of content providers are only one half of the equation. We’re also addressing the needs of content consumers with our product. In order to really solve the problems facing content creators we also have to address the problems faced by their readers. It’s a full ecosystem, a virtuous cycle — a whole new dimension of the Web.

Today our product team met with Shel Isreal to show him the alpha version of what we are building here at Radar Networks and get his feedback. Shel had a lot of good insights. We showed him our full product and explained the vision, and gave him a tour of the new dimension of the Web that we are building. We also showed him how content providers such as bloggers and other site creators, and content consumers, can benefit by joining this system. Then we asked him how he would describe it.

Shel suggested that one way to express the benefit of our product is that it helps content creators, like bloggers, become part of more conversations. "Conversation" is a key word for Shel, as many of you know. He views the Web as a network of conversations, not just a network of content. In a sense, content is a means to an end — conversation — rather than an end in itself. So from that perspective we are advancing the state-of-the-art in conversations (broadly speaking, not just in the sense of discussions, but in the sense of connecting people and information together in smarter ways). That’s an interesting take on what we are doing that I hadn’t really thought about.

Shel also suggested that even though we are still a ways from being
ready to launch the beta, he thought what we had was "so much better than anything he has seen" that we should start talking about it more — without getting into the
actual details of how we are doing it (gotta save something for later,
after all!).

I’ll explain more in future posts.

I’m very pleased to announce that two distinguished Silicon Valley veterans, Lew Tucker Ph.D. and Mike Clary, have joined Radar Networks (http://www.radarnetworks.com).

In addition, we have just launched a new version of the Radar Networks corporate website
with these details and more. It’s been a great few weeks at Radar: As
well as Lew and Mike, we’ve made a number of great new hires at other
levels of the company, including several new senior engineers, a search
architect, an additional UI designer, and our first office manager. On
top of that we’ve come up with several very interesting new algorithms
related to what we are doing over the last few weeks and our alpha is
making solid progress. We’re now around 15 people and growing and it
really feels like the company has shifted into a new stage of growth.
And we’re having a lot of fun!

Shel Israel and I just finished up working together for 10 days. I needed Shel’s perspective on what we are working on at Radar Networks. Shel lived up to his reviews as a brilliant thinker on strategic messaging, branding and positioning. So what are the 15 people at Radar Networks working on? It’s still a secret, but yes, it’s related to the Semantic Web, and yes, Shel has hinted on his blog at some of it. But it’s probably not what you think. And, no, it’s not semantic video blogging either. More hints later on. For now, if you are a blogger and you have a wish-list for what wikis or blogs could do next, feel free to submit your list in the comments on this post: I’m doing some informal market research…

[Corrected due to typo.]

Great news! Radar Networks, the venture I’ve been building, has received its first round of outside funding from Vulcan Capital. We are heavily in stealth mode.

First of all I know Clay Shirky, and he’s a good fellow. But he’s simply wrong about his claim that "tagging" (of the flavor that is appearing on del.icio.us — what I call "social tagging") is inherently better than the use of formal ontologies. Clay favors the tagging approach because it is bottom-up and emergent in nature, and he argues against ontologies because pre-specification cannot anticipate the future. But this is a simplistic view of both approaches. One could just as easily argue against tagging systems because they don’t anticipate the future — they are shortsighted, now-oriented systems that fail to capture the "big picture" or to optimally organize resources for the long-term. Their saving grace is that over time they do (hopefully) self-organize and prune out the chaff, but that depends both on the level of participation and the quality of that participation.

Tagging is certainly useful — and indeed collaborative authoring,
editing and filtering are powerful paradigms — but folksonomies (at
least present day ones) suffer from having too little formal structure
– tagging systems easily result in "metadata soup."  Ontologies are on
the other end of the spectrum — they are particulary useful for
accurately modeling the actual structure of the world, or of conceptual
domains — but admittedly in some cases their formal structure can be
overly rigid and specific. The benefit of tagging is primarily the
adaptive nature of the resulting taxonomies. The benefit of ontologies
is the rich, and unambiguous, semantics they define. Tagging systems
are useful when all that is needed is the ability to link items to
topics; ontologies are useful when what is needed is to rigorously
define or understand what is meant, or not meant, by particular
classes, fields and relationships — something that is essential for
good machine-processing of data.

One point that Clay makes, which I think is very interesting, is his
view that perhaps the world is moving from a graph-theory information
model (ontologies) to a set-theory model  (folksonomies) — but in
fact, under the surface this argument falls apart. OWL is nothing other
than a language for enabling extremely sophisticated set-theoretic
operations on information. In fact, if you actually look at the OWL
language itself, it is primarily comprised of set-theoretic statements.
I don’t really view graph-theory and set-theory as mutually exclusive
– in fact, they are highly connected, if not equivalent at a deep
level. But expressing information in graph form or set form does have
different benefits for certain types of information processing. In
particular, graphs can be beneficial when associative reasoning is
important — for example, when traversing links or networks between
nodes is key. Sets on the other hand are useful when relevance or
mutual membership are most important.

Clay discounts ontologies for many reasons. He has many arguments,
most of which have some merit, but fall short of convincing me (or
anyone in the field of knowledge representation). Indeed, tagging
systems are just special, highly simplistic cases of ontologies –
namely, they are ontologies with extremely basic semantics and almost
no constraints — they are even lower on the spectrum than taxonomies.
In fact, we could graph the spectrum of knowledge management as follows:

<—————————————————————>

Tags   Folders   Taxonomies   Databases   Ontologies

One of Clay’s early arguments against ontologies was that they are
merely systems for syllogistic logic — but in fact, that is simply not
the case. While the formal semantics of OWL doe support logical
inferencing and reasoning, that is not the only value of ontologies. In
fact, I think a much more important benefit of ontologies is simply
that they make the semantics of data structures explicit — which makes
it much easier to both process information, and integrate information
across different applications and representations. Ontologies are, in
my opinion, simply the next evolution of database schemas. Surely, Clay
would not argue that database schemas have no place in the world!

Another way of looking at ontologies and the semantic web is that
they do for the meaning of data what other markup languages have done
for the layout and structure of data. HTML provided a way to markup the
formatting of content. XML provided a way to markup the structure of
content. RDF and OWL provide a way to markup the meaning of
information. This is a logical progression, and it is something that
will really make the Web, desktop and enterprise easier to cope with.
Ontologies are not panaceas — but they are incredibly powerful when
used appropriately. And that is the operative word — they are not for
everything. Indeed, in cases where social tagging is sufficient,
ontologies may simply be overkill. But there are many, many cases where
social tagging simply does not, and cannot, have the semantic rigour
that is needed.

So what’s next? I think that ultimately we will see a synthesis of
these two approaches emerge. Imagine a folksonomy combined with an
ontology — a "folktology." In a folktology, users could
instantly propose or modify ontological classes and properties in the
same manner that they do with tags in tagging systems. The most popular
ontological constructs (the most-instantiated classes, or  slots on
classes, for example) would "rise to the top" and self-amplify, while
the less-instantiated ones would "fall to the bottom" over time. In
this way an emergent, self-organizing, and self-pruning ontology could
emerge within a community. Such a system would have the ease and
adaptability of a folksonomy plus the semantic richness and formal
structure of an ontology. I think ultimately a
<i>folktology</i> approach will be better than either
folksonomies or ontolgoies on their own.

This article proposes a design pattern for ontologies and the Semantic Web based on the concept of formally defined Roles as a means to richly express the semantics of relationships among entities in ontologies. Roles are special types of n-ary relations, and thus the use of Roles is a subset of the Semantic Web best-practices recommendation for N-Ary Relations.

The Semantic Web relies on ontologies – formal definitions of the
meaning of various concepts. For example, an ontology could define the
formal meaning of the term "Person" — specifically, that a "Person" is
a "Human" that has a "First Name" and a "Last Name" and has "Legal
Status," "Friends" and a "Gender" and many other attributes. Each of
these attributes could be further defined specifically — for example,
"a Friend" is a different Person who is "Socially-related" to the
former Person and "Has Met" that Person at least once, and "is Liked
by" and "Trusted by" that Person. Each of these predicates, such as
"Socially-related," "Trusted by," and "Has Met" may or may not be
further defined, depending on the structure of particular ontologies.

Most simple ontologies use binary relations to express predicates that connect things together. More complex and sophisticated ontologies, such as the ones I have developed for the Radar
Platform and my work with SRI and DARPA and the University of Texas Clib ontology project, instead only cast the most basic building-block
predicates with object and data type relations (in OWL). Instead, most relations (including even those that could be expressed with simple object properties) are defined using special classes called Roles. This moves much of the weight of expressing how classes interconnect from properties to Role classes.

While using Roles instead of simple object properties introduces certain minor complexities — such as the requirement to model N-ary relations, and thus Roles, such that they can be used in place of object properties to connect instances of classes — it results in even more important benefits. In particular, a major benefit is that the use of classes to represent Role relations enables far more expressive ontologies to be developed. This method is even more expressive than the potential use of additional facets on properties. While adding special additional facets to properties is certainly one way to augment the semantics of predicates, it still is not as richly expressive as simply using Role classes instead of properties for most relations. The use of Role classes enables ontolology designers to create rich ontologies of relations, such that every relation that is modeled by a Role can be formally defined as a concept with respect to other entities and relations in the ontology. In other words, it enables a much richer semantics to be defined for the domain.

I propose that the use of Role classes to define the semantics of various types of relations among entities (including among relations themselves) should be a Semantic Web Best Practice and should be adopted in all but the most simplistic ontologies. The rest of this article explains why I believe this in more detail.

By using a formal representation of Roles and their semantics, we
can create ontologies of Roles, each of which can have unique
properties, such as
relations to other Roles. This in turn will in the future make it
easier to map between different ontologies — unless the meaning of
relations is semantically defined in as formal a manner as the meaning
of classes, mapping will be quite difficult and ad-hoc.

For example, if we represent the predicate "hasFriend" using a mere
object property in OWL, it is simply a string with a meaning that could
be constrained somewhat by various OWL restrictions on its domain and
range, cardinality, and so forth. But if instead, a Role such as
"Friend of" was expressed as a class, then it could have various
properties of its own, such as "distance of relationship" (in
social degrees), "length of relationship" (in days), "Affinity level "
-  (a real number representing a measure of
how much the other party is liked by the Person occupying the Role),
"trust level" (a real number), "Context of first meeting" (an instance
of a Person, Group, Organization, Place, Project, Meeting or other
Event), etc.

Furthermore, since
"Friend of" is a Class rather than a simple property, it can be
connected to other Classes in the ontology — for example, classes defining other related Roles. For instance,
the class "Friend of" can be made a subclass of a more general class
called "Socially Related To" which itself can be a subclass of a more
general class, "Social Role" that encompasses all roles that social
agents can inhabit. "Social Role" can itself be a subclass of the most
general "Role" class that holds the basic properties of all Roles.

So for example, we might have a portion of our ontology that looks something like this class schematic:

Thing
    Agent
        Living Thing
            Human
        Software Agent
        Group
            Social Group
                Family Group
                Group of Friends
                Community
                Organization
                    Corporation
                    Team
    Living Thing
        Human
    Information Object
        Document
    Event
        Meeting
        Project
    Relation
        Action
        Hyperlink
        Role
            Information Role (may only be played by Information Objects)
                    Publication
                    Deliverable
                    Draft
                    Version
            Event Role (may only be played by Events)
                   Starting Event
                   Cause
                   Effect
                   Concluding Event
              …
            Agent Role (may only be played by Agents)
                Living Thing Role (may only be played by a Living Thing)
                    Human Social Role (may only be played by a Human)
                        Friend of/has Friend
                        Professional Role
                            Manager of/Managed by
                            Colleague of/Colleague of
                                Team-mate of/Team-mate of
                    Familial Role
                        Parent of/Child of
                            Sibling of
                    Group Agent Role
….

Note: The
properties of the class, Role should specify the basic semantics for
binding instances of other classes to and from Roles so that they can
"occupy"
those Roles such that all Things have  particular Role-properties so
that they may be either the "Source of" or "Target of" appropriate Role
instances. It should also specify additional properties that apply only
to Roles and for connecting them to possible "Inverse Role,"
"Equivalent Roles," "Non-Equivalent Roles," "Pre-Requisite Roles,"
"Compatible Roles," "Incompatible Roles," etc.  Thus for example we might have:

(Class Thing
    ….
    isRolePlayerOf (Role)
    isRoleTargetOf (Role)
    …)

(Class Role
    …
    hasRolePlayer (Thing)
    hasRoleTarget (Thing)
    hasInverseRole (Role)
    hasEquivalentRole (Role)
    hasNonEquivalentRole (Role)
    isRolePlayerOf (Role)
    isRoleTargetOf (Role)
    hasPreRequisiteRole (Role)
    hasCompatibleRole (Role)
    hasIncompatibleRole (Role)
    …)

<>

Note: In the above class and properties pseudocode, Roles can in fact play other Roles and be targets of other Roles. This is an additional benefit of using Roles that enables even more expressive power in our ontology. For example, it enables us to precisely model a statement such as, "Sue is the Manager-Of the Manager-Of Joe" by making an instance, A, of the Manager Role that hasRolePlayer = Sue and hasRoleTarget = another Manager Role instance, B, that hasRolePlayer = A, and hasRoleTarget = Joe. This kind of indirection cannot be easily modeled if we simply use properties, rather than Role classes, to represent relations. While this benefit may not be important to many applications, it is useful to have from the perspective of designing a fully expressive ontology, and in particular for text-mining and reasoning applications.

The use of Roles as a design pattern is a much richer approach to
modeling the world than
we see in simple ontologies such as FOAF ("Friend of a Friend") in
which the meaning of most predicates is not defined ontologically using
classes. By adopting the above suggested design pattern ontologies
can become richer, easier to develop and extend, and easier to reason
on and integrate in the future. It would be easy to add a Role class to FOAF, and would make the ontology far richer.

For more detailed examples of how to model Roles, as well as a terrific Upper Ontology to build on, I would suggest looking at the latest versions of the University of Texas Clib ontology, which, by the way, is open-source and based on the principles in this article. You can view all the current builds here. In particular, I would suggest looking at the OWL version, which is a scaled-down subset of the full ontology (which is in KM, a more expressive axiomatic language). I have contributed a large number of classes and relations to this version of the CLIB so feel free to ask me questions if you would like to discuss this further. Please note that this ontology is still evolving, so if you build on it, you might want to let us know and keep up with changes by checking the builds frequently.

The Ontology Problem is a fundamental challenge of the emerging Semantic Web. This problem is comprised of three key sub-problems, the Upper Ontology Problem, the Domain Ontology Problem, and the Ontology Integration Problem, described in detail below:

1. The Upper Ontology Problem

When representing the world with ontologies, we need certain basic
"building block" concepts before we can ontologically define
higher-level concepts. For example, before we can really define what we
mean by a "geographic region" we first need basic definitions of
building block concepts such as "planet," "geographic location," "set,"
"boundary," "container" and "content of," and perhaps "elevation,"
"longitude," "latitude" and so on. Until we have defined these building
blocks we cannot build a semantic definition of what a geographic
region really is. It turns out that at least when describing our
consensus reality, there is a relatively small set of building block
concepts that are needed by most ontologies. We can call a set of
building-block concepts an "Upper Ontology."

Upper Ontologies are harder to design than domain ontologies in a
certain respect — they are generally both more granular and more
macroscopic, and generally the concepts they define are more abstract
and often epistemological in nature. While someone may be a domain
expert in their own field and be able to design a fairly decent
ontology about their domain, designing a truly suitable Upper Ontology
is a different specialization altogether. This distinction is similar
to the difference between a programmer who designs compilers and
development environments, and a programmer who writes software on such
compilers and IDE’s. These two types of programmers generally have
different categories of skills and knowledge. Similarly, Upper
Ontologies and the skills needed to design them are quite different
from Mid-Level or Lower domain ontologies and the skillsets they
require.

The Upper Ontology Problem is simply that there is no
generally-accepted, comprehensive, standardized Upper Ontology in use
today. When developing a domain ontology, developers must therefore
either:

(a) Develop their own Upper Ontology first
(a big task that they shouldn’t have to undertake, and probably don’t
have time to complete),
(b) Use one of the various existing Upper
Ontologies such as SUO/SUMO, OpenCyc, or other proposed Upper
Ontologies (a choice which is difficult to make for a non-specialist
developer because they may not even know how to assess the relative
value of these different ontologies, and/or they may not have enough
knowledge about the respective languages in which various ontologies
are expressed to really understand them without extensive study first,
and worse, by choosing one such Upper Ontology all of their own
ontology’s next-level concepts will automatically become
"upper-ontology-dependent" and not necessarily compatible with the
other Upper Ontologies they did not choose),
(c) Or, finally, they
can decide to just not use an Upper Ontology (the choice made in
ontologies such as FOAF; a choice which makes things simpler for the
moment, but which also results in an "ontological light-cone" or
"ontology horizon" of sorts beyond which the concepts in the ontology
become ambiguous and essentially undefined.) None of these choices are
easy to make, nor optimal.

Domain-ontology developers should not have to worry about also
developing their own Upper Ontologies. Instead, either there should be
one truly good standard Upper Ontology, or there should at least exist
a meta-ontology that maps all the concepts in the most common Upper
Ontologies to one another so that it doesn’t matter which one is used.
But this hasn’t happened yet.

Note: A terrific
Upper Ontology that I highly recommend (disclaimer: I helped develop major parts of it), is the University of Texas Clib ontology, which, by the way, is open-source (it says GPL but will actually be LGPL soon). You can view all the current builds here. In particular, I would suggest looking at the OWL version,
which is a scaled-down subset of the full ontology (which is in KM, a
more expressive axiomatic language). I have contributed a large number
of classes and relations to this version of the CLIB so feel free to
ask me questions if you would like to discuss this further. Please note
that this ontology is still evolving, so if you build on it, you might
want to let us know and keep up with changes by checking the builds
frequently.

The Domain Ontology Problem

A few useful general-purpose mid-level and lower-level ("domain
level") ontologies exist. For example, FOAF is an ontology about people
and relationships, DOAP is a proposed ontology about projects, the
Dublin Core is an ontology of the most basic properties of library
resources, etc. There even highly detailed ontologies developed to
describe various medical domains, commerce domains and military
domains. However, it is safe to say that the vast majority of vertical
subject domains have yet to be modeled ontologically, let alone
released in an open manner.

There are simply so many knowledge niches in the world — even huge
ontologies containing tens of thousands of class definitions, such as
OpenCyc, are still relatively limited in their conceptual breadth,
depth and resolution. In order for all types of information and
knowledge to be expressible and accessible in the Semantic Web,
ontologies for all these specialized domains need to be developed and
made public ally available in some manner. Furthermore, they need to
somehow connect together via a solution to the above Upper Ontology
Problem so that they can be normalized and mapped to one another
easily. Until that happens the Semantic Web will still be incredibly
useful, but only for representing and accessing general knowledge or
working with domain-specific concepts that are defined by the small set
of currently existing domain ontologies.

The solution: More ontologies need to be created about new, vertical
domains, and mapped to common open Upper Ontologies. Easier said than
done! Before domain-ontologies will be created someone has to come up
with a compelling benefit for doing so — for example, applications or
services that make use of these domain-ontologies to solve problems
that real people actually have and need solutions to.

The Ontology Integration Problem 

As alluded to above, it is one thing to develop an ontology but
quite another to make it compatible with other existing ontologies.
This is the Ontology Integration Problem. This problem turns out to be
far more subtle than most people who currently write about the Semantic
Web have noted as of yet. Integrating ontologies is not as simple as
just mapping classes in one ontology to corresponding classes in
another ontology. Because it turns out that it is not merely the names
and properties of classes that are significant to defining their
meanings and mappings, but also their inheritance paths in their
respective ontologies. For example, consider these two ontology class
outlines:

• Ontology A
  • Thing
    • Legal Person
      • Human
      • Corporation
    • Living Thing
      • Person
    • Organization
      • Corporation
    • Professional Occupation
      • Lawyer
• Ontology B
  • Thing
    • Living Thing
      • Person
        • Legal Person
          • Lawyer
    • Non-Living Thing
      • Organization
        • Legal Organization
          • Corporation

If we mapped these ontologies to one another simply by virtue of
mapping "Person" in Ontology A to the class "Person" in Ontology B, we
would wrongly be in plying that Ontology A’s concept, "Person" is
equivalent to the Ontology B’s concept, "Person." However there is big
difference in actual meaning between what these two ontologies mean by
"Person." This difference comes from semantics implied by ontology
class inheritance differences between the two classes for "Person" in
these two ontologies. Ontology A uses "Person" to mean a human with
legal status in some legal system. Ontology B says that a Person is
simply some type of "Living Thing" but not necessarily a legal entity.
In other words, in Ontology A, "All Persons are Legal Entities" while
in Ontology B, "Some Persons may be Legal Entities," while others may
not be. Similarly, consider how to map between the concepts of "Legal Person" in the two ontologies, an even more hairy problem.

The difficulty in integrating these two ontologies is in figuring
out how to express the similarity and difference in meaning between
these two concepts of "Person." One answer is to create a new third
ontology that attempts to unify the concepts in both Ontology A and
Ontology B, which can then be used to map between them — this is a
kind of "semantic middleware" approach; it’s weakness is that it only
applies to the mapping between these two ontologies and cannot be
extended easily to map to additional ontologies, or for different
subject domains.

Another approach might be to instead develop a general semantics for
expressing inter-ontology mapping concepts — and then use this
meta-ontology to create instances that express mappings between classes
and properties in various ontologies. This approach is of particular
long-term value, however it is not simple to accomplish — gradations
in semantic intent are notoriously subtle and complex to codify, and to
my knowledge nobody has developed an ontology which attempts to
formalize them in an open, ontology-independent manner (although prior
work in OIL was in that direction).

Having worked on some very large ontology integration problems
(integrating two partially overlapping ontologies, each with several
thousand classes and properties defined, and expressed in different
ontology markup languages with different expressive power, for
example), I can tell you that the difficulty of such integration
increases exponentially to the number of concepts being integrated.

Because mapping between different ontologies is quite difficult –
even more difficult than designing new ontologies from scratch, most
ontology developers take the latter approach. Thus we have few mappings
between existing ontologies, and an increasing number of small,
non-integrated ontologies about different domains. While it is easy to
state that these various ontologies can be integrated such that they
eventually all connect together, the task of actual doing such
integration is difficult in practice. If the Semantic Web is really
going to one day "link together islands of meaning" in different places,
we must solve the Ontology Integration Problem.

The alternatives are unacceptable. If we don’t solve it, we will
either end up with: (a) lots of totally incompatible ontologies and
knowledge based on them (just more "data silos" which is precisely what
the Semantic Web was supposed to eliminate!), (b) an incomplete set of
partially-incorrect mappings between ontologies (because nobody has
time to map each ontology to every other ontology, and furthermore,
even if they tried, without adequate mapping semantics, such mappings
will contain partial-truths or even glaring errors and contradictions).

If the Ontology Integration Problem is not solved it will not be
possible to answer a semantic search query across the open Web for a
question such as "find all software products that work with Linux and
are open-source and are endorsed by people or companies I trust." Why
not? Because while there could be tons of raw RDF and OWL instance data
out on the Web that is relevant from various ontologies, unless it
either all uses the same ontology or all the ontologies that various
instances refer to are integrated, the query agent will have no way of
making sense of or normalizing the results. Of course, the query agent
could simply run the query on all data from all ontologies it knows
about, and then just present the results in a single list, sorted by
ontology — but as we’ve seen above, different ontologies might mean
different things by classes with the same names — and thus the results
returned may not really be relevant or well-ordered.

Another solution that has been proposed is to automate this process
by perhaps using learning and logic agents to analyze ontological
structures and/or the data-sets corresponding to various ontologies, in
order to automatically learn or derive rules and mappings that
integrate them. I personally doubt that the automated ontology mapping
approach will yield useful fruit anytime soon — there is still no
substitute for human domain-expertise in mapping between ontologies. It
simply requires too subtle an epistemological and semantic intelligence
for an automated program to do well.

I believe the solution will ultimately stem from a solution to the
Upper Ontology Problem — if we can solve that problem, then much of
the Ontology Integration Problem will go away as most ontologies will
automatically be inter-mapped at the Upper-Ontology Level at least. If
we had a standard Upper Ontology and furthermore, if this standard were
also to include meta-level concepts for mapping between ontologies and expressing differences in meaning between sets of classes in different ontologies,
then integrating ontolgies would certainly be easier.

Note: See Also, this related article, on how to design richer semantics using Roles as a design pattern in ontologies.

Draft 1.1 for Review (integrates some fixes from readers)
Nova Spivack (www.mindingtheplanet.net)

INTRODUCTION

This article presents some thoughts about the future of intelligence on Earth. In particular, I discuss the similarities between the Internet and the brain, and how I believe the emerging Semantic Web will make this similarity even greater.

DISTRIBUTED INTELLIGENCE

The Semantic Web enables the formal communication of a higher level of language — metalanguage. Metalanguage is language about language — language that encodes knowledge about how to interpret and use information. Metalanguages – particularly semantic metalanguages for encoding relationships between information and systems of concepts – enable a new layer of communication and processing. The combination of computing networks with semantic metalanguages represents a major leap in the history of communication and intelligence.

The invention of written language long ago changed the economics of communication by making it possible for information to be represented and shared independently of human minds. This made it less costly to develop and spread ideas widely across populations in space and time. Similarly, the emergence of software based on semantic metalanguages will dramatically change the economics not only of information distribution, but of intelligence — the act of processing and using information.

Semantic metalanguages provide a way to formally express, distribute and share the knowledge necessary to interpret and use information, independently of the human mind. In other words, they make it possible not just to write down and share information, but also to encode and share the background necessary for intelligently making use of that information. Prior to the invention of such a means to share this background knowledge about information, although information could be written and shared, the recipients of such information had to be intelligent and appropriately knowledgeable in advance in order to understand it. Semantic metalanguages remove this restriction by making it possible to distill the knowledge necessary to understand information into a form that can be shared just as easily as the information itself.

The recipients of information – whether humans or software – no longer have to know in advance (or attempt to deduce) how to interpret and use the information; this knowledge is explicitly coded in the metalanguage about the information. This is important for artificial intelligence because it means that expertise for specific domains does not have to be hard-coded into programs anymore — instead programs simply need to know how to interpret the metalanguage. By adding semantic metalanguage statements to information data becomes “smarter,” and programs can therefore become “thinner.” Once programs can speak this metalanguage they can easily import and use knowledge about any particular domain, if and when needed, so long as that knowledge is expressed in the metalanguage.

In other words, whereas basic written languages simply make raw information portable, semantic metalanguages make knowledge (conceptual systems) and even intelligence (procedures for processing knowledge) about information portable. They make it possible for knowledge and intelligence to be formally expressed, stored digitally, and shared independently of any particular minds or programs. This radically changes the economics of communicating knowledge and of accessing and training intelligence. It makes it possible for intelligence to be more quickly, easily and broadly distributed across time, space and populations of not only humans but also of software programs.

The emergence of standards for sharing semantic metalanguage statements that encode the meaning of information will catalyze a new era of distributed knowledge and intelligence on the Internet. This will effectively “make the Internet smarter.” Not just monolithic expert systems and complex neural networks, but even simple desktop programs and online software agents will begin to have access to a vast decentralized reserve of knowledge and intelligence.

The externalization, standardization and sharing of knowledge and intelligence in this manner, will make it possible for communities of humans and software agents to collaborate on cognition, not just on information. As this happens and becomes increasingly linked into our daily lives and tools, the "network effect" will deliver increasing returns. While today most of the intelligence on Earth still resides within human brains, In the near future, perhaps even within our lifetimes, the vast majority of intelligence will exist outside of human brains on the Semantic Web.

THE INTERNET IS A BRAIN AND THE WEB IS ITS MIND

Anyone familiar with the architecture and dynamics of the human nervous system cannot help but notice the striking similarity between the brain and the Internet. But is this similarity more than a coincidence – is the Internet really a brain in its own right – the brain of our planet? And is its collective behavior intelligent – does it constitute a global mind? How might this collective form of intelligence compare to that of an individual human mind, or a group of human minds?

I believe that the Internet (the hardware) is already evolving into a distributed global brain, and its ongoing activity (the software, humans and data) represents the cognitive process of an increasingly intelligent global mind. This global mind is not centrally organized or controlled, rather it is a bottom-up, emergent, self-organizing phenomenon formed from flows of trillions of information-processing events comprised of billions of independent information processors.

As with other types of emergent computing systems, for example John Conway’s familiar cellular automaton “The Game of Life,” on the Internet large scale homeostatic systems and seemingly intentional or guided information processes naturally emerge and interact within it. The emergence of sophisticated information systems does not require top-down design or control, it can happen in an evolutionary bottom-up manner as well.

Like a human brain, the Internet is a vast distributed computing network comprised of billions of interacting parallel processors. These processors include individual human beings as well as software programs, and systems of them such as organizations, which can all be referred to as "agents" in this system. Just as the computational power of the human brain as a whole is vastly greater than that of any of the individual neurons or systems within it, the computational power of the Internet is vastly beyond any of the individual agents it contains. Just as the human brain is not merely the sum of its parts, the Internet is more than the sum of its parts – like other types of distributed emergent computing systems, it benefits from the network effect. The power of the system grows exponentially as agents and connections between them are added.

The human brain is enabled by an infrastructure comprised of networks of organic neurons, dendrites, synapses and protocols for processing chemical and electrical messages. The Internet is enabled by an infrastructure of synthetic computers, communications networks, interfaces, and protocols for processing digital information structures. The Internet also interfaces with organic components however – the human beings who are connected to it. In that sense the Internet is not merely an inorganic system – it could not function without help from humans, for the moment at least. The Internet may not be organized in exactly the same form as the human brain, but it is at least safe to say it is an extension of it.

The brain provides a memory system for storing, locating and recalling information. The Internet also provides shared address spaces and protocols for using them. This enables agents to participate in collaborative cognition in a completely decentralized manner. It also provides a standardized shared environment in which information may be stored, addressed and retrieved by any agent of the system. This shared information space functions as the collective memory of the global mind.

Just as no individual neuron in the human brain could be said to have the same form or degree of intelligence as the brain as-a-whole – we individual humans cannot possibly comprehend the distributed intelligence that is evolving on the Internet. But we are part of it nonetheless, whether we know it or not.  The global mind is emerging all around us, and via us, is our creation but it is already becoming independent of us – truly it represents the evolution of a new form of meta-level intelligence that has never before existed on our planet.

Although we created it, the Internet is already far beyond our control or comprehension – it surrounds us and penetrates our world – it is inside our buildings, our tools, our vehicles, and it connects us together and modulates our interactions. As this process continues and the human body and biology begins to be networked into this system we will literally become part of this network – it will become an extension of our nervous systems and eventually, via brain-computer interfaces, it will be an extension of our senses and our minds. Eventually the distinction between humans and machines, and the individual and the collective, will gradually start to dissolve, along with the distinction between human and artificial forms of intelligence.

MEMES ARE EVOLVING MINDS OF THEIR OWN

The evolution of our planetary intelligence has been taking place for billions of years — it is a natural process, just like the evolution of human intelligence was long ago.  The Semantic Web is merely the next step in this process whereby communicable ideas (memes), having already evolved technologies to externalize themselves outside the human mind (i.e. books, recording, software, the Web, etc.) are starting to evolve the ability to propagate intelligently and interact without human intervention. In other words, although today memes are for the most part completely immobile and static unless perceived within a human brain, with the advent of the Semantic Web the cognitive processes for running memes will begin to spread outside the human brain, enabling memes to "run" without depending on humans.

This emerging planet-wide collective mind, of which we will be but parts, will evolve higher level meta-processes and structures that will vastly exceed our comprehension. Indeed this is already starting to happen — even today the self-organizing, chaotically emergent collective intelligence and information flows of the Internet exceed the power and understanding of any computer or brain on the planet. This new meta-level intelligence will be as far beyond human intelligence as the intelligence of the human brain is beyond that of its individual neurons.

THE INFRASTRUCTURE OF DISTRIBUTED INTELLIGENCE

The development of the global mind depends on the evolution of distributed systems that function as the global equivalent of consciousness, memory, learning, perception, introspection, planning, creativity, and behavior.

Distributed intelligence requires the decentralization of information and computation. The World Wide Web is a key catalyst for this evolutionary leap.  Before the Web there was no universally agreed-upon standard for publishing and accessing simple information – instead there were myriad incompatible, non-standardized competing proprietary formats. The lack of a common language made it difficult for applications to interoperate or understand one another’s data without explicit integration.

The significance of the Web is that its underlying metalanguage standards – HTML and HTTP – enable more widespread, interoperable and decentralized content production and access. Making it possible for agents anywhere in the system to publish and make use of information by any other agents in the system is an essential ingredient of a distributed intelligence. The Web is literally a World Wide File System – it is the memory function of the global mind.

If the Web enables the World Wide File System, the emergence of XML enables The World Wide Database.  XML enables agents in the system to define, store, retrieve, interact with, and interpret arbitrary data structures with arbitrary precision. Using XML any conceivable syntax and data schema can be defined and shared. XML adds more structure to the information in the memory of the global mind, enabling more sophisticated content and processes to be stored and accessed by agents in the system.

The recently emerging Semantic Web adds yet another layer of sophistication beyond XML. It enables agents in the system to begin to understand and reason about the meaning of information within the system. The Semantic Web enables software to work not merely with data but with concepts. Concepts are information structures that are connected to formal systems of ideas – in other words they are meaningful information.  The Semantic Web provides standards for transforming ordinary information structures into concepts that can be understood by software programs. Using metalanguages for defining semantics such as RDF and OWL, the Semantic Web makes it possible to connect data elements to concepts in formally defined systems of knowledge called ontologies. By doing this software programs are able to then reason intelligently about the information.

By connecting information to ontologies, programs can begin to process information more intelligently. For example, the content of a medical journal could be linked to a medical ontology that defines medical concepts and their interrelations. Using this ontology it would then be possible to do semantic searches of the journal that are far more intelligent than the primitive keyword searches that are currently used in most search systems today.  A semantic search for “information about the vascular system" would return articles and data records that refer to the heart, even though the word "heart" was not explicitly searched for. Furthermore, a semantic search for "organs connected to the heart" could make logical inferences across chains of concepts in the underlying medical ontology in order to return articles about the lungs, the liver, the kidneys, the brain, etc., even though none of those organs were explicitly named in the original query.

Smarter searches are just one of the many benefits of the Semantic Web. Beyond such basic applications, the Semantic Web makes it possible for software to automatically learn, reason, make suggestions, and manage tasks and processes more intelligently. What’s more, by providing a standardized language for describing systems of concepts and chains of reasoning, the Semantic Web makes it possible for programs to seamlessly share concepts and collaborate on reasoning tasks – in other words, it makes it possible not only for smarter computation within a given program, but it also enables smarter computation to take place between programs, making it possible for widespread distributed artificial intelligence to emerge on the Internet.

THE EVOLUTION OF METALANGUAGE

The Semantic Web is based on a higher level of language — metalanguage — language about language. Metalanguage is a form of communication that enables parties to rigorously express and share information about the meaning of information. In fact, metalanguage has existed since the dawn of humanity. For example, in the case of spoken language, humans communicate metalanguage by using tone, gesture, inflection, volume, and facial expressions. These cues convey vital information about the meaning of what we are communicating, making it possible for those we communicate with to more easily understand us. In written language very simple forms of metalanguage have also been in use for quite some time, such as for example, the formatting of text, the use of footnotes and diagrams. The way text is organized on a page, and the particular typefaces and styles used also constitute metalanguage expressions about the meaning of the text.

The Semantic Web provides metalanguage specifications and technologies that vastly increase the bandwidth and sophistication of metalanguage communication for all forms of digital media. For example, using metalanguages such as XML, RDF and OWL, the Semantic Web makes it possible to encode arbitrarily detailed knowledge about the structure, meaning, state, connections, reliability, sentiment, and policies of arbitrary chunks of information. In other words, a document can be encoded with metalanguage that adds layers of additional knowledge about the information it contains. These layers of information augment the text — they may provide definitions, links to other resources, information about the organization of information within the document, logical relations among concepts in the document, details about the history and license terms of the document, annotations from other readers, and even rules for interpreting, reasoning about, or using the document. What is important here is that this metalanguage is expressed in a manner that machines can understand.

In effect, semantic metalanguage gives computers access to layers of knowledge that previously could only exist or be utilized within the human brain. By making this metalanguage explicit and by standardizing it, it becomes possible not only to communicate it effectively between humans, but also between humans and programs, and even between programs and other programs.

The evolution from simple typography to SGML and HTML to XML and finally to the Semantic Web (RDF and OWL) can be viewed as a process of decoupling the interpretation of data from the agents that produce and consume the data. In other words not only the data itself, but also its interpretation, can now be stored outside of the agents of communication. HTML makes it possible for any program to correctly render data. XML makes it possible for any program to correctly parse and navigate the structure of data – for example to find a particular data element such as a field within a document. RDF and OWL make it possible for any program to understand what a particular data element means, and to reason about it.

If we look back to the dawn of humanity there was a time when humans were only able to communicate nonverbal or primitive verbal information. As richer forms of communication evolved sophisticated spoken languages and oral traditions emerged enabling the communication of more complex ideas. But spoken language had a major limitation – the distribution and access to information was dependent on being physically proximate enough to interact with particular individuals.

With the development of written languages however, it became possible to break through this limitation. Writing systems made it possible for ideas to be represented, stored and communicated independently of any particular individual, with less error, across greater distances in space and time than ever before. For the first time it was possible to learn something from someone else without them having to be present – anyone who could read the language and had sufficient background could interpret written characters into concepts. Next, with the advent of printing the economics of distributing and accessing written ideas reached a critical threshold of efficiency, enabling widely distributed communication and intellectual discourse.

Centuries later another critical threshold was crossed with the invention of long-distance communications networks such as teletypes, telephones, radio and television. These technologies made communication faster, richer, broader, and more ubiquitous and accessible than ever before. As recorded and recordable media emerged even these rich media experiences could be experienced asynchronously anywhere and at any time.

Next, the emergence of computers and computer networks made it possible for communications and information processes to be increasingly automated. At this point we begin to see something new – while previously only information could be represented outside of the human brain now even primitive forms of intelligence (information processing) could be represented and conducted outside of the human brain. The Internet and the World Wide Web are the logical extension of this process – they make it possible to distribute and access information, and to connect information and processors together, more widely than ever before – but they still rely on humans for the most part.

Without humans the Internet and Web of today would be nothing but a collection of relatively static information and dumb computer systems. But XML and the Semantic Web will change that by providing metalanguages that make it possible for humanlike intelligence to being to evolve and function outside of human brains. With advent of metalanguages humans are no longer necessary to create or interpret information. These technologies will enable the Web to actively and intelligently process information without human participation.

Metalanguages such as HTML, XML, RDF and OWL enable knowledge about information to be formally encoded into the information itself. As increasing levels of knowledge about data is encoded into the data, the data becomes more independent of humans -  it can be used by any agent anywhere.

HOW THE GLOBAL MIND THINKS

Semantic Web programs will share and process information intelligently, with or without the help of humans, by reading and writing metadata about data in a standardized way such that other programs can then reuse it. Programs will be able to leverage the knowledge that other programs create about the data they work with – even though these programs may not be directly integrated or even know about one another. In effect the Web becomes a gigantic shared knowledgebase that every program can read and write to.

Just as colonies of social insects such as ants and bees are able to perform intelligent collective behaviors without centralized control, the millions, or even billions, of humans and programs roaming independently through the Semantic Web, selectively reading, writing, annotating, linking, rating, and aggregating information, will perform collective intelligent behaviors without necessarily coordinating with one another or even knowing it. In other words the individual agents in such behaviors will participate in collective cognitive processes that transcend the comprehension of any individual.

Here’s how it might work: Imagine that a particular news article about a potential corporate merger exists on the Web. Intelligent agents – whether humans or software programs -  are then able to read this article and mark it up with semantic metadata in their particular areas of expertise. One agent specializes in identifying company names – whenever it sees the name of a company in an article it tags it with a link to the ontology definition of a corporation, as well as with metadata that links it to the Web site and other data records corresponding to the particular corporation it represents. Another agent specializes in recognizing people: whenever it sees the name of a person it tags it with a link to the ontology class for “person" and also with metadata that connects it to the home page for that person, articles about that person, friends and colleagues of that person, organizations that the person is affiliated with etc. Other agents that visit, or receive, the article could then tag it with their particular knowledge – some add metadata about links, others tag events, others add metadata about places, others add metadata about products and brands, others add metadata about technical terms and jargon, etc. 

We might even imagine that some of these agents are capable of generating new articles and data structures about the original article and linking them together – for example, one agent might generate a synopsis, another might translate it into another language, another might measure the opinions in the article, still another might generate a report based on the conclusions in the article. Because all of this knowledge is expressed using open semantic metadata standards, any program that later encounters any of it can make use of it in its own work, without having to be expressly programmed to do so.

This is already starting to happen in fact – For example, in the blogging community and communities of practice, which in an entirely bottom-up emergent manner, are naturally aggregating, annotating, linking, organizing and prioritizing information. Although there is no central guidance within such knowledge communities, their collective self-organizing behavior results in global information processes that appear to be intelligent. If one were to view the information dynamics of the Web from space – perhaps with a special sensor that could detect and measure these patterns as they emerged – would it not appear similar to the a functional brain imaging scan?

The Internet (the OS layer), the Web (the data layer), XML (the data schema and syntax layer), and the Semantic Web (the knowledge and reasoning layer) combine to provide the foundation for an increasingly intelligent distributed world-wide mind. They enable all the agents of the global mind to seamlessly share not just raw information, but even high-level concepts, knowledge and intelligent cognitive processes, in a manner that is open and independent of any individual system.

In particular the Semantic Web makes it possible to represent concepts such that they can be unambiguously interpreted and understood by any agent of the system. However, the success of this process will hinge on the development and adoption of open-standards-based, open-source ontologies, and mappings between them. This is already starting to take place, for example, FOAF, a simple ontology for describing social relationships, and SUMO, a standardized ontology of foundational concepts, among many others. I believe much of the initial development of these much-needed open-source ontologies will spring from the Weblog and RSS communities, where there is an increasing willingness (and need) on the part of participants to mark up and filter content with metadata.

CAN THE GLOBAL MIND PASS THE TURING TEST?

If the Internet is becoming a global mind, is there a way to test whether or not it is actually intelligent? Of course that first requires that one define intelligence – a notoriously fuzzy term! For the purposes of this article, we might define intelligence as “humanlike information processing.” One way to test for “humanlike intelligence” is to use the Turing Test – in which a human subject attempts to determine which of two “black boxes” is controlled by a human and which is a computer in a question-and-answer game.

An interesting modern-day spin on the classic Turing Test, might test large distributed online communities comprised of people and software programs, to see if such systems could be judged to be intelligent. It seems like a good bet that such systems – if hidden in a black box would be able to emulate “humanlike intelligence.”

I once tested this hypothesis in my own company many years ago. A difficult math problem was posed to me and to the best mathematician in our team. Whoever could answer it correctly the fastest would be judged as the best mathematician. I have never been much of a mathematician, but I still won this contest. My strategy was simply to farm out the problem to a number of the best mathematical brains I knew, integrate the answers, and package it up as a reply to the question.

My network of math-brains vastly outperformed the ability of my own brain or the brain of the math expert I was competing with. Not knowing how I solved the problem, those in the company would only be able to assume that I was a better mathematician. In point of fact however it was not a fair contest. "I" was not merely an individual but a vast collective super-brain comprised of several networked experts. The other guy was hopelessly outgunned.

This is an example of the power of distributed intelligence – the world of the future that is evolving on the Web right now. As the global brain continues to develop we will see individual humans, and even individual organizations, being dramatically outpaced by collective intelligences. One compelling example of how this is happening can be seen in the rise of open-source software development communities which are able to develop better code, faster, at less cost, and with broader adoption than has ever been possible by single entities.

READING THE GLOBAL MIND

If the Web is becoming a virtual mind of the planet is it possible to data-mine the Web in order to empirically measure, map, understand and even predict collective cognition? Can we empirically measure the Web in order to chart the past and present thinking of individuals, groups and communities, nations, or even of humanity-as-a-whole? By dong this can we learn to detect and track thoughts ("memes") as they emerge, spread, interact, develop and evolve in real-time? If we are able to empirically detect memes and develop a science of meme dynamics would this enable us to not only better understand the past and the present, but even to predict the future in a new way?

One approach to reading the global mind is to measure distributed cognitive trends by mining search engines results for  the frequency of search terms over time, such as Google’s Zeitgeist reports.  More recent approaches such as Daypop attempt to detect "word bursts" on the Web and "news bursts" among news articles.

Many academic and government research projects have explored the potential to data-mine news articles and other information sources in order to predict political events. For example, this project found that it is possible to predict conflicts such as wars as early as 6 to 8 weeks before they occur by data-mining news articles. There have also been projects to predict signs of political change, such as coups and election results, by data-mining political news.

Another interesting project describes a technique for statistically analyzing clusters of concepts that occur on the Web in order to attempt to find hot archetypes in the collective consciousness of humanity. The particular application of this system that the authors focus on is predicting terrorist events. Their system identifies hot themes, but requires a high degree of subjective interpretation in order to come up with predictions. While interesting, I am not sure the system can be used to reliably predict the future, although it certainly can help to understand the present. In any case, this project is significant in that it attempts to detect collective thoughts or archetypal patterns that transcend any individual mind or community. It’s definitely worth reading for those interesting in next-generation data-mining.

Another project that takes a completely different approach is the Global Consciousness Project which mines statistical deviations from randomness across a network of random number generators around the world and then correlates these deviations with global events — nobody knows why this works but the statistical data speaks for itself (this project may in fact point to yet another interesting connection between consciousness and quantum physics, similar to the famous double-slit experiment but on a global scale, but nobody really knows — all we know so far is that the data is sound.) This project might be described as an EEG for the planet. While it cannot provide insight into particular thoughts taking place in the global mind, it does provide a window into the activation and dynamics of the global mind.

In my own thinking on the subject, I have focused more on detecting and analyzing the higher-order distribution of memes in space and time. Memes are concepts that move across the global mind – they the building blocks of its collective thoughts. A meme might be as simple as a brand or an icon, or as sophisticated as a joke, a fact, tradition, fad, belief system, or a paradigm. I have spent some time speculating about a possible physics of ideas that might be able to empirically detect, measure and predict the dynamics and interactions among memes on the Web. My approach attempts to measure properties of memes in space and time in order to forecast their trajectories. For example, using this approach, one might be able to measure the geographic footprint, mass and velocity of a meme over time. With such data it then becomes possible to begin to measure the spread of ideas much like one might analyze the behavior of systems of particles, or the behavior of products and stocks within marketplaces.

The examples above represent just a small sample of the many research projects and technologies in this space. I would be very interested to hear of others of note.

ENTERPRISE MINDS

As the global mind develops it will initially be focused around making information more useable. But that will be just the beginning. Already a new generation of tools that will bring the power of distributed intelligence to the desktop and the enterprise are being developed in labs such as HP, Cycorp and Network Inference.

In addition to these projects, my company,Radar Networks has developed a complete platform in Java for developing and deploying Semantic Web applications.

The power of distributed intelligence made possible by the Semantic Web will dramatically evolve corporations – at least those that are not made extinct by it. In particular, it will enable workgroups and corporations to begin to distill and store not only their information but also their intelligence. As individuals and teams work, intelligent agents will learn from them. These agents will then be able to assist them in working more productively — they will help them search, organize and file information, track relevant news, better-leverage existing knowledge and resources, manage projects and tasks, share and access knowledge, and communicate and collaborate more productively with teammates, customers and partners. Similarly, smart agents will learn from corporations as-a-whole, and from their business interactions with employees, customers, suppliers and partners in order to dynamically streamline business processes and adapt to market changes intelligently.

By making organizational knowledge, learning and intelligence increasingly independent of the particular minds or programs within a given organization, all parts of these organizations as well as the whole organizations themselves will become more intelligent. As knowledge and intelligence about organizations become increasingly portable and reusable, organizations will evolve their own "group minds" and "enterprise minds." These distributed forms of intelligence will constitute a new level of structure, a new layer of organization. Such meta-level processes will help managers make smarter decisions by enabling them to better access the combined past and present knowledge and capabilities within their organizations and business relationships. They will also help organizations to notice opportunities or problems, and response to them more effectively.

Today there are many organizations that have realized that their primary product is knowledge. Tomorrow organizations will begin to realize that it is not just knowledge, but also intelligence, that is the key to their competitive advantage. Intelligence is the ability to utilize knowledge effectively.

Merely creating vast collections of knowledge that are inaccessible or simply not leveraged is of no benefit to anyone. What matters is that the knowledge is intelligently connected to business processes such that it measurably improves performance. What is necessary for this to happen is not merely the implementation of knowledge management systems, but rather the implementation of intelligent systems — a new way of creating and utilizing knowledge at all levels of the organization.

Knowledge must be intelligently integrated into every business activity, event, relationship, resource and tool. Furthermore the integration must be bidirectional — every business activity should be able to get knowledge from the enterprise and add knowledge back to it. By enabling this, with the right infrastructure and tools, organizations can literally begin to learn and improve based on their own collective experience. By providing all of the parts of an organization with access to the collective knowledge and intelligence of the system, the whole system can become more collectively intelligent.

At a higher-level, in order to enable more focused, goal-directed collective behaviors it is necessary to create control structures and adaptive feedback loops between the "parts" and the "wholes" within an organizations. What this means is that there needs to be a connection between the knowledge and intelligence taking place within each part, and the new meta-level knowledge and intelligence taking place at the level of the whole (such as a team or enterprise). The question then arises as to how to bring about such a connection? What connects the parts to the whole — what makes a collection of parts function as a whole, yet enables them to still maintain their individuality and independence? What enables the whole to function as one entity, despite being formed of myriad independent parts?

Traditional control structures such as top-down management hierarchies err on the side of the whole — they attempt to rigidly organize and control the parts of the organization in order to force them to conform into a cohesive whole. On the other extreme, more recent attempts to eliminate hierarchy and enable flatter, more "networked" organizations err on the side of the parts — they eliminate the hierarchical control structures altogether leaving nothing but chaotically interacting decentralized parts. Fortunately there is another alternative — there is a way to connect the parts and the whole without sacrificing either. The key is enabling richer self-knowledge.

In order to bring about synchrony between levels of a distributed organization there must be three essential ingredients: (1) The state of each part must be represented, (2) the state of the whole system (the combined system of all the parts) must be represented, (3) the parts must all have real-time access to all of these representations. By meeting these three requirements feedback becomes possible in several directions — between each part and every other part, and between each part and the system as-a-whole. These representations and feedback loops provide a vital function to distributed intelligences — they enable them to enact a simple form of self-awareness. Self-awareness is a vital ingredient of higher forms of intelligence. The richer a systems’ self-representation, the smarter and more effective it can be.

In order to accomplish self-awareness in a highly distributed organization, each part of the organization needs to have access to a self-representation of itself as well as of the whole system. Each part needs to be able to understand themselves and the system they are part of. By providing the parts of a distributed organization with access to information about both their own state and the state of the whole, the parts are empowered to adapt to the whole. By enabling this, the whole is also more able to adapt to the parts, because there is bidirectional feedback taking place between these levels. Rather than placing control structures at only the level of the whole, or at only the level of the parts, instead they are distributed across both levels.

The only way for this to be practical, economically feasible, or even technically possible, is by using emerging Semantic Web metalanguages. These metalanguages provide a common standard for sharing knowledge and intelligence at every level and across every part of an organization. Knowledge and intelligence are thus able to move freely across and between them and organizational learning takes place on the individual level, the group or sub-systems level, and the level of the whole system. And this learning is expressed, stored and shared in a single common metalanguage that is equally accessible to all. This is quite different from the case of present-day organizations in which there are different languages and formats for knowledge at different levels of the organization. For example, in most present-day organizations human knowledge and expertise is still locked inside individual human minds and totally dependent on them, group knowledge is stored on PCs and workgroup servers, and enterprise knowledge is stored in enterprise systems. Each of these systems speaks a different language and most are not directly integrated.

Numerous inefficiencies result from this. Why should it be so difficult to move a concept across an organization? Why does it require that data be translated and ported from one person to another, from one program to another? The reason it is so difficult today is that the interpretation of the data is not stored separately from the brains and programs that manipulate it — in other words, metalanguages are not being used. As a result, the intelligence of the organization is not portable — it is locked into silos such as people’s heads and particular applications that are explicitly programmed with particular skills and knowledge.

Those organizations that understand this are already starting to make use of metalanguages and Semantic Web technologies. Those that are first to begin exploring and deploying these "enterprise minds" will have a valuable head start that may provide them with crucial advantages in the marketplace. This is not unlike the advantages that Homo sapiens had over earlier primates. Larger and more advanced brains resulted in an increased capacity for language, communication and reasoning that ultimately enabled them to outperform less intelligent hominids. This same principle holds for organizations.

CONCLUDING THOUGHTS

The ideas in this essay are not unique to me – they are memes that are spreading on their own through the global mind. Many others such as the people involved with the Principia Cybernetica Project or my friend Howard Bloom have thought far more extensively than I have about these subjects. In writing this article I am merely providing a service to the global mind – that of aggregating, annotating and communicating these memes onward in a process that I cannot begin to comprehend. All I know is that the global mind is thinking about its own evolution and realizing that it is intelligent – and that I am just an infinitesimal part of that process. Yet, like you who are reading this, I somehow sense that what is taking place is incredibly important and will change our world and our species profoundly.