In typical Web-industry style we’re all focused minutely on the leading trend-of-the-year, the real-time Web. But in this obsession we have become a bit myopic. The real-time Web, or what some of us call “The Stream,” is not an end in itself, it’s a means to an end. So what will it enable, where is it headed, and what’s it going to look like when we look back at this trend in 10 or 20 years?

In the next 10 years, The Stream is going to go through two big phases, focused on two problems, as it evolves:

1. Web Attention Deficit Disorder. The first problem with the real-time Web that is becoming increasingly evident is that it has a bad case of ADD. There is so much information streaming in from so many places at once that it’s simply impossible to focus on anything for very long, and a lot of important things are missed in the chaos. The first generation of tools for the Stream are going to need to address this problem.
2. Web Intention Deficit Disorder. The second problem with the real-time Web will emerge after we have made some real headway in solving Web attention deficit disorder. This second problem is about how to get large numbers of people to focus their intention not just their attention. It’s not just difficult to get people to notice something, it’s even more difficult to get them to do something. Attending to something is simply noticing it. Intending to do something is actually taking action, expending some energy or effort to do something. Intending is a lot more expensive, cognitively speaking, than merely attending. The power of collective intention is literally what changes the world, but we don’t have the tools to direct it yet.

The Stream is not the only big trend taking place right now. In fact, it’s just a strand that is being braided together with several other trends, as part of a larger pattern. Here are some of the other strands I’m tracking:

• Messaging. The real-time Web aka The Stream is really about messaging in essence. It’s a subset of the global trend towards building a better messaging layer for the Web. Multiple forms of messaging are emerging, from the publish-and-subscribe nature of Twitter and RSS, to things like Google Wave, Pubsubhubub, and broadcast style messaging or multicasting via screencast, conferencing and media streaming and events in virtual worlds. The effect of these tools is that the speed and interactivity of the Web are increasing — the Web is getting faster. Information spreads more virally, more rapidly — in other words, “memes” (which we can think of as collective thoughts) are getting more sophisticated and gaining more mobility.
• Semantics. The Web becomes more like a database. The resolution of search, ad targeting, and publishing increases. In other words, it’s a higher-resolution Web. Search will be able to target not just keywords but specific meaning. For example, you will be able to search precisely for products or content that meet certain constraints. Multiple approaches from natural language search to the metadata of the Semantic Web will contribute to increased semantic understanding and representation of the Web.
• Attenuation. As information moves faster, and our networks get broader, information overload gets worse in multiple dimensions. This creates a need for tools to help people filter the firehose. Filtering in its essence is a process of attenuation — a way to focus attention more efficiently on signal versus noise. Broadly speaking there are many forms of filtering from automated filtering, to social filtering, to personalization, but they all come down to helping someone focus their finite attention more efficiently on the things they care about most.
• The WebOS.  As cloud computing resources, mashups, open linked data, and open API’s proliferate, a new level of aggregator is emerging. These aggregators may focus on one of these areas or may cut across them. Ultimately they are the beginning of true cross-service WebOS’s. I predict this is going to be a big trend in the future — for example instead of writing Web apps directly to various data and API’s in dozens of places, just write to a single WebOS aggregator that acts as middleware between your app and all these choices. It’s much less complicated for developers. The winning WebOS is probably not going to come from Google, Microsoft or Amazon — rather it will probably come from someone neutral, with the best interests of developers as the primary goal.
• Decentralization. As the semantics of the Web get richer, and the WebOS really emerges it will finally be possible for applications to leverage federated, Web-scale computing. This is when intelligent agents will actually emerge and be practical. By this time the Web will be far too vast and complex and rapidly changing for any centralized system to index and search it. Only massively federated swarms of intelligent agents, or extremely dynamic distributed computing tools, that can spread around the Web as they work, will be able to keep up with the Web.
• Socialization. Our interactions and activities on the Web are increasingly socially networked, whether individual, group or involving large networks or crowds. Content is both shared and discovered socially through our circles of friends and contacts. In addition, new technologies like Google Social Search enable search results to be filtered by social distance or social relevancy. In other words, things that people you follow like get higher visibility in your search results. Socialization is a trend towards making previously non-social activities more social, and towards making already-social activities more efficient and broader. Ultimately this process leads to wider collaboration and higher levels of collective intelligence.
• Augmentation. Increasingly we will see a trend towards augmenting things with other things. For example, augmenting a Web page or data set with links or notes from another Web page or data set. Or augmenting reality by superimposing video and data onto a live video image on a mobile phone. Or augmenting our bodies with direct connections to computers and the Web.

If these are all strands in a larger pattern, then what is the megatrend they are all contributing to? I think ultimately it’s collective intelligence — not just of humans, but also our computing systems, working in concert.

Collective Intelligence

I think that these trends are all combining, and going real-time. Effectively what we’re seeing is the evolution of a global collective mind, a theme I keep coming back to again and again. This collective mind is not just comprised of humans, but also of software and computers and information, all interlinked into one unimaginably complex system: A system that senses the universe and itself, that thinks, feels, and does things, on a planetary scale. And as humanity spreads out around the solar system and eventually the galaxy, this system will spread as well, and at times splinter and reproduce.

But that’s in the very distant future still. In the nearer term — the next 100 years or so — we’re going to go through some enormous changes. As the world becomes increasingly networked and social the way collective thinking and decision making take place is going to be radically restructured.

Social Evolution

Existing and established social, political and economic structures are going to either evolve or be overturned and replaced. Everything from the way news and entertainment are created and consumed, to how companies, cities and governments are managed will change radically. Top-down beaurocratic control systems are simply not going to be able to keep up or function effectively in this new world of distributed, omnidirectional collective intelligence.

Physical Evolution

As humanity and our Web of information and computatoins begins to function as a single organism, we will evolve literally, into a new species: Whatever is after the homo sapien. The environment we will live in will be a constantly changing sea of collective thought in which nothing and nobody will be isolated. We will be more interdependent than ever before. Interdependence leads to symbiosis, and eventually to the loss of generality and increasing specialization. As each of us is able to draw on the collective mind, the global brain, there may be less pressure on us to do things on our own that used to be solitary. What changes to our bodies, minds and organizations may result from these selective evolutionary pressures? I think we’ll see several, over multi-thousand year timescales, or perhaps faster if we start to genetically engineer ourselves:

• Individual brains will get less good at things like memorization and recall, calculation, reasoning, and long-term planning and action.
• Individual brains will get better at multi-tasking, information filtering, trend detection, and social communication. The parts of the nervous system involved in processing live information will increase disproportionately to other parts.
• Our bodies may actually improve in certain areas. We will become more, not less, mobile, as computation and the Web become increasingly embedded into our surroundings, and into augmented views of our environments. This may cause our bodies to get into better health and shape since we will be less sedentary, less at our desks, less in front of TV’s. We’ll be moving around in the world, connected to everything and everyone no matter where we are. Physical strength will probably decrease overall as we will need to do less manual labor of any kind.

These are just some of the changes that are likely to occur as a result of the things we’re working on today. The Web and the emerging Real-Time Web are just a prelude of things to come.

The next generation of Web search is coming sooner than expected. And with it we will see several shifts in the way people search, and the way major search engines provide search functionality to consumers.

Web 1.0, the first decade of the Web (1989 – 1999), was characterized by a distinctly desktop-like search paradigm. The overriding idea was that the Web is a collection of documents, not unlike the folder tree on the desktop, that must be searched and ranked hierarchically. Relevancy was considered to be how closely a document matched a given query string.

Web 2.0, the second decade of the Web (1999 – 2009), ushered in the beginnings of a shift towards social search. In particular blogging tools, social bookmarking tools, social networks, social media sites, and microblogging services began to organize the Web around people and their relationships. This added the beginnings of a primitive “web of trust” to the search repertoire, enabling search engines to begin to take the social value of content (as evidences by discussions, ratings, sharing, linking, referrals, etc.) as an additional measurment in the relevancy equation. Those items which were both most relevant on a keyword level, and most relevant in the social graph (closer and/or more popular in the graph), were considered to be more relevant. Thus results could be ranked according to their social value — how many people in the community liked them and current activity level — as
well as by semantic relevancy measures.

In the coming third decade of the Web, Web 3.0 (2009 – 2019), there will be another shift in the search paradigm. This is a shift to from the past to the present, and from the social to the personal.

Established search engines like Google rank results primarily by keyword (semantic) relevancy. Social search engines rank results primarily by activity and social value (Digg, Twine 1.0, etc.). But the new search engines of the Web 3.0 era will also take into account two additional factors when determining relevancy: timeliness, and personalization.

Google returns the same results for everyone. But why should that be the case? In fact, when two different people search for the same information, they may want to get very different kinds of results. Someone who is a novice in a field may want beginner-level information to rank higher in the results than someone who is an expert. There may be a desire to emphasize things that are novel over things that have been seen before, or that have happened in the past — the more timely something is the more relevant it may be as well.

These two themes — present and personal — will define the next great search experience.

To accomplish this, we need to make progress on a number of fronts.

First of all, search engines need better ways to understand what content is, without having to do extensive computation. The best solution for this is to utilize metadata and the methods of the emerging semantic web.

Metadata reduces the need for computation in order to determine what content is about — it makes that explicit and machine-understandable. To the extent that machine-understandable metadata is added or generated for the Web, it will become more precisely searchable and productive for searchers.

This applies especially to the area of the real-time Web, where for example short “tweets” of content contain very little context to support good natural-language processing. There a little metadata can go a long way. In addition, of course metadata makes a dramatic difference in search of the larger non-real-time Web as well.

In addition to metadata, search engines need to modify their algorithms to be more personalized. Instead of a “one-size fits all” ranking for each query, the ranking may differ for different people depending on their varying interests and search histories.

Finally, to provide better search of the present, search has to become more realtime. To this end, rankings need to be developed that surface not only what just happened now, but what happened recently and is also trending upwards and/or of note. Realtime search has to be more than merely listing search results chronologically. There must be effective ways to filter the noise and surface what’s most important effectively. Social graph analysis is a key tool for doing this, but in
addition, powerful statistical analysis and new visualizations may also be required to make a compelling experience.

(DRAFT 7. Work-In-Progress)

What is the universe and where does it come from?

There are two major schools of thought on this question:

1. Science: One is modern-day science, which takes the position that universe is strictly a physical phenomenon and that everything about can be explained by repeatable physical measurements, testable scientific theories, and the rules of math and logic.
2. Religion:
   The other school of thought is religion, which in general, takes the position that the universe comes from something non-physical that is ultimately unexplainable and beyond the reach of science.

In this paper we will take an intellectual adventure into the far fringes of both science and religion, to explore the question of whether or science and religion might be unified. Such a unification is an intellectual “Holy Grail” that could truly change the world. But is it even possible? I think it is, and I’ll propose the core of such a unification here.

The Possibility of Convergence

While there are clearly differences between the approaches and beliefs of the sciences and religions of the world, there are also more similarities than many would like to admit. Beyond that however, at the very deepest levels, they lead to similar logical conclusions and in fact intersect on certain fundamental points, whether their proponents know it or not.

In particular, the question of the origin and nature of the universe is where I believe science and religion converge. Whether one holds the view of science, the view of religion, or both, it turns out that there is a logical necessity for reaching the same final conclusions about the ultimate nature of reality.

Whether one starts from a scientific viewpoint and applies only the methods of science and logic, or one starts from a religious perspective and applies only the methods of religion and logic, either way the conclusion is the same. As long as one regards logic as a valid method of inquiry, the final answer is the same.

The Core Argument

So what is the answer? In short, everything is “nonoriginated.” This has a very specific meaning: the universe (or anything else that we might posit to exist) cannot logically originate from nothingness, from itself, or from some other fundamental thing.

Here’s how this conclusion is reached in a nutshell (I will explain this argument in more depth later in this article, as well as its many implications):

To claim that something originates from nothing is a contradiction.

To claim that something originates from itself is a contradiction.

To claim that something originates from something else leads to an infinite regress unless you claim there is a fundamental first-thing — but claiming there is a fundamental first-thing leads to a contradiction, so it’s not an option. An infinite regress on the other hand, is not really an origin.

Therefore none of the three above ways of originating are logically tenable, yet there is no other possible fourth alternative.

This then leaves only two possible conclusions about the universe (and anything else that is posited to exist):

1. The first option is that the universe is not really happening at all, because there’s no logical way for it to have originated.  But this is immediately contradictory to our experience. It is refuted by obvious, undeniable evidence — right in front of us we can see that something is happening — who knows what it is, but it would be absurd to deny that there is some kind of phenomena taking place.
2. The second option is that the universe is happening, although there is no origin for it (i.e. it is “nonoriginated”). It is not necessary for there to be an ultimate and final origin — no first cause, prime mover, fundamental particle, or first moment of creation. The universe must therefore be infinite in time, space, and levels of scale.

Option (1) is easily refuted. We are left with option (2) – Nonorigination.

But it is a bit strange to imagine a universe that has no beginning, no origin. How can the universe exist if it is truly beginningless? Without a first-cause what could ever have gotten it started? Without a final fundamental particle, what could things actually be made of? In fact, it is precisely because the universe is nonoriginated that it CAN appear at all. This will be explained further in this article.

We can see how this logic applies to the origin of the universe. How about God? Well if God exists then the same logic would apply: God must also be nonoriginated. Anything that is posited to exist must be nonoriginated.

This point of nonorigination is where science and religion intersect. Nonorigination is the ultimate nature of reality. It is not merely a concept — it is the actual nature of all things, and it has many profound implications. It points to a level of reality that is beyond the limits of space and time — and in this respect it is proof of what might be called the Divine, yet it is also completely compatible with the physical world and its laws.

There are several other key dimensions of nonorigination as well. Awareness is one of them. Awareness is the unique capacity of sentient beings to make observations. It plays an important role in making the universe happen, and is actually unified with nonorigination. Where there is nonorigination there MUST be awareness and vice-versa.

Likewise the process of cause-and-effect turns out to be a natural corollary to the nonorigination of the universe, and it’s powered by awareness, the act of making observations. If there were no such process, the universe could not work as it does; it would effectively be random.

I will explore these topics in a lot more detail below.

The unification of science and religion is not philosophy, it is logic. But how we interpret it, and what we do with it is a matter of personal preference and personal philosophy. This paper will not attempt to draw conclusions about what scientific or religious belief is best. That is up to you. Use the logical evidence however you see fit.

What Does the Universe Come From?

If one even merely posits the existence of the universe or even just the presence of a fundamental particle — then that immediately leads to further questions such as: Then where does that come from, what is it all really made of, and how could it all be taking place, what is space-time made of or located in, who or what designed this or how did it all happen so perfectly when it is statistically almost impossible?

Some people just can’t imagine that anything as vast as God could be possible, so they simply decide (without any real evidence) that God is impossible. Or they think that there could not be anything greater than or beyond the scope of the physical universe because they feel that the only things that can exist are physical things. To them, there is nothing but the physical, it is all a big machine, this is all there is — and for that reason they can’t believe in some kind of greater being or ultimate reality beyond space and time or the physical laws. But the grounds on which they claim God is not possible can also be used to claim the universe itself is not possible. If they believe in the possibility of the physical universe they also must accept the possibility of God by the same logic.

Here’s why: If the argument against the possibility of God is that it just isn’t possible for there to be something infinite, then that means either space and time are finite or they can’t exist either — the universe would not be possible because space and time are presently thought to be infinite.

Similarly, if the argument against the possibility of God is that there just couldn’t be anything beyond the physical universe, then even the physical universe could not exist — for if there were no possibility of anything greater than or beyond the universe then where is the physical universe taking place? What does it come from? What is it “in?” If it ever ends, what remains? This second argument is a bit of a difficult point so it bears further explanation.

Whenever you posit something, it logically has to either come from nothing, or from itself, or from something else. And at the time it exists it either has to depend on nothing, depend on itself, or depend on something else.

Stating that the universe comes from nothing or depends on nothing is problematic — it is in fact equivalent to saying that the universe comes from or depends on something beyond the universe: some primordial “nothingness.”

Stating that the universe comes from or depends on itself is circular and also a contradiction of sorts — in order for the universe to create itself or depend on itself it must already exist, and so this is impossible and not an option.

Yet stating that the universe comes from something else or depends on something else admits that there must be something beyond it to come from or depend on.

In other words, no matter what position one takes on the universe, it leaves open the possibility – indeed even the logical requirement – that there must be something before it, greater than it, deeper than it, beyond it, after it, etc.

Refuting Ideas that the Universe Comes from Nothingness

There are however some people who are not convinced by the above arguments. They hold tenaciously to the belief that the universe comes from some kind of primordial “nothingness” which they conceptualize as existing somehow on its own, either before or during the existence of the universe.

This belief in some kind of concrete “nothingness” has many problems. First of all to posit “nothingness” is to treat it as some kind of thing in fact — so it is self-contradictory from the start. Secondly, it is impossible to even imagine actual “nothingness” so labeling it, speaking of it, or positing that it exists is simply deluded. To posit it is not actually to posit it. To imagine it is not actually to imagine it. And in fact there is no way to even conceive of nothingness actually existing, for if it were to exist it would not be nothing. Finally, even if we ignore all these logical problems and still cling to the concept of nothingness, how could anything come from nothing? Let’s examine further.

If nothing really is “nothing” it could not contain anything that could serve as a cause or origin for anything else, let alone an entire universe. So it could not give rise to anything. In fact it would be a contradiction to assert the co-existence of nothing and something as well — so even if nothingness could somehow give rise to the universe it would have to be destroyed or eliminated at the moment the universe came into existence — but if that were the case how could it give rise to the universe — it could never overlap with the universe at all so how could it even be said to give rise to it?

For example the universe could not gradually emerge from nothingness since nothingness would be completely eliminated at the very first instant of the process of emergence, and then the process would be over since there would be no more nothingness left for the rest of it to emerge from.

Similarly the universe could not emerge all-at-once from nothingness either, because for that to happen there would at least have to be a moment in which nothing and the universe co-existed — the moment in which the universe emerged.

If we don’t allow for at least that one moment of co-existence before the universe replaces nothingness, then causality is not possible to establish: there would be no way to connect the emergence of the universe as coming out of or from a pure state of nothingness that existed before it — and so there would be no point in making this claim at all.

To say that one thing comes from another thing means we have to be able to show how they are connected, and for that to be possible they have to both exist at the same time, or there has to at least be some chain of events we can point to that connects them. But if nothing and something are truly mutually exclusive then that is simply not possible to establish.

All this effort is simply to show finally and totally that nothingness is a flawed concept, and to claim that something can come from nothingness is even more flawed.

Furthermore belief in the concept of nothingness actually refutes belief in the power of science. To believe in nothingness is mutually exclusive with a belief in the principles of science, for nothingness is not measurable, not verifiable in any way, and is therefore impenetrable to science. Therefore any scientist who claims that nothingness exists or that the universe came from nothingness is a hypocrite. Anyone who cites “nothingness” as the origin of the universe is not in fact being scientific, they are abandoning science. To claim that all space and time — and all science — springs from nothingness is akin to claiming that the physical world (and therefore the domain of science) depends upon something beyond the physical world and beyond domain of science, in other words on a domain that is traditionally the focus of religion.

If we say the universe sprang forth from nothingness that is like saying that science depends on something beyond the realm of science at the fundamental level, and if we say the opposite — that the universe has always existed or there is an infinite series of universes — that is also akin to saying that science depends on something beyond what science can ever explain — for infinity, while not a contradiction at least, is equally impenetrable to science.

Therefore there really is no possible origin of the universe that does not lead to a contradiction. But let’s explore all the alternatives to really make this clear.

Refuting Ideas that the Universe Comes from Itself

We have already seen that it is a mistake to claim that the universe came from nothingness, but if the universe didn’t spring forth magically from nothingness, then perhaps it came from itself? What would this mean? It would mean that the universe already existed before the universe existed and then somehow generated itself, from itself. That is circular reasoning, and it’s also a logical contradiction because if the universe already existed then it would be meaningless to speak of it “generating” itself – it already would have existed in the first place. There’s not much more that needs to be said about this. But I’ll say it anyway, just to make it perfectly clear that this is not an option.

Perhaps we might interpret the act of self-generation 0r “coming from itself” in a slightly modified manner. For example, the universe today comes from the early universe, and they are quite different. So saying the universe of today comes from the universe of the past is not saying that the universe today comes from itself, literally; rather it is saying it comes from something else: the early universe. That is certainly one way to wiggle out of the fallacy of something coming from itself, but it just leads to an infinite regress: the fallacy of something originating from something else. The next section explores why this is a fallacy.

Refuting Ideas that the Universe Comes from Something Else

If the universe doesn’t come from nothingness, or from itself, then what does it come from? If it comes from something else, then what does that thing come from? At some point there has to be a beginning to the process. But if there is a beginning then what is before it? Whatever was before it would have to be beyond the universe and would therefore be beyond the realm of science.

To state that the universe comes from something else is to say that something else (whatever it is) is the more fundamental level or prior state of the universe. In other words to state that the universe comes from something else is really saying the universe comes from the universe, at a deeper level or an earlier time, or a different place, or in a different state or form, or all of the above.

But all such statements are either claims that the universe, taken as a whole (all states of the universe over all time and space) comes from itself, or at worst it is a circular argument that simply pushes the problem down a level: what does that other more fundamental “something” that the universe depends on come from? Again we end up in an infinite regress or a contradiction.

On the other hand, if we claim that the universe is beginningless and nonoriginated — then what is the eternity in which this “beginninglessness” is taking place? What created this eternity?

To posit that there is an eternity “beyond” the universe, or that “contains” the universe (including space and time) is already to state that there is something beyond the realm of science, something outside the universe. But if we then claim that this “eternity” is some kind of more fundamental thing, we just end up in the same infinite regress as before — it is just a subtle concept of the universe coming from something else.

Another possibility might be to claim that eternity and the universe are the same thing. This is to say that the universe is infinite in scope — space and time are boundless and contain all there is. This is logically either equivalent to the claim that the universe comes from nothing, or from itself. Neither of those options is tenable as we have already seen.

If we posit that eternity comes from nothing that is a contradiction. If it is self-originated, that is circular and also a contradiction. If we say it comes from something else, then what does that other thing come from? We end up in an infinite series of greater eternities, each containing all the lesser ones, like a Russian doll — this is an infinite regress which fails to solve the problem. Or is there a highest level of eternity and if so, what prevents there from being greater levels of eternity beyond even that — what causes the boundary between one level of eternity and another to exist and if there is a boundary, what is on the other side of it? This leads to either a contradiction or an infinite regress once again. This line of reasoning also fails to answer the question.

If one claims that the universe contains all space and time, then are the container and what is contained finite or infinite in scope? If it is finite there must be some kind of edge, if it is infinite it implies something so inconceivably vast it is frankly mystical in scope and is logically equivalent to saying the universe comes from itself.

In short, if we claim the universe comes from something else that leads to circular arguments and contradictions, or an infinite regress. If we’re willing to accept circular arguments and logical contradictions or infinite regresses as satisfactory answers then that is not very different than accepting any other self-justified claims taken on faith, such as those made by religions or even those made in fairy-tales. In fact, any such claim is really a form of religious belief disguised as science. If we are willing to think this way — and ironically it turns out that most scientists are willing to think this way — then why not also believe in God or other religious ideas as well? It would be hypocritical not to.

Refuting Conceptions of an Originated God

It’s important to note that the same logic that refutes notions that the universe comes from nothing, itself, or something else, can also be applied to any claims that there is a God. If there is a God, then like the universe, it also cannot originate from nothing, itself, or something else without leading to logical fallacies. To claim that God came from nothingness is again the something-from-nothing argument that we know does not make sense under logical scrutiny.

To claim that God comes from God is circular reasoning and contradictory. To claim that God comes from something greater than God contradicts the very notion of God and/or leads to an infinite regress which just pushes the problem down to deeper levels — where does that infinite regress of ever greater Gods come from then?

Both the universe and the concept of God have the same existential status in fact. Neither one of them has an origin that we can actually find or name without ending up in a logical mess of contradictions and infinite regressions. In this respect they are quite similar.

The conclusion is that, like the universe, God or whatever we think of as God, must also be nonoriginated. There is no other logically tenable option.

Exploring Nonorigination

If neither any possible universe nor any possible God can be said to come from nothing, itself, or something else, then that leaves only two logical conclusions:

1. The first option is that these things are not possible and not happening at all since they can’t have originated — however that option is refuted by the fact that at least in the example of the universe, something is obviously and undeniably happening right now. The presence of the universe refutes the notion that it is impossible for something to exist that does not originate from nothing, itself or something else.
2. The second option is that such things could be possible, but in a “nonoriginated” manner. But what does this mean? In short, for something to be “nonoriginated” does not mean it is non-existent, it just means that it is not dependent on some initial set of causes and conditions. One way for something to exist in an nonoriginated manner is for it to be eternal, or at least beginningless.

Option (1) is refuted by the basic fact that we do observe something happening right now. Option (2) is the only remaining option, and is not refuted in any obvious manner.

But option (2) is mind-bending. How can something beginningless exist? How could it ever have come about if there were never any initial causes or conditions to start it? It’s the primordial chicken-and-the-egg problem.

And this is where things get interesting. Various scientific theories claim the universe either has an origin or is effectively nonoriginated. Likewise religions either claim the universe and/or God has an origin or is nonoriginated.

In the first case, the claim of an origin (such as theories in which the universe started from some physical event before which there was literally nothing, or in which there was nothing and then a Deity appeared and created the universe), we can prove logically that this leads to fallacies (because the origin cannot come from nothing, itself, or something else), so this view is simply wrong, or provisional at best; it’s not a final explanation.

In the second case, the claim of nonorgination, in which the universe is held to be beginningless and possibly endless (for example a never-ending sequence of Big-Bangs and Big-Crunches, or a timelessly existing realm), this begs the question of where did this never-ending sequence come from? How could it have ever started? What is it, what is eternity and what created eternity?

In either case however, whether we use science or religion to approach the problem of the origin of the universe, we end up at the same place in the end.

The path we may travel to get there is different, and certainly the language with which we express the conclusions is quite different, but the final result is the same. Logically speaking, the universe and God must both be either nonoriginated or created by something nonoriginated. It is the only logically tenable conclusion.

In other words whether universe is thought of as purely physical, or originating from God, the only logically tenable conclusion is that it is nonoriginated. And the same goes for God. We may believe that God is greater than the universe, in other words prior to it, and in this case God and the universe are not equivalent, however, upon final analysis, even in this configuration, the only logically tenable conclusion is nonorigination.

For example, if the universe is a physical thing that was created by God, yet God is nonoriginated, then by inference the universe is also ultimately nonoriginated (via God’s nonorigination). Although provisionally we can state that the universe originates from God, since God is in this case nonoriginated, the universe is ultimately nonoriginated, for no final origin can be found or logically established.

In summary, nonorigination is the single fundamental truth of both science and religion. It is the ultimate destination of all lines of reasoning. It is where they all converge.

Unification

And now, based on the above lines of reasoning, the final capstone on the argument.

If we posit that only the physical universe exists, then we have no other choice but to say the universe itself must be nonoriginated, in other words, it must be uncaused and unconditioned — neither coming from nothing or from something else.

There is no escape from this logical conclusion.

Nonorigination is always found to be the ultimate nature of whatever is posited to exist. It doesn’t matter how many levels of reality you think there are, as soon as you posit even one, it’s “turtles all the way down,” to quote the famous expression.

In other words, if you posit the universe resting on the back of something (for example, a giant turtle) then that something must in turn rest on the back of something else (another giant turtle, for example), and so on, endlessly. The only way to not have an endless pile of turtles resting on still deeper turtles is to posit a final fundamental turtle, but that makes no sense — for that turtle would be in free-fall, meaning the entire stack of turtles would have no foundation and would topple over.

What nonorigination really means however is that the stack of turtles can be infinite or finite – it really doesn’t matter and is equivalent — either way the entire stack itself, whether just 1 turtle our countless turtles, is nonoriginated. This is not to say that the stack depends on something else (some special subtle thing we call nonorigination), it is to say that the stack itself IS nonorigination. Nonorigination is NOT something separate from that which appears to exist.

This is very hard to accept conceptually, but it is a logical conclusion. The only way to deal with it intellectually, once you derive it and are convinced there is no way around it, is to simply accept it. The universe really is beyond conception — it really cannot ever be conceived. It’s infinite and its nature is inconceivable. This is not a mystical belief, it is in a fact a very refined logical view – a logical conclusion. It is the conclusion that there is no logical conclusion that accurately and validly describes the nature of the universe. In other words, it is the logical conclusion that the actual nature of the universe is beyond the limits of logic.

Now what’s interesting, and unifying, about this conclusion is that nonorigination is a logical and scientific kind of conclusion, and yet there is something about it that is inconceivable and wondrous about it. In fact nonorigination is curiously similar to what we think of when we speak of something “Divine.” It has many similar qualities to those we usually ascribe to divinity. For example, nonorigination is uncaused, unfindable, unexplainable, inconceivable, beyond to all space and time, beyond the limits of the mind, yet it is the nature of all things,  all things could be said to come from it, or have it’s nature – it is not separate from anything, yet no thing fully encompasses it.

Surely anything that has these qualities is not merely a “thing” — there is something amazingly beyond our common idea of a thing to it. Nonorigination could be said to be at once scientific and Divine — it is something infinitely beyond all conceptual limits — it is the point where everything converges.

Nonorigination says nothing about the day-to-day “relative level of the world” and how it functions — it is a statement about the ultimate nature of everything: the originlessness and fundamental essencelessness of whatever appears. Thus when speaking of nonorigination, we can make a conceptual distinction between the relative and ultimate levels of truth. They are both true, one does not contradict the other. The ordinary appearances that we label as “things” definitely appear and function as they normally do – nothing changes – yet we know that their ultimate nature is indescribably beyond what we ordinarily assume it to be. They are nonoriginated – totally ephemeral – like dreams.

Relative truth is a level of truth within limits — specifically it is a statement that holds true locally but not globally. Ultimate truth on the other had applies globally. In this case, within the reference frame of the universe alone, we can say that any effect we observe is originated from various causes and conditions, yet within the larger (global) frame of the origin of the entire universe, it is nonoriginated – it has no first cause. In any case, whether one chooses to accept this modal logic or not is a matter of personal preference.

Nonorigination is a very subtle truth because it neither asserts or refutes the universe and/or the Divine. In fact, what appears is free to appear and function — yet if we analyze it we find it is nonoriginated. That doesn’t mean there are no causes and effects in operation, it doesn’t mean the universe is random — in fact quite the contrary will be shown later in this article.

Beyond Four Logical Extremes

In Buddhism the ultimate nonoriginated, uncaused and unconditioned primordial nature of reality is said to be “unborn.” Since it has no cause it is never actually created or “born” as some thing, yet since it is also not literal nothingness, it is not entirely non-existent, for if it were nothingness it could not be something that we could even apply the labels of nonoriginated, uncaused and unconditioned to.

That which is nonoriginated is entirely free of all four possible logical possibilities:

1. Existence
2. Non-existence
3. Both existence and non-existence
4. Neither existence nor non-existence

It doesn’t exist because it is not originated. It doesn’t not-exist because it isn’t literally nothingness. It doesn’t both exist and not-exist because that is a logical contradiction and because we already refuted the extremes of existing and not-existing individually, therefore combining them doesn’t suddenly undo that refutation (for example, if you take two non-true statements and combine them you don’t get a true statement).

The fourth logical extreme is the hardest to overcome and there are a few different arguments to conquer it. First of all the assertion of something neither existing nor not-existing is also a contradiction, via double negatives: if it doesn’t exist then this is equivalent to not-existing, and if it doesn’t not-exist then this is equivalent to existing.

Another way to refute this extreme is by the fact that there is no other alternative to existing or not-existing: to exist is to be something, whereas to not-exist is to not be something. How could there be “something” which is neither something or not-something. If it is “something” that contradicts the prong of claim that it is neither “something” or not-something. Yet if it is “not something” then that contradicts the prong of the claim that it is neither something or “not-something.” In other words, to claim that something is neither something or not-something is contradictory from the very start.

The Nonorigination of Nonorigination

Once one is familiar with the concept of nonorigination it begins to feel familiar and in that lies a subtle trap: It is extremely important not to get stuck accidentally conceiving of nonorigination as a special kind of subtle thing. In fact, nonorigination, like everything else that we might posit to exist, is nonoriginated too. So it can’t be something. It also can’t be nothing. It’s actually free of of four logical extremes of being something or nothing. It’s not any of these four logical possibilities:

• Something
• Nothing
• Something and nothing
• Neither something nor nothing

There are no other logical possibilities than these four. Nonorigination cannot be said to be or not to be.

In fact, if we look for nonorigination we don’t find it on its own. For example, you cannot find the absence of something. The absence of that thing is literally the fact that you cannot find it. Nonorigination is the absence — in any moment of experience — of anything that can be found to exist, not-exist, exist and not-exist, neither exist nor not-exist. It is an absence of four logical ways of existing, not the presence of something else that could be labelled “nonorigination.”

But this absence is not merely a rhetorical or logical point — it really is the actual fundamental nature of reality. In other words, whatever the universe is — whatever appears to us — really does have this nature of nonorigination, this complete absence of existing, not-existing, both, or neither. This means the universe is far more unexplainable than can even be imagined.

The Primordial Nature of Reality

We have found that whatever there is, it must be nonoriginated. There is no other logical alternative. Even nonorigination is nonoriginated. So while there is no final isolated thing we can point to as nonorigination itself, the fact that whatever we can point to is always found to have a nature of being nonoriginated is a fundamental truth. In fact it is perhaps the fundamental truth. It’s the one logical conclusion that we always reach no matter what we analyze.  All roads lead to nonorigination.

Nonorigination is a truth that is even more true than a mathematical truth. Mathematical truths apply to this universe, this reality. But the truth of nonorigination applies to all possible universes, all possible realities. There is no reality that is beyond it. In this sense it is the most important, most fundamental, primordial truth. Because it is primordially true to this degree, it is perhaps one of the greatest, if not THE greatest, truth that anyone can ever realize.

If we say that the universe is nonoriginated, then it doesn’t exist the way that most scientists and even most religious thinkers imagine it to. While it’s not nothingness, it’s also not something, or any other alternative. This absence of having an existential status is in fact the way it really is, it is its primordial and ultimate nature. We can also say that this absence of existential status is the primordial nature of reality itself. There is no reality other than nonorigination in fact.

This means that reality itself is beyond the limits of existing and non-existing. This may defy common sense, or even feel impossible to imagine, yet it is the only logical option — it is inconceivable yet must be so. The fact that it cannot be conceived by the ordinary logical mind does not mean it is not possible. In fact, the inconceivability of nonorigination is its very nature. This barrier of inconcievability hides it from ordinary thought — keeping it effectively secret throughout the ages – yet it is not completely hidden. All the great religions and mystical traditions ultimately reveal it – indeed it is the great secret at the heart of every great spiritual tradition.

Many great religions all agree on this point at their highest levels of philosophy: Buddhism, Christianity, Judaism, Islam and Hinduism all agree at the purest conception of the Divine is really inconceivable and unnameable, and certainly primordial (not created or conditioned by anything else). At it’s very purest essence the universal truth of all religions, and even of science, is that there must be, and is, something uncreated and unconditioned at the root of reality.

Whether the universe is theorized to have sprung out of perfect randomness or nothingness, or it is an eternity, or there are infinite parallel universes, the only logically tenable way that the entire reference frame can exist is if it is nonoriginated. This nonoriginated, uncaused and unconditioned nature, is the primordial nature of reality — of the universe and/or the Divine — regardless of whether one believes in just one, or in both.

Likewise, if one pursues science relentlessly, never accepting a partial answer or mere concept or provisional finding for the ultimate truth – one will also eventually arrive at pure logic, and from there, it is inevitable that nonorigination will be found to be the only option.

So there we have it: the essence of the universe and the essence of the Divine are the same primordial nonoriginated reality. We can call that the universe, we can call it God, or we call it Buddha, Christ, Allah, Tao, or something else. It doesn’t matter what we call it really, it is nameless.

Freedom

If something is truly nonoriginated, in other words, uncaused and uncreated, then it is totally free. In particular it is free of all concepts and beliefs about it or anything else. It is free of all limitations. We cannot say that it has a particular name and no other name. We cannot say it can only be reached through one path and not others. We cannot say that it can only be served by obeying particular rules and not others. We cannot say that only some people have access to it while others don’t, or that anyone is closer to it than anyone else. This freedom of the ultimate nature of reality can be found equally in science and religion.

Who are we to say anything that would limit something that is totally uncaused and unconditioned? Something cannot be partially free. Either it is totally free or it is not free at all. There is no middle ground. If we truly believe in a conception of a “God” that is totally free, then we have to be careful not to impose further concepts onto it or onto ourselves or anyone else. The closer one is to knowing God, the less one can really say about God.

The same goes for science: we eventually must reach similar conclusions about the fabric of reality and the origin of the universe. We may be able to describe and predict all sorts of things about the physical universe, but the deeper or farther we look in space and time, the more it starts to appear increasingly indescribable, spontaneous and unconditioned.

At the smallest scales and the largest scales, and in fact at every scale in between, the origin and nature of the cosmos is and will always be a mystery. The best we can do is categorize it and glean some understandings about how it functions, but we’ll never be able to explain it. The universe, like God, is also beyond conception. It is either uncaused and unconditioned itself — which means it is free — or it depends on something that is uncaused and unconditioned. Either way, it is free from limitations.

Think about that for a moment. If the universe is free, or depends on something that is free, then either way, what takes place in the universe is ultimately uncaused and unconditioned, meaning the universe is effectively free in both cases.

What does “free” actually mean? It means literally that anything can happen. Any universe is possible. Any set of physical laws are possible. Any kind of world or event is possible. Anything at all is possible — even things which we can’t explain and which perhaps are contradictory to the physical laws (such as anomalies, miracles, etc.). This doesn’t necessary mean anything will actually happen or that everything that is possible already exists. It simply means anything is possible. There are no limits.

Observation

But then why do only particular things appear to happen, rather than other alternatives? Why do only some things happen rather than everything happening? Why does the universe appear to obey particular physical laws? Why don’t we observe miracles or other anomalies that contradict the physical laws (note: some people do claim they observe these phenomena, so we cannot say with certainty that they don’t happen at all…)? But in any case, why does the universe seem so rational and orderly if indeed absolutely anything is possible?

One school of thought on this question (the Many Worlds interpretation of quantum mechanics) answers that in fact everything does happen, but in parallel universes, all at once. So there’s no real choice being made — all possibilities from those that are consistent with the universe we know to those which are totally outlandish or seemingly impossible do happen, all at once.

Another school of thought claims that somehow the universe makes choices and that these choices come about whenever observations take place, and that they have something to do with probability — the universe is not precisely deterministic, but not entirely non-determinstic either. If that is the case, then the act of observing something essentially causes the universe to choose what actually happens from the set of all the things that could possibly happen.

But if the universe makes quantum mechanical choices at each moment of observation, then what comes first, the act of observation, or what is observed? What creates reality, what causes the choice that selects one possibility versus all the others? Is what appears literally caused by the observer, or is it there before being observed — does it cause the observer to observer it, or does the observer cause it to be observed? It’s unclear, according to quantum mechanics at least; It’s a chicken-and-the-egg kind of problem. In fact, the situation is better characterized as a kind of feedback loop, or a dance of sorts, that’s been going on forever.

The universe is ultimately free; anything can happen. But anything does not appear to happen, only some things happen. This is currently said to happen because of choices that are made when observations take place, at least on a subatomic level.

But while observation may cause or condition reality on the quantum scale, on the macroscopic level — the level of people and cars and houses and trees, and so forth — the act of observation does not seem to function in the same manner; it doesn’t cause things to happen. Or does it? The classic Zen koan, “If a tree falls in the woods and there is nobody there to hear it, does it make a sound?” addresses this question.

In fact, if there is no observer to hear the sound, how can we say there is a sound? When the tree falls it causes vibrations, but those vibrations only make a sound if they move the eardrum of something that can hear. If there is no observer, but only a recording device in the woods, there is a recording, but not yet a sound. The sound only can be said to exist when the recording device is actually used to play the recorded sound to an observer. Until that happens, the sound is not observed.

Quantum Mechanics

This strange fact is reflected in scientific experiments such as the famous “Double Slit Experiment” and many variations. In that experiment, the act of measuring the path that a photon takes causes it to appear to appear to behave like a particle, while if you don’t measure the path it appears to behave like a wave. In fact, this effect is even stranger — experiments have been done which seem to indicate that this effect can even go backwards in time. Even if you wait to measure the path the photon takes long after it has traveled through the experiment, that observation seems to effectively go backwards in time and cause the photon to retroactively behave one way or another, in the past.

Another famous thought-experiment which illustrates the interaction between observation and reality is the “Schroedinger’s Cat” example, in which a cat in a box is either dead or alive depending on whether a random event happens, but until you actually open the box you can’t know it’s actual status — and on a quantum level in fact, until the cat is observed you cannot really say it is either dead or alive; it exists in a kind of intermediate state. The moment of observation somehow causes the intermediate state to collapse into a particular quantum state. This is very odd stuff. And for a while it was thought to really only apply at very small scales, although more recently there is some evidence that similar logic may apply even at macroscopic scales.

What this all means is that there is something about observation that seems to cause the universe to make choices about what actually happens versus what could potentially happen. Another way of expressing this is that the universe — because it is totally free — has the freedom to make choices, and this happens through the act of observation.

This would also imply that the universe is intelligent and creative, and in fact, it would be fair to say this because the universe does produce and contain things that make observations (sentient beings like humans, for example), things that are intelligent and creative. If the universe can contain intelligent, creative beings, then certainly it must be vastly greater in scope – it must be vastly more intelligent and creative as a whole than the individual beings it contains, even if only in an emergent, collective manner. Or perhaps, as some have posited, the universe isn’t happening out there on it’s own but is in a very real sense, imagining itself through an unfolding process of creatively making observations via the beings within it. If this is the case then universe could be thought of as co-creating itself via the observations of the beings within it. Of course this leads to many logical contradictions and in the end, while fascinating to ponder, it does not tell us more than we already have discovered: The universe is nonoriginated, and so is everything within it.

The Improbability of the Universe

If the universe either is something totally free, or depends on something totally free, then either way, the universe is totally free.  It cannot be partially free for that is not freedom. That is to say there are no limitations on it. Anything can happen.

How then is it that we observe particular things and not everything happening? Why don’t each of us experience all possible parallel universes? Why is the universe the way it is, and not even slightly different? Why are things the way they are? We can look at physical things and use scientific knowledge to understand their trajectories and dynamics. That certainly helps us explain a little bit about those physical things. But it doesn’t tell us why the initial conditions were not different, or why the universe is such that the physical laws and physical constants are what they are.

Even a slight change in the structure or unfolding of the universe would have resulted in a vastly different outcome — the physical laws would be different, the physical constants would have different values, and this would result in different kinds of universes. Some would have very different properties than the one we live in. Some would support life, some would not. Some would have led to our planet and human beings, some would not. Some would have stars and galaxies, yet other extreme cases would burn out and collapse into giant black holes almost immediately, while other configurations would have led to the universe breaking into countless separate universes or literally exploding and then dissolving into countless separate black holes. And there are many other possibilities too. These claims may sound wild, but in fact they are predicted using our current scientific model — if we simply change the initial conditions of the early universe slightly.

So why did things turn out the way they did? And why does our universe seem perfectly balanced to support human life — or any life for that matter? There are so many possibilities for how the universe might have unfolded, and most of those possibilities do not result in a universe that could support human life at all. In fact the universe we live in is one of the more statistically improbable outcomes. The odds of our universe happening are infinitesimally small. So how did it happen?

Furthermore, at least on a quantum level it appears that until an act of observation takes place we cannot really say the universe makes a choice about what happens. So what about the early universe — before there were any human observers, or any living things at all to make observations? So what was made the first observation? Was there a “prime observer” at the first instant of the universe, and if not, how could it have come into being since on a quantum level without being observed it could not have had a particular state.

Or alternatively was there some other kind of outside observer that made the original observations of every ancient quantum interaction, enabling the universe to make choices, at least until living observers could evolve to make their own observations? Or, has the universe effectively made all those choices retroactively — for example, now that there are observers, has the effect of our present choices gone back in time and caused the universe to make all the necessary past choices to lead to the way things are today (that one is a mind-bender, but on a quantum level it is not unreasonable or impossible to consider — space and time are not obstacles on the quantum level. For more on this, read about the Anthropic Principle in physics and cosmology)

Perhaps only universes that can support life can therefore contain observers, and so only such universes can actually happen because without observers quantum level choices cannot be made — in other words, possible universes that don’t contain observers effectively cancel themselves out and never even happen, leaving only those universes that can and do support observers. This would at least eliminate a lot of possible universes and improve the odds of universes like ours ever happening. But there are still innumerable, literally countless, variations that are possible even within that set of observer-friendly universes. Why did it turn out that exactly one and only one of those possible universes — ours — is what happened?

Here’s another question that we have to consider as well: If observation is required for the universe to make choices and effectively collapse on various states out of the space of possible states it could be in, then either there was a first observer (which leads the contradiction that the first observer could not happen because it was not observed) or there has to be an infinite regression of observers, or we couldn’t have the present universe at all. Once again, we come to the logical problems we encountered earlier when discussing the universe and God. Either we end up in contradictions or regressions.

One possibility is that the universe is an observer of itself. We know that since the universe can contain observers (for example, humans), it is capable of making observations. So why should observations only happen on the human-scale. Perhaps there are larger systems within the universe that can make observations too? But even if we believe this it still doesn’t solve the problem — even if the universe can observe itself, what observes the universe? Alternatively, if we posit some kind of outside observer of the universe, then again, what observes that? In either case, we end up with a logical contradiction or an infinite regression.

Is there any way out?

Yes, there is one, and only one, way out of this labyrinth: It all comes down to consciousness.

Conscious Awareness

Just as we found that in order for the universe to exist either it must be nonoriginated, it also must be inherently observed. Without observation, nothing could happen, choices could not be made, at least according to quantum physics. Without observation, the universe would be an amorphous field of probabilities for potential events, but no actual events would ever take place. Observation is the key that transforms potential to actual.

But if this the case, what made the first observation that started it all? The answer is that there was no first observation. Instead, observation must be inherently unified with nonorigination. Just as we have used logic to establish there can be no first cause, there also can be no first observation. Nonorigination is the absence of an origin, including any original observation. There is no other alternative, at least if observation is necessary for the universe to exist, on a quantum mechanical level. Therefore there is no first cause, there is no first observation.

In other words, the universe does not require an outside observer or a first observer — yet observation does take place. The universe observes itself via its own creations — yet since the universe is nonoriginated, this self-observation is also nonoriginated. In other words, the observation of the universe by the universe is a relative-level phenomena which in fact is ultimately nonoriginated. Observation is necessary and does take place, yet it has no ultimate origin, it is free of all logical extremes.

So what is this mysterious capacity to observe? It seems to be pretty close to what we mean when we use the terms “consciousness” or “awareness.”

We humans have this special capacity to experience our minds and senses — to not only be aware of phenomena but to be aware of our own awareness — and it appears that animals and other forms of sentient life have this capacity too. We are able to observe and react to stimulus, but also to know it. We don’t just react automatically, like springs bouncing back from being compressed. We experience what we observe — we know — we are and we know that we are. We have a sense of our own being, we are aware that we are aware. This is observation in its most naked form.

Although anything can happen in theory, sentient beings — meaning beings who are aware such as ourselves — make observations. That is our function in the universe in fact –  and these observations have quantum level repercussions that actually cause the universe to choose particular actualities from the space of possibilities, which in turn feedback to affect the probabilities of our future observations. In a very real sense, observation creates reality. Through us, the universe observes itself. This means that the universe has the capacity of awareness, at least via the medium of sentient beings that are individually aware within it.

The universe could not appear at all, according to current quantum mechanical theory, without the act of observation, and yet the act of observation (aka awareness) is something totally mysterious and itself nonoriginated. Because awareness is nonoriginated, yet is what brings about the appearances of the universe, it plays a very special role in reality. It is like the flip side of nonorigination — it is inseparable from it, like the opposite face of a coin. Nonorigination could never appear in the form of the universe without awareness, and awareness could never be possible without nonorigination.

Whether you posit that the universe itself is aware independently from the sentient beings within it, or that it is only aware via the sentient beings it contains, the conclusion is the same: the universe is sentient, it is aware. Awareness — the essence of consciousness — has a very key role in the universe, and/or in whatever we think of as God. It is in fact THE key to it all. Awareness and nonorigination are not separate phenomena; they are interpenetrating yet distinct aspects of the same inconceivable primordial nature of reality.

Cause and Effect

From this discussion so far, we have concluded that the universe is nonoriginated. That is to say, the only logical option is that it exists in a nonoriginated manner — it does not arise from nothing, itself, or something else (OR if it arises from something else then that thing must be nonoriginated, or at least something at some point that is causally upstream from it has to be nonoriginated). For example if the universe comes from God, then either God must be nonoriginated, or that which God depends on has to be nonoriginated, and so on. The point is that the series of things and things that create them is finite, not infinite. There is no infinite regress.

This does not deny the operation of cause and effect within the universe, nor does it deny that there can be an infinite series of causes and effects that lead to or stem from any event within the universe. It only denies that there can be an infinite series of causes and effects the lead to the creation of the universe as-a-whole. In other words, on the relative level, within the universe, cause and effect can operate just as science (or even various religions) might predict. However, the universe as-a-whole is not caused, or eventually depends on something that is not caused.

Therefore the universe as we know it is not contradicted by claiming that it is nonoriginated. Nor is cause and effect contradicted by stating that ultimately the universe as-a-whole, or whatever is that which is nonoriginated, is totally and completely uncaused, unconditioned and therefore free. Furthermore, even though observers — individual sentient beings — within the universe are expressions of that primordial freedom (by virtue of being aware), they are still subject to the laws of cause and effect within the universe.

For example, a particular observer may make an observation, and in doing so they perturb the universe on a quantum level, which conditions what they end up observing. Observation is a cause. What is observed is partially an effect of the act of observation, and partially an effect of other causes and conditions that relate to it. When an observer makes an observation, together with the appropriate set of causes and conditions, a particular event is observed to take place. Similarly, that event then acts as a cause or condition for other observations and events to take place for that observer and/or other observers.

In this manner everything that happens within the universe is the result of a complex network of causes and conditions, in which observers play critical roles. Observers actually change the topology of the network (the patterns of linkages between various causes and conditions and observers) whenever they make observations. This ability to rewire the network by making observations is something that is unique to sentient beings — only true observers that are conscious are capable of causing this to happen.

In fact, without observers actively making observations we cannot truly say the network exists in any particular state — it could be in any of an infinite number of possible configurations representing any of an infinite number of possible timelines of universes. The act of observation is what triggers chains of cause and effect to “fire” (almost as if they were patterns of neurons and dendrites in the brain firing sequentially to generate various thoughts). When there is no observation taking place we might say that the universe is frozen in a kind of indeterminate state. Only when observations happen are particular chains of potential cause and effect in time and space activated, and thus particular events they bring about appear to take place.

The process of cause-and-effect changes the probabilities of various events, making them more or less likely to take place, that is, to be observed. And it is the act of observation itself which triggers the chain of cause and effect, which powers it, which makes it happen. This is how the universe works on a quantum level, and also perhaps how it works on other levels too (for example, the law of Karma in Buddhism is effectively this very process of cause and effect, or what is also called dependent-arising, taking place not only in the external physical world and the body, but within all sensory modalities and even within the mind).

But is cause-and-effect required for the universe to function the way it does? Is there an alternative?

Suppose that there were no cause-and-effect within the universe. Instead imagine what it would be like if everything happened randomly. In a totally random universe every event has an equal chance of happening, so either all events would happen at once, or none of them would. We don’t see either of these taking place however. Instead we see very non-random distributions of events taking place.

When you exert a force on an object it is highly likely to exert and equal and opposite reaction on you, and it is quite unlikely that it will do the opposite of that. But in a random universe both events would be equally likely, at least over all time and space and observers and possible universes. So if the all events are equally likely then we could not have the universe we experience, in which that is certainly
not the case.

One might move the problem down a level however by suggesting that perhaps this universe is only one universe in an infinite number of parallel or possible universes, which are all equally likely to happen, and we just got lucky somehow. We happen to be observers within this one, where things fall towards the force of gravity rather than being repelled by it, and so we are able to stand here on the planet and the planet retains its atmosphere, etc.

It’s fine to hold that view, however, even if one does, within this universe at least, it appears to be as if cause and effect is in operation. Whether cause and effect sequences are really happening sequentially over time and are influenced by the free will of observers, or they all happen all at once from the perspective of eternity and thus free will is illusory, what we experience would be the same. Thus these two alternatives are equivalent.

In this universe — which is the only one we observe — it appears to us as if cause and effect processes are unfolding over time, and for all intents and purposes, from our perspectives, whether causality unfolds creatively and non-deterministically over time and in part due to the free will of observers like ourselves influencing what we observe, or it’s all preordained in eternity, its equivalent.

What this means is that for this universe to happen, cause and effect is necessary. There may be other possible universe in the set of all possibilities which may not appear to contain processes that resemble cause and effect, but we are not experiencing any of them right now, nor can we even prove they exist. So from our perspectives it is as if they do not exist. Notably however, we cannot prove they do not exist either.

Now the question is how can a universe that appears to operate by cause and effect, within it, be nonoriginated? How could a universe full of causes and effects not have a cause? How can nonorigination and cause-and-effect be compatible? Isn’t that equivalent to claiming it is an effect (the univeres) that has no cause (nonorigination), and isn’t that therefore a logical contradiction? No. To make such a claim would indeed be a logical contradiction — an effect is the result of a cause and cannot exist without a corresponding cause. The solution is to not claim that the universe is an effect, nor to claim that nonorigination is a cause.

It is contradictory to assert the existence of an effect apart from its cause. Therefore the universe cannot be asserted to be an effect that has no cause. It is simply nonoriginated, it is not the result of anything. For it to be the result of something would contradict nonorigination, which we have already found is the only logical way that the universe can exist at all (because it can’t come from nothing, itself, or something else, so therefore it must either not exist at all, or it must exist in a nonorignated manner, and since it does appear to exist, it must exist in a nonoriginated manner).

Nonorigination requires that the entire universe is not a cause nor an effect. But although the entire universe is not a cause or an effect, it can appear to contain what look like, and function within it as, causes and effects — sequences of events that are causally linked over time and space in complex interdependent networks. This is a real mind-bender and will take some time to explain. Cause-and-effect is a relative level process — it is provisionally true — but on an ultimate level the process and everything within it is nonoriginated.

For example, we probe further, into any particular event, and we trace back its origins within the universe, and if space and time are infinite, then we may find an infinitely broad and deep network of causes and effects both upstream (leading to it) and downstream (stemming from it) in time. Since these sequences are infinite, they are from a logical perspective infinite regressions. To claim that any effect comes from an infinite series of causes and effects, is logically fallacious — we cannot prove such a claim since we cannot test infinity to see whether or not the series is truly infinite or not, or even what all the causes and effects in the alleged series even are.

Cause and Effect is Nonorigination

Therefore, from a logical level, even though causes and effects may appear within an infinite universe, they too must be nonoriginated — it is the only manner in which they can be said to exist without commiting a fallacy: They must exist in a manner that is free from four logical extremes. In other words, they cannot exist, not-exist, both exist and not-exist, or neither exist or not-exist.

They cannot exist because of infinite regression. They cannot not-exist because that is a logical contradiction and also conflicts with what we observe. Combining existing and not-existing is a logical contradiction. Rejecting both existing and not-existing leads to logical contradiction and also conflicts with what we observe. So while on a relative level the process cause-and-effect appears to operate, on the ultimate level of analysis, it is equivalent to being unoriginated, from our perspectives at least.

Another way of expressing the same thing is end result is that if the space and time are infinite, then the universe as well as its contents (including all causes, effects, observations, and observers) must be ultimately nonoriginated. And since it’s not possible to have a finite sequence of causes-and-effects (because that would mean that at least one cause or effect would not have a corresponding effect or case, which is not possible (because a cause and an effect are inseperable, it is a contradiction to claim you have one without the other), a finite universe of causes and effects is impossible. Therefore finite universes are impossible, since only universes that contain causes and effects would not be random.

Therefore our universe must be infinite, because we do observe processes of cause and effect, and it also must be nonoriginated (or be equivalent to something that is nonoriginated — for example be being part of an infinite series of causes and effects of universes or by being created by some kind of God’s free will, not by cause and effect (where God is by definition not orignated by anything else). These are the only logical possibilities.

The lines of reasoning in this section, and those above it, prove that lead us to conclude that only infinite universes in which cause and effect appear to operate are possible, and that such universes (and the causes and effects they contain) must be ultimately nonoriginated, and observed, in order to be said to occur.

In other words, cause and effect is nonorigination. Whatever appears to be generated by causes and effects is ultimately nonoriginated.

Nonorigination is Cause and Effect

The same is true in the reverse direction. We cannot say that something is nonoriginated unless there is some relative-level appearance of a thing to make that statement about. The notion that nonorigination could exist on it’s own without some subject or object that is nonoriginated is a contradiction. Nonorigination is a phenomenon that requires a complementary relative-level facet, namely whatever is being asserted to be nonoriginated. To assert nonorigination apart from anything else would be like positing a penny with no sides. A penny must have a heads and tails. It can’t be a penny without them.

Therefore where there is cause and effect is the result of nonorigination and observation, and where there is nonorigination and observation there is some phenomena — some event appearing to take place, and since phenomena do not happen randomly, the only alternative is that some combination causes and effects are at work.

It is the process of observations, causes and effects that makes some possible phenomena more or less likely than others at various locations in space and time. Without such a process all possible phenomena would be equally likely at all possible locations in space and time. That would not result in our universe, or anything like our universe, at least as far as we observers can know from our positions within space and time.

Perhaps one might argue that maybe if we could see eternity we might find that our universe was randomly generated as-a-whole, but that is not possible either — for if all universes were equally likely then they would either all happen at once or none of them would happen at all. The fact that this universe appears refutes the possibility that none of them happen at least. As for the possibility of them all happening at once, this is a possibility, but we can’t determine this for sure unless we can see eternity ourselves. From our perspective, and as far as we can know, only this one is happening.

Nonorigination is therefore equivalent to cause and effect, and vice-versa. The process of cause-and-effect is not refuted by nonorigination, indeed it is required by nonorigination, and vice-versa. The proof is that this universe is appearing and functioning the way it does.

Trinity

At each moment of our lives, of each moment of observation no matter how brief or precise — there is something else taking place that is NOT nothingness and NOT exactly whatever appears to us either.

For example when we observe a tree, we see the appearance of the tree visually. That appearance is there, at least as a mere visual image, not unlike an image in a dream. It may be a real image of a real tree, or a dream image of a dream tree — but that doesn’t matter, the two cases are equivalent for in fact we really cannot tell the difference at the moment of its appearance.

The image of the tree before us is of some thing which we may believe exists “out there” in the “real world” beyond our body and mind, and that it is really just a depiction of the object out there in the visual spectrum, formed by our particular sense organs and their abilities and limitations, and then rendered via the circuitry of our brains onto some kind of internal viewing screen, or to some further set of cognitive processes which then do things like interpret it, label it as a “tree” etc. That’s all fine — whether or not any of that is really what is taking place or not — at the very moment of an appearance appearing that is all hypothetical from our own perspective. All we can know at the moment of an appearance is that it is there in its own unique way, and that we know it.

The appearance is the object side of a moment of experience. The “we know it” part of the experience is the subject side. There are these two sides to every ordinary moment of experience. This is consciousness, a dualistic interpretation of what is taking place in every moment into having two poles of subject and object that are somehow two different things. Most people spend their lives experiencing everything — themselves, the outside world, others — in this dualistic mode of cognition. Note that dualism is not inherent, it is a conceptual interpretation of raw experience. Experience itself is not dualistic — there is no actual boundary that we can find between subject and object and we cannot separate them to have one without the other. This dualistic frame of mind is a deep-seated habit and unquestioned belief that is part of our “filter” of the world. It prevents us from knowing experience the way it actually is, and instead splits it like a prism splits a single beam of light, into multiple beams of “subject” and “object” halves of each moment.

It’s key to notice that the dualistic frame of mind — ordinary consciousness — is a kind of artificial division of the moment into two parts. It comes about because a misunderstanding on our own part of what is actually taking place in each moment. What we call the object side of experience is any appearance in any sensory modality or the mind. The subject side of experience is the label we give to the part of the moment that seems to be witnessing it, or being it.

In fact there are not really two things like this, divided and separate from one another. Instead there is only one thing taking place that has both of these aspects. What is taking place is nonorigination. It has two aspects: awareness and appearance. Actually this triad can be expressed in three formulas:

Nonorigination = awareness + appearance    (N = A + A’)

Appearance = Nonorigination – awareness     (A = N – A’)

Awareness = Nonorigination – appearance     (A = N – A’)

Each moment of experience combines all three of these together into a trinity — they are unified yet still distinct. This might in fact be The Ultimate Trinity of all trinities. Furthermore, if we focus on appearance we will find that it is nonorigination. If we focus on awareness we will find that it too is nonorigination. If we try to focus on nonorigination itself we never find it, instead we always find moments of awareness plus appearance. Yet if we then try to find the awareness or appearance on their own they dissolve back to nonorigination.

This Trinity is THE most important philosophical point of all. And I cannot take credit for it. Evertying I know about it or have said here is based on what I’ve learned from Buddhism and quantum mechanics. In particular there are thousands of years of highly developed Buddhist logical treatises on precisely this point.

What is Actually Happening

When things happen they don’t just appear out of nothingness.

There isn’t really any nothingness. Nothingness is impossible by virtue of the following proof: Something appears right now. Nothing and something are mutually exclusive.

Furthemore, even IF nothing was possible, it could never generate anything because there is no way to turn nothingness into something other than nothingness.

Instead of nothingness there is a kind of space of knowing or being — what might be called awareness. This space is not inherently personalized — it has no concepts or sense of I or of being an observer, etc. This awareness has the characteristic of being nonoriginated — we cannot find it or call it a concrete, truly-existing, isolated “thing.”

At the same time as there is any knowing or being, appearances spontaneously develop within its scope. For example, this is just like dreaming. In a dream there is the space of the mind and then within this space various appearances (and other sensory experiences, for example of sound, etc.) unfold. We then identify with a particular character or perspective in the dream and the appearance of its body — and we call that “I” or “self.” That is a habit — there is nothing inherently real about the character we see ourselves as in a dream — it is not really us, not really our body or our actual mind but rather just a dream image of a body and mind. We label it as “I” or “me” out of habit. In fact, our real body is alseep in bed and is not in the dream, and our real mind and self are having the dream they are not really in the dream. Or are they?

When we dream, dreams don’t appear out of nothing, they appear out of awareness.

The same goes for all the experiences (aka appearances in various sensory modalities) that we call a moment of “our universe.” At each moment of experience there is the space of awareness plus at least some appearance. Neither the awareness or the appearances are truly-existing or even separate, they are just two aspects of nonorigination.

Nonorigination — or what in Buddhism is called “emptiness” is not a final fundamental thing that can be grasped or found either — if you find it you find that it dissolves into awareness and appearances and these dissolve back into nonorigination, endlessly.

Time unfolds as the process of this infinite loop — the Trinity of nonorigination, awareness and appearance — iterating. We are always either looking at an appearance, our awareness, or nonorigination. In either case as soon as we make such an observation what we find is that these dissolve into their counterparts. As we keep observing we trigger the process of cause-and-effect which continues to perpetuate appearances and that is what powers the universe so to speak. The energy we put into it by making observations drives it to “run” this program so to speak, endlessly iterating new moments of experience that then trigger us to make further observations and so on.

On a quantum level, the process of enacting awareness, via simple acts of observation — is literally what causes the universe to make quantum decisions that jolt the quantum field of possibilities to “collapse” onto a single possibility whenever we look for it. This is analogous to being able to cause liquid water to suddenly freeze into ice by just looking at it. When we don’t look, it’s water, but when we do look it instantly freezes into a particular shape.

We can never really see it in its water form, it always freezes just when we look for it. But we can infer the water from the frozen shapes that appear. Even ice has has waterlike qualities — it’s clear, and it melts back into water when heated after all. If we look closely at any observation (any shape made of ice in this analogy), to find its nature, this is analogous to heating the ice we are looking at, which melts it back to liquid form.

Once it melts we can no longer see it (in this analogy) until we make the next observation as we continue to look for it again. Our next observation is conditioned by the previous observation — the network of probabilities for what can appear next are changed by the previous observation — and this causes it to follow from it, statistically, rather than to be completely random — this is the process of cause-and-effect in a nutshell. Therefore our acts of observation crystallize and perpetuate our experience in an ongoing, recursive process.

Each act of observation effectively loads the dice for the next act of observation and so changes the odds of the next possible dicerolls. If the world did not work this way it would be totally random. Since it’s not totally random — it does appear to behave in a non-random fashion, we are able to make various kinds of predictions, there is a certain amount of consistency over time, this is how the universe must and does work. Cause-and-effect makes the universe non-random and non-randomness of the universe results in cause-and-effect operating.

Metascience: What are the Possible Beliefs We Might Hold?

So far we have explored some very deep questions about the origin and nature of the universe and, if one believes in God, then of God too. We have found that all these questions converge on the same ultimate reality — the reality of nonorigination.

But while they may all converge on that point eventually, there are many different schools of thought within science and religion, and regarding how they relate to one another. So how do we choose what to believe in?

It is necessary to make such choices in order to simply function on a day-to-day level, to resolve difficult moral questions, and to figure out how to live or what to do in the future. Many people just accept the choice that is handed to them by their parents, or by authorities they trust. But if one has the freedom and presence of mind to question this themselves, then on what basis can an intelligent choice be made?

It’s difficult to make sense of the range of belief system choices available, and their biggest differences or main points. One could proceed on an extensive voyage of exploration — surveying every field of science and religion over decades (what I did by default). But the whole task might be a lot faster and more efficient if one had a map to start with.

I propose a field of thinking about what to believe that we might call “Metascience” in which we make maps to help people navigate possible belief systems more intelligently. In this approach we address big philosophical questions from a higher level, starting by enumerating the space of possible beliefs we could hold about them — rather than by starting with a particular choice of belief. (Note: Another word for Metascience might simply be philosophy or metaphysics. But Philosophy and more specifically, metaphysics, have gotten totally lost, irrelevant, and non-objective. It’s time for a refresh.).

So, regarding the choice of beliefs about the relatoinship between God and the universe — Instead of immediately diving into the rathole of arguing the specifics of any one particular belief system or position on the issues, first let’s at least try try to agree on what the set of possible beliefs and positions is, and on a way to enumerate them as elegantly and usefully as possible. Is a universally agreeable metascience possible? Can we come up with a way to enumerate all the possible belief systems about God and the universe that everyone can agree with?

A Categorization of All Possible Beliefs About The Universe and God

So here is my first attempt at mapping out the exhaustive metascientific enumeration of all possible philosophies regarding God and the Universe.

(A)  Hierarchical Approach: Either the universe or God is more fundamental and/or includes the other

• Theories in which the universe – or all time and space – take place within God’s mind and/or body and is subject to God’s laws and will
• Theories in which God exists as something within the universe, subject to it’s physical laws and conditions

(B) Dualistic Approach: The universe and God are two separate things

• Theories in which God is the first-cause, creator or “blind watchmaker” who started the universe and then detached from it
• Theories in which God is watching the universe from some place outside and separate from it and may or may not intervene
• Theories in which God and the universe are separate things that co-exist within an even higher-order universe and/or pantheon
• Theories in which either God or the universe is more potent or real than the other, and they are separate things

(C) Non-Dualistic Approach: The universe and God are one unified thing

• Theories in which the universe is a vast, intelligent, aware, sentient being of some sort (that we name “God”)
• Theories in which God is just a synonym or label for the universe, or vice-versa.
• The universe and God are a dichotomy; they are neither the same nor different. The universe and God are distinct but connected or merged together as one entity (e.g. God or the universe is considered to be the fundamental aspect and the other is considered to be relative aspect of the same dichotomy, the wave-particle duality, space-time, matter-energy, mind-body, one-many, etc.). Or in other words, theories in which God and the universe are two sides of the same coin so to speak — two distinct sides of the same thing
• Theories in which either God exists and the universe doesn’t, or the universe exists and God doesn’t

(D) Existential Approach: The universe and/or God is a provisionally existing thing

• Theories in which God or the universe has only a provisional kind of existence that when analyzed proves to reduce to a deeper level of existence, or to non-existence.
• Theories in which God or the universe is merely a conceptual construct or label for something that actually has no valid existence of its own (e.g. “the horns of a rabbit”)
• Theories in which God is a conceptual label for something that is impossible (e.g. “this statement is not true)
• Theories in which God is a fictional character in a story (e.g. the character,
  “Aslan” in the Chronicles of Narnia), or is a mental fabrication or projection of someone’s mind
• Theories in which the universe is fictional but taking place – a mere fantasy or dream or a mental fabrication or projection of someone’s mind — it doesn’t exist in reality, it only exists in each of our own perceptions or at least in someone’s mind.
• Theories such as nihilism which posit that there is actually nothing at all (a contradictory, and irrational assertion)

(E) Nonconceptual Approach: The universe and/or God is inconceivable

• Theories in which the universe and/or God is neither posited to exist, not-exist, both exist and not-exist, or neither exist nor not-exist (e.g. the Buddhist theory of “emptiness” or “freedom from four logical extreme views”)
• Theories in which God or the concept of the universe is a conceptualization of something real but inconceivable (e.g. “infinity” or “zero”)
• Theories in which God and/or the universe cannot be conceived of for some axiomatic reason, such as being transcendental, beyond the scope of thought or words, beyond logic, not in the material realm, higher-dimensional, beyond time and space, etc.

There are no other major categories that I can think of regarding the Universe and God. I believe this may be then an exhaustive list. But feel free to add your own thoughts in the comments below.

Are These Questions Worthwhile?

At this point, for the skeptics among us, we should ask whether it is even meaningful and worthwhile to try to unify science and religion.

It is certainly clear that science has value. But what about religion?

Firstly, much of the world’s population believes in some form of religion and these beliefs are at the root of much of what takes place in the world — culturally, politically, economically and more. For that reason, if nothing else, we really should have as deep an understanding of all the various conceptions about God as we can. But that’s just the start. In fact there are sound scientific and philosophical reasons for exploring the topic of God as well. The theory that God originated the universe is just a valid a hypothesis as any other theory — and may even be testable at some point in the future. It’s certainly no more outlandish than some of the more exotic and hard-to-test cosmological hypotheses put forth in recent decades.

In addition, many people (including even many scientists) have had personal experiences that indicate that there is some greater entity beyond the body, mind or individual self, and perhaps even beyond the physical limits of space and time. While not everyone has had such experiences, and there is no way to validate the experiences of others, the fact that such experiences are so common and so similar, is another data-point that makes this topic worthy of consideration both by those who claim to have had such experiences, and by those who claim to have not had them. They may be artifacts of the particular architecture of the human body and brain, or they may be pointing to a deeper reality that exists just as objectively as the physical world.

Finally, from a purely scientific perspective, the origin of the universe is a mystery, and therefore the possibility of God is as much an open question as it ever was. Science has been able to learn about how the universe works to some degree, and to map parts of it, and even to form conjectures about how it has developed — but where it comes from, how it started (if it even has a beginning at all), and even where it is located ultimately are a mystery. If one posits any kind of a beginning — such as a Big Bang — then that immediately begs the question of where did the Beginning come from?

Religion has certainly learned a lot from science over the millennia. But perhaps, ironically, science has as much to learn from religion in coming millennia, at least when it comes to understanding and exploring the farthest possible reaches of cosmology and the mind. The strange relationship between mind and matter may be what the next great scientific revolution will focus on.

Similarities Between Sciences and Religions

While science and religion may disagree on certain points, at the very deepest level, they may actually be more compatible than we might think. In fact, I would go so far as to propose that a grand unification of science and religion may come about in the future as we probe ever deeper into the edges of what we know about cosmology, subatomic physics, and even our understanding of consciousness and the mind.

The strangeness at the boundaries of science already points to a reality that goes beyond a strict division of mind and matter. For example, the simple act of observation seems to have an influence on what is actually measured to take place, according to the field of quantum mechanics. Similarly, at the borders of cosmology, questions still abound on the origin, structure, and fate of the universe. And in particular, given the improbability of a universe such as ours, which seems to be precisely balanced to support the emergence of intelligent life, how did this universe happen?

In many cases scientists are very careful to state that they simply don’t know certain things yet. But at the same time, as scientific theories come into vogue, they often get out of control. For example the theory of the Big Bang. This particular theory, like most other scientific theories, has gone from being a new and contentious proposal, to a major and mainstream scientific belief, to a term that even non-scientists embraced as fact, and now today there is new evidence that perhaps the Big Bang theory is flawed and/or totally incorrect.

In the field of the philosophy of science, which studies how scientific paradigms are born, how they develop and compete, and how they are overturned, there are many other examples (the view of the Newtonian universe versus the view of Relativity, for example, or various explanations for the quantum world, and more recently String Theory). As scientific belief systems emerge, their proponents sometimes develop a kind of faith in the veracity of their beliefs that is not yet justified by the evidence, or that can never be justified in some cases — this scientific faith is quite similar to religious faith. It’s a strong belief in an explanation of nature for which there is some evidence but not yet final proof.

In fact, in science, theories can only be falsified, they can never be established as permanent and final. One never knows if and when new evidence may emerge that overturns the received view, or points to a deeper understanding.

It should also be noted that it is not the case that science is rational and religion is not. In fact, most if not all religions claim that that at least some of their beliefs are verifiable by individuals who follow a rational and repeatable process (for example, do certain things and you will get certain results). In addition at least some religions also apply rigorous formal logic to support their viewpoints. Those religions that provide an experimental method (do certain things and anyone will get predictable results) and that also apply rigorous logic to their reasoning, are applying a form of scientific method. It may be a weak form of scientific method, but it is not irrational.

So while science and religion have very different methodologies, at least with regard to their answers to the really Big Questions, such as the origin and ultimate nature of the universe, they both require a certain amount of faith, and they are both rational processes to some degree.

Differences Between Sciences and Religions

However there are also certain key differences between sciences and religions. In particular, many religions are built from axioms (creation myths, dieties, stories, traditions, and rules) which are established tautologically (they are considered to be true because simply they are defined to be true). For example, those religions which found their belief systems on ancient manuscripts that are said to have come directly for God, are building their belief systems from axioms. Such texts are claimed to be axiomatically true and cannot be disputed for they are God’s Word.

Some relgions also make the claim that the only way to test and verify the truth of their beliefs is to first take them on faith as true. In other words, the only way to verify that x is true is to first believe that x is true, and then after you believe it, the evidence will start to emerge. In other words, not having faith — asking questions or having doubts — actually prevents one from discovering the truth. It is the act of having faith that actually opens the door, so to speak.

Putting faith first is the opposite of the scientific method. The scientific method starts with doubt. It invites questioning — nothing is too sacred to examine, and if some theory can’t stand up to scrutiny, or can’t be shown through experiment or logic to be true, then it can’t be said to be scientific fact. In fact, to accept that something is true without having doubts, but prior to having proof, would be a grave scientific error. This is a key difference between the methodologies of sciences and religions in general.

However, different though it may be from the scientific method, the religious approach seems to work. Billions of people throughout human history who have followed various religions have been able to verify, for themselves at least, the authenticity of their beliefs. Whether or not the stories in a certain religious text are literally true or only metaphorical or allegorical, the fact remains that the religious process of faith, devotion, prayer and personal growth do lead, in a predictable and repeatable manner, to profound religious experiences and in some cases even to unexplainable “miracles” at times (such as the many documented cases of spontaneous healings, for example). While this is certainly not the scientific method, it appears to work pretty well nonetheless.

It is not my intention to prove that the scientific method of “proof before faith” is better or worse than the religious approach of “faith before proof.” In fact, I think they both have their place, and they both work, for different purposes.

The Boundary Between Science and Religion is Fuzzier Than One Might Think

The boundary between where science ends and religion begins is fuzzy at best. In fact, they are so intimately connected at the deepest levels that perhaps they will one day turn out to be the same thing.

Already we have found that on the quantum scale there is an intimate and strange connection between conscious observation and what appears to happen. This is not well understood yet, but it is observed experimentally. Yet we don’t have any real understanding of what consciousness is, or how it interacts with what is observed. The sciences have very little understanding of the mind at all. In fact, many scientists don’t even believe there is a mind; they think the brain is a machine and the mind is a kind of illusion. There is no soul, no consciousness, no being at all. Yet others disagree. The jury is still out.

Religions on the other hand have been studying consciousness for millennia, and some are downright scientific about it. For example the ancient Hindu and Buddhist tantric sciences provide extremely detailed and sophisticated technologies for using the breath, posture, visualization, sound, and concentration to bring about extremely unusual states of body and mind (which have recently have been measured in scientific laboratories in a number of studies). Religions are in some ways way ahead of science when it comes to understanding the mind.

The mind is one of the places where science and religion are going to collide and most likely converge. Another is the ultimate nature of the universe — the nature of space and time. The boundary between science and religion becomes fuzzier as one begins to explore the mind, the relationship between mind and matter, and simply as one views the universe at the largest or smallest scales.

There have been many past attempts by scientists at proving and disproving the existence of God. In fact the question of God’s existence was once considered an acceptable topic of enquiry by scientists such as for example, Sir Isaac Newton, and many others. In the past science was concerned with all questions about nature — including questions about the nature of reality and the mind, and even the possibility of a soul. But in recent times the focus of mainstream science has shifted far away from such topics — which are now seen as almost taboo. But why should they be taboo? They are just as much a subject for enquiry as ever. God has not been proved to exist or not-exist by science, and therefore the jury is still out. The question is whether there is any way to prove that God exists or not? It may in fact be possible to do this, scientifically, eventually.

In any case, just as is the case for the question of God, there are many scientific questions that also have not been answered yet, especially in the fields of cosmology and theoretical physics. Where does the universe come from? What created it? What came before the Big Bang (if there was a Big Bang)? What medium is space-time taking place in right now, or if there is nothing beyond space time then how did it ever happen, what does it come from, how could there be nothing beyond it? Does the universe have any edges and if so what is outside them? If there are multiple universes, what separates them from each other, or are they connected and if so how? Do all possible states of all possible universes already exist or are they truly unfolding over time? Is everything predetermined by the physical laws, or is it all open to chance, or is there some level of intelligence and creativity taking place in the universe?

Even if science someday were able to describe and define everything there is to know about the physical universe, there would still be something more to know that could not be proved or described or defined. Godel’s famous Incompleteness proof established this on a formal logical level — there will always be gaps in our knowledge — of any formal systems we construct. No formal system can be both consistent and complete at the same time. We will never have perfect scientific knowledge of the universe. And even if we could, it would simply beg the question of what is beyond that — no matter what we say the universe is, the question will always come up: well, then where does it come from and how or why is it happening?

Whether through science or religion, all paths lead to the possibility of something inconceivably beyond what we know. And this is where the boundary between science and religions gets so fuzzy that it dissolves completely.

Making a Choice

Assuming we can all at least agree on the meta-level choices (the set of possible choices), we can then discuss possible criteria for comparing, testing, and even ranking the various possible choices available to us.

At the end of the process of course there may be no final best choice that everyone accepts (in fact, I can guarantee there will not be!), nor any agreement as to what are the best or correct criteria for choosing among them. But at least we can all at least agree on what the choices are and how they compare to one another in various ways.

This could go a long way to promoting and improving tolerance and understanding. Better yet, this kind of process might even lead to useful meta-level or inter-belief-system dialogues that may eventually lead to important discoveries and even grand unifications in the future.

However, for now, regardless of what belief system we prefer, we simply have to accept that the belief system we choose, if any, is a matter of personal choice (some might call that faith, others might call it aesthetic preference, others might call it a hunch or intuition) — at least until such time as someone comes up with a way to objectively prove to everyone else that there is only one correct choice. Until that time, even if we have our own favorite belief system choice, we still have to keep some measure of open-mindedness in the face of the set of other choices available and the fact that we can’t today prove objectively (to everyone) that we made the right choice.

At least however, we should be clear that if we are willing to believe anything about the universe, there are strong reasons why we therefore should keep an open mind with regard to the possibility of God. It is not that huge a leap in fact. If we are willing to accept that something as vast and inconceivable as the universe exists, then why not God too? We really don’t have much solid grounds for holding any beliefs about such things — to do so is really just an act of faith either way. We should not have illusions about that. Believing in scientific explanations of the cosmos is really not that much different than believing in religious ones.

The good news at least is that so long as our conception of God has the properties of being uncaused and unconditioned, we are likely to have made the right choice. This also means that all the great religions, at least at their cores, are in agreement — they are all worshiping the same ultimate God, regardless of what different names they use for it. You really can’t go wrong as long as you believe in an ultimate nature that is uncaused and uncreated. However — where you certainly CAN go wrong is in imposing any further beliefs on it. And many make that mistake.

Nonduality

I have shown in this article that if one believes in the physical universe described by science, then in fact there is a logical requirement that the universe is ultimately nonoriginated.

I have also shown that the same holds for belief in God — God is also logically required to be nonoriginated.

Therefore the universe and God have the same ultimate nature.

In addition I have shown that for the universe to make choices about what happens from the set of all possibilities, observation, and therefore awareness, is required. Furthermore the nature of sentient beings, and of God, is precisely this unique capacity of awareness. Both the universe and what we think of as God are characterized by the same nature of being nonoriginated and aware.

In fact, at this level, the ultimate nature is not very different from the core idea of what God is. On an ultimate level there is not really much of a distinction between the ultimate nature of the universe and the ultimate nature of God — it is just one ultimate reality. The universe and God may be one thing, or they may be two things, or only one and not the other may exist, but in any and all of these cases, there is still only one ultimate nature: nonoriginated awareness.

There is no escape from this logic. There is no question that somewhere down the line, we must finally accept that there is something greater than the universe — whatever we think the universe is — and the characteristics of that greater thing are in fact the one common theme of the conception of God across all religions. We can name it what we want, and certainly different religions do. We also may have different perspectives on it, and add all sorts of other details. But what all the great religions have in common is an ultimate nature that is essentially transcendental.

In other words, science and religion are two sides of the same coin. You really can’t have one without the other. They are a dichotomy, but not a duality. They are distinct yet unified.

We do however have the freedom to choose our relative level beliefs about science, and our religious tradition. This freedom is an expression of the primordial freedom of the awareness — our ability to choose what to observe — and this in turn is the ultimate nature of reality. Intellectual freedom is therefore not only irrepressible, it is a reflection of the nature of the universe, it is our birthright.

On the ultimate level everything is unified, but on the relative level, there is no one correct science or religion, there will always be different views, and they probably won’t always agree on all points, and this is perfectly in accord with the freedom of the universe, and each individual. So while science and religion may be unified on the ultimate level, they certainly are not unified on the relative level, and in fact even within each indivividual field of science and each religion, there are differing viewpoints and schools of thought. And this is good.

There is a menu of different belief systems in both arenas and various items on the menu are or are not compatible with one another, or with the beliefs of others. It’s really our personal choice to make. However, what should be clear from the above argument is we have to choose both a main course and a desert: science is undeniable, and religion is unavoidable, they are two sides of the same coin.

Science and religion are different on the relative level (though not as different as some might think), but they definitely converge at ultimate level and this convergence is not a matter of faith, it is a matter of logic. Therefore, regardless of whether we prefer science or religion, or any particular sect within either camp, at least we should not err on the side of thinking they are mutually exclusive.

Unifying Physics and Consciousness: The Next Scientific Revolution

If you pursue science to the very edges, you reach nonorigination. Similarly if you become as close as possible to the diety in any religious tradition, you reach nonorigination. Moreover, nonorigination is the nature of appearances and awareness, and vice-verse. They are never separated. It’s a trinity.

The ultimate nature of the universe, and the ultimate nature of God (if you believe in a God) – must logically be precisely the same. This nature unifies the physical world of seemingly “external” sensory experiences and seemingly “internal” mental events, with the unfindable yet undeniable dimension of awareness, and the unfindable yet logically required nature of being nonoriginated.

The beauty of this is that on the ultimate level there really is no question at all about whether or not the universe exists, or whether or not God exists — the appearances of primordially aware nonorigination is the truth — and it is the most amazing miracle of all. It is irrefutable, it is logically required, and it establishes a basis for authentic and universal spirituality. One can logically derive or directly experience this logical trinity through the vehicle of focusing on and logically analyzing any phenomena (the universe, the mind, God, etc.). When this trinity is recognized as the nature of reality, and directly experienced as such, that is the deepest scientific observation or religious experience possible.

The universe including the body and all other physical things in space and time, the conceptual mind and its mental realm of thoughts and emotions, and all possible real or imaginary deities, all have at their ultimate root, the same primordially nonoriginated awareness.

Proving this once and for all in a non-religiously couched manner — using pure logical reasoning — enables science to progress beyond its present day limitations to finally begin to make sense of the strangeness of the quantum world and of the role and nature of consciousness, and the ultimate nature of space and time.

The next frontier in science will not be simply be a deeper understanding of the physical world — it will be a broader and more integrated understanding that includes both the physical world and the realm of consciousness — the mental realm.

To fully explain and understand the physical world science must find ways to include and measure the crucial role of conscious observers. Each physical event has both sides on a quantum level: the side of the observer and the side of what is observed. Science has so far been focused exclusively on understanding the side of what is observed. But what is observed cannot fully be understood or explained without an equal measure of scientific understanding of the observer and the act of observation.

Similarly, the only way to fully understand consciousness is to include and measure the crucial relationship between consciousness and the process of appearance (namely cause and effect). Both the physical world and consciousness are nonoriginated — they are empty of having an origin, not having an origin, having both, or having neither.

We don’t have the tools for measuring or exploring consciousness yet, but we’re close. Experiments that show the impact of observation on reality are indicators that consciousness is a phenomenon that can affect the observable world. This means that consciousness is indirectly detectable via measurements of the physical world around observers. It may be that consciousness — the act of observing — cannot be directly measured or observed except on its own — by and “within” each individual — but may still me indirectly measured or detected via its affects on the quantum field in the environment when it is present.

By analogy, this is similar to how space is measured, so it is possible to imagine doing this for consciousness. In the case of space, we cannot see it, touch it, or measure it directly. We can only infer things about it by measuring other things — like the way light travels, or the way things move. These indirect measurements lead us to an understanding of space.

Similarly we may be able to triangulate on consciousness by measuring the effects of various physical changes on consciousness (as reported by a conscious observer) and/or by the effects of consciousness (some observer) on physical phenomena (such as the Double Slit experiment). This is definitely an interesting possibility for further exploration, and perhaps the next scientific revolution is waiting just over the horizon in this direction.

Our civilization has not even scratched the surface of this new frontier — a unified science of physics and consciousness. But we will soon. We have to. It is unavoidable. Our quest for knowledge and understanding will take us there whether we like it or not. Already there are cracks in our present scientific theories, and experiments are showing us gaps and contradictions in our theories that we cannot explain. And the light is spilling through them.

Video from my panel at DEMO Fall ’08 on the Future of the Web is now available.

I moderated the panel, and our panelists were:

Howard Bloom, Author, The Evolution of Mass Mind from the Big Bang to the 21st Century

Peter Norvig, Director of Research, Google Inc.

Jon Udell, Evangelist, Microsoft Corporation

Prabhakar Raghavan, PhD, Head of Research and Search Strategy, Yahoo! Inc.

The panel was excellent, with many DEMO attendees saying it was the best panel they had ever seen at DEMO.

Many new and revealing insights were provided by our excellent panelists. I was particularly interested in the different ways that Google and Yahoo describe what they are working on. They covered lots of new and interesting information about their thinking. Howard Bloom added fascinating comments about the big picture and John Udell helped to speak about Microsoft’s longer-term views as well.

Enjoy!!!

This is an older version of this article. The most recent version is located here:

http://www.readwriteweb.com/archives/future_of_the_desktop.php

—————

I have spent the last year really thinking about the future of the Web. But lately I have been thinking more about the future of the desktop. In particular, here are some questions I am thinking about and some answers I’ve come up so far.

(Author’s Note: This is a raw, first-draft of what I think it will be like. Please forgive any typos — I am still working on this and editing it…)

What Will Happen to the Desktop?

As we enter the third decade of the Web we are seeing an increasing shift from local desktop applications towards Web-hosted software-as-a-service (SaaS). The full range of standard desktop office tools (word processors, spreadsheets, presentation tools, databases, project management, drawing tools, and more) can now be accessed as Web-hosted apps within the browser. The same is true for an increasing range of enterprise applications. This process seems to be accelerating.

As more kinds of applications become available in Web-based form, the Web browser is becoming the primary framework in which end-users work and interact. But what will happen to the desktop? Will it too eventually become a Web-hosted application? Will the Web browser swallow up the desktop? Where is the desktop headed?

Is the desktop of the future going to just be a web-hosted version of the same old-fashioned desktop metaphors we have today?

No. There have already been several attempts at doing this — and they never catch on. People don’t want to manage all their information on the Web in the same interface they use to manage data and apps on their local PC.

Partly this is due to the difference in user experience between using files and folders on a local machine and doing that in “simulated” fashion via some Flash-based or HTML-based imitation of a desktop. Imitations desktops to-date have simply been clunky and slow imitations of the real-thing at best. Others have been overly slick. But one thing they all have in common: None of them have nailed it. The desktop of the future – what some have called “the Webtop” – still has yet to be invented.

It’s going to be a hosted web service

Is the desktop even going to exist anymore as the Web becomes increasingly important? Yes, there will have to be some kind of interface that we consider to be our personal “home” and “workspace” — but ultimately it will have to be a unified space that all our devices connect to and share. This requires that it be a hosted online service.

Currently we have different information spaces on different devices (laptop, mobile device, PC). These will merge. Native local clients could be created for various devices, but ultimately the simplest and therefore most likely choice is to just use the browser as the client. This coming “Webtop” will provide an interface to your local devices, applications and information, as well as to your online life and information.

Today we think of our Web browser running inside our desktop as an applicaiton. But actually it will be the other way around in the future: Our desktop will run inside our browser as an application.

Instead of the browser running inside, or being launched from, some kind of next-generation desktop web interface technology, it’s will be the other way around: The browser will be the shell and the desktop application will run within it either as a browser add-in, or as a web-based application.

The Web 3.0 desktop is going to be completely merged with the Web — it is going to be part of the Web. In fact there may eventually be no distinction between the desktop and the Web anymore.

The focus shifts from information to attention

As our digital lives shift from being focused on the old fashioned desktop to the Web environment we will see a shift from organizing information spatially (directories, folders, desktops, etc.) to organizing information temporally (feeds, lifestreams, microblogs, timelines, etc.).

Instead of being just a directory, the desktop of the future is going to be more like a feed reader or social news site. The focus will be on keeping up with all the stuff flowing in and out of the user’s environment. The interface will be tuned to help the user understand what the trends are, rather than just on how things are organized.

The focus will be on helping the user to manage their attention rather than just their information. This is a leap to the meta-level: A second-order desktop. Instead of just being about the information (the first-order), it is going to be about what is happening with the information (the second-order).

Users are going to shift from acting as librarians to acting as daytraders.

Our digital roles are already shifting from acting as librarians to becoming more like daytraders. In the PC era we were all focused on trying to manage the stuff on our computers — in other words, we were acting as librarians. But this is going to shift. Librarians organize stuff, but daytraders are focused on discovering and keeping track of trends. It’s a very different focus and activity, and it’s what we are all moving towards.

We are already spending more of our time keeping up with change and detecting trends, than on organizing information. In the coming decade the shelf-life of information is going to become vanishingly short and the focus will shift from storage and recall to real-time filtering, trend detection and prediction.

The Webtop will be more social and will leverage and integrate collective intelligence

The Webtop is going to be more socially oriented than desktops of today — it will have built-in messaging and social networking, as well as social-media sharing, collaborative filtering, discussions, and other community features.

The social dimension of our lives is becoming perhaps our most important source of information. We get information via email from friends, family and colleagues. We get information via social networks and social media sharing services. We co-create information with others in communities.

The social dimension is also starting to play a more important role in our information management and discovery activities. Instead of those activities remaining as solitary, they are becoming more communal. For example many social bookmarking and social news sites use community sentiment and collaborative filtering to help to highlight what is most interesting, useful or important.

It’s going to have powerful semantic search and social search capabilities built-in

The Webtop is going to have more powerful search built-in. This search will combine both social and semantic search features. Users will be able to search their information and rank it by social sentiment (for example, “find documents about x and rank them by how many of my friends liked them.”)

Semantic search will enable highly granular search and navigation of information along a potentially open-ended range of properties and relationships.

For example you will be able to search in a highly structured way — for example, search for products you once bookmarked that have a price of $10.95 and are on-sale this week. Or search for documents you read which were authored by Sue and related to project X, in the last month.

The semantics of the future desktop will be open-ended. That is to say that users as well as other application and information providers will be able to extend it with custom schemas, new data types, and custom fields to any piece of information.

Interactive shared spaces instead of folders

Forget about shared folders — that is an outmoded paradigm. Instead, the  new metaphor will be interactive shared spaces.

The need for shared community space is currently being provided for online by forums, blogs, social network profile pages, wikis, and new community sites. But as we move into Web 3.0 these will be replaced by something that combines their best features into one. These next-generation shared spaces will be like blogs, wikis, communities, social networks, databases, workspaces and search engines in one.

Any group of two or more individuals will be able to participate in a shared space that connects their desktops for a particular purpose. These new shared spaces will not only provide richer semantics in the underlying data, social network, and search, but they will also enable groups to seamlessly and collectively add, organize, track, manage, discuss, distribute, and search for information of mutual interest.

The personal cloud

The future desktop will function like a “personal cloud” for users. It will connect all their identities, data, relationships, services and activities in one virtual integrated space. All incoming and outgoing activity will flow through this space. All applications and services that a user makes use of will connect to it.

The personal cloud may not have a center, but rather may be comprised of many separate sub-spaces, federated around the Web and hosted by different service-providers. Yet from an end-user perspective it will function as a seamlessly integrated service. Users will be able to see and navigate all their information and applications, as if they were in one connected space, regardless of where they are actually hosted. Users will be able to search their personal cloud from any point within it.

Open data, linked data and open-standards based semantics

The underlying data in the future desktop, and in all associated services it connects, will be represented using open-standard data formats. Not only will the data be open, but the semantics of the data – the schema – will also be defined in an open way. The emerigng Semantic Web provides a good infrastructure for enabling this to happen.

The value of open linked-data and open semantics is that data will not be held prisoner anywhere and can easily be integrated with other data.

Users will be able to seamlessly move and integrate their data, or parts of their data, in different services. This means that your Webtop might even be portable to a different competing Webtop provider someday. If and when that becomes possible, how will Webtop providers compete to add value?

It’s going to be smart

One of the most important aspects of the coming desktop is that it’s going to be smart. It’s going to learn and help users to be more productive. Artificial intelligence is one of the key ways that competing Webtop providers will differentiate their offerings.

As you use it, it’s going to learn about your interests, relationships, current activities, information and preferences. It will adaptively self-organize to help you focus your attention on what is most important to whatever context you are in.

When reading something while you are taking a trip to Milan it may organize itself to be more contextually relevant to that time, place and context. When you later return home to San Francisco it will automatically adapt and shift to your home context. When you do a lot of searches about a certain product it will realize your context and intent has to do with that product and will adapt to help you with that activity for a while, until your behavior changes.

Your desktop will actually be a semantic knowledge base on the back-end. It will encode a rich semantic graph of your information, relationships, interests, behavior and preferences. You will be able to permit other applications to access part or all of your graph to datamine it and provide you with value-added views and even automated intelligent assistance.

For example, you might allow an agent that cross-links things to see all your data: it would go and add cross links to relevant things onto all the things you have created or collected. Another agent that makes personalized buying recommendations might only get to see your shopping history across all shopping sites you use.

Your desktop may also function as a simple personal assistant at times. You will be able to converse with your desktop eventually — through a conversational agent interface. While on the road you will be able to email or SMS in questions to it and get back immediate intelligent answers. You will even be able to do this via a voice interface.

For example, you might ask, “where is my next meeting?” or “what Japanese restaurants do I like in LA?” or “What is Sue’s Smith’s phone number?” and you would get back answers. You could also command it to do things for you — like reminding you to do something, or helping you keep track of an interest, or monitoring for something and alerting you when it happens.

Because your future desktop will connect all the relationships in your digital life — relationships connecting people, information, behavior, prefences and applications — it will be the ultimate place to learn about your interests and preferences.

Federated, open policies and permissions

This rich graph of meta-data that comprises your future desktop will enable the next-generation of smart services to learn about you and help you in an incredibly personalized manner. It will also of course be rife with potential for abuse and privacy will be a major function and concern.

One of the biggest enabling technologies that will be necessary is a federated model for sharing meta-data about policies and permissions on data. Information that is considered to be personal and private in Web site X should be recognized and treated as such by other applications and websites you choose to share that information with. This will require a way for sharing meta-data about your policies and permissions between different accounts and applicaitons you use.

The semantic web provides a good infrastructure for building and deploying a decentralized framework for policy and privacy integration, but it has yet to be developed, let alone adopted. For the full vision of the future desktop to emerge a universally accepted standard for exchanging policy and permission data will be a necessary enabling technology.

Who is most likely to own the future desktop?

When I think about what the future desktop is going to look like it seems to be a convergence of several different kinds of services that we currently view as separate.

It will be hosted on the cloud and accessible across all devices. It will place more emphasis on social interaction, social filtering, and collective intelligence. It will provide a very powerful and extensible data model with support for both unstructured and arbitrarily structured information. It will enable almost peer-to-peer like search federation, yet still have a unified home page and user-experience. It will be smart and personalized. It will be highly decentralized yet will manage identity, policies and permissions in an integrated cohesive and transparent manner across services.

By cobbling together a number of different services that exist today you could build something like this in a decentralized fashion. Is that how the desktop of the future will come about? Or will it be a new application provided by one player with a lot of centralized market power? Or could an upstart suddently emerge with the key enabling technologies to make this possible? It’s hard to predict, but one thing is certain: It will be an interesting process to watch.

I have been thinking a lot about social networks lately, and why there are so many of them, and what will happen in that space.

Today I had what I think is a "big realization" about this.

Everyone, including myself, seems to think that there is only room for one big social network, and it looks like Facebook is winning that race. But what if that assumption is simply wrong from the start?

What if social networks are more like automobile brands? In other words, there can, will and should be many competing brands in the space?

Social networks no longer compete on terms of who has what members. All my friends are in pretty much every major social network.

I also don’t need more than one social network, for the same reason — my friends are all in all of them. How many different ways do I need to reach the same set of people? I only need one.

But the Big Realization is that no social network satisfies all types of users. Some people are more at home in a place like LinkedIn than they are in Facebook, for example. Others prefer MySpace.  There are always going to be different social networks catering to the common types of people (different age groups, different personalities, different industries, different lifestyles, etc.).

The Big Realization implies that all the social networks are going to be able to interoperate eventually, just like almost all email clients and servers do today. Email didn’t begin this way. There were different networks, different servers and different clients, and they didn’t all speak to each other. To communicate with certain people you had to use a certain email network, and/or a certain email program. Today almost all email systems interoperate directly or at least indirectly. The same thing is going to happen in the social networking space.

Today we see the first signs of this interoperability emerging as social networks open their APIs and enable increasing integration. Currently there is a competition going on to see which "open" social network can get the most people and sites to use it. But this is an illusion. It doesn’t matter who is dominant, there are always going to be alternative social networks, and the pressure to interoperate will grow until it happens. It is only a matter of time before they connect together.

I think this should be the greatest fear at companies like Facebook. For when it inevitably happens they will be on a level playing field competing for members with a lot of other companies large and small. Today Facebook and Google’s scale are advantages, but in a world of interoperability they may actually be disadvantages — they cannot adapt, change or innovate as fast as smaller, nimbler startups.

Thinking of social networks as if they were automotive brands also reveals interesting business opportunities. There are still several unowned opportunities in the space.

Myspace is like the car you have in high school. Probably not very expensive, probably used, probably a bit clunky. It’s fine if you are a kid driving around your hometown.

Facebook is more like the car you have in college. It has a lot of your junk in it, it is probably still not cutting edge, but its cooler and more powerful.

LinkedIn kind of feels like a commuter car to me. It’s just for business, not for pleasure or entertainment.

So who owns the "adult luxury sedan" category? Which one is the BMW of social networks?

Who owns the sportscar category? Which one is the Ferrari of social networks?

Who owns the entry-level commuter car category?

Who owns equivalent of the "family stationwagon or minivan" category?

Who owns the SUV and offroad category?

You see my point. There are a number of big segments that are not owned yet, and it is really unlikely that any one company can win them all.

If all social networks are converging on the same set of features, then eventually they will be close to equal in function. The only way to differentiate them will be in terms of the brands they build and the audience segments they focus on. These in turn will cause them to emphasize certain features more than others.

In the future the question for consumers will be "Which social network is most like me? Which social network is the place for me to base my online presence?"

Sue may connect to Bob who is in a different social network — his account is hosted in a different social network. Sue will not be a member of Bob’s service, and Bob will not be a member of Sue’s, yet they will be able to form a social relationship and communication channel. This is like email. I may use Outlook and you may use Gmail, but we can still send messages to each other.

Although all social networks will interoperate eventually, depending on each person’s unique identity they may choose to be based in — to live and surf in — a particular social network that expresses their identity, and caters to it. For example, I would probably want to be surfing in the luxury SUV of social networks at this point in my life, not in the luxury sedan, not the racecar, not in the family car, not the dune-buggy. Someone else might much prefer an open source, home-built social network account running on a server they host. It shouldn’t matter — we should still be able to connect, share stuff, get notified of each other’s posts, etc. It should feel like we are in a unified social networking fabric, even though our accounts live in different services with different brands, different interfaces, and different features.

I think this is where social networks are heading. If it’s true then there are still many big business opportunities in this space.

There has been a lot of hype about artificial intelligence over the years. And recently it seems there has been a resurgence in interest in this topic in the media. But artificial intelligence scares me. And frankly, I don’t need it. My human intelligence is quite good, thank you very much. And as far as trusting computers to make intelligent decisions on my behalf, I’m skeptical to say the least. I don’t need or want artificial intelligence.

No, what I really need is artificial stupidity.

I need software that will automate all the stupid things I presently have to waste far too much of my valuable time on. I need something to do all the stupid tasks — like organizing email, filing documents, organizing folders, remembering things, coordinating schedules, finding things that are of interest, filtering out things that are not of interest, responding to routine messages, re-organizing things, linking things, tracking things, researching prices and deals, and the many other rote information tasks I deal with every day.

The human brain is the result of millions of years of evolution. It’s already the most intelligent thing on this planet. Why are we wasting so much of our brainpower on tasks that don’t require intelligence? The next revolution in software and the Web is not going to be artificial intelligence, it’s going to be creating artificial stupidity: systems that can do a really good job at the stupid stuff, so we have more time to use our intelligence for higher level thinking.

The next wave of software and the Web will be about making software and the Web smarter. But when we say "smarter" we don’t mean smart like a human is smart, we mean "smarter at doing the stupid things that humans aren’t good at." In fact humans are really bad at doing relatively simple, "stupid" things — tasks that don’t require much intelligence at all.

For example, organizing. We are terrible organizers. We are lazy, messy, inconsistent, and we make all kinds of errors by accident. We are terrible at tagging and linking as well, it turns out. We are terrible at coordinating or tracking multiple things at once because we are easily overloaded and we can really only do one thing well at a time. These kinds of tasks are just not what our brains are good at. That’s what computers are for – or should be for at least.

Humans are really good at higher level cognition: complex thinking, decisionmaking, learning, teaching, inventing, expressing, exploring, planning, reasoning, sensemaking, and problem solving — but we are just terrible at managing email, or making sense of the Web. Let’s play to our strengths and use computers to compensate for our weaknesses.

I think it’s time we stop talking about artificial intelligence — which nobody really needs, and fewer will ever trust. Instead we should be working on artificial stupidity. Sometimes the less lofty goals are the ones that turn out to be most useful in the end.

I’ve been thinking since 1994 about how to get past a fundamental barrier to human social progress, which I call “The Collective IQ Barrier.” Most recently I have been approaching this challenge in the products we are developing at my stealth venture, Radar Networks.

In a nutshell, here is how I define this barrier:

The Collective IQ Barrier: The potential collective intelligence of a human group is exponentially proportional to group size, however in practice the actual collective intelligence that is achieved by a group is inversely proportional to group size. There is a huge delta between potential collective intelligence and actual collective intelligence in practice. In other words, when it comes to collective intelligence, the whole has the potential to be smarter than the sum of its parts, but in practice it is usually dumber.

Why does this barrier exist? Why are groups generally so bad at tapping the full potential of their collective intelligence? Why is it that smaller groups are so much better than large groups at innovation, decision-making, learning, problem solving, implementing solutions, and harnessing collective knowledge and intelligence?

I think the problem is technological, not social, at its core. In this article I will discuss the problem in more depth and then I will discuss why I think the Semantic Web may be the critical enabling technology for breaking through the Collective IQ Barrier.

The Effective Size of Groups

For millions of years — in fact since the dawn of humanity — humansocial organizations have been limited in effective size. Groups aremost effective when they are small, but they have less collectiveknowledge at their disposal. Slightly larger groups optimize both effectiveness and access to resources such as knowledge and expertise. In my own experience working on many different kinds of teams, I think that the sweet-spot is between 20and 50 people. Above this size groups rapidly become inefficient andunproductive.

The Invention of Hierarchy

The solution that humans have used to get around this limitation in the effective size of groups is hierarchy.When organizations grow beyond 50 people we start to break them intosub-organizations of less than 50 people. As a result if you look atany large organization, such as a Fortune 100 corporation, you find ahuge complex hierarchy of nested organizations and cross-functionalorganizations. This hierarchy enables the organization to createspecialized “cells” or “organs” of collective cognition aroundparticular domains (like sales, marketing, engineering, HR, strategy,etc.) that remain effective despite the overall size of theorganization.

By leveraging hierarchy an organization of even hundreds ofthousands of members can still achieve some level of collective IQ as awhole. The problem however is that the collective IQ of the wholeorganization is still quite a bit lower than the combined collectiveIQ’s of the sub-organizations that comprise it. Even in well-structured, well-managed hierarchies, the hierarchy is still less thanthe sum of it’s parts. Hierarchy also has limits — the collective IQof an organization is also inversely proportional to the number ofgroups it contains, and the average number of levels of hierarchybetween those groups (Perhaps this could be defined more elegantly asan inverse function of the average network distance between groups inan organization).

The reason that organizations today still have to make suchextensive use of hierarchy is that our technologies for managingcollaboration, community, knowledge and intelligence on a collectivescale are still extremely primitive. Hierarchy is still one of the only and best solutions we have at our disposal. But we’re getting better fast.

Modern organizations are larger and far more complex than ever would have beenpractical in the Middle Ages, for example. They contain more people,distributed more widely around the globe, with more collaboration andspecialization, and more information, making more rapid decisions, thanwas possible even 100 years ago. This is progress.

Enabling Technologies

There have beenseveral key technologies that made modern organizations possible: the printing press,telegraph, telephone, automobile, airplane, typewriter, radio,television, fax machine, and personal computer. These technologies haveenabled information and materials to flow more rapidly, at less cost,across ever more widely distributed organizations. So we can see that technology does make a big difference in organizational productivity. The question is, can technology get us beyond the Collective IQ Barrier?

The advent of the Internet, and in particular the World Wide Webenabled a big leap forward in collective intelligence. These technologies havefurther reduced the cost to distributing and accessing information andinformation products (and even “machines” in the form of software codeand Web services). They have made it possible for collectiveintelligence to function more rapidly, more dynamically, on a wider scale, and at lesscost, than any previous generation of technology.

As a result of evolution of the Web we have seen new organizationalstructures begin to emerge that are less hierarchical, moredistributed, and often more fluid. For example, virtual teams that caninstantly form, collaborate across boundaries, and then dissolve backinto the Webs they come from when their job is finished. Thisprocess is now much easier than it ever was. Numerous hosted Web-basedtools exist to facilitate this: email, groupware, wikis, messageboards, listservers, weblogs, hosted databases, social networks, searchportals, enterprise portals, etc.

But this is still just the cusp of this trend. Even today with thecurrent generation of Web-based tools available to us, we are still notable to effectively tap much more of the potential Collective IQ of ourgroups, teams and communities. How do we get from where we are today(the whole is dumber than the sum of its parts) to where we want to bein the future (the whole is smarter than the sum of its parts)?

The Future of Productivity

The diagram below illustrates how I think about the past, present and future of productivity. In my view, from the advent of PC’s onwards we have seen a rapid growth in individual and group productivity, enabling people to work with larger sets of information, in larger groups. But this will not last — soon as we reach a critical level of information and groups of ever larger size, productivity will start to decline again, unless new technologies and tools emerge to enable us to cope with these increases in scale and complexity. You can read more about this diagram here.

http://novaspivack.typepad.com/nova_spivacks_weblog/2007/02/steps_towards_a.html

In the last 20 years the amount of information that knowledgeworkers (and even consumers) have to deal with on a daily basis has mushroomed by a factor of almost 10orders of magnitude and it will continue like this for several moredecades. But our information tools — and particular our tools forcommunication, collaboration, community, commerce and knowledgemanagement — have not advanced nearly as quickly. As a result thetools that we are using today to manage our information andinteractions are grossly inadequate for the task at hand: They were simply not designed tohandle the tremendous volumes of distributed information, and the rate of change ofinformation, that we are witnessing today.

Case in point: Email. Email was never designed for what it is beingused for today. Email was a simple interpersonal notification andmessaging tool and essentially that is what it is good for. But todaymost of us use our email as a kind of database, search engine,collaboration tool, knowledge management tool, project management tool,community tool, commerce tool, content distribution tool, etc. Emailwasn’t designed for these functions and it really isn’t very productive whenapplied to them.

For groups the email problem is even worse than it is for individuals –not only is everyone’s individual email productivity declining anyway,but collectively as groupsize increases (and thus group information size increases as well),there is a multiplier effect that further reduces everyone’semail productivity in inverse proportion to the size of the group.Email becomes increasingly unproductive as group size and informationsize increase.

This is not just true of email, however, it’s true of almost all theinformation tools we use today: Search engines, wikis, groupware,social networks, etc. They all suffer from this fundamental problem.Productivity breaks down with scale — and the problem is exponentially worse than it is for individuals in groups and organizations. But scale is increasing incessantly — that is a fact — and it will continue to do so for decades at least. Unless something is done about this we will simply be completely buried in our own information within about a decade.

The Semantic Web

I think the Semantic Web is a critical enabling technology that will help us get through this transition. It willenable the next big leap in productivity and collective intelligence.It may even be the technology that enables humans to flip the ratio so thatfor the first time in human history, larger groups of people canfunction more productively and intelligently than smaller groups. Itall comes down to enabling individuals and groups to maintain (andultimately improve) their productivity in theface of the continuing explosion in information and social complexitythat they areexperiencing.

The Semantic Web provides a richer underlying fabric for expressing,sharing, and connecting information. Essentially it provides a betterway to transform information into useful knowledge, and to share andcollaborate with it. It essentially upgrades the medium — in this case the Web and any other data that is connected to the Web — that we use for our information today.

By enriching the medium we can inturn enable new leaps in how applications, people, groups andorganizations can function. This has happened many times before in thehistory of technology.  The printing press is one example. The Web is a more recent one. The Web enriched themedium (documents) with HTML and a new transport mechanism, HTTP, forsharing it. This brought about one of the largest leaps in humancollective cognition and productivity in history. But HTML really onlydescribes formatting and links. XML came next, to start to provide away to enrich the medium with information about structure –the parts of documents. The Semantic Web takes this one step further –it provides a way to enrich the medium with information about the meaning of the structure — what are those parts, what do various links actually mean?

Essentially the Semantic Web provides a means to abstract andexternalize human knowledge about information — previously the meaningof information lived only in our heads, and perhaps in certainspecially-written software applications that were coded to understandcertain types of data. The Semantic Web will disrupt this situation by providingopen-standards for encoding this meaning right into the medium itself.Any application that can speak the open-standards of the Semantic Webcan then begin to correctly interpret the meaning of information, andtreat it accordingly, without having to be specifically coded tounderstand each type of data it might encounter.

This is analogous to the benefit of HTML. Before HTML everyapplication had to be specifically coded to each different documentformat in order to display it. After HTML applications could all juststandardize on a single way to define the formats of differentdocuments. Suddenly a huge new landscape of information becameaccessible both to applications and to the people who used them.The Semantic Web does something similar: It provides a way to makethe data itself “smarter” so that applications don’t have to know somuch to correctly interpret it. Any data structure — a document or adata record of any kind — that can be marked up with HTML to define its formatting, can also be marked up with RDFand OWL (the languages of the Semantic Web) to define its meaning.

Once semantic metadata is added, the document can not only bedisplayed properly by any application (thanks to HTML and XML), but itcan also be correctly understood by that application. For example theapplication can understand what kind of document it is, what it isabout, what the parts are, how the document relates to other things,and what particular data fields and values mean and how they map todata fields and values in other data records around the Web.

The Semantic Web enriches information with knowledge about what thatinformation means, what it is for, and how it relates to other things.With this in hand applications can go far beyond the limitations ofkeyword search, text processing, and brittle tabular data structures.Applications can start to do a much better job of finding, organizing,filtering, integrating, and making sense of ever larger and morecomplex distributed data sets around the Web.

Another great benefit ofthe Semantic Web is that this additional metadata can be added in atotally distributed fashion. The publisher of a document can add theirown metadata and other parties can then annotate that with their ownmetadata. Even HTML doesn’t enable that level of cooperative markup (exceptperhaps in wikis). It takes a distributed solution to keep up with ahighly distributed problem (the Web). The Semantic Web is just such adistributed solution.

The Semantic Web will enrich information and this in turn will enable people, groups and applications to work with information more productively. In particular groups and organizations will benefit the most because that is where the problems of information overload and complexity are the worst. Individuals at least know how they organize their own information so they can do a reasonably good job of managing their own data. But groups are another story — because people don’t necessarily know how others in their group organize their information. Finding what you need in other people’s information is much harder than finding it in your own.

Where the Semantic Web can help with this is by providing a richer fabric for knowledge management. Information can be connected to an underlying ontology that defines not only the types of information available, but also the meaning and relationships between different tags or subject categories, and even the concepts that occur in the information itself. This makes organizing and finding group knowledge easier. In fact, eventually the hope is that people and groups will not have to organize their information manually anymore — it will happen in an almost fully-automatic fashion. The Semantic Web provides the necessary frameworks for making this possible.

But even with the Semantic Web in place and widely adopted, moreinnovation on top of it will be necessary before we can truly breakpast the Collective IQ Barrier such that organizations can in practiceachieve exponential increases in Collective IQ. Human beings are only able to cope with a few chunks ofinformation at a given moment, and our memories and ability to processcomplex data sets are limited. When group size and data size growbeyond certain limits, we simply cannot cope, we become overloaded andjammed, even with rich Semantic Web content at our disposal.

Social Filtering and Social Networking — Collective Cognition

Ultimately, to remain productive in the face of such complexity wewill need help. Often humans in roles that require them to cope with large scales of information, relationships andcomplexity hire assistants, but not all of us can affordto do that, and in some cases even assistants are not able to keep upwith the complexity that has to be managed.

Social networking andsocial filtering are two ways to expand the number of “assistants” weeach have access to, while also reducing the price of harnessing the collective intelligence of those assistants to just about nothing. Essentially these methodologies enable people toleverage the combined intelligence and attention of large communitiesof like-minded people who contribute their knowledge and expertise for free. It’s a collective tit-for-tat form of altruism.

For example, Diggis a community that discovers the most interesting news articles. Itdoes this by enabling thousands of people to submit articles and voteon them. What Digg adds are a few clever algorithms on top of this for rankingarticles such that the most active ones bubble up to the top. It’s notunlike a stock market trader’s terminal, but for a completely differentclass of data. This is a great example of social filtering.

Anothergood example are prediction markets, where groups of people vote onwhat stock or movie or politician is likely to win — in some cases bybuying virtual stock in them — as a means to predict the future. Ithas been shown that prediction markets do a pretty good job of makingaccurate predictions in fact. In addition expertise referral serviceshelp people get answers to questions from communities of experts. Theseservices have been around in one form or another for decades and haverecently come back into vogue with services like Yahoo Answers. Amazonhas also taken a stab at this with their Amazon Mechanical Turk, whichenables “programs” to be constructed in which people perform the work.

I think social networking, social filtering, prediction markets,expertise referral networks, and collective collaboration are extremelyvaluable. By leveraging other people individuals and groups can stayahead of complexity and can also get the benefit of wide-areacollective cognition. These approaches to collective cognition arebeginning to filter into the processes of organizations and othercommunities. For example, there is recent interest in applying socialnetworking to niche communities and even enterprises.

The Semantic Webwill enrich all of these activities — making social networks andsocial filtering more productive. It’s not an either/or choice — thesetechnologies are extremely compatible in fact. By leveraging acommunity to tag, classify and organize content, for example, themeaning of that content can be collectively enriched. This is alreadyhappening in a primitive way in many social media services. TheSemantic Web will simply provide a richer framework for doing this.

The combination of the Semantic Web with emerging social networkingand social filtering will enable something greater than either on it’sown. Together, together these two technologies will enable much smarter groups, social networks, communities and organizations. But this still will not get us all the way past the Collective IQBarrier. It may get us close the threshold though. To cross thethreshold we will need to enable an even more powerful form ofcollective cognition.

The Agent Web

To cope with the enormous future scale andcomplexity of the Web, desktop and enterprise, each individual and group willreally need not just a single assistant, or even a community of humanassistants working on common information (a social filtering communityfor example), they will need thousands or millions of assistants working specificallyfor them. This really only becomes affordable and feasible if we canvirtualize what an “assistant” is.

Human assistants are at the top ofthe intelligence pyramid — they are extremely smart and powerful, and they are expensive — they  should not beused for simple tasks like sorting content, that’s just a waste oftheir capabilities. It would be like using a supercomputer array tospellcheck a document. Instead, we need to free humans up to do thereally high-value information tasks, and find a way to farm out thelow-value, rote tasks to software. Software is cheap or even free and it can be replicated as much asneeded in order to parallelize. A virtual army of intelligent agents is less expensive than a single human assistant, and much more suited to sifting through millions of Web pages every day.

But where will these future intelligent agents get their intelligence? In past attempts at artificial intelligence, researchers tried to buildgigantic expert systems that could reason as well as a small child forexample. These attempts met with varying degrees of success, but theyall had one thing in common: They were monolithic applications.

I believe that that future intelligent agents should be simple. They should not be advanced AI programs or expert systems. They should be capable of a few simple behaviors, the most important of which is to reason against sets of rules and semantic data. The basic logic necessary for reasoning is not enormous and does not require any AI — it’s just the ability to follow logical rules and perhaps do set operations. They should be lightweight and highly mobile. Insteadof vast monolithic AI, I am talking about vast numbers of very simpleagents that working together can do  emergent, intelligent operations en masse.

For example search — you might deploy a thousand agents to search all the sites about Italy for recipes and then assemble those results into a database instantaneously.  Or you might dispatch a thousand or more agents to watch for a job that matches your skills and goals across hundreds of thousands or millions of Websites. They could watch and wait until jobs that matched your criteria appeared, and then they could negotiate amongst themselves to determine which of the possible jobs they found were good enough to show you. Another scenario might be commerce — you could dispatch agents to find you the best deal on a vacation package, and they could even negotiate an optimal itinerary and price for you. All you would have to do is choose between a few finalist vacation packages and make the payment. This could be a big timesaver.

The above examples illustrate how agents might help an individual, but how might they help a group or organization? Well for one thing agents could continuously organize and re-organize information for a group. They could also broker social interactions — for example, by connecting people to other people with matching needs or interests, or by helping people find experts who could answer their questions. One of the biggest obstacles to getting past the Collective IQ Barrier is simply that people cannot keep track of more than a few social relationships and information sources at aany given time — but with an army of agents helping them, individuals might be able to cope with more relationships and data sources at once; the agents would act as their filters, deciding what to let through and how much priority to give it. Agents can also help to make recommendations, and to learn to facilitate and even automate various processes such as finding a time to meet, or polling to make a decision, or escalating an issue up or down the chain of command until it is resolved.

To make intelligent agents useful, they will need access to domain expertise. But the agents themselves will not contain any knowledge or intelligence of their own. The knowledge will exist outside on the Semantic Web, and so will the intelligence. Their intelligence, like their knowledge, will be externalized and virtualized in the form of axioms or rules that will exist out on the Web just like web pages.

For example, a set of axioms about travel could be published to the Web in the form of a document that formally defined them. Any agent that needed to process travel-related content could reference these axioms in order to reason intelligently about travel in the same way that it might reference an ontology about travel in order to interpret travel data structures. The application would not have to be specifically coded to know about travel — it could be a generic simple agent — but whenever it encountered travel-related content it could call up the axioms about travel from the location on the Web where they were hosted, and suddenly it could reason like an expert travel agent. What’s great about this is that simple generic agents would be able to call up domain expertise on an as-needed basis for just about any domain they might encounter. Intelligence — the heuristics, algorithms and axioms that comprise expertise, would be as accessible as knowledge — the data and connections between ideas and information on the Web.

The axioms themselves would be created by human experts in various domains, and in some cases they might even be created or modified by agents as they learned from experience. These axioms might be provided for free as a public service, or as fee-based web-services via API’s that only paying agents could access.

The key is that model is extremely scaleable — millions or billions of axioms could be created, maintained, hosted, accessed, and evolved in a totally decentralized and parallel manner by thousands or even hundreds of thousands of experts all around the Web. Instead of a few monolithic expert systems, the Web as a whole would become a giant distributed system of experts. There might be varying degrees of quality among competing axiom-sets available for any particular domain, and perhaps a ratings system could help to filter them over time. Perhaps a sort of natural selection of axioms might take place as humans and applications rated the end-results of reasoning using particular sets of axioms, and then fed these ratings back to the sources of this expertise, causing them to get more or less attention from other agents in the future. This process would be quite similar to the human-level forces of intellectual natural-selection at work in fields of study where peer-review and competition help to filter and rank ideas and their proponents.

Virtualizing Intelligence

What I have been describing is the virtualization of intelligence — making intelligence and expertise something that can be “published” to the Web and shared just like knowledge, just like an ontology, a document, a database, or a Web page. This is one of the long-term goals of the Semantic Web and it’s already starting now via new languages, such as SWRL, that are being proposed for defining and publishing axioms or rules to the Web. For example, “a non-biologicalparent of a person is their step-parent” is asimple axiom. Another axiom might be, “A child of a sibling of your parent is your cousin.” Using such axioms, an agent could make inferences and do simple reasoning about social relationships for example.

SWRL and other proposed rules languages provide potentialopen-standards for defining rules and publishing them to the Web sothat other applications can use them. By combining these rules withrich semantic data, applications can start to do intelligent things,without actually containing any of the intelligence themselves. The intelligence– the rules and data — can live “out there” on the Web, outside the code of various applications.

All theapplications have to know how to do is find relevant rules, interpret them, and apply them. Even the reasoning that may be necessary can be virtualized into remotely accessible Web services so applications don’t even have to do that part themselves (although many may simply include open-source reasoners in the same way that they include open-source databases or search engines today).

In other words, just as HTML enables any app to process and formatany document on the Web, SWRL + RDF/OWL may someday enable any application to reasonabout what the document discusses. Reasoning is the last frontier. Byvirtualizing reasoning — the axioms that experts use to reason aboutdomains — we can really begin to store the building blocks of humanintelligence and expertise on the Web in a universally-accessibleformat. This to me is when the actual “Intelligent Web” (what I callWeb 4.0) will emerge.

The value of this for groups and organizations is that they can start to distill their intelligence from individuals that comprise them into a more permanent and openly accessible form — axioms that live on the Web and can be accessed by everyone. For example, a technical support team for a product learns many facts and procedures related to their product over time. Currently this learning is stored as knowledge in some kind of tech support knowledgebase. But the expertise for how to find and apply this knowledge still resides mainly in the brains of the people who comprise the team itself.

The Semantic Web provides ways to enrich the knowledgebase as well as to start representing and saving the expertise that the people themselves hold in their heads, in the form of sets of axioms and procedures. By storing not just the knowledge but also the expertise about the product, the humans on the team don’t have to work as hard to solve problems — agents can actually start to reason about problems and suggest solutions based on past learning embodied in the common set of axioms. Of course this is easier said than done — but the technology at least exists in nascent form today. In a decade or more it will start to be practical to apply it.

Group Minds

Someday in the not-too-distant-future groups will be able toleverage hundreds or thousands of simple intelligent agents. Theseagents will work for them 24/7 to scour the Web, the desktop, theenterprise, and other services and social networks they are related to. They will help both the individuals as well as the collectives as-a-whole. They willbe our virtual digital assistants, always alert and looking for thingsthat matter to us, finding patterns, learning on our behalf, reasoningintelligently, organizing our information, and then filtering it,visualizing it, summarizing it, and making recommendations to us sothat we can see the Big Picture, drill in wherever we wish, and makedecisions more productively.

Essentially these agents will give groups something like their own brains. Today the only brains in a group reside in the skulls of the people themselves. But in the future perhaps we will see these technologies enable groups to evolve their own meta-level intelligences: systems of agents reasoning on group expertise and knowledge.

This will be a fundamental leap to a new order of collective intelligence. For the first time groups will literally have minds of their own, minds that transcend the mere sum of the individual human minds that comprise their human, living facets. I call these systems “Group Minds” and I think they are definitely coming. In fact there has been quite a bit of research on the subject of facilitating group collaboration with agents, for example, in government agencies such as DARPA and the military, where finding ways to help groups think more intelligently is often a matter of life and death.

The big win from a future in which individuals and groups canleverage large communities of intelligent agents is that they will bebetter able to keep up with the explosive growth of information complexity andsocial complexity. As the saying goes, “it takes a village.” There is just too much information, and too many relationships, changing too fast and this is only going to get more intense in years to come. The only way to cope with such a distributed problem is a distributed solution.

Perhaps by 2030 it will not be uncommon for Individuals and groups to maintain largenumbers of virtual assistants — agents that will help them keep abreast of themassively distributed, always growing and shifting information and sociallandscapes. When you really think about this, how else could we eversolve this? This is really the only practical long-term solution. But today it is still a bit of a pipedream; we’re not there yet. The key however is that we are closer than we’ve ever been before.

Conclusions

The Semantic Web provides the key enabling technology for all ofthis to happen someday in the future. By enriching the content of theWeb it first paves the way to a generation of smarter applications andmore productive individuals, groups and organizations.

The next majorleap will be when we begin to virtualize reasoning in the form ofaxioms that become part of the Semantic Web. This will enable a newgeneration of applications that can reason across information andservices. This will ultimately lead to intelligent agents that will be able to assist individuals,groups, social networks, communities, organizations and marketplaces sothat they can remain productive in the fact of the astonishinginformation and social network complexity in our future.

By adding more knowledge into our information, the Semantic Webmakes it possible for applications (and people) to use information moreproductively. By adding more intelligence between people,  information,and applications, the Semantic Web will also enable people andapplications to become smarter. In the future, these more-intelligentapps will facilitate higher levels of individual and collectivecognition by functioning as virtual intelligent assistants forindividuals and groups (as well as for online services).

Once we begin to virtualize not just knowledge (semantics) but alsointelligence (axioms) we will start to build Group Minds — groups that have primitive minds of their own. When we reach this point we will finally enable organizations to breakpast the Collective IQ Barrier: Organizations will start to becomesmarter than the sum of their parts. The intelligence of anorganization will not just be from its people, it will also come fromits applications. The number of intelligent applications in anorganization may outnumber the people by 1000 to 1, effectivelyamplifying each individual’s intelligence as well as the collectiveintelligence of the group.

Because software agents work all the time,can self-replicate when necessary, and are extremely fast and precise,they are ideally-suited to sifting in parallel through the millions or billions ofdata records on the Web, day in and day out. Humans and even groups ofhumans will never be able to do this as well. And that’s not what theyshould be doing! They are far too intelligent for that kind of work.Humans should be at the top of the pyramid, making the decisions,innovating, learning, and navigating.

When we finally reach this stage where networks of humans and smartapplications are able to work together intelligently for common goals,I believe we will witness a real change in the way organizations arestructured. In Group Minds, hierarchy will not be as necessary — the maximum effectivesize of a human Group Mind will be perhaps in the thousands or even themillions instead of around 50 people. As a result the shape of organizations in thefuture will be extremely fluid, and most organizations will be flat orcontinually shifting networks. For more on this kind of organization,read about virtual teams and networking, such as these books (by friends of mine who taught me everything I know about network-organization paradigms.)

I would also like to note that I am not proposing “strong AI” — a vision in which we someday makeartificial intelligences that are as or more intelligent thanindividual humans. I don’t think intelligent agents will individually be very intelligent. It will only be in vast communities of agents that intelligence will start to emerge. Agents are analogous to the neurons in the human brain — they really aren’t very powerful on their own.

I’m also not proposing that Group Minds will beas or more intelligent as the individual humans in groups anytime soon. I don’t think thatis likely in our lifetimes. The cognitive capabilities of an adult human are the product of millions of years of evolution. Even in the accelerated medium of the Web where evolution can take place much faster in silico, it may still take decades or even centuries to evolve AI that rivals the human mind (and I doubt such AI will ever be truly conscious, which means that humans, with their inborn natural consciousness, may always play a special and exclusive role in the world to come, but that is the subject of a different essay). But even if they will not be as intelligent as individual humans, Ido think that Group Minds, facilitated by masses of slightly intelligent agents and humans working in concert, can goa long way in helping individuals and groups become more productive.

It’s important to note that the future I am describing is notscience-fiction, but it also will not happen overnight. It will take atleast several decades, if not longer. But with the seeminglyexponential rate of change of innovation, we may make very large stepsin this direction very soon. It is going to be an exciting lifetime forall of us.

It’s been a while since I posted about what my stealth venture, Radar Networks, is working on. Lately I’ve been seeing growing buzz in the industry around the "semantics" meme — for example at the recent DEMO conference, several companies used the word "semantics" in their pitches. And of course there have been some fundings in this area in the last year, including Radar Networks and other companies.

Clearly the "semantic" sector is starting to heat up. As a result, I’ve been getting a lot of questions from reporters and VC’s about how what we are doing compares to other companies such as for example, Powerset, Textdigger, and Metaweb. There was even a rumor that we had already closed our series B round! (That rumor is not true; in fact the round hasn’t started yet, although I am getting very strong VC interest and we will start the round pretty soon).

In light of all this I thought it might be helpful to clarify what we are doing, how we understand what other leading players in this space are doing, and how we look at this sector.

Indexing the Decades of the Web

First of all, before we get started, there is one thing to clear up. The Semantic Web is part of what is being called "Web 3.0" by some, but it is in my opinion really just one of several converging technologies and trends that will define this coming era of the Web. I’ve written here about a proposed definition of Web 3.0, in more detail.

For those of you who don’t like terms like Web 2.0, and Web 3.0, I also want to mention that  I agree — we all want to avoid a rapid series of such labels or an arms-race of companies claiming to be > x.0. So I have a practical proposal: Let’s use these terms to index decades since the Web began. This is objective — we can all agree on when decades begin and end, and if we look at history each decade is characterized by various trends. 

I think this is reasonable proposal and actually useful (and also avoids endless new x.0′s being announced every year). Web 1.0 was therefore the first decade of the Web: 1990 – 2000. Web 2.0 is the second decade, 2000 – 2010. Web 3.0 is the coming third decade, 2010 – 2020 and so on. Each of these decades is (or will be) characterized by particular technology movements, themes and trends, and these indices, 1.0, 2.0, etc. are just a convenient way of referencing them. This is a useful way to discuss history, and it’s not without precedent. For example, various dynasties and historical periods are also given names and this provides shorthand way of referring to those periods and their unique flavors. To see my timeline of these decades, click here.

So with that said, what is Radar Networks actually working on? First of all, Radar Networks is still in stealth, although we are
planning to go beta in 2007. Until we get closer to launch what I can
say without an NDA is still limited. But at least I can give some
helpful hints for those who are interested. This article provides some hints, as well as what I hope is a helpful tutorial about natural language search and the Semantic Web, and how they differ. I’ll also discuss how Radar Networks compares some of the key startup ventures working with semantics in various ways today (there are many other companies in this sector — if you know of any interesting ones, please let me know in the comments; I’m starting to compile a list).

(click the link below to keep reading the rest of this article…)

Semantic Social Software: The Semantic Web for Consumers

Here at Radar Networks, we are building a next-generation Web-based
online service that will bring the Semantic Web to consumers and
professionals across the Web. This application is focused on enabling
the next generation of social software (note that
social software is not necessarily social networking — that is subset
of social software). It is an example of what "the
Intelligent Web" will be like. We are very excited about this service
and what it already does, but there’s still more to do before we
release it.

Our app is based on the Semantic Web. It will
enrich and facilitate more intelligent online relationships, community,
content, collaboration and even commerce. It will help to bring the
Semantic Web from research to reality by making it user-friendly,
accessible and most of all, directly useful and valuable, to ordinary
people. We are focused on providing value to consumers — not just developers or early-adopters. But like I said, I can’t really provide more details until we
get closer to launch.

Our Web 3.0 Applications Platform

In order to build our product we had to first build a new platform
to support the kinds of features and capabilities we designed — we
could not find any existing platform that could do what we wanted to
do. Existing platforms for the Semantic Web were too research-oriented
and did not provide the levels of scalability, performance and
ease-of-use that we required.

We have been working on this platform
over several years and several generations of our codebase. It is now
very robust and sophisticated. We believe it is also significantly more
scalable and performant than any platform we’ve seen in the Semantic
Web space to-date.

Our platform is a comprehensive, Java-based framework for semantic
web applications and services that has some similarities to Ruby on
Rails (although it is also very different from RoR and we are not going after the platform market — we’re really more focused on our application right now). Our platform also includes a lot of other technology such as our
extremely fast and scaleable storage layer for semantic data tuples,
powerful semantic query capabilities, and a range of algorithms for
analyzing data and doing intelligent things for users.

The platform
could be called a "Web 3.0" applications platform because it is
inherently based around RDF/OWL and the emerging Semantic Web. In
addition to the "Web 3.0" aspects of what we are doing, our platform
also makes heavy use of "Web 2.0" methods and technologies such as
AJAX, REST, widgets, and RSS/ATOM, to name a few.

What We are Not Doing: Natural Language Search

First of all, we at Radar Networks are NOT building a new search engine to compete with Google, like Powerset and TextDigger are doing — so we’re not competing with them. Companies like Powerset and TextDigger are working on natural language search. Natural language search is not equivalent to the Semantic Web, although the Semantic Web can certainly help that process.

Companies working specifically on natural language search are making
use of semantics, but at the word-level only. They use networks of words that are linked to synonyms, antonyms, homonyms and other variations. These are sometimes called semantic networks. Based on these networks of word meanings, they can understand the meaning of various words and expressions.

More sophisticated natural language search algorithms don’t just look at the words alone, they look at them in context, by analyzing the grammar and the rest of the content around them. The point of natural language search is ultimately to try to match the meaning of words in search queries to the content of various documents — and to do this better than Google, which basically just matches keywords without paying attention to the meaning of the words.

Essentially natural language search requires at least some level of artificial intelligence. Machine
understanding of natural language is a difficult problem and there has
been a lot of work on this over the last few decades. Today there are
many technologies that focus on this but the majority of them are based
on the assumption that software should do all the work to figure out
the meaning of information.

What We Are Doing: Semantic Web

In contrast to natural language search which focuses on trying to derive the meanings of words, the approach of the emerging Semantic Web makes use of metadata to encode the meaning of information.

In this approach, the meaning of the information can be explicitly
coded into the information just as HTML codes are added into content today — and this can be done by people or software, and even by communities. Once this meaning — or semantics — is explicitly encoded into content, it can then be re-used by other applications to make sense of the content. It’s worth noting that explicit semantics in content can also help natural language processing apps, as well as apps that don’t understand natural language.

In the Semantic Web approach, the meaning of the information is encoded using markup
languages such as RDF and OWL, which are W3C open standards. Words and concepts in the content of documents and data records can be marked up with RDF/OWL expressions to indicate what they mean — does a certain word or phrase such as "Lotus" for example, mean a software company, a software product, an exotic sportscar brand, or some other kind of concept? Without sophisticated natural language processing it is often difficult for software to determine this on its own. The Semantic Web provides markup codes that explicitly indicate the intended meaning of information in an unambiguous, machine-readable format.

Marking up content with additional metadata was possible before the Semantic Web using XML: you could just say <sportscar>Lotus</sportscar> but the problem is that the meaning of "sportscar" still had to be coded into applications in order for them know what it implies. With RDF/OWL that meaning can be formally encoded outside of applications in a set of definitions called an ontology. An ontology defines facts such as "a sportscar is a kind of car," "a car is a ground vehicle," "a car is a product," "a car is a device," "a sportscar is a recreational or competitive vehicle," etc.

Semantic Markup

By marking up content with OWL indicating that it is a sportscar, that meaning refers to the appropriate definitions in an ontology, from which any application that can read the ontology can then then infer these various specific intended meanings. The point here is that semantics are less ambiguous — they are explicitly encoded by the ontology which functions as a kind of more advanced data schema of sorts.

But this is really an oversimplification — OWL and ontologies can actually go a lot further than just defining the meaning of concepts — they can also define their logical relations as well. For example, how exactly are two things connected and are there any special restrictions on that connection? For instance, an ontology can define that a person’s sister must be female, or that a person can only have 1 biological mother, etc.

All kinds of apps can benefit from the extra hints about the meaning of
the information that can be provided by Semantic Web metadata around
content. For example, even a natural language search engine could do
less analysis and would need less intelligence, if it could leverage
existing semantic metadata that was already in content.

It’s important to note that applications and people don’t have to necessarily ever look at RDF or
OWL code (thank heavens!) — they can just work with objects and forms
like they already do on the Web and the underlying markup can be
created automatically for them. Nobody should have to look at raw RDF and OWL (unless they really want to), and the Semantic Web doesn’t force anyone to. For example, most of us don’t write HTML or XML or CSS by hand — but if we are using blogs or wikis or even posting listing on sites like job boards and auctions, we we are doing things that result in HTML, XML and CSS being created.

It should be clear from the above section that natural language search is a specific process that makes use of word-level semantics, but the Semantic Web is a broad set of technologies for defining the meaning of any kind of information (including, but not limited to words). The Semantic Web can help improve the process of natural language search, but today many natural language search algorithms do not make use of the Semantic Web or RDF/OWL data structures. However, as these technologies begin to converge (as they are here at Radar Networks, in fact) we will see new levels of accuracy become possible — the combination of traditional natural language processing and the richer semantics of RDF/OWL markup enables even more powerful machine-understanding and processing of text. That said, once again, I want to be clear that Radar Networks is not a search company — although we do use next-generation semantic search quite extensively in our application and platform.

Any application that can understand RDF/OWL can correctly interpret
the meaning of any content that is marked up with RDF/OWL metadata. If
a news article that mentions "Paris" many times is marked up with
RDF/OWL metadata then any app that can understand that metadata can for
example, correctly determine that the article is about the place Paris,
Texas, not the place Paris, France, and not the person Paris Hilton
either. The application doesn’t have to do any fancy natural language
processing to know this. Even a relatively "dumb" application that has
no ability to do natural language processing can still make sense of a
document if it can at least understand RDF/OWL.

So how does this explicit semantic markup in the form of
RDF/OWL metadata get into the document in the first place? Well it could have been added
automatically by some other software app that did  natural language
processing on it, or it could have resulted from newspaper editors and/or
even readers categorizing and/or tagging the document with tags for places, people, etc. in
a manner not unlike how they tag content in services like Flickr today.

The main point here is that adding the semantic metadata does not require
the apps that create or consume consume the content to understand natural language, nor does it require people to be XML coders — even regular end-users can help to define
the semantics of content by simply tagging it. The Semantic Web provides a much
richer and more expressive framework for doing this than is currently available in Web 2.0 "tags," but it’s not that far off either.

The Semantic Web can enhance word-level understanding and processing
of text in many ways, but note that it is not limited only to
word-level applications. The Semantic Web provides a way to make any
information more understandable to other applications — including data
records in databases, documents on the desktop and the Web, enterprise
data, photos, videos, music, and even Web services and software code.

Simple Examples of Semantics

For example, today there is a big problem in integrating data across
applications. In the enterprise for example, one application might define a record called a "Customer"
while another might call that concept by the term "Client." If a user
then searches for "Customers" they won’t necessarily also find records
for Clients. But using the Semantic Web the data records for Customers
and Clients can be mapped together so that applications can treat them
as equivalent. Any search for one will return the other as well. Not only can records be mapped to each other, but also the fields of those records can be
mapped together. For instance, the Customer record might have a field
named "Referred by" while the Client record might have a field called
"Introduced by" — these can be mapped together as well.

A similar example could apply to a consumer use-case — for example shopping: different stores describe the same product differently — with different terms. In one store a laptop is called a "laptop computer" and in another it is called a "portable computer," while another calls it a "desktop replacement." A search for any of these terms should return products that use any of these. Within a single commerce site this is not so hard, but what about searching across many commerce sites (which isn’t really even that easy to do at all today…)? If different commerce sites used the same underlying semantic metadata definitions to markup their various products, then users could search across their products with less trial-and-error, and they would get better results.

Of course the
technology for mapping between databases is not new — there are many
ways to do this — but the Semantic Web provides a way to do it that
may be more open and efficient in the long-run. Central to this approach is that an organization or
online service can use ontologies that centrally define key
concepts in a rigorous way. So instead of every different app and data
record having to be individually mapped to every other, they can
potentially all just map to the central ontology which functions as a
kind of semantic switchboard of sorts. All applications and queries can
use a common ontology (or set of them) to unify access to data records across many
different online services and databases. In a sense ontologies provide a way to define and share common languages for data, content, relationships and applications.

SPARQL and the Emerging Data-Web

More recently a new Semantic Web technology called SPARQL has also started to
emerge. SPARQL provides a common query language, like SQL, for querying
data that is stored in RDF. Any site or database that has RDF data and
that provides a SPARQL interface can be searched by any application
that speaks SPARQL. This means that the dream of "deep web search" is
finally going to become a reality. There is a huge amount of interest
in SPARQL at the moment and there are already a growing number of
SPARQL endpoints popping up around the Web. These new SPARQL endpoints
are to data what websites were to documents. It’s the beginning of what
some call "The Data Web" — which is the first step to the full-blown
Semantic Web. SPARQL is also a big piece of what we are doing.

Reasoning: The Next Frontier After Search

Another key benefit of using RDF/OWL is that these languages are
designed to support formal logical reasoning. By marking up information
with RDF/OWL sophisticated search and inferencing can then take place
around it. For example, by marking up various people and their social
connections it is then possible to infer for example, that Sue is
Jane’s cousin, that Bob and Dave are colleagues, and that product A is
incompatible with product B, etc.

This kind of logical reasoning and
inference is essential to enable the next-generation of the Web — an
Intelligent Web — where software and online services start to help
people work, communicate, socialize and shop more productively. For
example it will enable something beyond search — it will enable
services that provide answers or suggestions. This is not necessarily important for all applications today, but it
will become increasingly important in the future. Content that exists
in RDF/OWL essentially has a longer shelf-life and will be easier to
reuse, integrate and reason across in the future.

Differentiating The Players

The Semantic Web provides a comprehensive and growing framework of
technologies that enable the next evolution of the Web — it is therefore a
much broader and farther-reaching vision than natural language search,
even though that is certainly one area that it will benefit. Natural language search is really just about matching search queries to documents, by analyzing the meaning of words. The Semantic Web is about defining the meaning of data — any data — words, data records, documents, social relationships, product listings, etc. — and providing a way to query that data, integrate it, and reason across it.

In our own
application and platform we make use of a lot of natural-language
processing (NLP) and we also provide semantic search capabilities, but
our focus is on something quite different than searching the Web — yet
equally useful and important to everyone. Frankly, I’m glad we are not
working on search, as big an opportunity as that is — I think
competing directly with Google is a daunting task and not one I would
want to be on! Instead, we are providing a new environment in which
people can start to benefit from the power of the Semantic Web in areas
that Google is very weak in today or is not in at all in some cases;
it’s really quite orthogonal to Google and other search engines.

So from the above conversation it should be clear that we are
working on The Semantic Web, not just natural language search and
so we are quite different from companies like Powerset, Textdigger and
others who are working on word-level semantic understanding of text.
But what about Metaweb — how do
we differ from them? — Well from what we can glean so far, what we are
doing is also very different from them as well but perhaps not as
different as we are from Powerset.

Radar Networks and Metaweb are
frequently cited as the two main startups working to bring
semantically-driven Web 3.0 online services to consumers. My guess is
that there will be some similarities but even more differences. There
may even be opportunities for us to work together someday. But we’re
all still in stealth, so it’s hard to get very specific about our
similarities and differences today. One thing is for sure, 2007 is
going to be an exciting year for both our companies, and for the
emerging Web 3.0 generation of companies and products.

Web 3.0 is just beginning

In any case the next-evolution of the Web — what we call "The
Intelligent Web" (and what many are also calling "Web 3.0") is still in
the very early stages and I don’t think it will really hit big until
2010 (for a graphical timeline of how I think this will unfold, click here). In the meantime we are all putting the pieces in place.

Fortunately Web 3.0 is a big space with a lot of opportunity and there is
room for a many different players and business models to co-exist and
compete. The fact that there are now several ventures in this space is
a good thing for all of us, for as one person said to me the other day,
"a rising tide lifts all boats." I’m happy that there is enough action
for there to actually be some confusion for me to clear up! Only a year
ago it felt like we were the only commercial voice a wilderness of
academic research. Today VC’s are lining up to speak to us and the other
companies in the space, and we are literally having to keep them at bay
until we start our B round.

Solving Information Overload

The key realization behind all this recent interest in semantics is that keyword
search and traditional content and data representations are declining
in productivity. As the Web gets vaster and more complex, and as
consumers must work with a growing array of content and services,
productivity is seriously being threatened — not only in search, but
also in every other area of our digital lives. Most of us who work
intensively with knowledge and information already have a direct and
intuitive experience of how information overload has grown, even in the
last decade. Clearly something must be done about this or in another
few years we will all be buried in our own information.

The Semantic Web provides the best (and really the only) long-term
solution to information overload and complexity. By starting to add
richer semantics to data, and by enabling applications to start
leveraging this, it will make it possible to help people regain more of
their productivity and to make software smarter — without having to
attempt to create super-duper science fiction artificial intelligence.

It’s very important to keep in mind that The Semantic Web does not require that machines understand or reason as well as people — the semantics of the Semantic Web can be created by people and/or machines, and it doesn’t have to be perfect, it simply has to add hints that make content less ambiguous and more structured. By contrast, both the keyword approach of Google and the natural language search approach of companies like Powerset — if they are to keep up with the growing complexity of the Web — will require increasingly intelligent software, because basically in such systems the software has to do all the work by itself.

The Semantic Web actually is really more about leveraging the
collective intelligence of people and applications to enrich content — rather than trying
to make applications do all the work on their own — but this will be a lot clearer later in the process when there are several Semantic Web apps that demonstrate this.

Here at Radar Networks we have
been working towards this vision steadily — and we’re proud of the fact that we started working with semantics long before it was "cool" — we know this space inside out
and we think that our first application on our platform will be an "Aha
experience" for users.

It certainly has taken some time to bring the Semantic Web to
fruition, but when you think about it, Web 1.0 took about 5 years to
really get started, so it’s not without precedent. A new generation of
the Web is a big undertaking. For now, all of us working on anything having to do with "semantics"
or Web 3.0 need to work together to start mapping out this space and educating the marketplace so
that people (including the press and VC’s, and early-adopters) can
understand the companies and technologies more clearly. The rather
humorous irony for all of us, is that the meaning of the term "semantic" is still so
ambiguous today!

NOTES

• Master Copy can be found at this URL or http://tinyurl.com/yynb93
• Last Update: Tuesday, November 7, 2006, 10:17AM PST
• License — This article is distributed under the Creative Commons Deed. If you would like to distribute a version of thisarticle, please link back to http://www.mindingtheplanet.net from yourversion, thanks.
• Printable version — Click here to download the printable PDF version of this article
• Illustrated Version — See the version by the Lifeboat Foundation

Prelude

Many years ago, in the late 1980s, while I was still a college student, I visited my late grandfather, Peter F. Drucker, at his home in Claremont, California. He lived near the campus of Claremont College where he was a professor emeritus. On that particular day, I handed him a manuscript of a book I was trying to write, entitled, “Minding the Planet” about how the Internet would enable the evolution of higher forms of collective intelligence.

My grandfather read my manuscript and later that afternoon we sat together on the outside back porch and he said to me, “One thing is certain: Someday, you will write this book.” We both knew that the manuscript I had handed him was not that book, a fact that was later verified when I tried to get it published. I gave up for a while and focused on college, where I was studying philosophy with a focus on artificial intelligence. And soon I started working in the fields of artificial intelligence and supercomputing at companies like Kurzweil, Thinking Machines, and Individual.

A few years later, I co-founded one of the early Web companies, EarthWeb, where among other things we built many of the first large commercial Websites and later helped to pioneer Java by creating several large knowledge-sharing communities for software developers. Along the way I continued to think about collective intelligence. EarthWeb and the first wave of the Web came and went. But this interest and vision continued to grow. In 2000 I started researching the necessary technologies to begin building a more intelligent Web. And eventually that led me to start my present company, Radar Networks, where we are now focused on enabling the next-generation of collective intelligence on the Web, using the new technologies of the Semantic Web.

But ever since that day on the porch with my grandfather, I remembered what he said: “Someday, you will write this book.” I’ve tried many times since then to write it. But it never came out the way I had hoped. So I tried again. Eventually I let go of the book form and created this weblog instead. And as many of my readers know, I’ve continued to write here about my observations and evolving understanding of this idea over the years. This article is my latest installment, and I think it’s the first one that meets my own standards for what I really wanted to communicate. And so I dedicate this article to my grandfather, who inspired me to keep writing this, and who gave me his prediction that I would one day complete it.

This is an article about a new generation of technology that is sometimes called the Semantic Web, and which could also be called the Intelligent Web, or the global mind. But what is the Semantic Web, and why does it matter, and how does it enable collective intelligence? And where is this all headed? And what is the long-term far future going to be like? Is the global mind just science-fiction? Will a world that has a global mind be good place to live in, or will it be some kind of technological nightmare?

I’ve often joked that it is ironic that a term that contains theword “semantic” has such an ambiguous meaning for most people. Mostpeople just have no idea what this means, they have no context for it,it is not connected to their experience and knowledge. This is aproblem that people who are deeply immersed in the trenches of theSemantic Web have not been able to solve adequately — they have notfound the words to communicate what they can clearly see, what they areworking on, and why it matters for everyone. In this article I havetried, and hopefully succeeded, in providing a detailed introductionand context for the Semantic Web fornon-technical people. But even technical people working in the fieldmay find something of interest here as I piece together the fragmentsinto a Big Picture and a vision for what might be called “Semantic Web2.0.”

I hope the reader will bear with me as Ibounce around across different scales of technology and time, and fromthe extremes of core technology to wild speculation in order to tellthis story. If you are looking for the cold hardscience of it all, this article will provide an understanding but willnot satisfy your need for seeing the actual code; there are otherplaceswhere you can find that level of detail and rigor. But if you want tounderstand what it all really means and what the opportunity and futurelookslike – this may be what you are looking for.

I should also note that all of this is my personal view of what I’vebeen working on,and what it really means to me. It is not necessarily the official viewof the mainstream academic Semantic Web community — although there arecertainly many places where we all agree. But I’m sure that somereaders will certainly disagree or raise objections to some of myassertions, and certainly to my many far-flung speculations about thefuture. I welcome those different perspectives; we’re all trying tomake sense of this and the more of us who do that together, the more wecan collectively start to really understand it. So please feel free towrite your own vision or response, and please let me know so I can linkto it!

So with this Prelude in mind, let’s get started…

The Semantic Web Vision

The Semantic Web is a set of technologies which are designed toenable aparticular vision for the future of the Web – a future in which allknowledge exists on the Web in a format that software applications canunderstand andreason about. By making knowledge more accessible to software, softwarewillessentially become able to understand knowledge, think about knowledge,and createnew knowledge. In other words, software will be able to be moreintelligent –not as intelligent as humans perhaps, but more intelligent than say,your wordprocessor is today.

The dream of making software more intelligent has been around almost as longas software itself. And although it is taking longer to materialize than past experts hadpredicted, progress towards this goal is being steadilymade. At the same time, the shape of this dream is changing. It is becomingmore realistic and pragmatic. The original dream of artificial intelligence wasthat we would all have personal robot assistants doing all the work we don’twant to do for us. That is not the dream of the Semantic Web. Instead, today’sSemantic Web is about facilitating what humans do – it is about helping humansdo things more intelligently. It’s not a vision in which humans do nothing andsoftware does everything.

The Semantic Web vision is not just about helping software become smarter –it is about providing new technologies that enable people, groups,organizations and communities to be smarter.

For example, by providing individuals with tools that learn about what theyknow, and what they want, search can be much more accurate and productive.

Using software that is able to understand and automatically organize largecollections of knowledge, groups, organizations and communities can reachhigher levels of collective intelligence and they can cope with volumes ofinformation that are just too great for individuals or even groups tocomprehend on their own.

Another example: more efficient marketplaces can be enabled by software thatlearns about products, services, vendors, transactions and market trends andunderstands how to connect them together in optimal ways.

In short, the Semantic Web aims to make software smarter, not just for itsown sake, but in order to help make people, and groups of people, smarter. Inthe original Semantic Web vision this fact was under-emphasized, leading to theimpression that Semantic Web was only about automating the world. In fact, it isreally about facilitating the world.

The Semantic Web Opportunity

The Semantic Web is one of the most significant things to happen since theWeb itself. But it will not appear overnight. It will take decades. It willgrow in a bottom-up, grassroots, emergent, community-driven manner just likethe Web itself. Many things have to converge for this trend to really take off.

The core open standards already exist, but the necessary development tools haveto mature, the ontologies that define human knowledge have to come into beingand mature, and most importantly we need a few real “killer apps” to prove thevalue and drive adoption of the Semantic Web paradigm. The first generation ofthe Web had its Mozilla, Netscape, Internet Explorer, and Apache – and it alsohad HTML, HTTP, a bunch of good development tools, and a few killer apps andservices such as Yahoo! and thousands of popular Web sites. The same things arenecessary for the Semantic Web to take off.

And this is where we are today – this all just about to start emerging.There are several companies racing to get this technology, or applications ofit, to market in various forms. Within a year or two you will see mass-consumerSemantic Web products and services hit the market, and within 5 years therewill be at least a few “killer apps” of the Semantic Web. Ten years from nowthe Semantic Web will have spread into many of the most popular sites andapplications on the Web. Within 20 years all content and applications on theInternet will be integrated with the Semantic Web. This is a sea-change. A bigevolutionary step for the Web.

The Semantic Web is an opportunity to redefine, or perhaps to better define,all the content and applications on the Web. That’s a big opportunity. Andwithin it there are many business opportunities and a lot of money to be made. It’snot unlike the opportunity of the first generation of the Web. There areplatform opportunities, content opportunities, commerce opportunities, searchopportunities, community and social networking opportunities, and collaborationopportunities in this space. There is room for a lot of players to compete andat this point the field is wide open.

The Semantic Web is a blue ocean waiting to be explored. And like anyunexplored ocean its also has its share of reefs, pirate islands, hidden treasure, shoals,whirlpools, sea monsters and typhoons. But there are new worlds out there to be discovered,and they exert an irresistible pull on the imagination. This is an excitingfrontier – and also one fraught with hard technical and social challenges thathave yet to be solved. For early ventures in the Semantic Web arena, it’s notgoing to be easy, but the intellectual and technological challenges, and the potentialfinancial rewards, glory, and benefit to society, are worth the effort andrisk. And this is what all great technological revolutions are made of.

Semantic Web 2.0

Some people who have heard the term “Semantic Web” thrown around too muchmay think it is a buzzword, and they are right. But it is not just a buzzword –it actually has some substance behind it. That substance hasn’t emerged yet,but it will. Early critiques of the Semantic Web were right – the early visiondid not leverage concepts such as folksonomy and user-contributed content atall. But that is largely because when the Semantic Web was originally conceivedof Web 2.0 hadn’t happened yet. The early experiments that came out of researchlabs were geeky, to put it lightly, and impractical, but they are already beingfollowed up by more pragmatic, user-friendly approaches.

Today’s Semantic Web – what we might call “Semantic Web 2.0” is a kinder,gentler, more social Semantic Web. It combines the best of the original visionwith what we have all learned about social software and community in the last10 years. Although much of this is still in the lab, it is already starting totrickle out. For example, recently Yahoo! started a pilot of the Semantic Webbehind their food vertical. Other organizations are experimenting with usingSemantic Web technology in parts of their applications, or to store or mapdata. But that’s just the beginning.

The Google Factor

Entrepreneurs, venture capitalists and technologists are increasinglystarting to see these opportunities. Who will be the “Google of the SemanticWeb?” – will it be Google itself? That’s doubtful. Like any entrenchedincumbent, Google is heavily tied to a particular technology and worldview. Andin Google’s case it is anything but semantic today. It would be easier for anupstart to take this position than for Google to port their entireinfrastructure and worldview to a Semantic Web way of thinking.

If it is goingto be Google it will most likely be by acquisition rather than by internal origination. Andthis makes more sense anyway – for Google is in a position where they can just wait and buy the winner,at almost any price, rather than competing in the playing field. One thing to note however is that Google has at least one product offering that shows some potential for becoming a key part of the Semantic Web. I am speaking of Google Base, Google’s open database which is meant to be a registry for structured data so that it can be found in Google search. But Google Base does not conform to or make use of the many open standards of the Semantic Web community. That may or may not be a good thing, depending on your perspective.

Of course the downside of Google waiting to join the mainstream Semantic Web community until after the winner is announced is very large – once there is a winner it may be too late for Google to beat them. Thewinner of the Semantic Web race could very well unseat Google. The strategistsat Google are probably not yet aware of this but as soon as they seesignificant traction around a major Semantic Web play it will become of interestto them.

In any case, I think there won’t be just one winner, there will be severalmajor Semantic Web companies in the future, focusing on different parts of theopportunity. And you can be sure that if Google gets into the game, every majorportal will need to get into this space at some point or risk becomingirrelevant. There will be demand and many acquisitions. In many ways the Semantic Web will not be controlled by just one company — it will be more like a fabric that connects them all together.

Context is King — The Nature ofKnowledge

It should be clear by now that the Semantic Web is all about enablingsoftware (and people) to work with knowledge more intelligently. But what isknowledge? Knowledge is not just information. It is meaningful information – itis information plus context. For example, if I simply say the word “sem” toyou, it is just raw information, it is not knowledge. It probably has nomeaning to you other than a particular set of letters that you recognize and asound you can pronounce, and the mere fact that this information was stated byme.

But if I tell you that “sem” it is the Tibetan word for “mind” then suddenly,“sem means mind in Tibetan” to you. If I further tell you that Tibetans have about as many words for “mind” as Eskimos have for “snow,” this is further meaning. Thisis context, in other words, knowledge, about the sound “sem.” The sound is raw information. When it is given context itbecomes a word, a word that has meaning, a word that is connected to conceptsin your mind – it becomes knowledge. By connecting raw information to context,knowledge is formed.

Once you have acquired a piece of knowledge such as “sem means mind in Tibetan,” you may then also form further knowledgeabout it. For example, you may form the memory, “Nova said that ‘sem means mind in Tibetan.’” You mightalso connect the word “sem” to networks of further concepts you have about Tibet and your understanding of what the word “mind” means.

The mind is the organ of meaning – mind is where meaning is stored,interpreted and created. Meaning is not “out there” in the world, it is purelysubjective, it is purely mental. Meaning is almost equivalent to mind in fact.For the two never occur separately. Each of our individual minds has some way of internally representing meaning — when we read or hear a word that we know, our minds connect that to a network of concepts about it and at that moment it means something to us.

Digging deeper, if you are really curious,or you happen to know Greek, you may also find that a similar sound occurs inthe Greek word, sēmantikós – which means “having meaning” and in turn is the root of the English word “semantic”which means “pertaining to or arising from meaning.” That’s an odd coincidence!“Sem” occurs in Tibetan word for mind, and the English and Greek words that allrelate to the concepts of “meaning” and “mind.” Even stranger is that not only do these words have a similar sound, they have a similar meaning.

With all this knowledge at yourdisposal, when you then see the term “Semantic Web” you may be able to inferthat it has something to do with adding “meaning” to the Web. However, if youwere a Tibetan, perhaps you might instead think the term had something to dowith adding “mind” to the Web. In either case you would be right!

Discovering New Connections

We’ve discovered a new connection — namely that there is an implicit connectionbetween “sem” in Greek, English and Tibetan: they all relate to meaning andmind. It’s not a direct, explicit connection – it’s not evident unless you digfor it. But it’s a useful tidbit of knowledge once it’s found. Unlike the direct migration of the sound “sem” from Greek to English,there may not have ever been a direct transfer of this sound from Greek toSanskrit to Tibetan. But in a strange and unexpected way, they are all connected. This connectionwasn’t necessarily explicitly stated by anyone before, but was uncovered byexploring our network of concepts and making inferences.

The sequence of thought about “sem”above is quite similar to kind of intellectual reasoning and discovery that theactual Semantic Web seeks to enable software to do automatically.  How is this kind of reasoning and discovery enabled? The Semantic Web providesa set of technologies for formally defining the context of information. Just asthe Web relies on a standard formal specification for “marking up” informationwith formatting codes that enable any applications that understand those codesto format the information in the same way, the Semantic Web relies on newstandards for “marking up” information with statements about its context – itsmeaning – that enable any applications to understand, and reason about, the meaning of those statements in the same way.

By applying semantic reasoning agents to large collections of semantically enhanced content, all sorts of new connections may be inferred, leading to new knowledge, unexpected discoveries and useful additional context around content. This kind of reasoning and discovery is already taking place in fields from drug discovery and medical research, to homeland security and intelligence. The Semantic Web is not the only way to do this — but it certainly will improve the process dramatically. And of course, with this improvement will come new questions about how to assess and explain how various inferences were made, and how to protect privacy as our inferencing capabilities begin to extend across ever more sources of public and private data. I don’t have the answers to these questions, but others are working on them and I have confidence that solutions will be arrived at over time.

Smart Data

By marking up information with metadata that formally codifies its context, we can make the data itself “smarter.” The data becomes self-describing. When you get a piece of data you also get the necessary metadata for understanding it. For example, if I sent you a document containing the word “sem” in it, I could add markup around that word indicating that it is the word for “mind” in the Tibetan language.

Similarly, a document containing mentions of “Radar Networks” could contain metadata indicating that “Radar Networks” is an Internet company, not a product or a type of radar technology. A document about a person could contain semantic markup indicating that they are residents of a certain city, experts on Italian cooking, and members of a certain profession. All of this could be encoded as metadata in a form that software could easily understand. The data carries more information about its own meaning.

The alternative to smart data would be for software to actually read and understand natural language as well as humans. But that’s really hard. To correctly interpret raw natural language, software would have to be developed that knew as much as a human being. But think about how much teaching and learning is required to raise a human being to the point where they can read at an adult level. It is likely that similar training would be necessary to build software that could do that. So far that goal has not been achieved, although some attempts have been made. While decent progress in natural language understanding has been made, most software that can do this is limited around particular vertical domains, and it’s brittle — it doesn’t do a good job of making sense of terms and forms of speech that it wasn’t trained to parse and make sense of.

Instead of trying to make software a million times smarter than it is today, it is much easier to just encode more metadata about what our information means. That turns out to be less work in the end. And there’s an added benefit to this approach — the meaning exists with the data and travels with it. It is independent of any one software program — all software can access it. And because the meaning of information is stored with the information itself, rather than in the software, the software doesn’t have to be enormous to be smart. It just has to know the basic language for interpreting the semantic metadata it finds on the information it works with.

Smart data enables relatively dumb software to be smarter with less work. That’s an immediate benefit. And in the long-term as software actually gets smarter, smart data will make it easier for it to start learning and exploring on its own. So it’s a win-win approach. Start with by adding semantic metadata to data, end up with smarter software.

Making Statements About the World

Metadata comes down to making statements about the world in a manner that machines, and perhaps even humans, can understand unambiguously. The same piece of metadata should be interpreted in the same way by different applications and readers.

There are many kinds of statementsthat can be made about information to provide it with context. For example, youcan state a definition such as “person” means “a human being or a legalentity.” You can state an assertion such as “Sue is a human being.” You canstate a rule such that “if x is a human being, then x is a person.”

From thesestatements it can then be inferred that “Sue is a person.” This inference is soobvious to you and me that it seems trivial, but most software today cannot dothis. It doesn’t know what a person is, let alone what a name is. But ifsoftware could do this, then it could for example, automatically organizedocuments by the people they are related to, or discover connections betweenpeople who were mentioned in a set of documents, or it could find documentsabout people who were related to particular topics, or it could give you a listof all the people mentioned in a set of documents, or all the documents relatedto a person.

Of course this is a very basicexample. But imagine if your software didn’t just know about people – it knewabout most of the common concepts that occur in your life. Your software wouldthen be able to help you work with your documents just about as intelligentlyas you are able to do by yourself, or perhaps even more intelligently, becauseyou are just one person and you have limited time and energy but your softwarecould work all the time, and in parallel, to help you.

Examples and Benefits

How could the existence of the Semantic Web and all the semantic metadata that defines it be really useful toeveryone in the near-term?

Well, for example, the problem of email spam would finally be cured:your software would be able to look at a message and know whether it wasmeaningful and/or relevant to you or not.

Similarly, you would never have to file anything by hand again. Your software could atuomate all filing and information organization tasks for you because it would understand your information and your interests. It would be able to figure out when to file something in a single folder, multiple folders, or new ones. It would organize everything — documents, photos, contacts, bookmarks, notes, products, music, video, data records — and it would do it even better and more consistently than you could on your own. Your software wouldn’t just organize stuff, it would turn it into knowledge by connecting it to more context. It could this not just for individuals, but for groups, organizations and entire communities.

Another example: search would bevastly better: you could search conversationally by typing in everyday naturallanguage and you would get precisely what you asked for, or even what youneeded but didn’t know how to ask for correctly, and nothing else. Your searchengine could even ask you questions to help you narrow what you want. You wouldfinally be able to converse with software in ordinary speech and it would understandyou.

The process of discovery would be easier too. You could have software agent that worked as your personal recommendation agent. It would constantly be looking in all the places you read or participate in for things that are relevant to your past, present and potential future interests and needs. It could then alert you in a contextually sensitive way, knowing how to reach you and how urgently to mark things. As you gave it feedback it could learn and do a better job over time.

Going even further with this,semantically-aware software – software that is aware of context, software thatunderstands knowledge – isn’t just for helping you with your information, itcan also help to enrich and facilitate, and even partially automate, yourcommunication and commerce (when you want it to). So for example, your software could help you with your email. It would be able to recommend responses to messages for you, or automate the process. It would be able to enrich your messaging anddiscussions by automatically cross-linking what you are speaking about withrelated messages, discussions, documents, Web sites, subject categories,people, organizations, places, events, etc.

Shopping and marketplaces wouldalso become better – you could search precisely for any kind of product, withany specific attributes, and find it anywhere on the Web, in any store. You could post classified ads and automatically get relevant matches according to your priorities, from all over the Web, or only from specific places and parties that match your criteria for who you trust. You could also easily invent a new custom datastructure for posting classified ads for a new kind of product or service and publishit to the Web in a format that other Web services and applications couldimmediately mine and index without having to necessarily integrate with yoursoftware or data schema directly.

You could publish an entiredatabase to the Web and other applications and services could immediately startto integrate your data with their data, without having to migrate your schemaor their own. You could merge data from different data sources together to create new data sources without having to ever touch or look at an actual database schema.

Bumps on the Road

The above examples illustrate thepotential of the Semantic Web today, but the reality on the ground is that the technology isstill in the early phases of evolution. Even for experienced software engineersand Web developers, it is difficult to apply in practice. The main obstaclesare twofold:

(1) The Tools Problem:

There are very few commercial-gradetools for doing anything with the Semantic Web today – Most of the tools forbuilding semantically-aware applications, or for adding semantics toinformation are still in the research phase and were designed for expertcomputer scientists who specialize in knowledge representation, artificialintelligence, and machine learning.

These tools require a largelearning curve to work with and they don’t generally support large-scaleapplications – they were designed mainly to test theories and frameworks, notto actually apply them. But if the Semantic Web is ever going to becomemainstream, it has to be made easier to apply – it has to be made moreproductive and accessible for ordinary software and content developers.

Fortunately, the tools problem isalready on the verge of being solved. Companies such as my own venture, RadarNetworks, are developing the next generation of tools for building Semantic Webapplications and Semantic Web sites. These tools will hide most of thecomplexity, enabling ordinary mortals to build applications and content thatleverage the power of semantics without needing PhD’s in knowledge representation.

(2) The Ontology Problem:

The Semantic Web providesframeworks for defining systems of formally defined concepts called “ontologies,”that can then be used to connect information to context in an unambiguous way. Withoutontologies, there really can be no semantics. The ontologies ARE the semantics,they define the meanings that are so essential for connecting information tocontext.

But there are still few widely used or standardized ontologies. Andgetting people to agree on common ontologies is not generally easy. Everyonehas their own way of describing things, their own worldview, and let’s face itnobody wants to use somebody else’s worldview instead of their own.Furthermore, the world is very complex and to adequately describe all the knowledgethat comprises what is thought of as “common sense” would require a very largeontology (and in fact, such an ontology exists – it’s called Cyc and it is solarge and complex that only experts can really use it today).

Even to describe the knowledge ofjust a single vertical domain, such as medicine, is extremely challenging. Tomake matters worse, the tools for authoring ontologies are still very hard touse – one has to understand the OWL language and difficult, buggy ontologyauthoring tools in order to use them. Domain experts who are non-technical andnot trained in formal reasoning or knowledge representation may find theprocess of designing ontologies frustrating using current tools. What is needed are commercial quality tools for buildingontologies that hide the underlying complexity so that people can just pourtheir knowledge into them as easily as they speak. That’s still a ways off, butnot far off. Perhaps ten years at the most.

Of course the difficulty ofdefining ontologies would be irrelevant if the necessary ontologies alreadyexisted. Perhaps experts could define them and then everyone else could justuse them? There are numerous ontologies already in existence, both on thegeneral level as well as about specific verticals. However in my own opinion,having looked at many of them, I still haven’t found one that has the rightbalance of coverage of the necessary concepts most applications need, andaccessibility and ease-of-use by non-experts. That kind of balance is arequirement for any ontology to really go mainstream.

Furthermore, regarding the presentcrop of ontologies, what is still lacking is standardization. Ontologists havenot agreed on which ontologies to use. As a result it’s anybody’s guess whichontology to use when writing a semantic application and thus there is a highdegree of ontology diversity today. Diversity is good, but too much diversityis chaos.

Applications that use differentontologies about the same things don’t automatically interoperate unless theirontologies have been integrated. This is similar to the problem of databaseintegration in the enterprise. In order to interoperate, different applicationsthat use different data schemas for records about the same things, have to bemapped to each other somehow – either at the application-level or the data-level.This mapping can be direct or through some form of middleware.

Ontologies canbe used as a form of semantic middleware, enabling applications to be mapped atthe data-level instead of the applications-level. Ontologies can also be usedto map applications at the applications level, by making ontologies of Webservices and capabilities, by the way. This is an area in which a lot ofresearch is presently taking place.

The OWL language can expressmappings between concepts in different ontologies. But if there are manyontologies, and many of them partially overlap, it is a non-trivial task toactually make the mappings between their concepts.

Even though concept A inontology one and concept B in ontology two may have the same names, and evensome of the same properties, in the context of the rest of the concepts intheir respective ontologies they may imply very different meanings. So simplymapping them as equivalent on the basis of their names is not adequate, theirconnections to all the other concepts in their respective ontologies have to beconsidered as well. It quickly becomes complex. There are some potential waysto automate the construction of mappings between ontologies however – but theyare still experimental. Today, integrating ontologies requires the help ofexpert ontologists, and to be honest, I’m not sure even the experts have itfigured out. It’s more of an art than a science at this point.

Darwinian Selection of Ontologies

All that is needed for mainstream adoption to begin is for a largebody of mainstream content to become semantically tagged andaccessible. This will cause whatever ontology is behind that content to become popular.

When developers see that there is significant content andtraction around aparticular ontology, they will use that ontology for their ownapplicationsabout similar concepts, or at least they will do the work of mappingtheir ownontology to it, and in this way the world will converge in a Darwinianfashionaround a few main ontologies over time.

These main ontologies will then beworth thetime and effort necessary to integrate them on a semantic level,resulting in acohesive Semantic Web. We may in fact see Darwinian natural selection take place not just at the ontology level, but at the level of pieces of ontologies.

A certain ontology may do a good job of defining what a person is, while another may do a good job of defining what a company is. These definitions may be used for a lot of content, and gradually they will become common parts of an emergent meta-ontology comprised of the most-popular pieces from thousands of ontologies. This could be great or it could be a total mess. Nobody knows yet. It’s a subject for further research.

Making Sense of Ontologies

Since ontologies are so important,it is helpful to actually understand what an ontology really is, and what itlooks like. An ontology is a system of formally defined related concepts. Forexample, a simple ontology is this set of statements such as this:

A human is a living thing.

A person is a human.

A person may have a first name.

A person may have a last name.

A person must have one and only onedate of birth.

A person must have a gender.

A person may be socially related toanother person.

A friendship is a kind of socialrelationship.

A romantic relationship is a kindof friendship.

A marriage is a kind of romanticrelationship.

A person may be in a marriage withonly one other person at a time.

A person may be employed by anemployer.

An employer may be a person or anorganization.

An organization is a group ofpeople.

An organization may have a productor a service.

A company is a type organization.

We’ve just built a simple ontologyabout a few concepts: humans, living things, persons, names, socialrelationships, marriages, employment, employers, organizations, groups,products and services. Within this system of concepts there is particular logic,some constraints, and some structure. It may or may not correspond to yourworldview, but it is a worldview that is unambiguously defined, can becommunicated, and is internally logically consistent, and that is what isimportant.

The Semantic Web approach providesan open-standard language, OWL, for defining ontologies. OWL also provides fora way to define instances of ontologies. Instances are assertions within theworldview that a given ontology provides. In other words OWL provides a meansto make statements that connect information to the ontology so that softwarecan understand its meaning unambiguously. For example, below is a set ofstatements based on the above ontology:

There exists a person x.

Person x has a first name “Sue”

Person x  has a last name “Smith”

Person x has a full name “Sue Smith”

Sue Smith was born on June 1, 2005

Sue Smith has a gender: female

Sue Smith has a friend: Jane, who isanother person.

Sue Smith is married to: Bob, anotherperson.

Sue Smith is employed by Acme, Inc, a company

Acme Inc. has a product, Widget2.0.

The set of statements above, plusthe ontology they are connected to, collectively comprise a knowledge basethat, if represented formally in the OWL markup language, could be understoodby any application that speaks OWL in the precise manner that it was intendedto be understood.

Making Metadata

The OWL language provides a way tomarkup any information such as a data record, an email message or a Web pagewith metadata in the form of statements that link particular words or phrasesto concepts in the ontology. When software applications that understand OWLencounter the information they can then reference the ontology and figure outexactly what the information means – or at least what the ontology says that itmeans.

But something has to add thesesemantic metadata statements to the information – and if it doesn’t add them or adds thewrong ones, then software applications that look at the information will getthe wrong idea. And this is another challenge – how will all this metadata getcreated and added into content? People certainly aren’t going to add it all byhand!

Fortunately there are many ways tomake this easier. The best approach is to automate it using special softwarethat goes through information, analyzes the meaning and adds semantic metadataautomatically. This works today, but the software has to be trained or providedwith rules and that takes some time. It also doesn’t scale cost-effectively tovast data-sets.

Alternatively, individuals can beprovided with ways to add semantics themselves as they author information. Whenyou post your resume in a semantically-aware job board, you could fill out aform about each of your past jobs, and the job board would connect that data toappropriate semantic concepts in an underlying employment ontology. As anend-user you would just fill out a form like you are used to doing;under-the-hood the job board would add the semantics for you.

Another approach is to leveragecommunities to get the semantics. We already see communities that are addingbasic metadata “tags” to photos, news articles and maps. Already a few simpletypes of tags are being used pseudo-semantically: subject tags and geographicaltags. These are primitive forms of semantic metadata. Although they are notexpressed in OWL or connected to formal ontologies, they are at leastsemantically typed with prefixes or by being entered into fields or specificnamespaces that define their types.

Tagging by Example

There may also be another solution to the problem of how to add semantics to content in the not to distant future. Once asuitable amount of content has been marked up with semantic metadata,it may be possible, through purely statistical forms of machinelearning, for software to begin to learn how to do a pretty good job ofmarking up new content with semantic metadata.

For example, if thestring “Nova Spivack” is often marked up with semantic metadata statingthat it indicates a person, and not just any person but a specificperson that is abstractly represented in a knowledge base somewhere,then when software applications encounter a new non-semanticallyenhanced document containing strings such as “Nova Spivack” or”Spivack, Nova” they can make a reasonably good guess that thisindicates that same specific person, and they can add the necessarysemantic metadata to that effect automatically.

As more and more semanticmetadata is added to the Web and made accessible it constitutes a statisticaltraining set that can be learned and generalized from. Although humansmay need to jump-start the process with some manually semantic tagging,it might not be long before software could assist them and eventuallydo all the tagging for them. Only in special cases would software needto ask a human for assistance — for example when totally new terms orexpressions were encountered for the first several times.

The technology for doing this learning already exists — and actually it’s not very different from how search engines like Google measure the community sentiment around web pages. Each time something is semantically tagged with a certain meaning that constitutes a “vote” for it having that meaning. The meaning that gets the most votes wins. It’s an elegant, Darwinian, emergent approach to learning how to automatically tag the Web.

One this is certain, if communities were able to tagthings with more types of tags, and these tags were connected to ontologies andknowledge bases, that would result in a lot of semantic metadata being added tocontent in a completely bottom-up, grassroots manner, and this in turn would enable this process to start to become automated or at least machine-augmented.

Getting the Process Started

But making the userexperience of semantic tagging easy (and immediately beneficial) enough that regular people will do it, is a challenge that has yet to be solved.However, it will be solved shortly. It has to be. And many companies andresearchers know this and are working on it right now. This does have to be solved to get the process of jump-starting the Semantic Web started.

I believe that the Tools Problem – the lack of commercial grade tools forbuilding semantic applications – is essentially solved already (although theproducts have not hit the market yet; they will within a few years at most).The Ontology Problem is further from being solved. I think the way this problemwill be solved is through a few “killer apps” that result in the building up ofa large amount of content around particular ontologies within particular onlineservices.

Where might we see this content initially arising? In my opinion it will most likely be within vertical communities of interest, communities of practice, and communities of purpose. Within such communities there is a need to create a common body of knowledge and to make that knowledge more accessible, connected and useful.

The Semantic Web can really improve the quality of knowledge and user-experience within these domains. Because they are communities, not just static content services, these organizations are driven by user-contributed content — users play a key role in building content and tagging it. We already see this process starting to take place in communities such as Flickr, del.icio.us, the Wikipedia and Digg. We know that communities of people do tag content, and consume tagged content, if it is easy and beneficial enough for to them to do so.

In the near future we may see miniature Semantic Webs arising around particular places, topics and subject areas, projects, and other organizations. Or perhaps, like almost every form of new media in recent times, we may see early adoption of the Semantic Web around online porn — what might be called “the sementic web.”

Whether you like it or not, it is a fact that pornography was one of the biggest drivers of early mainstream adoption of personal video technology, CD-ROMs, and also of the Internet and the Web.

But I think it probably is not necessary this time around. While, I’m sure that the so-called “sementic web” could become better from the Semantic Web, it isn’t going to be the primary driver of adoption of the Semantic Web. That’s probably a good thing — the world can just skip over that phase of development and benefit from this technology with both hands so to speak.

The World Wide Database

In some ways one could think of theSemantic Web as “the world wide database” – it does for the meaning of data records what theWeb did for the formatting documents. But that’s just the beginning. It actually turnsdocuments into richer data records. It turns unstructured data into structureddata. All data becomes structured data in fact. The structure is not merelydefined structurally, but it is defined semantically.

In other words, it’s notmerely that for example, a data record or document can be defined in such a wayas to specify that it contains a certain field of data with a certain label ata certain location – it defines what that field of data actually means in anunambiguous, machine understandable way. If all you want is a Web of data,XML is good enough. But if you want to make that data interoperable and machineunderstandable then you need RDF and OWL – the Semantic Web.

Like any database,the Semantic Web, or rather the myriad mini-semantic-webs that will comprise it,have to overcome the challenge of data integration. Ontologies provide a betterway to describe and map data, but the data still has to be described andmapped, and this does take some work. It’s not a magic bullet.

The Semantic Webmakes it easier to integrate data, but it doesn’t completely remove the dataintegration problem altogether. I think the eventual solution to this problemwill combine technology and community folksonomy oriented approaches.

The Semantic Web in HistoricalContext

Let’s transition now and zoom out to see the bigger picture. The Semantic Webprovides technologies for representing and sharing knowledge in new ways. Inparticular, it makes knowledge more accessible to software, and thus to otherpeople. Another way of saying this is that it liberates knowledge fromparticular human minds and organizations – it provides a way to make knowledgeexplicit, in a standardized format that any application can understand. This isquite significant. Let’s put this in historical perspective.

Before the invention of the printing press, there were two ways to spreadknowledge – one was orally, the other was in some symbolic form such as art orwritten manuscripts. The oral transmission of knowledge had limited range and ahigh error-rate, and the only way to learn something was to meet someone whoknew it and get them to tell you. The other option, symbolic communicationthrough art and writing, provided a means to communicate knowledgeindependently of particular people – but it was only feasible to produce a fewcopies of any given artwork or manuscript because they had to be copied byhand. So the transmission of knowledge was limited to small groups or at leastsmall audiences. Basically, the only way to get access to this knowledge was tobe one of the lucky few who could acquire one of its rare physical copies.

The invention of the printing press changed this – for the first timeknowledge could be rapidly and cost-effectively mass-produced and mass-distributed.Printing made it possible to share knowledge with ever-larger audiences. Thisenabled a huge transformation for human knowledge, society, government,technology – really every area of human life was transformed by thisinnovation.

The World Wide Web made the replication and distribution of knowledge eveneasier – With the Web you don’t even have to physically print or distributeknowledge anymore, the cost of distribution is effectively zero, and everyonehas instant access to everything from anywhere, anytime. That’s a lot betterthan having to lug around a stack of physical books. Everyone potentially haswhatever knowledge they need with no physical barriers. This has been anotherhuge transformation for humanity – and it has affected every area of humanlife. Like the printing press, the Web fundamentally changed the economics ofknowledge.

The Semantic Web is the next big step in this process – it will make all theknowledge of the human race accessible to software. For the first time,non-human things (software applications) will be able to start working withhuman knowledge to do things (for humans) on their own. This is a big leap – aleap like the emergence of a new species, or the symbiosis of two existingspecies into a new form of life.

The printing press and the Web changed the economics of replicating,distributing and accessing knowledge. The Semantic Web changes the economics ofprocessing knowledge. Unlike the printing press and the Web, the Semantic Webenables knowledge to be processed by non-human things.

In other words, humans don’t have to do all the thinking on their own, theycan be assisted by software. Of course we humans have to at least first createthe software (until we someday learn to create software that is smart enough tocreate software too), and we have to create the ontologies necessary for thesoftware to actually understand anything (until we learn to create software thatis smart enough to create ontologies too), and we have to add the semanticmetadata to our content in various ways (until our software is smart enough todo this for us, which it almost is already). But once we do the initial work ofmaking the ontologies and software, and adding semantic metadata, the systemstarts to pick up speed on its own, and over time the amount of work we humanshave to do to make it all function decreases. Eventually, once the system hasencoded enough knowledge and intelligence, it starts to function withoutneeding much help, and when it does need our help, it will simply ask us andlearn from our answers.

This may sound like science-fiction today, but in fact it a lot of this isalready built and working in the lab. The big hurdle is figuring out how to getthis technology to mass-market. That is probably as hard as inventing thetechnology in the first place. But I’m confident that someone will solve iteventually.

Once this happens the economics of processing knowledge will truly bedifferent than it is today. Instead of needing an actual real-live expert, theknowledge of that expert will be accessible to software that can act as theirproxy – and anyone will be able to access this virtual expert, anywhere,anytime. It will be like the Web – but instead of just information beingaccessible, the combined knowledge and expertise of all of humanity will alsobe accessible, and not just to people but also to software applications.

The Question of Consciousness

The Semantic Web literally enables humans to share their knowledge with eachother and with machines. It enables the virtualization of human knowledge andintelligence. With respect to machines, in doing this, it will lend machines“minds” in a certain sense – namely in that they will at least be able tocorrectly interpret the meaning of information and replicate the expertise ofexperts.

But will these machine-minds be conscious? Will they be aware of themeanings they interpret, or will they just be automatons that are simplyfollowing instructions without any awareness of the meanings they areprocessing? I doubt that software will ever be conscious, because from what Ican tell consciousness — or what might be called the sentient awareness ofawareness itself as well as other things that are sensed — is an immaterialphenomena that is as fundamental as space, time and energy — or perhaps evenmore fundamental. But this is just my personal opinion after having searchedfor consciousness through every means possible for decades. It just cannot befound to be something, yet it is definitely and undeniably taking place.

Consciousness can be exemplified through the analogy of space (but unlikespace, consciousness has this property of being aware, it’s not a mere lifelessvoid). We all agree space is there, but nobody can actually point to itsomewhere, and nobody can synthesize space. Space is immaterial andfundamental. It is primordial. So is electricity. Nobody really knows whatelectricity is ultimately, but if you build the right kind of circuit you canchannel it and we’ve learned a lot about how to do that.

Perhaps we may figure out how to channel consciousness like we channelelectricity with some sort of synthetic device someday, but I think that ishighly unlikely. I think if you really want to create consciousness it’s mucheasier and more effective to just have children. That’s something ordinarymortals can do today with the technology they were born with. Of course whenyou have children you don’t really “create” their consciousness, it seems to bethere on its own. We don’t really know what it is or where it comes from, orwhen it arises there. We know very little about consciousness today.Considering that it is the most fundamental human experience of all, it isactually surprising how little we know about it!

In any case, until we truly delve far more deeply into the nature of themind, consciousness will be barely understood or recognized, let aloneexplained or synthesized by anyone. In many eastern civilizations there aremulti-thousand year traditions that focus quite precisely on the nature ofconsciousness. The major religions have all universally concluded thatconsciousness is beyond the reach of science, beyond the reach of concepts,beyond the mind entirely. All those smart people analyzing consciousness for solong, and with such precision, and so many methods of inquiry, may have a pointworth listening to.

Whether or not machines will ever actually “know” or be capable of beingconscious of that meaning or expertise is a big debate, but at least we can allagree that they will be able to interpret the meaning of information and rulesif given the right instructions. Without having to be conscious, software willbe able to process semantics quite well — this has already been proven. It’sworking today.

While consciousness is and may always be a mystery that we cannot synthesize– the ability for software to follow instructions is an established fact. Inits most reduced form, the Semantic Web just makes it possible to providericher kinds of instructions. There’s no magic to it. Just a lot of details. Infact, to play on a famous line, “it’s semantics all the way down.”

The Semantic Web does not require that we make conscious software. It justprovides a way to make slightly more intelligent software. There’s a bigdifference. Intelligence is simply a form of information processing, for themost part. It does not require consciousness — the actual awareness of what isgoing on — which is something else altogether.

While highly intelligentsoftware may need to sense its environment and its own internal state andreason about these, it does not actually have to be conscious to do this. Theseoperations are for the most part simple procedures applied vast numbers of timeand in complex patterns. Nowhere in them is there any consciousness nor doesconsciousness suddenly emerge when suitable levels of complexity are reached.

Consciousness is something quite special and mysterious. And fortunately forhumans, it is not necessary for the creation of more intelligent software, noris it a byproduct of the creation of more intelligent software, in my opinion.

The Intelligence of the Web

So the real point of the Semantic Web is that it enables the Web to becomemore intelligent. At first this may seem like a rather outlandish statement,but in fact the Web is already becoming intelligent, even without the SemanticWeb.

Although the intelligence of the Web is not very evident at first glance,nonetheless it can be found if you look for it. This intelligence doesn’t existacross the entire Web yet, it only exists in islands that are few and farbetween compared to the vast amount of information on the Web as a whole. Butthese islands are growing, and more are appearing every year, and they arestarting to connect together. And as this happens the collective intelligenceof the Web is increasing.

Perhaps the premier example of an “island of intelligence” is theWikipedia, but there are many others: The Open Directory, portals such as Yahooand Google, vertical content providers such as CNET and WebMD, commercecommunities such as Craigslist and Amazon, content oriented communities such asLiveJournal, Slashdot, Flickr and Digg and of course the millions of discussionboards scattered around the Web, and social communities such as MySpace andFacebook. There are also large numbers of private islands of intelligence onthe Web within enterprises — for example the many online knowledge andcollaboration portals that exist within businesses, non-profits, andgovernments.

What makes these islands “intelligent” is that they are places where people(and sometimes applications as well) are able to interact with each other tohelp grow and evolve collections of knowledge. When you look at them close-upthey appear to be just like any other Web site, but when you look at what theyare doing as a whole – these services are thinking.They are learning, self-organizing, sensing their environments, interpreting,reasoning, understanding, introspecting, and building knowledge. These are theactivities of minds, of intelligent systems.

The intelligence of a system such as the Wikipedia exists on several levels– the individuals who author and edit it are intelligent, the groups that helpto manage it are intelligent, and the community as a whole – which isconstantly growing, changing, and learning – is intelligent.

Flickr and Digg also exhibit intelligence. Flickr’s growing system of tagsis the beginnings of something resembling a collective visual sense organ onthe Web. Images are perceived, stored, interpreted, and connected to conceptsand other images. This is what the human visual system does. Similarly, Digg isa community that collectively detects, focuses attention on, and interpretscurrent news. It’s not unlike a primitive collective analogue to the humanfacility for situational awareness.

There are many other examples of collective intelligence emerging on theWeb. The Semantic Web will add one more form of intelligent actor to the mix –intelligent applications. In the future, after the Wikipedia is connected tothe Semantic Web, as well as humans, it will be authored and edited by smartapplications that constantly look for new information, new connections, and newinferences to add to it.

Although the knowledge on the Web today is still mostly organized withindifferent islands of intelligence, these islands are starting to reach out andconnect together. They are forming trade-routes, connecting their economies,and learning each other’s languages and cultures. The next-step will be forthese islands of knowledge to begin to share not just content and services, butalso their knowledge — what they know about their content and services. The SemanticWeb will make this possible, by providing an open format for the representationand exchange of knowledge and expertise.

When applications integrate their content using the Semantic Web they willalso be able to integrate their context, their knowledge – this will make thecontent much more useful and the integration much deeper. For example, when anapplication imports photos from another application it will also be able toimport semantic metadata about the meaning and connections of those photos.Everything that the community and application know about the photos in theservice that provides the content (the photos) can be shared with the servicethat receives the content. Better yet, there will be no need for customapplication integration in order for this to happen: as long as both servicesconform to the open standards of the Semantic Web the knowledge is instantlyportable and reusable.

Freeing Intelligence from Silos

Today much of the real value of the Web (and in the world) is still lockedaway in the minds of individuals, the cultures of groups and organizations, andapplication-specific data-silos. The emerging Semantic Web will begin to unlockthe intelligence in these silos by making the knowledge and expertise theyrepresent more accessible and understandable.

It will free knowledge and expertise from the narrow confines of individualminds, groups and organizations, and applications, and make them not only moreinteroperable, but more portable. It will be possible for example for a personor an application to share everything they know about a subject of interest aseasily as we share documents today. In essence the Semantic Web provides acommon language (or at least a common set of languages) for sharing knowledgeand intelligence as easily as we share content today.

The Semantic Web also provides standards for searching and reasoning moreintelligently. The SPARQL query language enables any application to ask forknowledge from any other application that speaks SPARQL. Instead of merekeyword search, this enables semantic search. Applications can search forspecific types of things that have particular attributes and relationships toother things.

In addition, standards such as SWRL provide formalisms for representing andsharing axioms, or rules, as well. Rules are a particular kind of knowledge –and there is a lot of it to represent and share, for example proceduralknowledge, and logical structures about the world. An ontology provides a meansto describe the basic entities, their attributes and relations, but rulesenable you to also make logical assertions and inferences about them. Withoutgoing into a lot of detail about rules and how they work here, the importantpoint to realize is that they are also included in the framework. All forms ofknowledge can be represented by the Semantic Web.

Zooming Way, Waaaay Out

So far in this article, I’ve spenta lot of time talking about plumbing – the pipes, fluids, valves, fixtures,specifications and tools of the Semantic Web. I’ve also spent some time onillustrations of how it might be useful in the very near future to individuals,groups and organizations. But where is it heading after this? What is thelong-term potential of this and what might it mean for the human race on ahistorical time-scale?

For those of you who would prefer not to speculate, stop reading here. Forthe rest of you, I believe that the true significance of the Semantic Web, on along-term timescale is that it provides an infrastructure that will enable theevolution of increasingly sophisticated forms of collective intelligence. Ultimatelythis will result in the Web itself becoming more and more intelligent, untilone day the entire human species together with all of its software andknowledge will function as something like a single worldwide distributed mind –a global mind.

Just the like the mind of a single human individual, the global mind will bevery chaotic, yet out of that chaos will emerge cohesive patterns of thoughtand decision. Just like in an individual human mind, there will be feedbackbetween different levels of order – from individuals to groups to systems ofgroups and back down from systems of groups to groups to individuals. Becauseof these feedback loops the system will adapt to its environment, and to itsown internal state.

The coming global mind will collectively exhibit forms of cognition andbehavior that are the signs of higher-forms of intelligence. It will form andreact to concepts about its “self” – just like an individual human mind. Itwill learn and introspect and explore the universe. The thoughts it thinks maysometimes be too big for any one person to understand or even recognize them –they will be comprised of shifting patterns of millions of pieces of knowledge.

The Role of Humanity

Every person on the Internet will be a part of the global mind. Andcollectively they will function as its consciousness. I do not believe some newform of consciousness will suddenly emerge when the Web passes some thresholdof complexity. I believe that humanity IS the consciousness of the Web anduntil and unless we ever find a way to connect other lifeforms to the Web, orwe build conscious machines, humans will be the only form of consciousness ofthe Web.

When I say that humans will function as the consciousness of the Web I meanthat we will be the things in the system that know. The knowledge of theSemantic Web is what is known, but what knows that knowledge has to besomething other than knowledge. A thought is knowledge, but what knows thatthought is not knowledge, it is consciousness, whatever that is. We can figureout how to enable machines to represent and use knowledge, but we don’t knowhow to make them conscious, and we don’t have to. Because we are alreadyconscious.

As we’ve discussed earlier in this article, we don’t need conscious machines, we just need more intelligent machines.Intelligence – at least basic forms of it – does not require consciousness. It may be the case that the very highest forms of intelligence require or are capable of consciousness. This may mean that software will never achieve the highest levels of intelligence and probably guaranteesthat humans (and other conscious things) will always play a special role in theworld; a role that no computer system will be able to compete with. We providethe consciousness to the system. There may be all sorts of other intelligent,non-conscious software applications and communities on the Web; in fact therealready are, with varying degrees of intelligence. But individual humans, andgroups of humans, will be the only consciousness on the Web.

The Collective Self

Although the software of the Semantic Web will not be conscious we can say that system as a whole contains or is conscious to the extent that human consciousnesses are part of it. And like most conscious entities, it may also start to be self-conscious.

If the Web ever becomes a global mind as I am predicting, will it have a“self?” Will there be a part of the Web that functions as its central self-representation?Perhaps someone will build something like that someday, or perhaps it will evolve.Perhaps it will function by collecting reports from applications and people inreal-time – a giant collective zeitgeist.

In the early days of the Web portals such as Yahoo! provided this function — they were almost real-time maps of the Web and what was happening. Today making such a map is nearly impossible, but services such as Google Zeitgeist at least attempt to provide approximations of it. Perhaps through random sampling it can be done on a broader scale.

My guess is that the global mind will need a self-representation at somepoint. All forms of higher intelligence seem to have one. It’s necessary forunderstanding, learning and planning. It may evolve at first as a bunch ofcompeting self-representations within particular services or subsystems withinthe collective. Eventually they will converge or at least narrow down to just afew major perspectives. There may also be millions of minor perspectives thatcan be drilled down into for particular viewpoints from these top-level “portals.”

The collective self, will function much like the individual self – as amirror of sorts. Its function is simply to reflect. As soon as it exists theentire system will make a shift to a greater form of intelligence – because forthe first time it will be able to see itself, to measure itself, as a whole. Itis at this phase transition when the first truly global collective self-mirroring function evolves, that we can say that the transition from a bunch of cooperating intelligent parts toa new intelligent whole in its own right has taken place.

I think that the collective self, even if it converges on a few majorperspectives that group and summarize millions of minor perspectives, will becommunity-driven and highly decentralized. At least I hope so – because theself-concept is the most important part of any mind and it should be designedin a way that protects it from being manipulated for nefarious ends. At least Ihope that is how it is designed.

Programming the Global Mind

On the other hand, there are times when a little bit of adjustment or guidance iswarranted – just as in the case of an individual mind, the collective selfdoesn’t merely reflect, it effectively guides the interpretation of the pastand present, and planning for the future.

One way to change the direction ofthe collective mind, is to change what is appearing in the mirror of thecollective self. This is a form of programming on a vast scale – When thisprogramming is dishonest or used for negative purposes it is called “propaganda,” but there are cases whereit can be done for beneficial purposes as well. An example of this today ispublic service advertising and educational public television programming. Allforms of mass-media today are in fact collective social programming. When yourealize this it is not surprising that our present culture is violent andmessed up – just look at our mass-media!

In terms of the global mind, ideally one would hope that it would be able tolearn and improve over time. One would hope that it would not have the collective equivalent of psycho-social disorders. To facilitate this, just like any form of higherintelligence, it may need to be taught, and even parented a bit. It also mayneed a form of therapy now and then. These functions could be provided by thepeople who participate in it. Again, I believe that humans serve a vital and irreplaceablerole in this process.

How It All Might Unfold

Now how is this all going to unfold? I believe that there are a number ofkey evolutionary steps that Semantic Web will go through as the Web evolvestowards a true global mind:

1. Representing individual knowledge. The first step is to make individuals’knowledge accessible to themselves. As individuals become inundated withincreasing amounts of information, they will need better ways of managing it,keeping track of it, and re-using it. They will (or already do) need”personal knowledge management.”

2. Connecting individual knowledge. Next, once individual knowledge isrepresented, it becomes possible to start connecting it and sharing it acrossindividuals. This stage could be called “interpersonal knowledgemanagement.”

3. Representing group knowledge. Groups of individuals also need ways ofcollectively representing their knowledge, making sense of it, and growing itover time. Wikis and community portals are just the beginning. The Semantic Webwill take these “group minds” to the next level — it will make the collective knowledge ofgroups far richer and more re-usable.

4. Connecting group knowledge. This step is analogous to connectingindividual knowledge. Here, groups become able to connect their knowledge togetherto form larger collectives, and it becomes possible to more easily access andshare knowledge between different groups in very different areas of interest.

5. Representing the knowledge of the entire Web. This stage — what might becalled “the global mind” — is still in the distant future, but atthis point in the future we will begin to be able to view, search, and navigatethe knowledge of the entire Web as a whole. The distinction here is thatinstead of a collection of interoperating but separate intelligentapplications, individuals and groups, the entire Web itself will begin tofunction as one cohesive intelligent system. The crucial step that enables thisto happen is the formation of a collective self-representation. This enablesthe system to see itself as a whole for the first time.

How it May be Organized

I believe the global mind will be organized mainly in the form of bottom-up and lateral, distributed emergent computation andcommunity — but it will be facilitated by certain key top-down services thathelp to organize and make sense of it as a whole. I think this future Web willbe highly distributed, but will have certain large services within it as well– much like the human brain itself, which is organized into functionalsub-systems for processes like vision, hearing, language, planning, memory,learning, etc.

As the Web gets more complex there will come a day when nobody understandsit anymore – after that point we will probably learn more about how the Web isorganized by learning about the human mind and brain – they will be quitesimilar in my opinion. Likewise we will probably learn a tremendous amountabout the functioning of the human brain and mind by observing how the Webfunctions, grows and evolves over time, because they really are quite similarin at least an abstract sense.

The internet and its software and content is like a brain, and the state ofits software and the content is like its mind. The people on the Internet arelike its consciousness. Although these are just analogies, they are actuallyuseful, at least in helping us to envision and understand this complex system. Asthe field of general systems theory has shown us in the past, systems at verydifferent levels of scale tend to share the same basic characteristics and obeythe same basic laws of behavior. Not only that, but evolution tends to convergeon similar solutions for similar problems. So these analogies may be more thanjust rough approximations, they may be quite accurate in fact.

The future global brain will require tremendous computing and storageresources — far beyond even what Google provides today. Fortunately as Moore’s Law advances thecost of computing and storage will eventually be low enough to do thiscost-effectively. However even with much cheaper and more powerful computingresources it will still have to be a distributed system. I doubt that therewill be any central node because quite simply no central solution will be ableto keep up with all the distributed change taking place. Highly distributed problemsrequire distributed solutions and that is probably what will eventually emergeon the future Web.

Someday perhaps it will be more like a peer-to-peer network, comprised ofapplications and people who function sort of like the neurons in the human brain.Perhaps they will be connected and organized by higher-level super-peers orsuper-nodes which bring things together, make sense of what is going on andcoordinate mass collective activities. But even these higher-level serviceswill probably have to be highly distributed as well. It really will bedifficult to draw boundaries between parts of this system, they will all beconnected as an integral whole.

In fact it may look very much like a grid computing architecture – in whichall the services are dynamically distributed across all the nodes such that atany one time any node might be working on a variety of tasks for differentservices. My guess is that because this is the simplest, most fault-tolerant,and most efficient way to do mass computation, it is probably what will evolvehere on Earth.

The Ecology of Mind

Where we are today in this evolutionary process is perhaps equivalent to therise of early forms of hominids. Perhaps Austrolapithecus or Cro-Magnon, ormaybe the first Homo Sapiens. Compared to early man, the global mind is like the rise of 21^stcentury mega-cities. A lot of evolution has to happen to get there. But itprobably will happen, unless humanity self-destructs first,which I sincerely hope we somehow manage to avoid. And this brings me to afinal point. This vision of the future global mind is highly technological;however I don’t think we’ll ever accomplish it without a new focus on ecology.

Ecology probably conjures up images of hippies and biologists, or maybehippies who are biologists, or at least organic farmers, for most people, but infact it is really the science of living systems and how they work. And anysystem that includes living things is a living system. This means that the Webis a living system and the global mind will be a living system too. As a living system, the Web is an ecosystem and is alsoconnected to other ecosystems. In short, ecology is absolutely essential tomaking sense of the Web, let alone helping to grow and evolve it.

In many ways the Semantic Web and the collective minds, and the global mind,that it enables, can be seen as an ecosystem of people, applications,information and knowledge. This ecosystem is very complex, much like naturalecosystems in the physical world. An ecosystem isn’t built, it’s grown, andevolved. And similarly the Semantic Web, and the coming global mind, will notreally be built, they will be grown and evolved. The people and organizationsthat end up playing a leading role in this process will be the ones thatunderstand and adapt to the ecology most effectively.

In my opinion ecology is going to be the most important science anddiscipline of the 21^st century – it is the science of healthysystems. What nature teaches us about complex systems can be applied to everykind of system – and especially the systems we are evolving on the Web. Inorder to ever have a hope of evolving a global mind, and all the wonderfullevels of species-level collective intelligence that it will enable, we have tonot destroy the planet before we get there. Ecology is the science that cansave us, not the Semantic Web (although perhaps by improving collectiveintelligence, it can help).

Ecology is essentially the science of community – whether biological,technological or social. And community is a key part of the Semantic Web atevery level: communities of software, communities of people, and communities ofgroups. In the end the global mind is the ultimate human community. It is thereward we get for finally learning how to live together in peace and balancewith our environment.

The Necessity of Sustainability

The point of this discussion of the relevance of ecology to the future ofthe Web, and my vision for the global mind, is that I think that it is clearthat if the global mind ever emerges it will not be in a world that is anythinglike what we might imagine. It won’t be like the Borg in Star Trek, it won’t belike living inside of a machine. Humans won’t be relegated to the roles ofslaves or drones. Robots won’t be doing all the work. The entire world won’t becoated with silicon. We won’t all live in a virtual reality. It won’t be one ofthese technological dystopias.

In fact, I think the global mind can only come to pass in a much greener,more organic, healthier, more balanced and sustainable world. Because it willtake a long time for the global mind to emerge, if humanity doesn’t figure outhow to create that sort of a world, it will wipe itself out sooner or later,but certainly long before the global mind really happens. Not only that, butthe global mind will be smart by definition, and hopefully this intelligencewill extend to helping humanity manage its resources, civilizations andrelationships to the natural environment.

The Smart Environment

The global mind also needs a global body so to speak. It’s not going to bean isolated homunculus floating in a vat of liquid that replaces the physicalworld! It will be a smart environment that ubiquitously integrates with ourphysical world. We won’t have to sit in front of computers or deliberatelylogon to the network to interact with the global mind. It will be everywhere.

The global mind will be physically integrated into furniture, houses,vehicles, devices, artworks, and even the natural environment. It will sensethe state of the world and different ecosystems in real-time and alert humansand applications to emerging threats. It will also be able to allocateresources intelligently to compensate for natural disasters, storms, andenvironmental damage – much in the way that the air traffic control systemsallocates and manages airplane traffic. It won’t do it all on its own, humansand organizations will be a key part of the process.

Someday the global mind may even be physically integrated into our bodiesand brains, even down the level of our DNA. It may in fact learn how to curediseases and improve the design of the human body, extending our lives, sensorycapabilities, and cognitive abilities. We may be able to interact with it bythought alone. At that point it will become indistinguishable from a limitedfrom of omniscience, and everyone may have access to it. Although it will onlyextend to wherever humanity has a presence in the universe, within thatboundary it will know everything there is to know, and everyone will be able toknow any of it they are interested in.

Enabling a Better World

By enabling greater forms of collective intelligence to emerge we really arehelping to make a better world, a world that learns and hopefully understandsitself well enough to find a way to survive. We’re building something thatsomeday will be wonderful – far greater than any of us can imagine. We’re helpingto make the species and the whole planet more intelligent. We’re building thetools for the future of human community. And that future community, if it ever arrives,will be better, more self-aware, more sustainable than the one we live intoday.

I should also mention that knowledge is power, and power can be used forgood or evil. The Semantic Web makes knowledge more accessible. This puts more power in the hands of the many, not just the few. As long as we stick to this vision — we stick to making knowledge open and accessible, using open standards, in as distributed a fashion as we can devise, then the potential power of the Semantic Web will be protected against being coopted or controlled by the few at the expense of the many. This is where technologists really have to be socially responsible when making development decisions. It’s important that we build a more open world, not a less open world. It’s important that we build a world where knowledge, integration and unification are balanced with respect for privacy, individuality, diversity and freedom of opinion.

But I am not particularly worried that the Semantic Web and the future globalmind will be the ultimate evil – I don’t think it is likely that we will end upwith a system of total control dominated by evil masterminds with powerfulSemantic Web computer systems to do their dirty work. Statistically speaking, criminal empires don’t last very long because theyare run by criminals who tend to be very short-sighted and who also surroundthemselves with other criminals who eventually unseat them, or theyself-destruct. It’s possible that the Semantic Web, like any other technology,may be used by the bad guys to spy on citizens, manipulate the world, and doevil things. But only in the short-term.

In the long-term either our civilization will get tired of endlesssuccessions of criminal empires and realize that the only way to actuallysurvive as a species is to invent a form of government that is immune to beingtaken over by evil people and organizations, or it will self-destruct. Eitherway, that is a hurdle we have to cross before the global mind that I envisioncan ever come about. Many civilizations came before ours, and it is likely thatours will not be the last one on this planet. It may in fact be the case that adifferent form of civilization is necessary for the global mind to emerge, andis the natural byproduct of the emergence of the global mind.

We know that the global mind cannot emerge anytime soon, and therefore, ifit ever emerges then by definition it must be in the context of a civilizationthat has learned to become sustainable. A long-term sustainable civilization is a non-evil civilization. And that is why I think it is a safebet to be so optimistic about the long-term future of this trend.

Change This, a project that helps to promote interesting new ideas so that they get noticed above the noise level of our culture has published my article on “A Physics of Ideas” as one of their featured Manifestos. They use an innovative PDF layout for easier reading, and they also provide a means for readers to provide feedback and even measure the popularity of various Manifestos. I’m happy this paper is getting noticed finally — I do think the ideas within it have potential. Take a look.

Note: This experiment is now finished.

GoMeme 2.0 — Copy This GoMeme From This Line to The End of this article, and paste into your blog. Then follow the instructions below to fill it out for your site.

Steal This Post!!!! This is a GoMeme– a new way to spread an idea along social networks. This is the second generation meme in our experiment in spreading ideas. To find out what a GoMeme is, and how this experiment works, or just to see how this GoMeme is growing and discuss it with others, visit the Root Posting and FAQ for this GoMeme at www.mindingtheplanet.net .

By adding this GoMeme to your Weblog you can get higher Google rankings for your site, and help your friends get higher Google rankings too. Your blog will be linked from every other blog that discovers this GoMeme downstream from your blog (from your readers, their readers, and so on). And that will raise your Google rankings in proportion to the number of downstream bloggers that get this GoMeme from you and post it to their blogs. The more people who blog the GoMeme from your blog, the better your Google rankings will get.

By hosting this meme on your blog, you will also be participating in an experiment to generate a distributed Blog survey and test how memes spread through social networks. The dataset from this experiment is public, open and decentralized — every blog that participates hosts their own data about their own blog. Anyone can then get the whole dataset by just searching Google for this unique string: 98818912959q This code is the “global unique identifier,” or GUID for this GoMeme — it marks every web page that participates in this GoMeme so that it can later be found with all the others. (Note it may take a week or longer before Google indexes your blog, so be patient).

Disclaimer

This is purely an experiment and is just for fun. We are really just curious to see what will happen and this is not a commercial project. Participation is voluntary. We don’t mean to annoy anyone. However, if you don’t have much curiosity, or at least a sense of humor, you may find this experiment to be upsetting. In that case, you might try drinking a good strong cup of coffee. If after that you are still unhappy with us, just don’t read any further and have a great day! (If you don’t want your blog to get better Google rankings, that’s purely your choice!) On the other hand, if you are interested in exploring new technologies and pushing the envelope, then keep reading and we look forward to your participation in this experiment. We also request that participants in this experiment refrain from spamming anyone with this GoMeme. To spread it, just put it on your blog; that should be enough.

INSTRUCTIONS FOR ADDING THIS GOMEME TO YOUR OWN SITE

Step 1 First, to add your site to this experiment, copy the GoMeme to your site from the “Copy This GoMeme From Here” heading above to the End of this article. Please copy this whole article and try not to alter the text so that it is authentic for the people who get it from your blog. If you would like to come up with your own catchy headline, or anything else to help your GoMeme spread, feel free! Let’s see what works best.

Step 2: Now, fill in your answers to these Required Survey Fields (Note: Replace the answers below with your own answers). These will later be automatically data-mined by bots to compile the survey results.

(1) I found this GoMeme at URL: http://www.mindingtheplanet.com

(2) I found this GoMeme on date (day/month/year):03/08/04

(3) I found this GoMeme at time (in GMT format): 01:00:00

(4) I foundit via “Newsreader Software” or “Browsing the Web” or “Searching the Web” or “An E-Mail Message”: Browsing the Web

(5) I posted this GoMeme at my URL (use a hyperlink): Minding the Planet

(6) I posted this on date (day/month/year): 03/08/04

(7) I posted this at time (in GMT format): 01:00:00

(8) My posting location is (city, state, country): San Francisco, California, USA

Step 3: If you’re feeling very altruistic today, also fill in these optional survery fields (Replace the answers below with your own answers):

(9) My Weblog is hosted by: Typepad

(10) My age is: 35

(11) My gender is: Male

(12) My occupation is: Internet Entrepreneur, Ontologist, Writer

(13) I use the following RSS/Atom reader software: Typepad, FeedDemon

(14) I use the following software to post to my blog: Typepad, Zempt

(15) I have been blogging since (day, month, year): 05/08/03

(16) My web browser is: IE

(17) My operating system is: Windows XP

Step 4:Now add an entry for your site after the last entry in the PATH LIST below:
Your entry should be of the form: line number, URL, hyperlink, optional personal GUID for your blog.

(Note: If you would like to track all postings of the Meme that result from your posting of it, once Google has indexed them, you may add your own optional GUID after your hyperlink on your line of the Path List — just make sure it is short, unique, and doesn’t return any results on Google — for example “mysitename137a2r28″. Also note, if the path list gets too long, you should still try to include the whole path in your blog — even if you have to put the list on a continuation page rather than the excerpt for your posting — and make sure others copy the whole GoMeme along with your Path List when they get the GoMeme from you — If they don’t copy it, your blog and your upstream blogs won’t be linked from their blogs).

PATH LIST

1. http://www.mindingtheplanet.net Minding The Planet, mindingtheplanet14798
2. (your Path List entry goes here in the form URL, hyperlink, GUID. Also, please add a new line after this one, for the next person.)

The End

You did it! Now spread it! If all goes well and others find this GoMeme from your blog, you should see some interesting results. Please comment back on the original post and tell us how you’re doing or what you observe, if anything noteworthy happens.

by Nova Spivack
Originally published on July 28, 2004; Updated on October 10, 2011

http://novaspivack.com

Should there be a Constitutional Separation of Corporation and State?

Today our American democracy faces a new threat to its integrity, a threat even greater than terrorism in the long-term. This threat is the corporation. In this essay I propose that it may be time to introduce a new principle into our democracy and a new amendment to our Constitution – a formal “Separation of Corporation and State.”

To illustrate this point, consider an earlier “separation” that has been essential to our democracy — the Separation of Church and State. What would America be like if the Constitution did not provide for the separation of Church and State? Would it be a nation that protects and celebrates freedom, equality and pluralism? Or would it be a nation, not so unlike those presently under the sway of fundamentalism, run by religious lobbies, religious police, and fanatical extremists?

I have nothing against religion – in fact I am religious myself – but I don’t think religion should have anything to do with government, or vice-versa. This is in fact one of the key ideas in our Constitution. Many of our Founding Fathers were deeply religious, but they recognized the need to make a clear distinction between their religious ideals and their political ideals. Thus over time a Constitutional separation of Church and State was formed — a separation that would not only protect the integrity and objectivity of government, but also that of religious institutions.

However, although they were well-aware of the risks of mixing politics and religion, our nation’s early Constitutional scholars were not as concerned with the risks of mixing politics and business. And why should they have been? At the time corporations were not nearly as independent or influential as monarchies and the Church. They were not considered threats. It would not be until later in the Industrial Age that corporations became a serious political force to reckon with. But one might well wonder whether our Constitution would have included protections against corporate influence had corporations been more of a force at the time it was devised.

Today corporations are becoming the single most powerful force shaping our societies and governments. While corporations have great potential to benefit society and even governments, they are entirely selfish entities – they have no accountability to the public, and no responsibility to ensure the public good. A government that is influenced by corporations can easily become a government that caters to corporations, a government that is effectively run by corporations. Such a government is not representative of its people anymore. It is therefore not a democracy.

Corporate influence on government, if not carefully regulated, is a threat to democracy. It is a threat to the American way of life. This threat to democracy may not be as dramatic as terrorism, but in the long-term it may be far more damaging to society. In fact this threat was foreseen by some of our most visionary leaders:

“I see in the near future a crisis approaching that unnerves me and causes me to tremble for the safety of my country. … corporations have been enthroned and an era of corruption in high places will follow, and the money power of the country will endeavor to prolong it’s reign by working upon the prejudices of the people until all wealth is aggregated in a few hands and the Republic is destroyed.”
– Abraham Lincoln

“The liberty of a democracy is not safe if the people tolerate the growth of private power to a point where it becomes stronger than their democratic State itself. That, in its essence, is Fascism — ownership of government by an individual, by a group or by any controlling private power.”
– Franklin D. Roosevelt

Because this threat was impossible to envision at the time our nation was formed, our Constitution was not designed with specific countermeasures and as a result our leaders, our government, our democracy, and our citizens, are presently without protection from political influence and manipulation by corporate interests. The danger of this is that our government may be run by corporations, or at least key decisions may be based on commercial interests. But is it democratic for national decisions to be driven by corporations that are only responsible to their shareholders? Are We The People represented by the corporate decision-makers and politicians they fund?

Are we living in a true democracy when many of our highest elected officials continue to receive money from, and hold stock in, large corporations they formerly worked for, or may work for when they are out of office? Are we living in a true democracy when our leaders are able to award lucrative no-bid contracts to their former employers? Are we living in a true democracy when public policy is influenced by corporate-backed political lobbies that spend millions of dollars to influence key decisions and elections? Are we living in a true democracy when the same people who start and run our wars also benefit financially from lucrative military industrial contracts? Is this ethical? Is this what our Founding Fathers intended, or is our Shining City on the Hill starting to get a bit tarnished?

I ask you then: Is it time to modify the Constitution to specifically provide for a formal “Separation of Corporation and State” in our democracy? And if we don’t take action, can our democracy survive?

One viewpoint on the matter is that we should not enforce a specific Separation of Corporation and State but rather seek to provide ethical guidelines to corporations and politicians — in other words, we should simply trust politicians and business people to maintain ethical boundaries and act appropriately. But can we really rely on them to self-regulate? Can we trust the foxes to guard the hens? After all if politicians require corporate endorsements and funding, or at least the absence of corporate interference, to win elections and stay in power, and if corporations in turn require political influence to cut costs, increase profits and beat the competition, can we really trust them to not do deals with one another?

As America and the world enter the twenty first century there appears to be a blurring of the distinction between capitalism and democracy. Many Americans, let alone others around the world, may not even be aware that there is any distinction at all! In fact, capitalism is not a form of government – it is an economic framework while democracy is not an economic framework, it is a social system. They are not one entity, they are two complementary systems. While they are often found together and have the potential for profound symbiosis (and in fact cannot really thrive without one another), neither is a sufficient substitute for the other.

For example running a corporation exclusively according to the rules of democracy is probably not good for the bottom line, but neither is running a nation exclusively according to the rules of capitalism good for society. These two forces must be balanced appropriately. In a corporation, democracy must take second place (although I argue elsewhere that perhaps corporations should be at least more democratic than they presently are). In a society however, democracy must take first place; it must never be overwhelmed by capitalist interests.

If there was no separation of Church and State in America, both our government and religious institutions would suffer. Similarly, in the case of the tension between capitalism and democracy, the only viable, sustainable, and effective path is to maintain a very precise balance. If this balance is not maintained, neither democracy nor capitalism can function with full effectiveness and everyone loses in the long-run. Short-term thinkers may gain temporary benefits by taking advantage of imbalances of this nature, but only at the expense of the many, and ultimately even at their own expense.

From the perspective of John Stuart Mill, who advocated the philosophy of “the greatest good for the greatest number,” we must not give in to the temptation to seek short-term gain at the expense of long-term sustainability. Ensuring that this does not happen is essential to the sustainability both of democracies and free markets. Unrestrained capitalism is a cancer – ultimately it consumes everything in its path.

At the same time, unrestrained democracy can easily devolve into socialism and economic gridlock – the death of the free market economy, and stunted growth. Only a very delicate, precise, and carefully enforced balance between capitalism and democracy can ensure both long-term homeostasis AND growth – a sustainable civilization.

The issue of the Separation of Corporation and State runs deep – it is not only our problem, it is everyone’s problem because America is now leading the world. Our American democracy is the template for new democracies, an example for others to follow. And now that we are in the business of seeding new democracies it is even more important that we practice what we preach. What kind of democracies are we really making in other countries? And what kind of democracy are we ourselves really living in now? What kind of standard – what kind of a template – are we providing for others who would emulate us?

What makes this nation so great is that it stands for something – it always has. We stand for freedom, we stand for equality, we stand for justice, we stand for tolerance, we stand for opportunity, we stand for human rights, we stand for democratic principles – and in fact, we stand for balance.

Balance between opposing agendas, opposing priorities, opposing points of view, has always been the heart of our nation’s underlying philosophy. This willingness to live by, and fight for, these basic rights and principles is what has made us great, what has given us moral authority on the world stage. It is also what has made the idea of America – our cultural meme – so contagious. If we forget this balance or fail to preserve it, we may lose everything we have worked for, everything we have attained, and the whole world will lose alongside us. What a lost opportunity that would be.

Americans need to think about this issue carefully. The very heart of American democracy and capitalism is balance. To preserve this balance, we must adapt and evolve our nation in the face of change. Today that balance is threatened – some would argue it is already gone – due to corporate influence over the political process. In other words, our nation is at risk of losing its heart.

The question is not therefore, “should there be a Separation of Corporation and State” but rather “how can we realistically and practically ensure a Separation of Corporation and State?” Should we add new protections to the Constitution in some way? Should we legislate? Should we simply “let the market sort it out” or trust our leaders and corporations to self-regulate and do the right thing?

I am a dedicated capitalist; I have benefited from the free market and I believe in self-organization, creative chaos, and bottom-up emergent solutions to complex distributed problems. So I would not advocate restraining capitalism to such an extent that it loses its edge. Capitalism is a reflection of nature, of evolution itself – a basic creative process that leads to innovation, growth, optimization, and development that can benefit individuals and societies in incalculable ways.

Without capitalism democracies lack energy and cannot thrive, grow, innovate and reproduce. Yet at the same time, I believe deeply in democracy and the basic principles that America stands for. Without democracy – true democracy – capitalism becomes malignant, destructive, and cannibalistic.

I would not want to live in a non-capitalist society – how boring, how complacent, how uninspiring and uncreative that would be. But neither would I want to live in a world controlled by corporations that are solely conditioned by profit motives – that would be a world raped of every natural resource, polluted to the point of being uninhabitable, commercialized and dumbed-down to the point of total conformity and cultural decay — a world completely for sale and thus completely sold out.

Because neither of these extreme futures — democracy without capitalism, or capitalism without democracy, is acceptable, I believe it is time to really address this issue of the Separation of Corporation and State as a society, and as a marketplace. Because if we don’t find a new balance between capitalism and democracy we will lose both.

But is it too late? Is it futile to address this issue? Some would argue the Great Sell-Out happened long ago. Others might even go so far as to suggest that it is not even a meaningful question anymore — that nations are no longer the primary actors in the world, but rather that we have already begun evolving a new world order that transcends nations altogether – a world governed by interacting transnational corporations – what we might call corpocracies that are the new units of civilization. But I hope that’s not the case. I believe we still have a chance at restoring the balance we’ve lost.

It is not too late to save democracy. We can and must evolve our democratic system to adapt and survive in a world of giant global corporations. While it is impossible to prevent interactions between government and corporations, or between our political leaders and corporate entities, we may be able to find ways to protect governments and politicians from corporate influence.

What would be some concrete steps to implement this proposed separation of corporation and state? As a first step, I think there should be a serious effort to revise or eradicate the concept of corporate personhood.

Beyond that, we could perhaps require that government officials sever their financial relationships to corporations while they serve in office, and perhaps even for a year or more after their service ends (provided the government still pays them during that grace period). For example it might be considered unethical and unacceptable for a top government official to leave office and immediately go to work for a major lobbying firm, or to receive huge payments for speaking or doing other favors for corporations, at least within some period of time after they serve in office.

In the case of certain high elected or appointed officials such as presidents, vice-presidents, members of Congress and the House of Representatives, cabinet members, chief regulators, and Supreme Court justices, the rules might even be a bit stricter. For instance, in the case of Supreme Court justices for instance, it might be time to require that not only they, but even their spouses, should have no financial connections to corporate influences.

A more moderate approach would be to allow financial connections to corporations while serving in a top government role, but simultaneously to more tightly regulate and monitor them — even for some time after a person serves at a high level in government.

We could also apply stricter controls to corporations and how they fund political lobbies and campaigns, and how they promote and sell products and services to the government. What these controls might actually be, and how to police them, is a topic for further thinking and debate.

These are just a few example ideas, and I’m sure much better solutions could be proposed. Beyond merely pointing out the imperative we face, and providing some examples, I do not have the answer, I do not know the formula for the balance we need to create. This is a question for people far more qualified and knowledgeable than myself to address – a question for our political leaders, our business leaders, our political scientists and Constitutional scholars, and our community activists.

But one thing is certain: The separation of corporation and state, or lack thereof, is an issue which will have the most profound effect on our nation, our society, and the rest of the world. It is perhaps the key challenge that America must address as we enter the twenty-first century.

by Nova Spivack, http://www.novaspivack.com

Original: July 8, 2004

Revised: February 5, 2005; February 28, 2010

(Permission to reprint or share this article is granted, with a citation to this Web Page: http://www.novaspivack.com/science/a-physics-of-ideas-measuring-the-physical-properties-of-memes)

This paper provides an overview of a new approach to measuring the physical properties of ideas as they move in real-time through information spaces and populations such as the Internet. It has applications to information retrieval and search, information filtering, personalization, ad targeting, knowledge discovery and text-mining, knowledge management, user-interface design, market research, trend analysis, intelligence gathering, machine learning,organizational behavior and social and cultural studies.

Introduction

In this article I propose the beginning of what might be called a physics of ideas. My approach is based on applying basic concepts from classical physics to the measurement of ideas — or what are often called memes — as they move through information spaces over time.

Ideas are perhaps the single most powerful hidden forces shaping our lives and our world. Human events are really just the results of the complex interactions of myriad ideas across time, space and human minds. To the extent that we can measure ideas as they form and interact, we can gain a deeper understanding of the underlying dynamics of our organizations, markets, communities, nations, and even of ourselves. But the problem is, we are still remarkably primitive when it comes to measuring ideas. We simply don’t have the tools yet and so this layer of our world still remains hidden from us.

However, it is becoming increasingly urgent that we develop these tools. With the evolution of computers and the Internet ideas have recently become more influential and powerful than ever before in human history. Not only are they easier to create and consume, but they can now move around the world and interact more quickly, widely and freely. The result of this evolutionary leap is that our information is increasingly out of control and difficult to cope with, resulting in the growing problem of information overload.

There are many approaches to combating information overload, most of which are still quite primitive and place too much burden on humans.  In order to truly solve information overload, I believe that what is ultimately needed is a new physics of ideas — a new micro-level science that will enable us to empirically detect, measure and track ideas as they develop, interact and change over time and space in real-time, in the real-world.

In the past various thinkers have proposed methods for applying concepts from epidemiology and population biology to the study of how memes spread and evolve across human societies. We might label those past attempts as “macro-memetics” because they are chiefly focused on gaining a macroscopic understanding of how ideas move and evolve. In contrast, the science of ideas that I am proposing in this paper is focused on the micro-scale dynamics of ideas within particular individuals or groups, or within discrete information spaces such as computer desktops and online services and so we might label this new physics of ideas as a form of “micro-memetics.”

To begin developing the physics of ideas I believe that we should start by mapping existing methods in classical physics to the realm of ideas. If we can treat ideas as ideal particles in a Newtonian universe then it becomes possible to directly map the wealth of techniques that physicists have developed for analyzing the dynamics of particle systems to the dynamics of idea systems as they operate within and between individuals and groups.

The key to my approach is to empirically measure the meme momentum of each meme that is active in the world. Using these meme momenta we can then compute the document momentum of any document that contain those memes. The momentum of a meme is a measure of the force of that meme within a given space, time period, and set of human minds (a “context”). The momentum of a document is the force of that document within a given context.

Once we are able to measure meme momenta and document momenta we can then filter and compare individual memes or collections of memes, as well as documents or collections of documents, according to their relative importance or “timeliness” in any context.

Using these techniques we can empirically detect the early signs of soon-to-be-important topics, trends or issues; we can measure ideas or documents to determine how important they are at any given time for any given audience; we can track and graph ideas and documents as their relative importances change over time in various contexts; we can even begin to chart the impact that the dynamics of various ideas have on real-world events. These capabilities can be utilized in next-generation systems for knowledge discovery, search and information retrieval, knowledge management, intelligence gathering and analysis, social and cultural research, and many other purposes.

The rest of this paper describes how we might attempt to do this, some applications of these techniques, and a number of further questions for research.

Background

Before I go into the details of my proposal, a little background maybe relevant. In 1993 I worked as an analyst at Individual, Inc. Individual’s business was to provide filtered strategic business intelligence to the top decision-makers of major corporations. In that job I was part of a sophisticated information filter. Individual used artificial intelligence to automatically collect news and other content from thousands of sources in real-time. Their system then filtered this  information into news feeds tailored to the strategic interests of their customers.

It was a two-phase system. First the computers sorted incoming content into topic-oriented buckets. Next these buckets of potentially interesting articles were routed to a team of human analysts with expertise in the relevant topic areas. The analysts would go through the articles in the buckets to prioritize them, remove duplicates or items that had come through in previous articles as well as items that did not belong, and add in any items that should be included. Finally the analysts would place the most strategically relevant articles from these various buckets into newsfeeds for each customer. Thus the humans were a very important part of the algorithm — they provided the intuition, knowledge, prioritization and trend detection capabilities of the system. This combination of machine and human filtering resulted in very high-quality strategic newsfeeds for their customers.

As one of Individual’s analysts, what this meant in practical terms was that every night from about 8 PM until 1 AM I had to personally read through around 1600 news articles. My beat was emerging technology, software, broadband, online-services, multimedia and satellite applications. It was a challenge to merely read through, let alone make sense of, such a volume of information every night.Furthermore, not only did I have to figure out what was important and how to prioritize it for each of the approximately 20 global corporations that I filtered for, but I also had to remember if I had ever seen and published anything about a given subject before in the previous year. By trial and error I gradually evolved a solution to this problem and this in turn led me to formulate the ideas that are the foundation of this paper.

The human brain is incredibly adept at recognizing patterns — and in particular we are tuned to detect subtle changes in size, mass and velocity. Many examples of this can be found in nature — for example in frogs. Frogs have interesting visual systems. They are tuned to focus on things that move. They are most sensitive to size and velocity, but they also notice changes in velocity. Things that are small and that don’t move are not of particular interest to them. Things that move in erratic ways are most interesting. But human brains are far more sophisticated — we don’t merely detect the size and velocity of things, we track changes in momentum. Momentum relates the “mass” or “size” of things to the way in which they change or move over time. What is important about momentum is that a low-mass thing moving quickly can have just as large a momentum as a large-mass thing moving slowly. In other words, we can detect small but “hot” emerging trends as well as large but gradual trends. We are extremely sensitive to momentum.

What I realized at Individual back in 1993 was that the way I figured out what articles to prioritize was not so different from how a frog finds flies to eat — but more sophisticated. I realized that I filter information according to the momenta of ideas — how the various memes in the articles I was reading were growing and moving through space and time in the culture I lived in and the communities I was interested in.

Human brains are highly sophisticated momentum detectors — our brains are constantly filtering billions of inputs and patterns in real-time and computing their momenta in order to differentiate signal from noise and to attenuate to what is most important at any given time. Furthermore as trends in the world emerge,grow, peak and fade away, so do their momenta, and we are able to very sensitively detect these changes in momentum in real-time,adjusting our priorities and attention accordingly. There is nothing magical about this process: it can be modeled mathematically,  and  therefore there is good reason to believe that computers can eventually be made to do this as well.

Memes

The Physics of Ideas is the science of micro-memetics — a science of the micro-level dynamics of individual memes. It is therefore necessary to define what we mean by the term “meme” (pronounced “meem”)? — basically, a meme is any replicable idea. More formally, a decent definition of a meme is:

“/meem/ [coined on analogy with `gene' by Richard Dawkins] n. An idea considered as a {replicator}, esp. with the connotation that memes parasitize people into propagating them much as viruses do. Used esp. in the phrase `meme complex’ denoting a group of mutually supporting memes that form an organized belief system, such as a religion. This lexicon is an (epidemiological) vector of the `hacker subculture’ meme complex; each entry might be considered a meme. However, `meme’ is often misused to mean `meme complex’. Use of the term connotes acceptance of the idea that in humans (and presumably other tool- and language-using sophonts) cultural evolution by selection of adaptive ideas has superseded biological evolution by selection of hereditary traits. Hackers find this idea congenial for tolerably obvious reasons.” (Definition from: The Hacker’s Dictionary)

Memes are essential to the way the human brain processes ideas and how it decides what is important. We are basically “meme processors” — we are “life-support systems for memes” to put it another way. To use a computer analogy, our physical bodies are like the hardware and operating system, and our minds — the dynamical activity and state of this hardware — are like the software applications and content running on the hardware. Our minds could be viewed as systems of interacting memes — complex systems of ideas that interact within us, and across our relationships.

Memes are capable of spreading across human social relationships via human interactions, and via human usage of static storage vehicles such as printed media, audio or video, and digital storage media — they are highly “communicable.” (And soon, as I have proposed in other articles, with the coming Semantic Web memes will be able to spread and interact without needing humans at all — machines will be able to process them on their own).

The Media is the Mirror

Before we can measure the physical properties of memes, we need a way to identify the memes we are interested in analyzing. We can identify memes by analyzing textual media such as document collections, wire services, and the Web.

The memes within text appear to be dormant — they are frozen digital representations. They do not move or reproduce on their own — they need help from humans (for the moment). But by inference, static textual representations of memes provide a mirror of the actual “active memes” that are taking place in the minds of the people who author and consume that media. What this indicates is that by analyzing textual media we are not merely looking at the memetic properties of text, we are looking at the memetic properties of people’s minds and of organizations, societies and cultures. In a sense, by selectively choosing the right media we can make a virtual focus group — we can see what people in this group are thinking.

The media is a mirror of our minds and cultures. By analyzing suitably selected information sources (for example, “all news articles from USA newspapers”) we can effectively focus on a reflection of the memes that are actually present within the minds of humans in a particular place, time, industry, community, demographic, etc. The more we know about the information sources, the more we can infer about the memes we find, and thus the memes taking place within the minds of the people who interact with those information sources.

The simplest approach to identifying memes in textual media is to simply pre-specify a list of memes we are interested in and to then search for any matching strings. For example we might be interested in measuring memes related to a particular trend, such as “Java technology,” so we could compile a list of terms related to Java and then use search techniques to locate all instances of those terms. We can then measure their properties.

A more sophisticated approach than specifying interesting memes in advance is to discover them empirically by analyzing text to see what’s there. To do this we might automatically identify nouns or noun-phrases and then measure their dynamics to see whether they are interesting enough to warrant further analysis. There are many existing computational liguistics techniques for isolating parts of speech and linguistic expressions.

Each of these nouns or phrases is a potential meme (we may consider them to all be actual memes or we may filter for only those memes that exhibit dynamics in space and time that meet our threshold for what constitutes “interesting” or “memelike” behavior. Another, more brute-force approach, would be to simply analyze every noun and phrase in a document or corpus for any that exhibit “memelike” dynamics in order to discover memes empirically instead of specifying them and then gathering their stats.

We can use various standard methods from text-mining and natural language processing to do a smarter job of identifying memes (for example, we can use stemming to consolidate various forms of the same word, we can use translation to consolidate expressions of the same meme in different languages, and we can use conceptual clustering and even ontologies to consolidate different memes that are equivalent to the same underlying meme). But for now, we can start by identifying memes in a simple way — the same way we might identify “topics” or “keywords” in a document. Once we can do this we can then measure the physical properties of those memes as they move through time and various spaces of interest.

(Note: We don’t necessarily have to analyze every document in a corpus to gather valid statistics for memes within it. We can use random sampling techniques for arbitrary degrees of accuracy if we wish to optimize for faster results and less computation. Instead of analyzing every occurance of each meme, we can analyze a statisically valid sample of the corpus.)

The Physics of Ideas

I suggest that the physics of ideas will be quite similar, if not equivalent to, the physics of the natural world. Everything in the universe emerges from the same underlying laws, even memes. The intellectual processes taking place within our own minds, as well as across our relationships and social organizations are similar to the dynamics of particle systems, fluid flows, gasses, and galaxies. We should therefore be able to map existing physical knowledge to the memescape, the dimension of memes.

Here are a set of basic measurements of the physical properties of memes and documents:

(Author’s Note, February 28, 2010: My latest thinking on this topic has evolved considerably from when this article was originally written in 2005. Instead of viewing memes as classical particles, I now think it is probably more accurate and useful to model them as physical waves or fields. At any given location (a media outlet, or a geographic place, or a person or document) every meme can be represented as a vector at any given time. In any case, regardless of the particular physical model we choose to map to memetics, the key point here is that it should be possible to make such a mapping from physics to memetics. This is a testable hypothesis. For example, select a certain mapping and generate some measurements about the higher order dynamics of memes, and then see if we can make testable predictions from those. Through such a process it should be possible to experimentally test and verify whatever mapping we choose, to find mappings that are most useful and accurate. Once we choose a mapping from physics to memetics that works, it could be an extraordinarily powerful tool for making sense of what is going on in the world, and particularly on the Web. I leave it to the physicists among us to come up the correct model, mappings, and experiments. In addition, since the original date of publication, social media has become an enormous playing field for memes and particularly rich source of data for measuring and mapping meme dynamics. In addition to documents we may also think of people and their lifestreams as sources of memetic data for measuring memes. Below is the original proposed mapping — which primarily was a classical physical model, focused on documents only.)

- Absolute meme mass = how “large” the meme is. There are various ways to come up with a measure of mass for memes and I don’t claim to have come up with the only, or even the best, way to do so. This is still a subject for further investigation. However, to begin, one approach at least is to interpret the mass as the total number of times a meme is mentioned in the corpus since the beginning of time to the present. However, it has been pointed out that this interpretation will cause the mass to increase over time. Still, it may be a useful interpretation, and in this paper I will use it provisionally. Another and perhaps better possibility, is to quantify the relative importance of particular memes in advance (for example by having analysts rate the terms that are most important to them) and to use these values as the mass of those memes.  Note: When computing meme mass, we can choose to count repeat mentions or ignore them — doing so has slightly different effects on the algorithm. We can also, if we wish, get more fancy and look at clusters of memes (via semantic network indexing or entity extraction, for example) that relate to the same concepts in order to compute “concept-cluster momenta” but that is not required.

- Absolute meme velocity = how fast the meme is moving in the corpus in the present time interval = The rate of occurrences (or “mentions”) of the meme per unit time (minutes, hours, days, etc.) in a given time interval.

- Absolute meme momentum = the force or importance of the meme in the corpus = the meme’s absolute mass x the meme’s absolute velocity

- Relative meme mass = the mass of a meme within a subset of documents or data in the corpus representing some set of interests. (Note: we call a subset of mutually co-relevant documents a “reference frame” or a “context.”) such as a set of interests, a particular period in time, etc. (rather than in the entire corpus).

- Relative meme velocity = the velocity of a meme within a reference frame.

- Relative meme momentum = the relative meme mass X the relative meme velocity.

On the basis of these we can then compute derivatives such as:

- Absolute meme acceleration = how the absolute meme velocity is changing in the entire corpus = The change in absolute velocity per unit time of the meme in the corpus.

- Relative meme acceleration = the change in relative velocity of a meme.

- Absolute meme impulse = the change in importance per unit time = the change in a meme’s absolute momentum.

- Relative meme impulse = the change of a meme’s relative momentum.

Next, we use the above concepts to look at sets of memes, for example documents:

- Absolute document momentum = the force or importance of a document in the entire corpus = the sum of the absolute momenta of each meme that occurs in the document.  (Note: we may choose to count or ignore repeat occurrences of an article in different locations or at different times — this has different effects).

- Relative document momentum = the force or importance of a document within a reference frame = the sum of the relative meme momenta in the document. This is a more contextually sensitive measure of document momentum — it couples momentum more tightly with a context, such as a particular query or time interval, or demographic segment.  (Note: we may choose to count or ignore repeat occurrences of an article in different locations or at different times — this has different effects).

- Hybrid document momentum = a measure of momentum that combines both relative and absolute measurements = either relative mass X absolute velocity or absolute mass X relative velocity.

How To Analyze a Corpus Using These Methods

We can then apply the above measurements to entire corpora (collections of documents). This enables us to empirically rank the ideas occurring in the corpus in any interval of time. Furthermore it enables us to rank and prioritize documents in the corpus according to their momenta within any time interval — in other words, how representative they are of “important” or “timely” ideas within any time interval.

To do this, first we must create an index of stats for all memes we are interested in. We can use the above mentioned techniques for identifying memes to do this. For each meme we identify, we create a record in our index that lists the stats we find for it by source location and time. We then analyze our text sources and update the records in this table (for a historical analysis we do this all at once; for a real-time analysis we do it continuously on an ongoing basis or in batches). As new instances of memes are found we append the corresponding records in the index.

We can now use these statistics to plot memes and documents according to our measurements of meme and document mass and velocity. This enables us to segment the memes or documents according to the various possible configurations of these dimensions. Each of these configurations has a useful meaning, for example a document with low absolute mass, moderate or high relative mass, high absolute velocity and high relative velocity contains “newly emerging trends of interest to the current context” whereas a document with high absolute mass, low relative mass, high absolute velocity and low relative velocity contains “established large trends that are not very relevant to the current context.”

By looking at the impulse (the change in momentum) we can also chart the direction of these trends (increasing or decreasing). Memes that have high positive impulse are becoming more “important” than those with lower impulses. This enables us to determine whether memes are “heating up” or “cooling off” — a meme is heating up if it is important and timely and has positive impulse.

Thus documents that have high document momenta contain memes that have high meme momenta — in other words they are representative of whatever ideas happen to be most important now. Tomorrow, when the momenta of various memes may have changed, the same documents might have different document momenta.

These techniques provide a way to rank documents that is in some respects like Google’s algorithm, except that it works for all types of information — not just information that is highly interlinked with hotlinks or citations but even for flat text — and it is capable of arbitrary resolution in time and space. For example, Google is basically estimating document popularity — or effectively, endorsements implied by citations — for each query. Google determines the rank of a page in a set of results by estimating the community endorsement of that page as implied by the number of relevant pages that link to it. Using the proposed physics of ideas however we can accomplish the same thing in a different and possibly better way — we can now compute the ‘potential community value’ of a document — without actually requiring links in order to figure that out. Instead, we can determine the relative strength of the ideas (the memes) that are present in the document and compare them to the memes that are present in the community of documents that are relevant to the keywords in our query.

For example, we do a query for “space tourism” and get back 6,830,000 documents in Google. Next we compute the above stats for each of those documents. We then rank the documents returned by this query according to their relative document momenta. This has the effect of ranking the documents according to the strengths of memes that are particularly of interest to the community represented by the query results. Thus it enables us to rank the resulting documents for our “space tourism” query to favor those documents that contain the highest momentum memes relative to set of memes that matter to the community — in other words the documents that contain ideas that are most “timely for the community” would appear higher. So this is a way to figure out not just what is relevant but what is important or in other words timely at a given point in time to people with a given set of interests.

Example Applications

Using the above techniques we can use momentum to provide a more sensitive way to filter any collection of information objects for which we can gather stats representing mass and velocity. There are numerous useful applications of doing this. Below I describe some of them.

Filtering E-Mail

For example, one might filter their e-mail using meme and document momenta in order to automatically view messages, people and topics with high momentum, low momentum, growing or declining momentum, etc. One could also use these techniques to data-mine the articles in a news feed or corpus for those that contain the “hottest trends.” It could be used to automatically detect “emerging hot topics,” “people to watch,” “companies to watch,” “products or brands to watch” etc. When ever you send a message the system measures the memes in that message and updates a special meme-stats index called “my interests” which just has the meme-stats for memes in messages you send. All incoming e-mail messages you receive can then be ranked according to their document momenta with respect to the meme momenta in the “my interests” index. This e-mail filter is automatically adaptive — as you send messages it learns what your current interest priorities are and this is reflected in changing meme momenta, even as your interests shift over time. These updated momenta are then used to filter incoming mail. So your mail filter learns what is important to you as you work and adapts to focus on your current priorities and interests, without you having to teach it. It just learns and adapts to model your current interests as you work.

Media Analysis

Beyond just that, these techniques can be used to perform more precise media analysis — for example they can be applied to measure the success of an advertising or marketing campaign by correlating the campaign placements with changes in momentum of the memes for the brand or product in the media.

Predicting Changes to a Stock Price

We can also use these techniques to make predictions — for example, we can correlate meme-momenta for memes related to a company with technical properties of its financials and stock price and then make predictions about price changes by analyzing news articles to detect changing meme-momenta related to the company. We can also do pure statistical correlations between meme momentuma and stock momenta for example. The financial news media is like a mirror reflecting what is taking place in the markets — but investors also use this mirror to decide what to do in the markets. So by measuring what appears in this mirror we can predict what investors are likely to do next.

Prioritizing Search Results and Implicit Query Expansion

We can also use these techniques to prioritize Internet search results — or any search results for that matter. For example, a set of Web documents can be prioritized by their document momenta, such that those that represent the memes that are currently the hottest can score higher — in other words, documents that are currently more timely can score higher than those that are less timely, and documents that are more timely yet less relevant (on a keyword level) can be ranked higher than those that are less timely but more keyword-relevant.

For example, suppose you search for “Asian restaurant.” If the meme “Vietnamese food” is currently in vogue in the media, meaning that it has higher momentum currently, then documents about restaurants that contain “Asian” or “restaurant” and that contain “Vietnamese food” will score higher than those that only mention “Asian” or “restaurant’ and “Chinese food” (assuming that Chinese food currently has a lower momentum). But this could change later as trends change. In other words, although we searched for “Asian food” we ended up getting documents ranked not merely by the keywords “Asian food” but by what topics related to Asian food have highest momentum today. This is a form of “implicit query expansion” and “implicit filtering.” In other words the system can prioritize search results for you according to the present momenta or in other words, the timeliness, of memes that occur in them. So it can show you the documents that are likely to be most important to you NOW in light of current trends and events, versus just the documents that have the best keyword relevancy.

Market Research

To make things even more interesting, we can add additional arguments to our “Rank of item” function and our meme-stats table — for example, not just a measure of mentions but also a measure of “hits” — hits on a meme increase whenever a document containing the meme is viewed. We can also add another dimension to represent the spatial distribution of memes. This will enable us to track the vectors of memes through time and space. We can do this by associating each source (each publisher) with a geographic location. We then segment our meme-stats table by geography to break out the momentum of each meme in each geographic region. This enables us to do things like filter documents by “how important they are to people in New York.”

By adding further dimensions — such as demographic profiles gleaned for example from the reader-surveys of publishers we can also segment by demographics, so we can even filter documents by “how important they are in the last month to professional, Democratic party affiliated, college educated, women in New York City who earn a median household income of $100,000.”

By adding still one more dimension to measure “sentiment” for each mention of a meme (as a function of the positive or negative language occurring near it or better yet, about it), we can even start to rank memes according to the percent of members of a given population that support or oppose them.In other words, this system can be used to empirically measure what polls and focus groups do informally. The notion here is that by selecting media sources that are representative of the community you are interested in understanding, you can then view memes and meme data relative to that group. You can also do this in the other direction, simply look to discover what memes have interesting stats for the group your are interested in. Another use of this technology might be to analyze intellectual history by computing meme-stats from historical documents or past news articles.

We can also leverage the fact that meme dynamics can be corellated with those of other memes to determine dynamical dependencies amongst them. This enables us to determine that some memes postively or negatively reinforce others. It also enables us to discover sets of related memes — such that we can learn that stats on a given meme should be inherited by related “child memes” in an automatically or manually generated taxonomy of memes.

Measuring and Mapping Ideas in the Semantic Web

We could also reference metadata about the semantics of various memes we can even filter for various types of memes — such as “memes related to vehicles” or “memes representing people” or “memes representing products ,” etc. This enables us to start measuring ideas as they occur and interact on the emerging Semantic Web — but not just particular memes, even conceptual systems of memes that are interacting or somehow ontologically related. By linking with an ontology, for example, we can track the momentum of all memes related to “American cars” versus those for “German cars.” The ontology enables inferences that help us find all memes that represent types of cars and classify them by nationality of manufacture.

Intelligence Analysis

These techniques might even be used to detect signs of potential terrorism, and to “get inside the minds” of various people or groups of interest — simply analyze the meme-stats for memes in documents they create or view to automatically generate a profile of the main ideas currently occupying their minds. Next by tracking this over time you can start to plot trajectories and make predictions. Intelligent agents can then be trained to notice “interesting” patterns in these trajectories and alert analysts as needed.

Advertising Targeting

The same methods could be used to better target advertisements or recommendations to users. Knowing what memes are currently most important to a party enables better personalization and targeting. In this case a Web site could track what memes are hottest for a given user account — derived from what pages they view and what messages they write or respond to. This data could then be used to augment the users’ interest profile with more dimensions of detail about each interest — such as how timely it is to the user, what particular nuances are specifically interesting, what their sentiment is. This could result in less irrelevance and spam for users and better results for marketers.

Knowledge Discovery

Now what gets interesting is the above methods can be used on both directions. We can use them to ask questions about memes we are interested in and we can also use to empirically discover memes we should be interested in within any corpus. So for example we can just empirically compute meme momenta and document momenta in any collection of information and then filter for whatever dynamics we are interested in, for example, “hot new emerging trends to watch.”

A New Kind of Portal

Using these methods it is possible to build a new kind of portal that provides a window into the collective mind of the planet (or any community of interest). It would show what people within the desired segment think is important over time. We could watch an animation on it of how memes for “Jihad” have spread, or for how those for a technology like “Java” have spread versus those for “Microsoft .Net,” or how a particular war is currently viewed by the public in different states or different demographic segments. A user could “drill down” into any meme to see it’s stats, all articles where it was mentioned, and related items on the Web, and maybe even products etc.

Open Questions & Directions for Further Research

It is important to note that these simple physical concepts could be taken much further. For example, using the above approach we should be able to determine the “gravity of a meme” or of a document or any set of memes or documents. We can then start to model the shape of memetic manifolds — the shape of space-time for ideas. We can also start to look at systems of memes as fields. Perhaps there may even be applications of fluid dynamics, relativity theory, or even quantum mechanics to what is taking place in the memescape — but today we are just taking baby-steps, just as Newton and the early natural philosophers did long before us. We need to begin to simply have the ability to measure memes and their basic interactions before we can go on to higher levels of analysis. I leave it to the physicists among us to take this to the next level of formalism — would anyone like to try their hand at formalizing the above proposed equations for the physics of ideas, or perhaps proposing even better ones?

There are a number of open questions I am still thinking about that suggest opportunities to refine these techniques. In particular, should we normalize documents somehow so that large documents don’t have an unfair advantage over small documents (because large documents have more terms in them and thus have higher document momenta)?

Another question is whether or not we should rank documents first by relevance to query, and then within each “relevancy band” further rank by document momentum within that band? This has the effect of limiting the impact of momentum versus relevancy — which may be useful if relevancy is considered to be more important. For example the top 100 most relevant documents are ranked by relevancy and then within that set they are ranked by document momentum and displayed, next the second 100 most relevant documents are ranked by relevancy and then within that set they are ranked by document momentum and displayed, etc.

Another question is whether there is an ideal set of priorities for the various measurement dimensions above with which to rank documents for general searches. We can let users choose their own priorities of course, for example, by letting users set their priorities for various memetic dimensions, we can then tailor our ranking for their needs. Are they just looking for all documents that are relevant to a query, or are they really trying to find documents that are representative of the most timely issues relevant to a query? We might enable users to set their weights for the absolute and relative measurements of documents in order to view different rankings of search results. Better yet, we could simply provide them with natural language filters to apply, such as “Filter for documents that contain currently hot topics related to this query.” In other words they can set priorities for the above dimensions in order to favor one dimension over another — so they might decide that query relevance is most important, document mass is second and velocity is least important. This would translate to a constraint such that it would be more difficult for documents with low relevance to be ranked higher than documents with high relevancy just because they have higher momenta On the other hand, they might want to favor momenta — for example if they really want to find documents that mention the latest trends related to a query — in which case we would favor document mass and/or velocity above document relevancy in our ranking. I am still thinking about the best way to handle these tradeoffs. Letting the user set their priorities is one way — but it may be possible to do a good job of satisfying most people with a particular set of default priorities. What is the best set of default priorities for general use?

There is also the question of how to best represent the “footprint of a meme” in geographic space. We can detect mentions of memes and using the above methods we may be able to associate each mention with a particular geography (the geographic region of the publisher and/or the intended audience — if the source has an audited audience demographic survey — as most publications that sell advertising do — then it is easy to associate any memes that occur within its content with particular geography and demography). Now the question is suppose we are tracking a particular meme — can we determine its geographic trajectory over time? Can we determine the vector of each meme at each sector in a geographic map? And can we represent that in an animated map for exampe, perhaps with something like a fluid flow animation?

Another open area to study is to analyze the higher order distributions of memes in order to automatically detect memes that are “interesting” (ie. not “noise” according to our priorities). One easy way to do this is to automatically ignore any memes that have a random distribution. We may also want to de-emphasize memes that have regular distributions — such as memes for which the dynamics have been the same for a reasonanble period of time. In other words, we want to filter for memes that have dynamics which deviate from being predictable or stable (randomness and regularity are both predictable). My hypothesis is that the really interesting memes — the memes that represent important emerging trends or current hot issues — will exhibit high volatility. For example, imagine for a moment that we are tracking memes related to “digital music” — if we look back in time there will be a point where the word “Napster” suddenly appears — at first it is a relatively “small” meme but gradually it spreads and gains momentum. Then there is a critical point where it begins to grow exponentially. Then it probably levels off for a while or even inflects after the initial hype phase ends. Next another dramatic increase in momentum should be seen around the time of the music industry’s lawsuits against Napster. Then following the resolution of these we should see Napster fall off dramatically. Later we see momentum increase again as the new commercial version of Napster is announced. This type of pattern is what we are looking for. Can we characterize these patterns well enough that we can detect them automatically?

Perhaps one way to do this is by training a neural network to recognize the types of patterns that interest us — we could do this for example by taking historical content (such as the last 10 years of the Associate Press) and then telling a neural net what memes are most important to us. The neural net can then learn from this training data. We can then run the neural net on current or more recent news and let it guess what is important to us based on the patterns of past important trends. We can rate these guesses to provide further feedback to improve learning. This approach could be used to train intelligent agents that specialize in detecting particular types of trends — for example, we could train agents to alert us when a major new technology trend is about to erupt, or when we should invest in a technology stock, or when a company we track is experiencing a major change of some sort, or to tells us when a new competing product emerges or when an existing competing product overtakes our own product, etc. We could also potentially train agents to recognize the early signs of important cultural or political issues, significant changes in sentiment or focus for a given community we are interested in, or even signs of emerging threats.

Are There Ideal Meme Distributions?

Perhaps one of the most interesting questions I have thought about in relation to the physics of ideas is whether or not there are perhaps “ideal distributions” of memes that get the best response from humans? In other words, do the higher order distributions of memes that become major trends, or that get the most attention in noisy environments, have similar characteristics? If it turns out that this is the case then it could provide a powerful new technique for advertising, information filtering, and even for user-interface design. I believe we can analyze memes to answer this question. Here’s how we might do it:

Approach 1: We choose a representative set of memes for major trends. We analyze their higher order distributions in the media. We then attempt to figure out whether these distributions have anything in common that we can isolate. We then search the media for other memes that have distributions with similar properties and test whether they are in fact major trends. We can provide feedback by scoring the output of these trials and using an evolutionary algorithm to evolve successively better filters. Eventually through such a process we can evolve an agent that is good at discovering major trends in the media.

Approach 2: We can do a perceptual psychology experiment to discover and evolve memes that get the most attention. Create a noisy environment in any sensory modality — let’s use visual information for the moment. Put 100 human subjects in a room and show them a computer generated slideshow. Our slideshow consists of 100 images. We change slides rapidly. Each slide is shown many times in the course of the slideshow, with a frequency according to one of many different distributions we wish to test. For example, one slide is shown such that it has low mass, low velocity — a low momentum. Another is shown to have high momentum. Others are shown to vary such that their momentum inflects and is volatile. We can test a number of different momentum curves in this manner — such as linear or nonlinear momentum growth, etc. At the end of the slideshow we give each subject all the slides and ask them to prioritize them in order of most important to least important — we ask them to tell us what they think the most important slides in the slideshow were. This effectively tests the various distributions we ran in the experiment to see which ones had the strongest cognitive effect on the subjects. Two weeks or a month later we repeat this rating test to see which distributions have the strongest long-term effect as well. By doing this experiment many times with many distributions we can experimentally determine which memetic distributions have the strongest cognitive impact. The next step would be to test whether the distributions we discover are applicable across sensory modalities — for example, do the distributions we found for vision also work for the auditory system. My hypothesis is that they do hold across modalities. If this is the case then we have discovered a key underying meta-pattern in the human perceptual system — the pattern by which humans recognize what to tune their attention to.

There is another interesting and related question to the above experiments: Do certain distributions retain attention better than others? The human perceptual system attenuates to signals very quickly — we tune out anything regular or predictable and focus on identifying novelty. But what is “novelty?” Any new meme that occurs is novel at first, but whether or not it remains novel or gets tuned out is another question. Which meme distributions do NOT get tuned out as quickly, or ever? Is there an optimal way to vary the distribution of a meme such that it continues to remain novel? In thinking about this, are there any meta-patterns to the memes that have gotten your attention in the past? For example, is there something about the way that particular technology trends or celebrities have moved through the media that made them appear to be hotter and more important to you? Having high momentum at a given time is part of this, but it may in fact be the change in momentum over time — the “meme impulse” — that really makes the difference. For example in my own experience I notice that trends that exhibit exponential growth in momentum quickly get my attention — but as soon as the growth becomes predictable I lose interest. So it seems that the trends that retain my interest the best are the ones that have more variable graphs — graphs that are neither random nor regular. Is there an ideal balance between randomness and order? What patterns have this balance — can we quantify this and define it more concretely?

A better understanding of the cognitive effects of various higher order distributions of memes in various human sensory modalities could be particularly useful for advertisers, marketers, and user-interface designers. An advertiser or marketer could use this knowledge to design campaigns that get the most attention and that are not “tuned out” by people as quickly. A user-interface designer could use this information to design interfaces for manging changing information in which the signal-to-noise ratio is optimized so that users can quickly focus on just the most important changing information — for example the information display of a stock-trading terminal, executive information system, military situation room, or fighter jet cockpit user-interface could perhaps be improved using these principles.

Concluding Remarks

Given that memes are now among the most powerful “hidden” forces shaping our individual minds, our relationships, organizations and our world, wouldn’t it be great if we could really measure them and analyze them empirically?

That is what I hope the basic techniques provided above will help to catalyze. By making this hidden layer visible we can gain a much better understanding of our world. Let me know if you end up using these techniques for anything interesting (and hopefully you will make your ideas open-source too so everyone can benefit).

What these basic techniques provide is a way to measure the movement of ideas in time and space. For example, we can track the trajectories of ideas in our workspaces, our teams, enterprises, cities, nations or interest-communities. We can also track them across geography or any other set of dimensions.

Because we can compute basic physical properties of memes we can start to apply Newtonian physics to analyze them. Perhaps by doing so we can really develop a “Physics of Memetics” with which we may begin to predict the outcomes of interactions among memes, the future trajectories of memes, and the influence changes to memes have on events in the so-called “real world” and vice-versa. With this in hand we could potentially teach systems to learn to detect memetic patterns of interest to us — for example the early “fingerprints” in the media that indicate the outcome of a proposed act of legislation or a vote, or a stock price, or a political change. We could also use it to detect emerging cultural trends, and to measure and compare the dynamics of brands or competing technologies in various markets in order to predict winners.

By putting this information into the public domain I hope to see these techniques in use as widely as possible. They will provide dramatic benefits in managing large volumes of information, improving knowledge worker and team productivity, and in discovering and measuring trends in communities.

Ultimately, I would like to see this embodied in a “grand cultural project” — a real-time map of the memetic dynamics taking place around the globe. This map would be filterable in order to show relative memetic dynamics in different places, communities, etc., and to show how various memes are spreading and interacting over time around the world. The data would be open and accessible via an open API so that all services that manage information could provide information to it and query it for stats when needed.

Draft 1.1 for Review (integrates some fixes from readers)
Nova Spivack (www.mindingtheplanet.net)

INTRODUCTION

This article presents some thoughts about the future of intelligence on Earth. In particular, I discuss the similarities between the Internet and the brain, and how I believe the emerging Semantic Web will make this similarity even greater.

DISTRIBUTED INTELLIGENCE

The Semantic Web enables the formal communication of a higher level of language — metalanguage. Metalanguage is language about language — language that encodes knowledge about how to interpret and use information. Metalanguages – particularly semantic metalanguages for encoding relationships between information and systems of concepts – enable a new layer of communication and processing. The combination of computing networks with semantic metalanguages represents a major leap in the history of communication and intelligence.

The invention of written language long ago changed the economics of communication by making it possible for information to be represented and shared independently of human minds. This made it less costly to develop and spread ideas widely across populations in space and time. Similarly, the emergence of software based on semantic metalanguages will dramatically change the economics not only of information distribution, but of intelligence — the act of processing and using information.

Semantic metalanguages provide a way to formally express, distribute and share the knowledge necessary to interpret and use information, independently of the human mind. In other words, they make it possible not just to write down and share information, but also to encode and share the background necessary for intelligently making use of that information. Prior to the invention of such a means to share this background knowledge about information, although information could be written and shared, the recipients of such information had to be intelligent and appropriately knowledgeable in advance in order to understand it. Semantic metalanguages remove this restriction by making it possible to distill the knowledge necessary to understand information into a form that can be shared just as easily as the information itself.

The recipients of information – whether humans or software – no longer have to know in advance (or attempt to deduce) how to interpret and use the information; this knowledge is explicitly coded in the metalanguage about the information. This is important for artificial intelligence because it means that expertise for specific domains does not have to be hard-coded into programs anymore — instead programs simply need to know how to interpret the metalanguage. By adding semantic metalanguage statements to information data becomes “smarter,” and programs can therefore become “thinner.” Once programs can speak this metalanguage they can easily import and use knowledge about any particular domain, if and when needed, so long as that knowledge is expressed in the metalanguage.

In other words, whereas basic written languages simply make raw information portable, semantic metalanguages make knowledge (conceptual systems) and even intelligence (procedures for processing knowledge) about information portable. They make it possible for knowledge and intelligence to be formally expressed, stored digitally, and shared independently of any particular minds or programs. This radically changes the economics of communicating knowledge and of accessing and training intelligence. It makes it possible for intelligence to be more quickly, easily and broadly distributed across time, space and populations of not only humans but also of software programs.

The emergence of standards for sharing semantic metalanguage statements that encode the meaning of information will catalyze a new era of distributed knowledge and intelligence on the Internet. This will effectively “make the Internet smarter.” Not just monolithic expert systems and complex neural networks, but even simple desktop programs and online software agents will begin to have access to a vast decentralized reserve of knowledge and intelligence.

The externalization, standardization and sharing of knowledge and intelligence in this manner, will make it possible for communities of humans and software agents to collaborate on cognition, not just on information. As this happens and becomes increasingly linked into our daily lives and tools, the "network effect" will deliver increasing returns. While today most of the intelligence on Earth still resides within human brains, In the near future, perhaps even within our lifetimes, the vast majority of intelligence will exist outside of human brains on the Semantic Web.

THE INTERNET IS A BRAIN AND THE WEB IS ITS MIND

Anyone familiar with the architecture and dynamics of the human nervous system cannot help but notice the striking similarity between the brain and the Internet. But is this similarity more than a coincidence – is the Internet really a brain in its own right – the brain of our planet? And is its collective behavior intelligent – does it constitute a global mind? How might this collective form of intelligence compare to that of an individual human mind, or a group of human minds?

I believe that the Internet (the hardware) is already evolving into a distributed global brain, and its ongoing activity (the software, humans and data) represents the cognitive process of an increasingly intelligent global mind. This global mind is not centrally organized or controlled, rather it is a bottom-up, emergent, self-organizing phenomenon formed from flows of trillions of information-processing events comprised of billions of independent information processors.

As with other types of emergent computing systems, for example John Conway’s familiar cellular automaton “The Game of Life,” on the Internet large scale homeostatic systems and seemingly intentional or guided information processes naturally emerge and interact within it. The emergence of sophisticated information systems does not require top-down design or control, it can happen in an evolutionary bottom-up manner as well.

Like a human brain, the Internet is a vast distributed computing network comprised of billions of interacting parallel processors. These processors include individual human beings as well as software programs, and systems of them such as organizations, which can all be referred to as "agents" in this system. Just as the computational power of the human brain as a whole is vastly greater than that of any of the individual neurons or systems within it, the computational power of the Internet is vastly beyond any of the individual agents it contains. Just as the human brain is not merely the sum of its parts, the Internet is more than the sum of its parts – like other types of distributed emergent computing systems, it benefits from the network effect. The power of the system grows exponentially as agents and connections between them are added.

The human brain is enabled by an infrastructure comprised of networks of organic neurons, dendrites, synapses and protocols for processing chemical and electrical messages. The Internet is enabled by an infrastructure of synthetic computers, communications networks, interfaces, and protocols for processing digital information structures. The Internet also interfaces with organic components however – the human beings who are connected to it. In that sense the Internet is not merely an inorganic system – it could not function without help from humans, for the moment at least. The Internet may not be organized in exactly the same form as the human brain, but it is at least safe to say it is an extension of it.

The brain provides a memory system for storing, locating and recalling information. The Internet also provides shared address spaces and protocols for using them. This enables agents to participate in collaborative cognition in a completely decentralized manner. It also provides a standardized shared environment in which information may be stored, addressed and retrieved by any agent of the system. This shared information space functions as the collective memory of the global mind.

Just as no individual neuron in the human brain could be said to have the same form or degree of intelligence as the brain as-a-whole – we individual humans cannot possibly comprehend the distributed intelligence that is evolving on the Internet. But we are part of it nonetheless, whether we know it or not.  The global mind is emerging all around us, and via us, is our creation but it is already becoming independent of us – truly it represents the evolution of a new form of meta-level intelligence that has never before existed on our planet.

Although we created it, the Internet is already far beyond our control or comprehension – it surrounds us and penetrates our world – it is inside our buildings, our tools, our vehicles, and it connects us together and modulates our interactions. As this process continues and the human body and biology begins to be networked into this system we will literally become part of this network – it will become an extension of our nervous systems and eventually, via brain-computer interfaces, it will be an extension of our senses and our minds. Eventually the distinction between humans and machines, and the individual and the collective, will gradually start to dissolve, along with the distinction between human and artificial forms of intelligence.

MEMES ARE EVOLVING MINDS OF THEIR OWN

The evolution of our planetary intelligence has been taking place for billions of years — it is a natural process, just like the evolution of human intelligence was long ago.  The Semantic Web is merely the next step in this process whereby communicable ideas (memes), having already evolved technologies to externalize themselves outside the human mind (i.e. books, recording, software, the Web, etc.) are starting to evolve the ability to propagate intelligently and interact without human intervention. In other words, although today memes are for the most part completely immobile and static unless perceived within a human brain, with the advent of the Semantic Web the cognitive processes for running memes will begin to spread outside the human brain, enabling memes to "run" without depending on humans.

This emerging planet-wide collective mind, of which we will be but parts, will evolve higher level meta-processes and structures that will vastly exceed our comprehension. Indeed this is already starting to happen — even today the self-organizing, chaotically emergent collective intelligence and information flows of the Internet exceed the power and understanding of any computer or brain on the planet. This new meta-level intelligence will be as far beyond human intelligence as the intelligence of the human brain is beyond that of its individual neurons.

THE INFRASTRUCTURE OF DISTRIBUTED INTELLIGENCE

The development of the global mind depends on the evolution of distributed systems that function as the global equivalent of consciousness, memory, learning, perception, introspection, planning, creativity, and behavior.

Distributed intelligence requires the decentralization of information and computation. The World Wide Web is a key catalyst for this evolutionary leap.  Before the Web there was no universally agreed-upon standard for publishing and accessing simple information – instead there were myriad incompatible, non-standardized competing proprietary formats. The lack of a common language made it difficult for applications to interoperate or understand one another’s data without explicit integration.

The significance of the Web is that its underlying metalanguage standards – HTML and HTTP – enable more widespread, interoperable and decentralized content production and access. Making it possible for agents anywhere in the system to publish and make use of information by any other agents in the system is an essential ingredient of a distributed intelligence. The Web is literally a World Wide File System – it is the memory function of the global mind.

If the Web enables the World Wide File System, the emergence of XML enables The World Wide Database.  XML enables agents in the system to define, store, retrieve, interact with, and interpret arbitrary data structures with arbitrary precision. Using XML any conceivable syntax and data schema can be defined and shared. XML adds more structure to the information in the memory of the global mind, enabling more sophisticated content and processes to be stored and accessed by agents in the system.

The recently emerging Semantic Web adds yet another layer of sophistication beyond XML. It enables agents in the system to begin to understand and reason about the meaning of information within the system. The Semantic Web enables software to work not merely with data but with concepts. Concepts are information structures that are connected to formal systems of ideas – in other words they are meaningful information.  The Semantic Web provides standards for transforming ordinary information structures into concepts that can be understood by software programs. Using metalanguages for defining semantics such as RDF and OWL, the Semantic Web makes it possible to connect data elements to concepts in formally defined systems of knowledge called ontologies. By doing this software programs are able to then reason intelligently about the information.

By connecting information to ontologies, programs can begin to process information more intelligently. For example, the content of a medical journal could be linked to a medical ontology that defines medical concepts and their interrelations. Using this ontology it would then be possible to do semantic searches of the journal that are far more intelligent than the primitive keyword searches that are currently used in most search systems today.  A semantic search for “information about the vascular system" would return articles and data records that refer to the heart, even though the word "heart" was not explicitly searched for. Furthermore, a semantic search for "organs connected to the heart" could make logical inferences across chains of concepts in the underlying medical ontology in order to return articles about the lungs, the liver, the kidneys, the brain, etc., even though none of those organs were explicitly named in the original query.

Smarter searches are just one of the many benefits of the Semantic Web. Beyond such basic applications, the Semantic Web makes it possible for software to automatically learn, reason, make suggestions, and manage tasks and processes more intelligently. What’s more, by providing a standardized language for describing systems of concepts and chains of reasoning, the Semantic Web makes it possible for programs to seamlessly share concepts and collaborate on reasoning tasks – in other words, it makes it possible not only for smarter computation within a given program, but it also enables smarter computation to take place between programs, making it possible for widespread distributed artificial intelligence to emerge on the Internet.

THE EVOLUTION OF METALANGUAGE

The Semantic Web is based on a higher level of language — metalanguage — language about language. Metalanguage is a form of communication that enables parties to rigorously express and share information about the meaning of information. In fact, metalanguage has existed since the dawn of humanity. For example, in the case of spoken language, humans communicate metalanguage by using tone, gesture, inflection, volume, and facial expressions. These cues convey vital information about the meaning of what we are communicating, making it possible for those we communicate with to more easily understand us. In written language very simple forms of metalanguage have also been in use for quite some time, such as for example, the formatting of text, the use of footnotes and diagrams. The way text is organized on a page, and the particular typefaces and styles used also constitute metalanguage expressions about the meaning of the text.

The Semantic Web provides metalanguage specifications and technologies that vastly increase the bandwidth and sophistication of metalanguage communication for all forms of digital media. For example, using metalanguages such as XML, RDF and OWL, the Semantic Web makes it possible to encode arbitrarily detailed knowledge about the structure, meaning, state, connections, reliability, sentiment, and policies of arbitrary chunks of information. In other words, a document can be encoded with metalanguage that adds layers of additional knowledge about the information it contains. These layers of information augment the text — they may provide definitions, links to other resources, information about the organization of information within the document, logical relations among concepts in the document, details about the history and license terms of the document, annotations from other readers, and even rules for interpreting, reasoning about, or using the document. What is important here is that this metalanguage is expressed in a manner that machines can understand.

In effect, semantic metalanguage gives computers access to layers of knowledge that previously could only exist or be utilized within the human brain. By making this metalanguage explicit and by standardizing it, it becomes possible not only to communicate it effectively between humans, but also between humans and programs, and even between programs and other programs.

The evolution from simple typography to SGML and HTML to XML and finally to the Semantic Web (RDF and OWL) can be viewed as a process of decoupling the interpretation of data from the agents that produce and consume the data. In other words not only the data itself, but also its interpretation, can now be stored outside of the agents of communication. HTML makes it possible for any program to correctly render data. XML makes it possible for any program to correctly parse and navigate the structure of data – for example to find a particular data element such as a field within a document. RDF and OWL make it possible for any program to understand what a particular data element means, and to reason about it.

If we look back to the dawn of humanity there was a time when humans were only able to communicate nonverbal or primitive verbal information. As richer forms of communication evolved sophisticated spoken languages and oral traditions emerged enabling the communication of more complex ideas. But spoken language had a major limitation – the distribution and access to information was dependent on being physically proximate enough to interact with particular individuals.

With the development of written languages however, it became possible to break through this limitation. Writing systems made it possible for ideas to be represented, stored and communicated independently of any particular individual, with less error, across greater distances in space and time than ever before. For the first time it was possible to learn something from someone else without them having to be present – anyone who could read the language and had sufficient background could interpret written characters into concepts. Next, with the advent of printing the economics of distributing and accessing written ideas reached a critical threshold of efficiency, enabling widely distributed communication and intellectual discourse.

Centuries later another critical threshold was crossed with the invention of long-distance communications networks such as teletypes, telephones, radio and television. These technologies made communication faster, richer, broader, and more ubiquitous and accessible than ever before. As recorded and recordable media emerged even these rich media experiences could be experienced asynchronously anywhere and at any time.

Next, the emergence of computers and computer networks made it possible for communications and information processes to be increasingly automated. At this point we begin to see something new – while previously only information could be represented outside of the human brain now even primitive forms of intelligence (information processing) could be represented and conducted outside of the human brain. The Internet and the World Wide Web are the logical extension of this process – they make it possible to distribute and access information, and to connect information and processors together, more widely than ever before – but they still rely on humans for the most part.

Without humans the Internet and Web of today would be nothing but a collection of relatively static information and dumb computer systems. But XML and the Semantic Web will change that by providing metalanguages that make it possible for humanlike intelligence to being to evolve and function outside of human brains. With advent of metalanguages humans are no longer necessary to create or interpret information. These technologies will enable the Web to actively and intelligently process information without human participation.

Metalanguages such as HTML, XML, RDF and OWL enable knowledge about information to be formally encoded into the information itself. As increasing levels of knowledge about data is encoded into the data, the data becomes more independent of humans -  it can be used by any agent anywhere.

HOW THE GLOBAL MIND THINKS

Semantic Web programs will share and process information intelligently, with or without the help of humans, by reading and writing metadata about data in a standardized way such that other programs can then reuse it. Programs will be able to leverage the knowledge that other programs create about the data they work with – even though these programs may not be directly integrated or even know about one another. In effect the Web becomes a gigantic shared knowledgebase that every program can read and write to.

Just as colonies of social insects such as ants and bees are able to perform intelligent collective behaviors without centralized control, the millions, or even billions, of humans and programs roaming independently through the Semantic Web, selectively reading, writing, annotating, linking, rating, and aggregating information, will perform collective intelligent behaviors without necessarily coordinating with one another or even knowing it. In other words the individual agents in such behaviors will participate in collective cognitive processes that transcend the comprehension of any individual.

Here’s how it might work: Imagine that a particular news article about a potential corporate merger exists on the Web. Intelligent agents – whether humans or software programs -  are then able to read this article and mark it up with semantic metadata in their particular areas of expertise. One agent specializes in identifying company names – whenever it sees the name of a company in an article it tags it with a link to the ontology definition of a corporation, as well as with metadata that links it to the Web site and other data records corresponding to the particular corporation it represents. Another agent specializes in recognizing people: whenever it sees the name of a person it tags it with a link to the ontology class for “person" and also with metadata that connects it to the home page for that person, articles about that person, friends and colleagues of that person, organizations that the person is affiliated with etc. Other agents that visit, or receive, the article could then tag it with their particular knowledge – some add metadata about links, others tag events, others add metadata about places, others add metadata about products and brands, others add metadata about technical terms and jargon, etc. 

We might even imagine that some of these agents are capable of generating new articles and data structures about the original article and linking them together – for example, one agent might generate a synopsis, another might translate it into another language, another might measure the opinions in the article, still another might generate a report based on the conclusions in the article. Because all of this knowledge is expressed using open semantic metadata standards, any program that later encounters any of it can make use of it in its own work, without having to be expressly programmed to do so.

This is already starting to happen in fact – For example, in the blogging community and communities of practice, which in an entirely bottom-up emergent manner, are naturally aggregating, annotating, linking, organizing and prioritizing information. Although there is no central guidance within such knowledge communities, their collective self-organizing behavior results in global information processes that appear to be intelligent. If one were to view the information dynamics of the Web from space – perhaps with a special sensor that could detect and measure these patterns as they emerged – would it not appear similar to the a functional brain imaging scan?

The Internet (the OS layer), the Web (the data layer), XML (the data schema and syntax layer), and the Semantic Web (the knowledge and reasoning layer) combine to provide the foundation for an increasingly intelligent distributed world-wide mind. They enable all the agents of the global mind to seamlessly share not just raw information, but even high-level concepts, knowledge and intelligent cognitive processes, in a manner that is open and independent of any individual system.

In particular the Semantic Web makes it possible to represent concepts such that they can be unambiguously interpreted and understood by any agent of the system. However, the success of this process will hinge on the development and adoption of open-standards-based, open-source ontologies, and mappings between them. This is already starting to take place, for example, FOAF, a simple ontology for describing social relationships, and SUMO, a standardized ontology of foundational concepts, among many others. I believe much of the initial development of these much-needed open-source ontologies will spring from the Weblog and RSS communities, where there is an increasing willingness (and need) on the part of participants to mark up and filter content with metadata.

CAN THE GLOBAL MIND PASS THE TURING TEST?

If the Internet is becoming a global mind, is there a way to test whether or not it is actually intelligent? Of course that first requires that one define intelligence – a notoriously fuzzy term! For the purposes of this article, we might define intelligence as “humanlike information processing.” One way to test for “humanlike intelligence” is to use the Turing Test – in which a human subject attempts to determine which of two “black boxes” is controlled by a human and which is a computer in a question-and-answer game.

An interesting modern-day spin on the classic Turing Test, might test large distributed online communities comprised of people and software programs, to see if such systems could be judged to be intelligent. It seems like a good bet that such systems – if hidden in a black box would be able to emulate “humanlike intelligence.”

I once tested this hypothesis in my own company many years ago. A difficult math problem was posed to me and to the best mathematician in our team. Whoever could answer it correctly the fastest would be judged as the best mathematician. I have never been much of a mathematician, but I still won this contest. My strategy was simply to farm out the problem to a number of the best mathematical brains I knew, integrate the answers, and package it up as a reply to the question.

My network of math-brains vastly outperformed the ability of my own brain or the brain of the math expert I was competing with. Not knowing how I solved the problem, those in the company would only be able to assume that I was a better mathematician. In point of fact however it was not a fair contest. "I" was not merely an individual but a vast collective super-brain comprised of several networked experts. The other guy was hopelessly outgunned.

This is an example of the power of distributed intelligence – the world of the future that is evolving on the Web right now. As the global brain continues to develop we will see individual humans, and even individual organizations, being dramatically outpaced by collective intelligences. One compelling example of how this is happening can be seen in the rise of open-source software development communities which are able to develop better code, faster, at less cost, and with broader adoption than has ever been possible by single entities.

READING THE GLOBAL MIND

If the Web is becoming a virtual mind of the planet is it possible to data-mine the Web in order to empirically measure, map, understand and even predict collective cognition? Can we empirically measure the Web in order to chart the past and present thinking of individuals, groups and communities, nations, or even of humanity-as-a-whole? By dong this can we learn to detect and track thoughts ("memes") as they emerge, spread, interact, develop and evolve in real-time? If we are able to empirically detect memes and develop a science of meme dynamics would this enable us to not only better understand the past and the present, but even to predict the future in a new way?

One approach to reading the global mind is to measure distributed cognitive trends by mining search engines results for  the frequency of search terms over time, such as Google’s Zeitgeist reports.  More recent approaches such as Daypop attempt to detect "word bursts" on the Web and "news bursts" among news articles.

Many academic and government research projects have explored the potential to data-mine news articles and other information sources in order to predict political events. For example, this project found that it is possible to predict conflicts such as wars as early as 6 to 8 weeks before they occur by data-mining news articles. There have also been projects to predict signs of political change, such as coups and election results, by data-mining political news.

Another interesting project describes a technique for statistically analyzing clusters of concepts that occur on the Web in order to attempt to find hot archetypes in the collective consciousness of humanity. The particular application of this system that the authors focus on is predicting terrorist events. Their system identifies hot themes, but requires a high degree of subjective interpretation in order to come up with predictions. While interesting, I am not sure the system can be used to reliably predict the future, although it certainly can help to understand the present. In any case, this project is significant in that it attempts to detect collective thoughts or archetypal patterns that transcend any individual mind or community. It’s definitely worth reading for those interesting in next-generation data-mining.

Another project that takes a completely different approach is the Global Consciousness Project which mines statistical deviations from randomness across a network of random number generators around the world and then correlates these deviations with global events — nobody knows why this works but the statistical data speaks for itself (this project may in fact point to yet another interesting connection between consciousness and quantum physics, similar to the famous double-slit experiment but on a global scale, but nobody really knows — all we know so far is that the data is sound.) This project might be described as an EEG for the planet. While it cannot provide insight into particular thoughts taking place in the global mind, it does provide a window into the activation and dynamics of the global mind.

In my own thinking on the subject, I have focused more on detecting and analyzing the higher-order distribution of memes in space and time. Memes are concepts that move across the global mind – they the building blocks of its collective thoughts. A meme might be as simple as a brand or an icon, or as sophisticated as a joke, a fact, tradition, fad, belief system, or a paradigm. I have spent some time speculating about a possible physics of ideas that might be able to empirically detect, measure and predict the dynamics and interactions among memes on the Web. My approach attempts to measure properties of memes in space and time in order to forecast their trajectories. For example, using this approach, one might be able to measure the geographic footprint, mass and velocity of a meme over time. With such data it then becomes possible to begin to measure the spread of ideas much like one might analyze the behavior of systems of particles, or the behavior of products and stocks within marketplaces.

The examples above represent just a small sample of the many research projects and technologies in this space. I would be very interested to hear of others of note.

ENTERPRISE MINDS

As the global mind develops it will initially be focused around making information more useable. But that will be just the beginning. Already a new generation of tools that will bring the power of distributed intelligence to the desktop and the enterprise are being developed in labs such as HP, Cycorp and Network Inference.

In addition to these projects, my company,Radar Networks has developed a complete platform in Java for developing and deploying Semantic Web applications.

The power of distributed intelligence made possible by the Semantic Web will dramatically evolve corporations – at least those that are not made extinct by it. In particular, it will enable workgroups and corporations to begin to distill and store not only their information but also their intelligence. As individuals and teams work, intelligent agents will learn from them. These agents will then be able to assist them in working more productively — they will help them search, organize and file information, track relevant news, better-leverage existing knowledge and resources, manage projects and tasks, share and access knowledge, and communicate and collaborate more productively with teammates, customers and partners. Similarly, smart agents will learn from corporations as-a-whole, and from their business interactions with employees, customers, suppliers and partners in order to dynamically streamline business processes and adapt to market changes intelligently.

By making organizational knowledge, learning and intelligence increasingly independent of the particular minds or programs within a given organization, all parts of these organizations as well as the whole organizations themselves will become more intelligent. As knowledge and intelligence about organizations become increasingly portable and reusable, organizations will evolve their own "group minds" and "enterprise minds." These distributed forms of intelligence will constitute a new level of structure, a new layer of organization. Such meta-level processes will help managers make smarter decisions by enabling them to better access the combined past and present knowledge and capabilities within their organizations and business relationships. They will also help organizations to notice opportunities or problems, and response to them more effectively.

Today there are many organizations that have realized that their primary product is knowledge. Tomorrow organizations will begin to realize that it is not just knowledge, but also intelligence, that is the key to their competitive advantage. Intelligence is the ability to utilize knowledge effectively.

Merely creating vast collections of knowledge that are inaccessible or simply not leveraged is of no benefit to anyone. What matters is that the knowledge is intelligently connected to business processes such that it measurably improves performance. What is necessary for this to happen is not merely the implementation of knowledge management systems, but rather the implementation of intelligent systems — a new way of creating and utilizing knowledge at all levels of the organization.

Knowledge must be intelligently integrated into every business activity, event, relationship, resource and tool. Furthermore the integration must be bidirectional — every business activity should be able to get knowledge from the enterprise and add knowledge back to it. By enabling this, with the right infrastructure and tools, organizations can literally begin to learn and improve based on their own collective experience. By providing all of the parts of an organization with access to the collective knowledge and intelligence of the system, the whole system can become more collectively intelligent.

At a higher-level, in order to enable more focused, goal-directed collective behaviors it is necessary to create control structures and adaptive feedback loops between the "parts" and the "wholes" within an organizations. What this means is that there needs to be a connection between the knowledge and intelligence taking place within each part, and the new meta-level knowledge and intelligence taking place at the level of the whole (such as a team or enterprise). The question then arises as to how to bring about such a connection? What connects the parts to the whole — what makes a collection of parts function as a whole, yet enables them to still maintain their individuality and independence? What enables the whole to function as one entity, despite being formed of myriad independent parts?

Traditional control structures such as top-down management hierarchies err on the side of the whole — they attempt to rigidly organize and control the parts of the organization in order to force them to conform into a cohesive whole. On the other extreme, more recent attempts to eliminate hierarchy and enable flatter, more "networked" organizations err on the side of the parts — they eliminate the hierarchical control structures altogether leaving nothing but chaotically interacting decentralized parts. Fortunately there is another alternative — there is a way to connect the parts and the whole without sacrificing either. The key is enabling richer self-knowledge.

In order to bring about synchrony between levels of a distributed organization there must be three essential ingredients: (1) The state of each part must be represented, (2) the state of the whole system (the combined system of all the parts) must be represented, (3) the parts must all have real-time access to all of these representations. By meeting these three requirements feedback becomes possible in several directions — between each part and every other part, and between each part and the system as-a-whole. These representations and feedback loops provide a vital function to distributed intelligences — they enable them to enact a simple form of self-awareness. Self-awareness is a vital ingredient of higher forms of intelligence. The richer a systems’ self-representation, the smarter and more effective it can be.

In order to accomplish self-awareness in a highly distributed organization, each part of the organization needs to have access to a self-representation of itself as well as of the whole system. Each part needs to be able to understand themselves and the system they are part of. By providing the parts of a distributed organization with access to information about both their own state and the state of the whole, the parts are empowered to adapt to the whole. By enabling this, the whole is also more able to adapt to the parts, because there is bidirectional feedback taking place between these levels. Rather than placing control structures at only the level of the whole, or at only the level of the parts, instead they are distributed across both levels.

The only way for this to be practical, economically feasible, or even technically possible, is by using emerging Semantic Web metalanguages. These metalanguages provide a common standard for sharing knowledge and intelligence at every level and across every part of an organization. Knowledge and intelligence are thus able to move freely across and between them and organizational learning takes place on the individual level, the group or sub-systems level, and the level of the whole system. And this learning is expressed, stored and shared in a single common metalanguage that is equally accessible to all. This is quite different from the case of present-day organizations in which there are different languages and formats for knowledge at different levels of the organization. For example, in most present-day organizations human knowledge and expertise is still locked inside individual human minds and totally dependent on them, group knowledge is stored on PCs and workgroup servers, and enterprise knowledge is stored in enterprise systems. Each of these systems speaks a different language and most are not directly integrated.

Numerous inefficiencies result from this. Why should it be so difficult to move a concept across an organization? Why does it require that data be translated and ported from one person to another, from one program to another? The reason it is so difficult today is that the interpretation of the data is not stored separately from the brains and programs that manipulate it — in other words, metalanguages are not being used. As a result, the intelligence of the organization is not portable — it is locked into silos such as people’s heads and particular applications that are explicitly programmed with particular skills and knowledge.

Those organizations that understand this are already starting to make use of metalanguages and Semantic Web technologies. Those that are first to begin exploring and deploying these "enterprise minds" will have a valuable head start that may provide them with crucial advantages in the marketplace. This is not unlike the advantages that Homo sapiens had over earlier primates. Larger and more advanced brains resulted in an increased capacity for language, communication and reasoning that ultimately enabled them to outperform less intelligent hominids. This same principle holds for organizations.

CONCLUDING THOUGHTS

The ideas in this essay are not unique to me – they are memes that are spreading on their own through the global mind. Many others such as the people involved with the Principia Cybernetica Project or my friend Howard Bloom have thought far more extensively than I have about these subjects. In writing this article I am merely providing a service to the global mind – that of aggregating, annotating and communicating these memes onward in a process that I cannot begin to comprehend. All I know is that the global mind is thinking about its own evolution and realizing that it is intelligent – and that I am just an infinitesimal part of that process. Yet, like you who are reading this, I somehow sense that what is taking place is incredibly important and will change our world and our species profoundly.

See the rest of this article for a detailed description of how to build a working network automaton….

>

This diagram (click to see larger version) illustrates why I believe technology evolution is moving towards what I call the Metaweb. The Metaweb is emerging from the convergence of the Web, Social Software and the Semantic Web.

This article discusses new research in how the brain makes buying decisions and other choices — what is now called “neuromarketing”. Neuromarketing researchers seek to discover, and influence, the neurological forces at work inside the mind of potential customers. According to the article, most decisions are made subconsciously and are not necessarily rational at all – in fact they may be primarily governed by emotions and other more subtle cognitive factors such as identity and sense of self. For example, when studied under a functional MRI, the reward centers of brains of subjects who were given “The Pepsi Challenge” lit up when they tasted Pepsi, but Coke actually lit up the parts of the brain responsible for “sense of self” — a much deeper response. In other words, the Coke brand is somehow connected to deeper neurological structures than Pepsi.

Neuromarketing is interesting — it’s actually something I’ve been thinking about on my own in an entirely different context. What I am interested in is the question of “What makes people decide that a given meme is ‘hot’?” Each of us is immersed in a sea of memes — we are literally bombarded with thousands or even millions of ideas, brands, products and other news every day — But how do we decide which ones are “important,” “cool,” and “hot?” What causes the human brain to pick out certain of these memes at the expense of the others? In other words, how do we differentiate signal from noise, and how do we rank memetic signals in terms of their relative “importance?” Below I discuss some new ideas about how memes are perceived and ranked by the human brain.

Let’s call an individual instance of a meme at a particular time and place an “occurance.” An occurance might be a mention of the meme in the media, or in a direct or overheard conversation, or an advertisement, etc. What’s interesting is that the occurance patterns of memes do not have the same space-time dynamics. For example, some memes — such as mentions of common nouns — behave pretty much like “noise”. They occur in our experience with random distributions. Other memes, such as a steadily growing or declining trend, have linear dynamics — the number of occurances per unit of time is either constant or gains or declines in frequency with fairly predictable dynamics. Then there are nonlinear memes that behave chaotically — they exhibit erratic growth, sudden inflections, and are hard to predict.

Memes with random dynamics are quickly tuned out by the human perceptual system and by the brain. Memes with linear dynamics are treated differently by the human brain depending on whether they are unchanging, gaining or declining in frequency. Unchanging frequency is quickly tuned out as “background” noise. Memes that linearly gain or decline in frequency are attenuated as signal in proportion to their slope: The steeper the slope of frequency change, the more they are regarded as “signal.” Memes with nonlinear dynamics however are perhaps the most potentially interesting to the human nervous system — this is because it appears that humans are tuned to attend to “novelty” above all else. Nonlinear dynamics tend to be more novel than random or linear dynamics because they are hard to predict — the trajectories of nonlinear memes are “full of surprises” — they can suddenly change frequency and inflect. But not all nonlinear dynamics are equally interesting to humans — at the extreme of nonlinearity they are just random noise.

So the “ideal” dynamics is somewhere between chaos and order. What trajectories are optimally balanced between chaos and order? If we can figure this out, this is the key to a whole new science of marketing and advertising. Why is that? Because for example, it would enable us to mathematically optimize the frequency of an ad campaign in order to spend less money while actually generating better results in the minds of the target audience.

What’s interesting about this is that the way most advertising and marketing campaigns are conducted may in fact be almost maximally non-optimal. For example, showing an ad with regular, random or linear frequency over space and time in a market will quickly desensitize subjects — their brains will start to tune the ad out as “noise.” In other words, the brain attenuates to ad campaigns over time. So to get more attention, advertisers and marketers should vary their ad campaigns to make them more novel, more interesting, and less like “noise” or “background.” But how should they vary them, on what schedule, and in what manner, in order to generate the maximum effect? This is the interesting question to explore. What advertising distributions are perceived as most “interesting,” “memorable,” “positive,” and “important” to the human brain?

I believe it is possible to empirically test for the memetic dynamics that are MOST effective at getting attention in noisy perceptual environments by humans. By doing this we can, in a double-blind manner, figure out just what patterns work best. An experiment of this nature could be conducted in the following manner:
1. Create a mosiac of 100 frames, each capable of displaying an image
2. At each step in time (where a step is an intervale of time of fixed length, for example .1 seconds), show an image in every frame of the mosaic.
3. The particular image that is shown in any given frame at any step in time is determined by a computer.
4. At the start of the experiment the computer assigns a different “trajectory” to each of a set of 100 images. A trajectory can include frequency distribution over time as well as x/y coordinate trajectory over time.
5. Randomly select a set of test subjects to watch the slideshow for 1 hour.
6. After the slideshow, ask the subjects to rank the images they saw in order of “importance”
7. Two weeks later, without showing the subjects the slideshow again, ask the subjects to rank the images again in order of importance to see which images are retained as important for longer.
8. Test to see whether images that were shown with particular trajectories are consistently ranked as more or less important by subjects.
9. Run the experiment many times with different sets of subjects and many different candidate trajectories.
10. For each experimental run, the computer can use the trajectories that scored best in previous runs in order to narrow in on the best trajectories.

Over time this experiment might yield a set of trajectories that are most effective in getting and retaining attention in noisy perceptual environments. This would be a very useful discovery — it could be applied in many fields to overcome “information overload.” It would not only be of use to advertisers and marketers, but also to lobbyists, analysts in homeland security and intelligence, pilots in noisy cockpit environments, and even to help knowledge workers manage their email, prioritize search results, and monitor news more effectively.

I am having an interesting conversation with Howard Bloom, author, memeticist, historian, scientist, and social theorist. We have been discussing network models of the universe and the underlying “metapatterns” that seem to unfold at every level of scale. Below is my reply to his recent note, followed by his note which is extremely well written and interesting…

————
From: Nova Spivack
To: Howard Bloom
Subject: Re: Graph Automata — Is the Universe Similar to a Social Network?

Howard, what a great reply!

Indeed the metapattern you point out seems to happen at all levels of scale. I am looking for the underlying Rule that generates this on abstract graphs — networks of nodes and arcs.

In thinking about this further, I think we live in a “Social Universe.” What binds the universe together, and causes all structure and dynamics at every level of scale, is communication along relationships. Communication takes place via relationships. And relationships in turn develop based on the communication that takes place across them.

Relationships and communications take place between locations in the manifold of spacetime, as well as between fundamental particles, cells, people, ideas, network devices, belief systems, organizations, economies, civilizations, ecosystems, heavenly bodies, galaxies, superclusters, or entire universes. Whether you call it “gravitation” and “repulsion” and other forces are really just emergent properties of the dynamics of relationships and communications. It’s really all very self-similar.

I believe that we can make an abstract model of this — just a graph comprised of nodes connected by arcs — where the nodes (and possibly the arcs too) have states, and information may travel across them. Then, at each moment in time, we may apply simple local rules to modify the states of nodes and arcs in this network based on their previous states and the states of their neighbors.

For example, at time t+1, the state of each node is a function of the states of the nodes within some number of arcs from it and states of the arcs along each path to each of those nodes. Also, at time t+1, the state of each arc is a function of the states of the nodes it connects and perhaps also the states of the arcs of those nodes.

A node represents an entity — for example a particle or a person or a stock symbol. The state of a node can be a single number or an array of numbers representing many different variables, depending on our simulation. Arcs represent communications channels, through which nodes measure one another. The state of each arc encodes the strength of the relationship — the communication channel — it represents. Measurements can only happen via relationships — for a measurement to take place some information must travel from the thing being measured to the thing that measures it. In my abstract model I use a directed graph — so each relationship (arc) is only one-way from one node to another node. Thus a “bidirectional relationship” is a case where two nodes, x and y, are connected by two arcs xy and yx.

I also start with a maximally connected graph — every single node has one relationship arc to every other node and one from every other node. This allows for every node to potentially make a relative measurement of every other node according to its “perspective” on the relationship.

At every step in the simulation, every node x measures the state of every other node y via the relationship from that other node y to x — but the measurement is conditioned by the state of the arc along which it takes place such that in some cases it is enahnced dramatically, or dampened to the point where it is simply not strong enough to matter. When a measurement is dampened to that degree it is equivalent to there being “no relationship” between the nodes.

Thus although there are always virtual relationships between all entities, only some relationships are “actual” in the sense that they are strong enough to enable measurement to take place. And this changes over time, based on how the entities interact. Our rule should evolve the strength of relationships based on the measurements that take place across them.

The philosophy of this model is based on the insight that a relationship is in fact the most fundamental thing in the universe — even more fundamental than particles or locations in space-time. This is very much philosophically in the camp of Liebniz as opposed to Newton.

In fact, in my model, both nodes and arcs are actually relationships — a “node” is represented by an arc that loops back on itself — it is something that measures itself — a circular relationship; an arc is a relationship that does not loop back on itself — a relationship that connects one node to another. Therefore there are really just relationships in this model but they are interpreted differently depending on their shape — an “entity” is a node, a self-relationship, a communication channel is an arc — an other-relationship. I mention this only because of its elegance — it makes it possible ultimately to have a single rule that operates only on arcs at each step in the simulation (since nodes are arcs too in this conception), rather than having different rules to compute node states and arc states.

The most basic act in the universe is to measure something via a relationship. A measurement is therefore the most fundamental unit of communication. A series of measurements that take place between two entities is an interaction — a process of communication. Relationships are communication channels (arcs) that affect the measurements that travel across them: strong channels may enhance measurements, weak ones may dampen them. So the measurements that nodes make of each other are conditioned by the arcs that mediate them. Likewise, the state of a relationship, and therefore its effectiveness as a communication channel, may change based on the measurements that take place across it over time.

This model is essentially very similar to a neural network, and in fact a modified neural network algorithm may be just what we are looking for. I would not be surprised if in fact we could empirically discover this rule by looking for a pattern in the way relationships and interactions develop among neurons in the brain, people in social networks, memes in belief systems, services on the Internet, stocks in economies, stars in galaxies, etc. As you point out, gravitation between stars is similar to the attraction between people. And relationships between people are not so different from topological connections between locations in space-time, or the forces that bind particles together.

Using networks to model these various phenomena is not merely interesting, it may be essential to discovering a unified theory of the universe — there may actually be a metapattern to all “social networks” that helps us to discover the key underlying laws of the universe, at every level of scale. And that is something our civilization has not done yet — we have not found a general theory of structure and dynamics that applies equally well at every level of scale, in every context. Quantum mechanics is still not unified with Relativity, let alone with Biology, Society, Ecology, Economics, etc.

I think this “metaunification” will be easier to accomplish if we use the same basic model to represent structure and dynamics at every level of scale. Currently very different models and languages are used by thinkers to represent systems at different levels of scale — and this is one of the reasons we have not achieved much unification to date. We need to get everyone speaking the same language — using the same modelling tools — so it is easier to map between discoveries in different domains. Network models are ideal for this purpose.

I believe that an empirical study of existing social networks on different levels of scale is one route to finding the general pattern we are looking for: All social networks — at all levels of scale — should obey certain laws that we can discover through observation and then generalize into a general theory. Another approach is purely through mathematics — it should be possible to derive an abstract mathematics of social networks. Finally there is also the computational approach — simply generate and test different social network rules, and perhaps even use a genetic algorithm to evolve an optimal one. Perhaps that is the computation that our universe is running?

— Nova Spivack

—–Original Message—–
From: Howard Bloom
To: Nova Spivack
Subject: Re: Graph Automata — Is the Universe Similar to a Social Network?

Nova–Fancy running into you here on paleopsych, The International Paleopsychology Project email list.

Pavel Kurakin and I, curiously enough, are looking at the basic patterns underlying social connections among quantum particles, insects, and, implicitly, humans for an upcoming paper. Pavel is with the Keldysh Institute of Applied Mathematics of the Russian Academy of Science.

Something you’ve said hits a nerve: “the network seeks to help each node optimally balance its connectivity against information overload”. Bear with me while I seem to go way out beyond left field. I’m working on a book called Reinventing Capitalism: Putting Soul In the Machine–A Quick Re-Vision of Western History. One of the chapters is called “Marketing Meaning—Moses And The Slogan”.

It’s about the way in which Moses marketed his new religion…the way in which he drummed it into the head of his Chosen People. If you can believe the Bible and Sigmund Freud’s brilliant analysis of Moses and of the basics of community building in his Moses and Monotheism, Moses boiled his entire system of belief down to a slogan–”Hear, oh Israel, the Lord The God, the Lord is one.” Then the sociologically wise prophet told his followers to hang this bumper-sticker distillation on their doorposts so they would see it in the morning when they walked out of the house and in the evening when they walked back in again. He ordered the men to bind the slogan to their wrists, arms, and foreheads twice a day. And he apparently made darned sure that this catch-phrase was repeated a lot.

Today we call this sort of thing branding. Why is it so necessary to us human beings? Why do we need to have just a small Olympus of stars and leaders we can gossip about? Why do we focus on brands like Coke, Pepsi, and Dr. Pepper, but toss lesser known brands aside? Why do we show interest in only two or three presidential primary candidates–Kerry, Dean, and Clark–much to the consternation of the other five or six?

We have only seven slots for immediate memory in the brain. This limits the information we can handle. It limits the number of choices we can comprehend..or tolerate. (From the description of another Reinventing Capitalism chapter: “A little choice is freedom. Too much choice is agony.”)

So to get through to us, you have to make it simple and you have to make it stick. You have to repeat it over and over again until we get it. Once we’ve gotten it, we can slide it from consciousness to habit (from explicit memory to implicit memory) and concentrate on something else.

What does this cellular automata-style rule of individual capacity mean when writ large in group behavior? It means that we need to do a lot of quorum sensing. We need to go along with the herd. We need to pay attention to what everyone else is paying attention to. We need to buck it and criticize it if we want, but to fixate on it one way or the other. If George Bush Jr and the Iraq War are the topics of the day, we can hate Bush, love Bush, hate the war, love the war, but not get sidetracked by detailed examinations of who the Chechen rebels are.

We go with the flow of popularity. We follow fads, even if we only come along for the ride and criticize them.

This rule pretty much applies to the cosmos, too. Gas whisps and dust clouds in the early cosmos went where the action was. They congregated around self-forming swirls called galaxies. Then they aggregated even further in suns and planets. The general rule was this: To he who hath it shall be given. From he who hath not, even what he hath shall be taken away.

Why the repetition of this rule on two very different levels–the gravity wars that led to galaxy formation and the popularity wars, the wars of social gravity, that determine who will be the candidates in a presidential election and who will be the hot rock and TV stars of the day? Humans aggregate to limit the flow of information. But do specks of interstellar dust interpret information, too?

Specks of dust do respond to attraction and repulsion cues. If they’re negatively charged they avoid other negatively charged things. They move in the patterns dictated by magnetic waves surrounding stars and furling in galaxies. And, of course, they pick up on the come-hither cues of gravity.

But surely interstellar dust flecks and whisps of gas can’t work to optimize their information flow. They can’t gravitate around stars because of a need to keep their seven slots of memory from radical overflow. Nor can they use those stars to keep them entertained–to assure that they don’t suffer the pain of boredom that comes from information underload.

Whither comes this common pattern that keeps the big getting bigger yet provides a few key choices? Why do embryonic stars in a star nest or cluster compete like starlets in Hollywood to become the next big center of attraction?

There’s a primal pattern, an evolutionarily stable strategy, what I’ve been calling an Ur-pattern, rearing its head here on many levels of emergence. But how does this similarity appear and why?

— Howard

In this article I discuss some insights about optimization of social networks. Basically I suggest that “trust is not preserved” along relationship paths of more than 3 hops. In other words, social networks should never forward messages beyond 3 hops. Doing so makes the communication of that message effectively arbitrary, adding noise to the system and degrading utility for users.

The Law of Density

The value of a social network, to each member, is inversely proportional to the density of the network, defined as the average number of relationships per member. As a network becomes denser, it becomes harder to manage — each member experiences more “social overload” and there are an increasing number of routes between members. As density increases the likelihood that any two members of the network are connected by at least one path increases, and furthermore the distance of such paths decreases.

Social networks, if they are successful, asymptotically tend towards maximal density — networks in which everyone is linked to everyone else by 1 hop, at which point there is no more social benefit of belonging to the network.

The Law of Sparsity